Notes on Veilid

Today I took a serious look into Veilid as a possible network provider for Hermes, my long-running delay-tolerant overlay network project. What I found continued to intrigue and yet ultimately disappoint me, at least in its current state.

I first heard of Veilid in 2023 through their DefCon 31 talk. Initially, I was half-relieved and half-worried that they had solved the problem I had wanted to solve: creating a platform that I could build CRDT-based end-to-end encrypted peer-to-peer applications on top of. Then I saw that they relied on nodes with public IP addresses and open firewalls, used WebSockets as their transport between “peers”, and that they prioritized IP anonymization through onion routing. These design decisions made me concerned about the performance of Veilid for applications in contrast to simply hosting a relay server. Six times higher latency from onion routing combined with the potential bottleneck of having the minority of nodes with open firewalls be the entrypoint for the vast majority of onion circuits makes Veilid a risky bet from the perspective of an application developer who wants performance that rivals a standard store and forward relay server setup.

Fast forward to a few weeks ago, I heard that Veilid added hole-punching support for native clients (i.e. clients not running in a web browser) and so I decided to look into them again, hoping that their holepunch implementation might fix the bottleneck issue. What I found is a half-baked implementation, especially when it came to supporting hard-nat easy-nat hole-punching, not to mention the lack of rigorous network testing infrastructure to validate their implementation. To their credit, they are very clear that Veilid is still unstable and under heavy development, but what they’ve prioritized over the past few years has me worried about their long-term ability to solve the truly difficult issues at hand; I especially worry about resilience against byzantine actors in the network and the optimization of their system. I’m concerned that the high-latency onion routing will result in a degraded enough experience that people instead rely on other solutions for realtime or near-realtime collaborative applications; exactly the kind of application that seems most suited to a peer-to-peer overlay network.

Overall, while I applaud their ambition to provide byzantine object and secret storage, networking, and onion routing in a single project, I fear that they are far from being done with any one of these aspects of Veilid, let alone close to reaching a stable and resilient release especially in the face of byzantine network nodes. Though I realize that even three years later I cannot yet build on top of their network for my projects, I continue to hope that someday it will be practical for me to do so. I want to be very clear that this is not an indictment of the Veilid project’s overall aspirations but rather an evaluation of its state today. I hope that in the next few years they can solve the performance and security oversights that their current design has - possibly through network simulation testing - as they continue to strive for a stable version.