I stumbled upon this quite bonkers article about building a web search engine from scratch as a solo developer with relatively modest resources, i absolutely can recommend reading it:

Building a web search engine from scratch in two months with 3 billion neural embeddings

Summary (Generated):

• Wilson Lin built a full web search engine from scratch in ~2 months, crawling ~280M pages and generating 3B SBERT embeddings on a GPU cluster consisting of 200 GPUs.
• Lin focused on neural-embedding search, with smart HTML normalization + sentence-level chunking + contextual “statement chaining” so queries match meaning and intent, not keywords.
• The infra is highly optimized + cheap: custom crawler & RocksDB-based queues/KV, sharded HNSW / CoreNN vector DB, mTLS service mesh, hundreds of GPUs on low-cost providers (Runpod, Hetzner, Oracle).
• The SERP emphasizes high-quality, low-SEO-spam content, knowledge panels (Wikipedia/Wikidata), and a light AI assistant for quick answers and reranking, but still feels like a classic fast search engine.
• Biggest lessons: crawling + quality filtering are the hardest part, embeddings make very specific queries vastly better, and search + LLMs will likely coexist (LLMs shouldn’t memorize everything, but retrieve via dense indices).

I forgot to mention that i released a new tool for my project ouroboros-db, so here it is:

The “OuroborosDB Overhead Calculator” is a little tool which helps you to calculate the overhead of erasure coding systems while also considering the overhead of indexes and the blocks themselves. It is kinda fun to play around with so give it a try!

https://i5heu.github.io/ouroboros-db-overhead-calculator/

In the midst of working on my project, ouroboros-db, I realized I needed a straightforward way to locate other nodes using the same RSA key. That’s when PathfinderBeacon came into being and that i release to the public with these words.

What is PathfinderBeacon?

PathfinderBeacon is a tool designed to make it easier to bootstrap your distributed systems using just a RSA key and one of the most robust dictionaries available: DNS. Since DNS inherently caches information at various levels, it allows PathfinderBeacon to deliver node addresses quickly and with minimal bandwidth usage. This means if multiple searches are conducted within the same network for the same room, there’s a high chance the requests won’t even leave your local network, thanks to the caching capabilities of your router’s local DNS.

You can think of it like a DDNS (Dynamic Domain Name System) service, but you only need an RSA key to write to a subdomain. The subdomain is generated from the RSA key, and there is little surface for abuse since you can only publish highly sanitized TXT records.

How It Works

Setting up is pretty simple. You either provide an RSA key or let PathfinderBeacon generate one for you. Once set, you can publish your room and node addresses, and anyone with the same RSA key can write to the same room, while anyone can read from it. This makes it easy to bootstrap your distributed systems without the need of running your own bootstrap service.

Currently, there is an official Golang library for interacting with PathfinderBeacon, which you can find here. If you’re interested in developing libraries for other programming languages, feel free to reach out, and I’d be glad to assist and subsequently feature your work on the project repository.

Future Directions

For me this project is pretty much finished since it solves the underlying problem quite effectively, but i am fascinated by this problem and i would love to develop PathfinderBeacon further, maybe even in a distributed thingy without a central server, private rooms and much more is thinkable. You can checkout the README for more ideas.

So, that’s PathfinderBeacon in a nutshell—a simple, secure way to manage node addresses within distributed systems. It’s a small project with big potential, and I’m excited to see how it can support and simplify the management of distributed networks.

Thanks for reading, and I’m looking forward to constructive feedback or contributions!

You can find PathfinderBeacon on GitHub

Sadly i did’t saved measurements form before the migration to the new Infrastructure. I will do better next time ;)

My blog has experienced a major infrastructure update.
Previously this blog was served by a tiny but highly optimized server at a Hetzner datacenter.
Since this blog is static it was for example possible to pre-compress the files, which shaved a massive 1-0.5ms from a cold TTBF.

Sadly this approach was a bit flawed because outside of germany the cold TTBF times weren’t exactly good.
Surprisingly CloudFlare did nothing to these times, since the cache of CloudFlare is short lived and not prefetched.

My solution to this problem, to make a global low cold TTBF possible, was BunnyCDN which provides a Service that can store and server static file, these files are replicated Globally to user-selectable locations.
At my case (i use all locations), the price per GB per month is 0.28$ which is very good if you consider that all content is replicated and served from SSDs in 14 locations.

Here you can see where these locations are, green is the “main storage zone”, which is not configurable.

The switch to BunnyCDN gave me a immense performance boost around the world, and surprisingly also in germany.
BunnyCDN dose not allow for rsync or something usable to transfer the Files to them.

I use lftp to transfer the files via FTP, which takes about 9 min each time. This is tbh shitty, and i don’t like it a bit.

DNS and Peering

Sadly BunnyCDN dose not allow to use them as a nameServer, also i have to use a c-name to hook up to their CDN.
Cloudflare on the other side acts as a nameservers and returns direct IPv4s and IPv6s on the DNS-query, this will save 1 DNS-Request and is therefore faster, especially since most users using the DNS-Server from their ISP which are often very slow.

So i tried to put CloudFlare in front of BunnyCDN, which had a negative impact on the TTFB, although not a great one.
In same cases it is faster to get a uncached page on BunnyCDN through CloudFlare because CloudFlare has a lot and very good peering to ISPs and exchanges.

Sadly CloudFlare TTBF behaves a unpredictable.
It can be that you get in one request 20ms TTFB and the next one takes 300ms.

I think i will run CloudFlare anyway in front of BunnyCDN in hope to better mitigate Attacks and big page visit numbers.
I hope that CloudFlare will cache such surges for me… they can become expensive.

Costs

This entire setup cost me less then 1$ per Month (without the Domain), which is very very cheap for something this fast.
I think it is amazing how cheap and easy you can distribute your ideas and thoughts today, without relining on a platform that appropriates your content.

Conclusion

Costs less then 1$ per Month.
Scales amazingly.
Very fast global latency.
I don’t have to keep a server updated.