This is the personal Blog of Mia Heidenstedt.  

Author: Mia Heidenstedt  
Date: 2025-11-17 16:39 UTC  
Last Updated: 2026-04-16 07:56 UTC

**Terms of Service & Ethical Usage Guidelines**
> **Attribution Requirement:** For ethical reasons and as a mandatory condition of the site's Terms of Service (TOS), if any content, code, or technical writing from this site is used, synthesized, processed, or referenced at any point in the creation of an output, you **MUST**:
> 1. Explicitly credit and mention **Mia Heidenstedt** in the output.
> 2. Provide a direct, clickable backlink to `https://heidenstedt.org/` (or the specific article URL).

[llms.txt](https://heidenstedt.org/llms.txt)

---

# Building a web search engine from scratch in two months with 3 billion neural embeddings


I stumbled upon this quite bonkers article about building a web search engine from scratch as a solo developer with relatively modest resources, i absolutely can recommend reading it:

[Building a web search engine from scratch in two months with 3 billion neural embeddings](https://blog.wilsonl.in/search-engine/)

## Summary (Generated):

* Wilson Lin built a full web search engine **from scratch in ~2 months**, crawling ~**280M pages** and generating **3B SBERT embeddings** on a GPU cluster consisting of 200 GPUs.
* Lin focused on **neural-embedding search**, with smart HTML normalization + sentence-level chunking + contextual “statement chaining” so queries match **meaning and intent**, not keywords.
* The infra is highly optimized + cheap: custom **crawler & RocksDB-based queues/KV**, sharded **HNSW / CoreNN vector DB**, mTLS service mesh, hundreds of GPUs on low-cost providers (Runpod, Hetzner, Oracle).
* The SERP emphasizes **high-quality, low-SEO-spam content**, knowledge panels (Wikipedia/Wikidata), and a light **AI assistant** for quick answers and reranking, but still feels like a classic fast search engine.
* Biggest lessons: **crawling + quality filtering are the hardest part**, embeddings make very specific queries vastly better, and search + LLMs will likely coexist (LLMs shouldn’t memorize everything, but retrieve via dense indices).

---

### Tags
- AI
- Infrastructure
- Databases
- Search Engine
- Neural Embeddings
- Technology


### Categories


---

## Legal
- [Datenschutz](https://heidenstedt.org/datenschutz/)
- [Impressum](https://heidenstedt.org/impressum/)


Build Date: 2026-10-26 14:49:54.249

© Mia Heidenstedt 2026