Labi's protocol library indexes 9,749 open-access lab protocols. Every one of them is a real, individually addressable page — good for search, good for the scientists who land on exactly the method they needed.
It also turned out to be exactly the kind of thing that blows through a hosting quota you didn't know you had.
What happened
Our hosting plan bills by how many unique pages get freshly generated in a month. We were well past the limit — not because of real visitor traffic, but because of how search engines index a site this size: they systematically crawl every URL in a sitemap, and every deploy resets the cache those pages were sitting in. Deploy often enough with a catalog that large, and the crawl traffic alone adds up fast.
What we changed
- Stopped telling search engines every page had "changed today" on every deploy — a leftover sitemap detail that was inviting more re-crawling than necessary.
- Capped how much of the catalog we expose to crawlers at once, releasing it in batches instead of all 9,749 at once — better for hosting costs, and honestly better SEO practice too.
- Started batching our own deploys instead of shipping every small fix immediately, since each deploy is what resets the cache in the first place.
None of this changes what you see as a visitor. It's the unglamorous, load-bearing kind of work that mostly matters when you don't notice it.
More on what we're building next soon.