Build a 500k Domain Search Engine in a Weekend

Host

Welcome back to another episode of Tech Talk! Today, we have a fascinating story about building a search engine, but not just any search engine—this one’s personal and built for makers. Our guest is Marlin, who managed to create a search engine for 500,000 domains over a weekend for just ten bucks! Marlin, thanks for joining us!

Expert

Thanks for having me! I'm excited to share how this all came together.

Host

So, let’s start at the beginning. You mentioned that you were frustrated with existing search engines. Can you tell us what led to this late-night coding session?

Expert

Absolutely! I was tired of searching for portfolios, unique art projects, and interesting software only to be buried under tons of corporate content. It felt like there was so much creativity out there, but it was hard to find.

Host

That makes a lot of sense. It’s like trying to find a needle in a haystack, right? So, how did you decide what you wanted to build?

Expert

Exactly! I wanted a personal search engine that focused on the kind of work I cared about—art, code, poetry—without all the clutter. My plan was to create a crawler that only looked at homepages and then summarized them.

Host

That sounds like a huge undertaking! You mentioned you cataloged over 560,000 homepages in just a few days. What was your approach?

Expert

I started with a clear goal in mind. I knew I didn’t want to index the entire web, just a select few domains that mattered to me. I also limited what data I was collecting to keep things simple—no full HTML pages, just key metadata.

Host

That’s a smart strategy! It’s like packing light for a trip. So, can you break down the technical side for us? What processes did you use?

Expert

Sure! I had a few different components. First, there was a fetcher that grabbed the webpage data. Then a worker that processed this data through a small local model to summarize it and categorize it. I also had a steward to manage any suspicious domain activity.

Host

You really made the most of your resources! How did you ensure your search engine didn’t just become another corporate tool?

Expert

Well, I intentionally excluded certain features like multi-tenancy and recrawl scheduling. My goal was to keep it personal and focused. I even created a blocklist to filter out unwanted domains.

Host

That’s really impressive! Now, I have to ask, what were some of the challenges you faced?

Expert

One major challenge was managing the raw page text without letting it build up. With so many domains, it could quickly eat up space. So, I made sure to wipe the text once the model processed it.

Host

That sounds like quite the balancing act! And all of this for under ten dollars? What kept the costs so low?

Expert

I rented a GPU for a short time and relied on my own PC for most of the processes. It’s amazing what you can do with just a bit of tech know-how and some creativity!

Host

Incredible! This is such an inspiring story, Marlin. If someone wanted to create their own search engine, what’s your biggest piece of advice?

Expert

Start small, focus on what truly matters to you, and don’t be afraid to limit the scope of your project. It’ll help you avoid getting overwhelmed.

Host

Great advice! Thank you so much for sharing your journey with us, Marlin. I think our listeners will find this really inspiring!

Expert

Thank you for having me! I hope it encourages others to explore their ideas!

Host

And that’s a wrap for today’s episode! If you enjoyed this conversation, don’t forget to subscribe for more tech stories! See you next time!