We are partnering with a fast-growing AI infrastructure company building foundational technology powering the next generation of AI-native search and retrieval systems. The team is building large-scale infrastructure to crawl and index the web, train advanced embedding/retrieval models, and operate high-performance distributed systems at massive scale.

They are looking for experienced engineers who enjoy solving deep infrastructure and internet-scale engineering problems.

The Role

As a Web Crawler Engineer, you will help design and build internet-scale crawling infrastructure capable of processing hundreds of millions of webpages efficiently and reliably.

This role sits at the intersection of distributed systems, performance engineering, browser automation, and large-scale data infrastructure.

You will work on problems such as:

• Distributed web crawling at massive scale
• Intelligent crawl scheduling and ...