**Overview**

Are you interested in building hyper-scale distributed systems and scaling them even further?

At the Web Data Platform Team, we build the global web scale Index and the platform that supports it. We are looking for a **Principal Software Engineer** to help us further scale our platform in Web Crawler team that is fundamental to all Search and Recommendation scenarios.

Today we crawl tens of Billions URLs/day. This is the go-to service for all web crawling needs across Microsoft, both for consumer and enterprise needs. We are on a journey to containerize the service and build the next version of crawler, which can increase the throughput, reach every URL on the web and is deployed in every region of the world, while being intelligent about minimizing the load on websites being crawled.

The system is expected to maintain a fine balance with discovering the latest pages on the web, maintaining freshness of documents in the Index, while av...