Crawl budget

Crawl budget describes the amount of resources a search engine allocates to crawling a given website within a given period. On large sites, a limited crawl budget can mean that not all pages are visited equally often. Optimization is about guiding the bots toward the most important pages and avoiding waste on irrelevant URLs.

On a large webshop, crawl budget can be saved by blocking endless filter URLs that the bots would otherwise work through.