Crawling

Crawling is the process by which search engine bots systematically move through the internet and follow links to discover pages. During crawling, the page's content is retrieved so it can later be assessed and potentially indexed. A page that cannot be crawled has a hard time being found by search engines.

By providing a logical link structure, it becomes easier for search engines to crawl all the important pages.