Robots.txt

What robots.txt is

Robots.txt is a small text file that sits at the root of a website at the address example.com/robots.txt. It’s the first thing a search engine looks for when it visits a site, and it tells the robots which parts of the site they may crawl, and which they should stay away from. The file controls access, not indexing, and that difference is worth keeping straight.

A typical robots.txt looks like this:

User-agent: *
Disallow: /cart/
Disallow: /my-account/
Allow: /

Sitemap: https://example.com/sitemap-index.xml

Here’s what the lines mean. User-agent: * applies to all robots. Disallow blocks a folder or path. Allow grants access, even inside an otherwise blocked folder. And Sitemap points the search engine to your sitemap, which is a good habit to keep.

How to check your own robots.txt

1. Type your address followed by /robots.txt into the browser, for example yourshop.com/robots.txt.
2. Read what’s listed under Disallow. If something important is blocked, for example an entire product folder, that could be why those pages aren’t being found.
3. Test individual URLs in Google Search Console, which has a robots.txt tool that shows whether a given page is blocked.

The most important pitfall

A Disallow in robots.txt prevents crawling, but not necessarily indexing. If other links point to the page, Google may still show it in the results without a description, because it isn’t allowed to read the content. If you want to keep a page completely out of search results, the right tool is a noindex meta tag on the page itself, not a Disallow in robots.txt. A classic mistake is blocking a page in robots.txt while also setting noindex on it. In that case the robot never gets to read the noindex, because it’s blocked from entering in the first place. The two shouldn’t be used together.