Robots meta noindex vs. robots.txt

Noindex and robots.txt solve two different tasks, even though they are often confused. Robots.txt controls whether a page may be crawled, while noindex controls whether a page may be indexed. A page blocked in robots.txt therefore cannot have its noindex signal read, which can lead to unintended results.

To keep a page entirely out of search results, noindex is generally used rather than blocking it in robots.txt.