Robots.txt

A text file used to instruct web robots (like search engine crawlers) how to crawl pages on their website.