A text file used to instruct web robots (like search engine crawlers) how to crawl pages on their website.
Back to Glossary Index Page