Robots.txt is a plain text format file that is placed in the root folder of your website. This Robots.txt file helps search engines to index your site more appropriately.
With this Robots.txt file, you can tell search engines like Google which pages on your site should be excluded from indexing.