This file is important for both SEO purposes as well as to protect sensitive information from being indexed by bots.
A robots.txt file is not necessary for every website, but it can be helpful in cases where you want to block access to certain parts of your site, such as login pages that require a user name and password.
This section will cover the best practices of using Robots.txt files, and provide you with some tips for how you might use them on your website. If you are unfamiliar with robots.txt files, this section will help to put them into perspective.
The first step of using a robots.txt file is deciding which pages on your website you would like the web crawler to interact with, and what they can do when they visit those pages (if anything).
By following these guidelines, you have given the web crawler a "foot in the door," so to speak. This is where robots.txt files are helpful because they provide you with a list of permissions that control how your website is crawled by bots.
The default permissions found in most robots.txt files include prohibiting all crawling and indexing, as well as allowing indexing of certain pages like
1) The bot's name.
2) The bot's URL or command line parameters.
3) The bot's description.
The 3 things that you must include in your bots txt file are the bot’s name, the bot’s URL or command line parameters, and the bot’s description.
The robot's txt file is not just a tool for webmasters, but it also helps with SEO in the sense that it provides instructions to Google's crawler which will then follow those instructions when crawling your site.