A robot.txt file is essential in helping search engine robots navigate through your site. This file tells the robot where it should and should not look on your web server.
After your home page the robot file is usually the most accessed on your web server. All search engine robots try to access this file before they crawl your site.
This file is very impotant in search engine optimisation and can help to train search engine robots when to access your site i.e. tell them how often you do updates or add new pages.
Lastly, there are many files and pages on your site that you may never want to be access or indexed by search engines. The robots.txt file can tell the robot what not to index too!
Don't have one? Contact us and we will be glad to help and advise you.
Click to go back to Elite Business Solutions' Blog