What is robot.txt?: robots.txt is a text file, created to build a restriction on the website downloader boats of several search engines. You can disallow image crawling, post URL crawling, specific post crawling by this file. [Read more →]