Robots.txt file is a file that tells the search engines which pages to access and index on our website and which pages not to by search engine crawlers. Below i have mentioned the basic structure for robots.txt file. (Note: You have to save your file name as robots.txt only)
User-agent: *
Allow: /
Sitemap: www.yourwebsite.com/sitemap.xml
No comments:
Post a Comment