robots.txt and Sitemap Explained: The Foundation for Google

robots.txt and sitemap are two simple text files that have a big impact on how Google crawls and understands your website. Both are among the first things a basic SEO check should look at.

What does robots.txt do?

This file at your-domain.com/robots.txt tells search engine crawlers which parts of the website they're allowed to visit and which they aren't.

A typical example

• User-agent: *
• Allow: /
• Disallow: /admin/
• Disallow: /internal/
• Sitemap: https://your-domain.com/sitemap.xml

This allows crawlers everywhere except the admin and internal sections, while also pointing to the sitemap.

The most dangerous mistake

"Disallow: /" with nothing else blocks the entire website from every search engine. This sometimes happens by accident when a development setting gets carried over to production.

What does the sitemap do?

The XML sitemap lists all your important pages in one place - including the last modified date. This helps Google find new or changed content much faster instead of having to discover it purely through links.

How they work together

robots.txt controls what may be crawled. The sitemap shows what should be crawled. Without a robots.txt block, Google usually finds the sitemap automatically anyway - but it's still worth referencing it directly in robots.txt and additionally submitting it in Google Search Console.

Check it yourself in seconds

Just open your-domain.com/robots.txt and your-domain.com/sitemap.xml in your browser. An error message or blank page means one of the two files is missing.