XML Sitemaps and robots.txt in WordPress

A sitemap is a list of URLs you want search engines to know about, with optional metadata about when each changed. It helps discovery, particularly for pages with few internal links, and it gives you a way to see in Search Console how many of your submitted pages were actually indexed.

It is not a ranking factor, and being in a sitemap doesn't guarantee indexing. A page with no internal links and thin content won't be indexed just because it's listed.

Its diagnostic value is the underrated part. When Search Console says you submitted 400 URLs and 120 are indexed, that gap is a question worth answering — and you can only ask it if the sitemap contains what you think it does.

• Canonical URLs only. If a page canonicalises to another URL, the canonical one goes in the sitemap and the other doesn't.
• Indexable pages only. Anything with a noindex must not be listed — that's a contradictory instruction.
• URLs returning 200. No redirects, no 404s.
• Pages you'd be happy for someone to land on from a search result.
• Accurate lastmod dates. Falsifying them so everything looks freshly updated teaches search engines to ignore the field.

Both core's sitemaps and the major SEO plugins generate entries for everything by default, and 'everything' is more than most sites want indexed.

• Tag archives with one or two posts each. Thin pages that compete with the posts themselves.
• Author archives on a single-author site, which duplicate the blog index.
• Date archives, which are almost never useful as landing pages.
• Attachment pages — a page per uploaded image, containing an image and nothing else. These should be disabled or redirected, not indexed.
• Custom post types registered by plugins for internal use: form entries, popups, layout blocks, slider items.
• Paginated archive pages beyond the first, depending on how your plugin handles them.
• The test for each: would you be happy for a search visitor to land there? If not, exclude it from the sitemap and noindex it.

These are confused constantly and the distinction has real consequences.

robots.txt controls crawling. A disallowed URL isn't fetched. But Google can still index it — from links elsewhere — and show it in results with no description, because it was never allowed to read the page.

noindex controls indexing. The page is fetched, the directive is read, and the page is kept out of the index.

The critical consequence: if you block a page in robots.txt, Google can never read its noindex tag. Blocking and noindexing the same page means the noindex is invisible and the page may stay indexed indefinitely. To remove something from the index, allow crawling and use noindex.

• Stopping crawlers wasting time on genuinely useless paths — faceted filter combinations, internal search results, infinite calendar pages.
• Pointing at your sitemap with a Sitemap: line.
• Controlling specific crawlers that misbehave, though the well-behaved ones are rarely the problem.
• It is not for hiding anything. robots.txt is public, and listing a path you want kept private is an advertisement.

Old advice, still repeated widely, says to block /wp-content/ and /wp-includes/. Don't. Google needs your CSS and JavaScript to render the page and assess it — blocking them means Google sees an unstyled, broken version of your site.

WordPress generates a virtual robots.txt if no physical file exists, and it's sensible by default. It also adds a site-wide Disallow when Settings → Reading → 'Discourage search engines' is ticked, which is worth knowing because that checkbox is the single most common cause of a site not being indexed at all.

If you need a custom robots.txt, either use your SEO plugin's editor or place a physical file — but be aware a physical file overrides the virtual one entirely, including the Sitemap line WordPress would have added.

• A sitemap file can hold 50,000 URLs and must be under 50MB uncompressed. Past that you need multiple files and an index.
• Split by type — posts, pages, products, categories — which makes the Search Console coverage report far more useful because you can see which type isn't being indexed.
• Keep the index file at a predictable URL and reference it from robots.txt.
• Regenerate on publish rather than on a schedule, so new content is discoverable immediately.
• Consider an image sitemap for an image-heavy site, and a news sitemap only if you actually qualify for Google News.

Submit the sitemap index in Search Console and Bing Webmaster Tools once. After that the useful work is monitoring rather than resubmitting.

The Pages report tells you how many submitted URLs are indexed and, more usefully, why the others aren't: 'Crawled - currently not indexed', 'Discovered - currently not indexed', 'Excluded by noindex tag', 'Duplicate without user-selected canonical'. Each of those is a different problem with a different fix.

A large gap between submitted and indexed usually means either thin content, or a sitemap listing pages that were never meant to be indexed. Both are worth fixing, and the second is easier.

• Open your sitemap index and look at what's in it. Is every post type there one you want indexed?
• Spot-check a few URLs from each sitemap file: do they return 200, are they canonical, do they lack a noindex?
• Open robots.txt. Is anything blocked that shouldn't be, particularly CSS and JavaScript?
• Check the Sitemap: line is present and correct.
• Confirm 'Discourage search engines' is unticked in Settings → Reading.
• Check Search Console's Pages report for the submitted-versus-indexed gap and the reasons given.