An XML sitemap is written not for people but for crawlers: programs that fetch web pages automatically, including the crawlers AI systems rely on for content. Crawlers usually discover pages through links, both within a website and from other websites. A sitemap supplements this: crawlers that support the format can pick up every URL it lists without having to find it through links first.
How an XML sitemap is structured
The format is defined in the Sitemap protocol on sitemaps.org and is supported by Google and Microsoft, among others. The file is written in XML, a markup language in which every piece of information sits between named markers called tags. The protocol recommends placing the file directly under the domain, for example at /sitemap.xml. For each page, the file contains an entry; its two most important pieces of information are:
- loc: the page’s full address, starting with the transfer protocol (usually https) and the domain. This is required.
- lastmod: the date of the last change, for example 2026-09-14 for September 14, 2026. This is optional.
A complete entry looks like this, with “page address” standing for the full address: “<url><loc>page address</loc><lastmod>2026-09-14</lastmod></url>”. All entries sit inside one enclosing tag called urlset. A single file may contain at most 50,000 URLs and be no larger than 50 MB uncompressed; larger websites split their URLs across several sitemaps and list those in a sitemap index file.
Which URLs and dates belong in it
In Bing’s Webmaster Guidelines, Microsoft advises listing only canonical URLs. The canonical URL is the one preferred address of a page when the same content can be reached under several addresses, for example with and without “www” or with added parameters. According to Microsoft, the sitemap should also reflect the website’s current structure, and deleted or redirected URLs should be removed promptly.
Under the protocol, the lastmod date must state when the page itself last changed, not when the sitemap was generated. Microsoft recommends giving both date and time in ISO 8601 format, the international standard for dates and times, for example 2026-09-14T09:30:00+02:00, where +02:00 is the offset from Coordinated Universal Time (UTC). According to Google, lastmod should reflect the last significant update to a page; a change to the main content, structured data, or links, for example, generally counts as significant, but an updated copyright year does not. Google says it uses lastmod only if the value is consistently and verifiably accurate. If a content management system (CMS), the software used to run a website, stamps every page with the current date each time it generates the sitemap, lastmod therefore sends no useful signal. How a page shows its publication and update dates to people and machines is covered under content freshness.
How crawlers find the sitemap
Microsoft names two ways to make a sitemap known. The first is a line in robots.txt: “Sitemap:” followed by the full address of the sitemap file. The line is not one of robots.txt’s own access rules but comes from the Sitemap protocol; the robots.txt standard, RFC 9309, explicitly allows crawlers to interpret such additional lines. It is not tied to the group of rules for any particular crawler, can appear anywhere in the file, and can be repeated for several sitemaps. The second way is to submit the sitemap in Bing Webmaster Tools, Microsoft’s tool for website owners.
Alongside the sitemap, Microsoft recommends IndexNow, a protocol a website can use to notify Bing and other participating services immediately about individual URLs that were added, updated, or removed.
What an XML sitemap means for AI answers
Microsoft explicitly connects sitemaps with AI answers. In a July 2025 post on the Bing Webmaster Blog, Microsoft writes that freshness signals influence how quickly updates are reflected in AI-generated answers. According to the post, the lastmod date helps Bing decide which URLs to fetch again and which to skip because their content hasn’t changed. Bing’s Webmaster Guidelines, which also cover Copilot, add that proper sitemap use helps Bing keep its index accurate and up to date and helps pages stay reliably eligible as a basis for AI answers (grounding). According to Microsoft, a sitemap does not guarantee visibility.
Google also uses XML sitemaps to discover pages but does not describe a specific connection to its AI features. OpenAI, Anthropic, and Perplexity say nothing about sitemaps in the documentation of their crawlers. A sitemap only announces pages: whether a crawler may and can fetch a listed page still depends on robots.txt, the server, and other factors that together make up a website’s crawlability. Since many CMSs generate the file automatically, providing it with accurate dates and naming it in robots.txt usually takes little effort.