How do I get a list of every page URL on a website?

If the site publishes XML sitemaps — and most sites of any size do — you do not have to crawl it. The list already exists, the site maintains it for search engines, and reading it takes seconds instead of hours.

The steps are always the same: fetch robots.txt and look for its Sitemap: lines; if there are none, try the common paths (/sitemap.xml, /sitemap_index.xml). What comes back is either a list of URLs or a sitemap index pointing at more sitemap files, which you follow down. Large sitemaps are usually gzipped, so unpack them. Then take every <loc>.

A real run

{
  "startUrls": ["https://nodejs.org"],
  "maxUrls": 2000,
  "flatten": true,
  "includeSitemapReport": true
}

{
  "url": "https://nodejs.org/learn/command-line/accept-input-from-the-command-line-in-nodejs",
  "lastmod": "2026-08-30",
  "changefreq": "weekly",
  "priority": "0.8",
  "sourceSitemap": "https://nodejs.org/learn/sitemap.xml",
  "website": "https://nodejs.org"
}

{
  "website": "https://nodejs.org",
  "recordType": "sitemap-report",
  "sitemapUrl": "https://nodejs.org/learn/sitemap.xml",
  "discoveredVia": "robots.txt",
  "status": "ok",
  "urlCount": 88,
  "note": ""
}

{
  "website": "https://nodejs.org",
  "recordType": "sitemap-report",
  "sitemapUrl": "https://nodejs.org/sitemap.xml",
  "discoveredVia": "robots.txt",
  "status": "ok",
  "urlCount": 1645,
  "note": ""
}

1,645 URLs from /sitemap.xml and 88 from /learn/sitemap.xml, both named in robots.txt — 1,733 rows, one per URL, each carrying the lastmod, changefreq and priority its sitemap declared, and the file it came from. The report rows at the end are what makes a short result explainable: if a crawl comes back with less than you expected, they say which file was read and what it contributed.

When the site has no sitemap

It happens, including on sites you would not expect. The useful answer then is “there is no sitemap” — not an empty list, and not a crash. Same day, same tool, pointed at a site that publishes none:

{
  "website": "https://svelte.dev",
  "recordType": "sitemap-report",
  "status": "no sitemap found",
  "urlCount": 0,
  "note": "robots.txt has no Sitemap: line, and none of the common sitemap paths exist. The site may not publish a sitemap, or it is at a non-standard URL you can pass directly."
}

If you know the sitemap lives somewhere non-standard, pass that URL directly as the start URL and it will be parsed the same way.

Counting a site instead of listing it

“How many pages does this site have?” is a different question, and a cheaper one. With reportOnly: true every sitemap is still read, but no URLs are returned — you get the per-file counts and the total. Nothing is charged for it, because nothing was delivered.

{ "startUrls": ["https://nodejs.org"], "reportOnly": true }

{
  "website": "https://nodejs.org",
  "recordType": "sitemap-report",
  "sitemapUrl": "all files above",
  "status": "total for this website",
  "urlCount": 1733,
  "note": "1733 URL(s) across 2 sitemap file(s). Report only: no URLs were returned and nothing was charged."
}

Filtering, so you pay for the part you want

includeRegex and excludeRegex are applied while reading, so the rows you do not want are never returned and never charged: "includeRegex": "/blog/" gives you the blog and nothing else. lastmodAfter keeps only pages whose sitemap says they changed on or after a date — the cheap way to re-crawl or re-index just what moved.

What this does not do

• It is not a crawler. You get what the site declares. A page that is published but missing from every sitemap will not appear, and no sitemap reader will find it.
• lastmod is a claim, not a measurement. It is whatever the site wrote in the file. Some sites keep it honest, some stamp it with today's date on every deploy. We return it as given and do not second-guess it.
• It does not fetch the pages. No page content, no titles, no status codes — this returns the list. Fetching is the next tool's job.

What it costs

$0.0005 per URL returned: the run above, 1,733 URLs, would cost $0.87. Counting a site costs nothing, a site with no readable sitemap costs nothing, and a run stops at 25,000 URLs, so the worst case is bounded at $12.50 whatever you point it at.