Discover URLs from nested XML sitemaps, then optionally check HTTP status and indexability

Sitemap Scraper & Analyzer reads public sitemap.xml and nested sitemap index files, extracts URL inventories without a headless browser, and reports structure and freshness metrics (top directories, depth distribution, file types, lastmod, changefreq). Optional checkStatus sends HEAD requests to test whether discovered URLs return HTTP 200. This is sitemap XML discovery for technical SEO — not a known URL-list HTTP status pass, and not page HTML extraction.

From $4.00 per 1,000 results

Open Sitemap Scraper & Analyzer on Apify

Nested XML sitemap discovery, not a known URL list and not page HTML

Use this page when the job is to parse sitemap XML (including nested indexes), produce a URL inventory, and optionally HEAD-check availability. Use Bulk URL Status Checker when you already have URLs and need status codes, redirect chains, and timing. Use Website Content Extractor when you already have page URLs and need cleaned docs, pricing, policy, or product HTML.

This Actor Bulk URL Status Checker Website Content Extractor
Intent Discover URLs from nested XML sitemaps; optional HEAD 200 / indexability check HTTP status, redirect-chain, timing, and broken-URL QA on a known list Clean docs, product, pricing, policy, and help-center HTML
Primary input Required sitemapUrls (sitemap.xml or a domain that auto-discovers /sitemap.xml) Known page URLs in urls (max 1000) Known page URLs in urls (max 200)
What it reads Sitemap XML and sitemap index files; optional HEAD on discovered URLs when checkStatus is true Live HTTP(S) responses for those URLs; it does not crawl pages Live page HTML for those URLs
Deliverable URL inventory plus structure and freshness analysis; optional HTTP 200 sampling Status code, final URL, redirect chain, response time, severity Markdown, text, or HTML plus title and optional metadata
Not this job Status QA of a URL list you already have; page-body extraction Discovering URLs from sitemap XML Parsing sitemap.xml to find URLs

The live Store description is: Extract thousands of URLs from complex nested XML sitemaps to audit website structure, HTTP status codes, and indexability for technical SEO. The live schema description is: Parse and analyze sitemap.xml files. Discover URL structure, update patterns, and site architecture. Supports sitemap index files. The published SEO description adds: verify HTTP response codes via HEAD requests, and audit directory depth. Optional status/indexability checking is the checkStatus HEAD-for-200 flag (default false). The live schema does not publish robots, canonical, or noindex fields; those are not this Actor’s sitemap-XML job.

Use cases

README Who/Why table: SEO agencies (sitemap completeness and structure), content strategists (update patterns and stale pages), web developers (verify sitemap structure before search engine submission), and competitive analysts (map competitor site architecture from public sitemaps). Users can schedule runs to monitor website structure changes, feed fresh URLs into search engines, or supply seed lists to other tools as an initial step before heavier article or contact-details extractors.

How is Sitemap Scraper & Analyzer different from Bulk URL Status Checker and Website Content Extractor?

This Actor reads sitemap.xml and nested sitemap index files to discover URL inventories, site structure, and update patterns. Required input is sitemapUrls. Optional checkStatus sends HEAD requests to test whether discovered URLs return HTTP 200. Bulk URL Status Checker checks a known URL list you already have (urls, maximum 1000) for live HTTP status, redirect chains, response timing, and broken-URL findings; it does not crawl pages and it does not parse sitemap XML to find URLs. Website Content Extractor fetches known page URLs (maximum 200) and returns cleaned markdown, text, or HTML; it does not parse sitemaps. Use this Actor to discover URLs from nested XML sitemaps, then optionally check status and indexability. Use Bulk URL Status Checker when the URL list is already known. Use Website Content Extractor when you need page body content.

What input is required?

sitemapUrls is required: URLs of sitemap.xml files to analyze. Auto-discovers /sitemap.xml if you provide just a domain. Schema editor is stringList. Schema prefill is https://github.com/sitemap.xml and https://example.com. Optional live schema fields are maxUrls (integer, default 5000, minimum 100, maximum 50000; maximum URLs to process from each sitemap), checkStatus (boolean, default false; send HEAD requests to check if URLs return 200), maxChargeUsd (number, default 1, minimum 0, maximum 100), delivery (dataset or webhook, default dataset), webhookUrl (used when delivery is webhook; Slack, Discord, or any HTTP endpoint), concurrency (integer, default 3, minimum 1, maximum 10), and dryRun (boolean, default false). additionalProperties is false. The live schema description is: Parse and analyze sitemap.xml files. Discover URL structure, update patterns, and site architecture. Supports sitemap index files.

Field Type Default Notes
sitemapUrls array required (prefill https://github.com/sitemap.xml, https://example.com) Sitemap.xml URLs, or a domain that auto-discovers /sitemap.xml
maxUrls integer 5000 Maximum URLs to process from each sitemap. Schema min 100, max 50000.
checkStatus boolean false HEAD requests to check if URLs return 200. Slower; finds dead links.
maxChargeUsd number 1 Safety cap (min 0, max 100). Results beyond the cap stay as no-charge limit_reached rows.
delivery string dataset dataset or webhook
webhookUrl string — Used when delivery is webhook. Slack, Discord, or any HTTP endpoint.
concurrency integer 3 Parallel requests. Schema min 1, max 10. Higher may trigger rate limits.
dryRun boolean false Run without saving results or sending webhooks

Published README Input Example (matches the schema):

{
  "sitemapUrls": ["https://apify.com/sitemap.xml"],
  "maxUrls": 5000,
  "checkStatus": false,
  "concurrency": 3
}

Published Store example run input:

{
  "sitemapUrls": [
    "https://apify.com/sitemap.xml"
  ],
  "maxUrls": 100,
  "checkStatus": false,
  "concurrency": 3,
  "delivery": "dataset",
  "dryRun": false
}

Key Features from the README: nested sitemap index support; structure analysis (top directories, depth distribution, file extensions); update pattern detection (lastmod distribution, changefreq analysis); optional HEAD sampling for dead links; architecture insights from the sitemap alone; bulk processing of multiple sitemaps per run. README Tips: keep concurrency ≤ 5 when auditing production sites; enable dryRun before a paid schedule; run a tiny URL count first, then scale. The published FAQ states requests use a standard User-Agent and honor site rate limits, and that an unreachable input URL records an error row while other URLs keep processing.

Run Sitemap Scraper & Analyzer on Apify

Which README input examples are not live schema fields?

README examples labeled Single domain audit, Bulk competitor sitemap snapshot, and Lastmod-only diff for drift send domains, expandIndexes, onlyLastmodChanged, and sinceDays. Those names are not in the live input schema. additionalProperties is false, so do not send them. The schema-valid README Input Example is sitemapUrls, maxUrls, checkStatus, and concurrency. The published Store example run input is sitemapUrls, maxUrls, checkStatus, concurrency, delivery, and dryRun. maxChargeUsd is in the live schema and is omitted from that README input table and from the published Store example run input.

What does a result contain?

The published README output table lists meta, results, and per-result sitemapUrl, finalUrl, status, analysis, error, and checkedAt. The published README sample is one sitemap object (https://apify.com/sitemap.xml) with status ok and an analysis object: type urlset, totalUrls 1247, structure.topDirectories with path/count/percentage, structure.depthDistribution, and updateFrequency.lastModRange plus urlsWithLastmod. finalUrl, error, and checkedAt are in the field table and not shown on that sample. The sample is a README illustration, not a live coverage guarantee. There is no separate published dataset/output JSON schema on the Store page beyond that README table and sample.

Published README output example:

{
  "sitemapUrl": "https://apify.com/sitemap.xml",
  "status": "ok",
  "analysis": {
    "type": "urlset",
    "totalUrls": 1247,
    "structure": {
      "topDirectories": [
        { "path": "/store", "count": 890, "percentage": 71 },
        { "path": "/blog", "count": 156, "percentage": 13 }
      ],
      "depthDistribution": { "1": 45, "2": 890, "3": 312 }
    },
    "updateFrequency": {
      "lastModRange": { "oldest": "2023-01-15", "newest": "2026-02-20" },
      "urlsWithLastmod": 1100
    }
  }
}

The README positions this as the sitemap-XML step before heavier extractors. Related Store Actors named in the README (not extra landings on this site) include Bulk URL Status Checker for a known URL list, Meta Tag Analyzer for head metadata, and Website Content Extractor / Article Content Extractor / Contact Details Extractor as downstream seed targets.

How do dataset, webhook, dry-run, and the charge cap work?

delivery defaults to dataset (saves to Apify Dataset; recommended). Set delivery to webhook and provide webhookUrl to send results to a URL. webhookUrl works with Slack, Discord, or any HTTP endpoint. dryRun true runs without saving results or sending webhooks, for testing. maxChargeUsd defaults to 1 as a safety cap; results beyond the cap are kept in output as no-charge limit_reached rows. Store Quickstart: start with store-input.example.json to analyze one public sitemap with a small URL cap; then store-input.templates.json with Quickstart (Dataset) for a fast structural audit, Large Site Audit for deeper coverage and status checks, or Webhook Alert for change-driven monitoring.

README Tips also: use webhook delivery for recurring cron runs; results are dataset-first; use Apify API run-sync-get-dataset-items for JSON in CI. Recurring runs can use Apify Schedules (hourly, daily, weekly) combined with webhook delivery.

How is Sitemap Scraper & Analyzer priced?

Billing is pay per event. The published Store card is from $4.00 / 1,000 results. Live Store events are Actor Start at $0.001 (charged when the Actor starts; number of events depends on Actor memory, one event per GB, minimum one event) and result at $0.004 (single result in the default dataset; the pricing page lists Result as $4.00 / 1,000). Platform usage is listed as included on the pricing page. The README Cost section still quotes actor-start $0.01 plus dataset-item $0.003 ($3.01 for 1,000 items); live Store billing currently uses the $4.00 / 1,000 card and the $0.001 / $0.004 events.

From $4.00 per 1,000 results ($0.004 per default-dataset result; Actor Start $0.001)

See Sitemap Scraper & Analyzer pricing on Apify

Limits to keep in mind

Open Sitemap Scraper & Analyzer on Apify

Related pages