POST /api/webparser/sitemap
Price: 1 credit
Fetch URLs from website sitemap
Discover and extract all URLs from a website sitemap. Supports include/exclude pattern filters. Some sites block sitemap access (e.g. reddit.com) — returns 0 URLs in that case.
access-token string requiredtimeout integer — Max scrapping execution timeout (in seconds) (default: 300; min: 20; max: 1500)url string required — Website URL to fetch sitemap from (format: uri; examples: "https://www.example.com", "https://blog.example.com"; minLength: 1; maxLength: 2083)include_patterns array nullable — Regex patterns for URL paths to include (all URLs if not specified) (examples: ["^/blog/","^/docs/"])exclude_patterns array nullable — Regex patterns for URL paths to exclude (examples: ["\\.pdf$","/admin/"])same_host_only boolean — Only include URLs from the same host as the base URL (default: true)respect_robots boolean — Check robots.txt and respect disallowed URLs (default: true)count integer nullable — Maximum number of URLs to return (min: 1)return_details boolean — Return detailed sitemap entries (lastmod, changefreq, priority) instead of just URLs (default: false)@type string (default: "SitemapResult")urls array required — List of URLs found in sitemap(s)entries array nullable — Detailed sitemap entries (if return_details=True)@type string (default: "SitemapEntry")loc string required — URL of the pagelastmod string nullable — Last modification date (ISO 8601 format)changefreq string nullable — Change frequency (always, hourly, daily, weekly, monthly, yearly, never)priority number nullable — Priority (0.0-1.0) (min: 0; max: 1)total_found integer required — Total number of URLs foundsitemap_locations array required — Sitemap URLs that were processed422 — The request body did not validate Check the fields against this schema. A URN with the wrong prefix is the most common cause.408 — The request ran past its time limit Raise `timeout` in the request body, up to the maximum this endpoint documents. Lowering `count` or turning off the `with_*` flags also helps, because less work finishes sooner.412 — No sitemap found for the given URL. Retrying will not help: either the entity does not exist, or the input points at a different one.429 — Too many requests: a rate limit or a usage window is exhausted When the response carries an X-Retry-After header, wait that many seconds and retry: the same number is in the body as `detail.retry_after`, and the limit clears once that window passes. The message in the body names the limit that was hit.500 — Something broke on our side Retrying will not help. If it keeps happening, send us the X-Request-ID from the response headers.529 — Rate limit reached, or the endpoint is overloaded Wait at least 30 seconds, then retry.X-Error — Error message text (present only on error)X-Request-ID — Unique request identifierX-Execution-Time — Execution time in secondsX-Result-Count — How many records the body carries. 0 means an empty result, which is a normal answer and not by itself an error. A non-zero count can come back together with X-Error when the failure happened partway through — read this header and X-Error independently.X-Total-Available-Results — How many records exist for this query, when the endpoint can say. On a `dry_run` request this is the answer and the body is empty. It saturates: the endpoint's documented maximum means 'at least that many', any smaller number is exact.X-Warning — Present when the request body carried keys this endpoint does not document. They were ignored, so any filter you meant to apply through them did not apply. Check the spelling against this schema and retry.X-Retry-After — Seconds to wait before retrying. Present only on 429.