/webparser/parse

POST /api/webparser/parse

Price: 1 credit

Parse and clean HTML from web page

How to use it

Parse and extract clean content from any web page URL. Supports minimal extraction and tag filtering. Does NOT support JavaScript-rendered SPAs (e.g. platform.openai.com, Vercel/Next.js apps) — these return empty content. Works best with static HTML pages, blogs, docs. For JavaScript-heavy pages use /webparser/render instead.

Parameters

• access-token string required

Request body

• timeout integer — Max scrapping execution timeout (in seconds) (default: 300; min: 20; max: 1500)
• url string required — URL of the page to parse (format: uri; examples: "https://www.example.com", "https://blog.example.com/article"; minLength: 10; maxLength: 2083)
• include_tags array nullable — CSS selectors of elements to include (keep only these) (examples: ["article",".content","#main-content"])
• exclude_tags array nullable — CSS selectors or wildcard masks of elements to exclude. Examples: '.sidebar', '.advertisement', '*promo*', '*banner*' (examples: [".sidebar",".advertisement","*promo*"])
• only_main_content boolean — Extract only main content of the page (heuristic algorithm) (default: false)
• remove_comments boolean — Remove HTML comments (default: true)
• resolve_srcset boolean — Convert image srcset to src (selects the largest image) (default: true)
• return_full_html boolean — Return full HTML document (True) or only body content (False) (default: false)
• min_text_block integer — Minimum text block size for main content detection (in characters) (default: 200; min: 0)
• remove_base64_images boolean — Remove base64-encoded images (reduces output size) (default: true)
• strip_all_tags boolean — Remove all HTML tags and return plain text only (default: false)
• extract_contacts boolean — Extract links, emails, and phone numbers from the page (default: false)
• same_origin_links boolean — Only extract links from the same domain (used with extract_contacts) (default: false)
• social_links_only boolean — Only extract social media links (LinkedIn, Twitter/X, Facebook, Instagram, etc.) (default: false)
• extract_minimal boolean — Use minimal extraction (only links, title, emails, phones if set) (default: false)

Response

• @type string (default: "WebParserResult")
• cleaned_html string required — Cleaned HTML
• url string required — URL of the original page
• meta_description string nullable — Meta description
• metadata object — Additional metadata
• links array nullable — Extracted URLs from the page
• emails array nullable — Extracted email addresses
• phones array nullable — Extracted phone numbers

Errors

• 422 — The request body did not validate Check the fields against this schema. A URN with the wrong prefix is the most common cause.
• 408 — The request ran past its time limit Raise `timeout` in the request body, up to the maximum this endpoint documents. Lowering `count` or turning off the `with_*` flags also helps, because less work finishes sooner.
• 412 — Failed to fetch URL or page not found. Retrying will not help: either the entity does not exist, or the input points at a different one.
• 415 — The URL serves a document rather than a web page (PDF, Office file, image, CSV, archive).
• 429 — Too many requests: a rate limit or a usage window is exhausted When the response carries an X-Retry-After header, wait that many seconds and retry: the same number is in the body as `detail.retry_after`, and the limit clears once that window passes. The message in the body names the limit that was hit.
• 500 — Something broke on our side Retrying will not help. If it keeps happening, send us the X-Request-ID from the response headers.
• 529 — Rate limit reached, or the endpoint is overloaded Wait at least 30 seconds, then retry.

Response headers

• X-Error — Error message text (present only on error)
• X-Request-ID — Unique request identifier
• X-Execution-Time — Execution time in seconds
• X-Result-Count — How many records the body carries. 0 means an empty result, which is a normal answer and not by itself an error. A non-zero count can come back together with X-Error when the failure happened partway through — read this header and X-Error independently.
• X-Total-Available-Results — How many records exist for this query, when the endpoint can say. On a `dry_run` request this is the answer and the body is empty. It saturates: the endpoint's documented maximum means 'at least that many', any smaller number is exact.
• X-Warning — Present when the request body carried keys this endpoint does not document. They were ignored, so any filter you meant to apply through them did not apply. Check the spelling against this schema and retry.
• X-Retry-After — Seconds to wait before retrying. Present only on 429.