Website URL map and sitemap API (WEB_MAP) - querying.ai

WEB_MAP lists a site’s addresses without reading their content. One task reads the start page for its links, robots.txt for the sitemaps it names (or /sitemap.xml and /sitemap_index.xml when it names none), and those sitemaps. Use it to pick pages for WEB_SCRAPE or to scope a WEB_CRAWL.


`taskType`	`WEB_MAP`
Input	`url` (required); `limit`, `includePaths`, `excludePaths`, `includeSubdomains`, `country`
Credits	1 per task
Country	Defaults to `US`.
Location	Not supported (`422 LOCATION_UNSUPPORTED`).

Request

curl -X POST https://api.querying.ai/v1/async/task \
  -H "Authorization: Bearer $QUERYING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "taskType": "WEB_MAP", "payload": { "url": "https://www.example.com", "limit": 500, "includePaths": ["/blog/**"] } }'

Parameters

Field	Required	Description
`url`	Yes	The site’s start address. Same rules as `WEB_SCRAPE`: public `http` or `https`, default port, up to 2,048 characters.
`limit`	No	How many addresses to return, an integer from 1 to 10,000. Defaults to 1,000.
`includePaths`	No	Keep only addresses whose path matches one of these patterns. Up to 20 patterns, each starting with `/` and at most 200 characters.
`excludePaths`	No	Drop addresses whose path matches one of these patterns. Same limits. `excludePaths` wins over `includePaths`.
`includeSubdomains`	No	`true` adds the site’s subdomains. Defaults to `false`.
`country`	No	The region the site is requested from. Defaults to `US`.

In a pattern, * matches within one path segment and ** matches across segments; every other character matches itself. /blog/** also matches /blog. Paths are matched percent-decoded. The site is the host the start address lands on after redirects, with or without www..

Response

For example:

{
  "url": "https://www.example.com",
  "urls": [
    { "url": "https://www.example.com/", "source": "page" },
    { "url": "https://www.example.com/blog/launch", "source": "sitemap", "lastmod": "2026-10-01" }
  ],
  "sitemaps": ["https://www.example.com/sitemap.xml"],
  "truncated": false
}

Field	Description
`url`	The start address you sent.
`urls`	The start page, the links on it, then sitemap entries. Each address appears once, without fragments or tracking parameters (`utm_*`, `gclid`, `fbclid` and similar).
`urls[].source`	`page` for a link on the start page, `sitemap` for a sitemap entry.
`urls[].lastmod`	The sitemap’s last-modified date, when it gives one.
`sitemaps`	The sitemap files read, in reading order.
`truncated`	`true` when the list stopped before the site’s addresses ran out: at `limit`, at the 50-sitemap-file cap, or at the 90-second time budget.

Limits and errors

The task fails with PAGE_UNAVAILABLE when no address matches includePaths and excludePaths, or when the start page cannot be read and the site lists no sitemap.
Invalid options return 422 VALIDATION_ERROR naming the field.
WEB_MAP ignores markdown, rawResponse and searchQueries. It rejects html and analysis with 422 and citationAnalysis with 400; see unsupported flags.
Other request errors are listed in Errors.

Web crawl to read the pages as Markdown
Web scrape