> ## Documentation Index
> Fetch the complete documentation index at: https://docs.querying.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Website URL map and sitemap API (WEB_MAP)

> List a site's addresses from its start page links and its sitemaps, filtered by path.

`WEB_MAP` lists a site's addresses without reading their content. One task reads the start page for its links, `robots.txt` for the sitemaps it names (or `/sitemap.xml` and `/sitemap_index.xml` when it names none), and those sitemaps. Use it to pick pages for [`WEB_SCRAPE`](/engines/web-scrape) or to scope a [`WEB_CRAWL`](/engines/web-crawl).

| | |
| - | - |
| `taskType` | `WEB_MAP` |
| Input | `url` (required); `limit`, `includePaths`, `excludePaths`, `includeSubdomains`, `country` |
| Credits | 1 per task |
| Country | Defaults to `US`. |
| Location | Not supported (`422 LOCATION_UNSUPPORTED`). |

## Request

```bash theme={"theme":"css-variables"}
curl -X POST https://api.querying.ai/v1/async/task \
  -H "Authorization: Bearer $QUERYING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "taskType": "WEB_MAP", "payload": { "url": "https://www.example.com", "limit": 500, "includePaths": ["/blog/**"] } }'
```

## Parameters

| Field | Required | Description |
| - | - | - |
| `url` | Yes | The site's start address. Same rules as `WEB_SCRAPE`: public `http` or `https`, default port, up to 2,048 characters. |
| `limit` | No | How many addresses to return, an integer from 1 to 10,000. Defaults to 1,000. |
| `includePaths` | No | Keep only addresses whose path matches one of these patterns. Up to 20 patterns, each starting with `/` and at most 200 characters. |
| `excludePaths` | No | Drop addresses whose path matches one of these patterns. Same limits. `excludePaths` wins over `includePaths`. |
| `includeSubdomains` | No | `true` adds the site's subdomains. Defaults to `false`. |
| `country` | No | The region the site is requested from. Defaults to `US`. |

In a pattern, `*` matches within one path segment and `**` matches across segments; every other character matches itself. `/blog/**` also matches `/blog`. Paths are matched percent-decoded.

The site is the host the start address lands on after redirects, with or without `www.`.

## Response

For example:

```json theme={"theme":"css-variables"}
{
  "url": "https://www.example.com",
  "urls": [
    { "url": "https://www.example.com/", "source": "page" },
    { "url": "https://www.example.com/blog/launch", "source": "sitemap", "lastmod": "2026-10-01" }
  ],
  "sitemaps": ["https://www.example.com/sitemap.xml"],
  "truncated": false
}
```

| Field | Description |
| - | - |
| `url` | The start address you sent. |
| `urls` | The start page, the links on it, then sitemap entries. Each address appears once, without fragments or tracking parameters (`utm_*`, `gclid`, `fbclid` and similar). |
| `urls[].source` | `page` for a link on the start page, `sitemap` for a sitemap entry. |
| `urls[].lastmod` | The sitemap's last-modified date, when it gives one. |
| `sitemaps` | The sitemap files read, in reading order. |
| `truncated` | `true` when the list stopped before the site's addresses ran out: at `limit`, at the 50-sitemap-file cap, or at the 90-second time budget. |

## Limits and errors

* The task fails with `PAGE_UNAVAILABLE` when no address matches `includePaths` and `excludePaths`, or when the start page cannot be read and the site lists no sitemap.
* Invalid options return `422 VALIDATION_ERROR` naming the field.
* `WEB_MAP` ignores `markdown`, `rawResponse` and `searchQueries`. It rejects `html` and `analysis` with `422` and `citationAnalysis` with `400`; see [unsupported flags](/engines/response-options.md#unsupported-flags).
* Other request errors are listed in [Errors](/concepts/errors.md).

## Related

* [Web crawl](/engines/web-crawl.md) to read the pages as Markdown
* [Web scrape](/engines/web-scrape.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.