> ## Documentation Index
> Fetch the complete documentation index at: https://docs.querying.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Website crawler to Markdown API (WEB_CRAWL)

> Crawl a site from a start address and get each page as Markdown, billed per page returned.

`WEB_CRAWL` follows a site's links breadth first from a start address and reads each page the way [`WEB_SCRAPE`](/engines/web-scrape) does. You get every page's Markdown, metadata and images in one result.

| | |
| - | - |
| `taskType` | `WEB_CRAWL` |
| Input | `url` (required); `maxPages`, `maxDepth`, `maxChars`, `includePaths`, `excludePaths`, `includeSubdomains`, `country` |
| Credits | 1 per page returned. The task holds `maxPages` credits at submit. |
| Country | Defaults to `US`. |
| Location | Not supported (`422 LOCATION_UNSUPPORTED`). |

## Request

```bash theme={"theme":"css-variables"}
curl -X POST https://api.querying.ai/v1/async/task \
  -H "Authorization: Bearer $QUERYING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "taskType": "WEB_CRAWL", "payload": { "url": "https://www.example.com/docs", "maxPages": 25, "maxDepth": 2, "includePaths": ["/docs/**"] } }'
```

## Parameters

| Field | Required | Description |
| - | - | - |
| `url` | Yes | The start address. Same rules as `WEB_SCRAPE`: public `http` or `https`, default port, up to 2,048 characters. The start page is always read, whatever the patterns and `robots.txt` say. |
| `maxPages` | No | Pages to read, the start page included. An integer from 1 to 100. Defaults to 10. |
| `maxDepth` | No | Link hops from the start page, an integer from 0 to 5. Defaults to 2. `0` reads the start page alone. |
| `maxChars` | No | Caps each page's `markdown`, an integer from 1,000 to 50,000. Defaults to 20,000. |
| `includePaths`, `excludePaths` | No | Path patterns, as in [`WEB_MAP`](/engines/web-map): up to 20 each, starting with `/`, at most 200 characters. `*` matches within one segment, `**` across segments. `excludePaths` wins. |
| `includeSubdomains` | No | `true` follows links to the site's subdomains. Defaults to `false`. |
| `country` | No | The region pages are requested from. Defaults to `US`. |

The crawl follows a link when it stays on the site (with or without `www.`), matches the path patterns, names a web page rather than a file (PDFs, images, archives and other files are skipped), and is allowed by the site's `robots.txt` for every crawler.

## Response

For example:

```json theme={"theme":"css-variables"}
{
  "url": "https://www.example.com/docs",
  "pages": [
    {
      "url": "https://www.example.com/docs",
      "finalUrl": "https://www.example.com/docs",
      "depth": 0,
      "statusCode": 200,
      "title": "...",
      "markdown": "# ...",
      "images": [],
      "truncated": false
    }
  ],
  "failed": [{ "url": "https://www.example.com/docs/old", "depth": 1, "error": "the site answered 404 for this page" }],
  "stopReason": "max-pages"
}
```

| Field | Description |
| - | - |
| `pages` | The pages read, in the order the crawl found them. Each has `WEB_SCRAPE`'s fields plus `depth`, its link hops from the start page. |
| `failed` | Pages that could not be read, each with `url`, `depth` and the site's reason as `error`. |
| `stopReason` | Why the crawl ended. See below. |

| `stopReason` | Meaning |
| - | - |
| `done` | No link was left to follow. |
| `max-pages` | `maxPages` pages were read. |
| `failures` | 10 pages in a row could not be read, or the total number of failed pages reached `maxPages`. |
| `time` | The crawl's time ran out. New pages stop starting after about 4 minutes. |

## Credits

When you submit, the task holds 1 credit per page of `maxPages` (25 credits in the example above). When it completes, it is charged 1 credit for each page in `pages`, and the rest of the hold is released. Pages in `failed` are not charged. A crawl that returns 3 of 25 pages costs 3 credits. See [Credits and limits](/concepts/credits-and-limits.md) for how holds work.

## Limits and errors

* When the start page cannot be read, the task fails with `PAGE_UNAVAILABLE`. Any other page that cannot be read goes to `failed` and the crawl continues.
* Invalid options return `422 VALIDATION_ERROR` naming the field.
* `WEB_CRAWL` ignores `markdown`, `rawResponse` and `searchQueries`. It rejects `html` and `analysis` with `422` and `citationAnalysis` with `400`; see [unsupported flags](/engines/response-options.md#unsupported-flags).
* Other request errors are listed in [Errors](/concepts/errors.md).

## Related

* [Web map](/engines/web-map.md) to list a site's addresses without reading them
* [Web scrape](/engines/web-scrape.md) for one page


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.