Website crawler to Markdown API (WEB_CRAWL)

WEB_CRAWL follows a site’s links breadth first from a start address and reads each page the way WEB_SCRAPE does. You get every page’s Markdown, metadata and images in one result.


`taskType`	`WEB_CRAWL`
Input	`url` (required); `maxPages`, `maxDepth`, `maxChars`, `includePaths`, `excludePaths`, `includeSubdomains`, `country`
Credits	1 per page returned. The task holds `maxPages` credits at submit.
Country	Defaults to `US`.
Location	Not supported (`422 LOCATION_UNSUPPORTED`).

Request

curl -X POST https://api.querying.ai/v1/async/task \
  -H "Authorization: Bearer $QUERYING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "taskType": "WEB_CRAWL", "payload": { "url": "https://www.example.com/docs", "maxPages": 25, "maxDepth": 2, "includePaths": ["/docs/**"] } }'

Parameters

Field	Required	Description
`url`	Yes	The start address. Same rules as `WEB_SCRAPE`: public `http` or `https`, default port, up to 2,048 characters. The start page is always read, whatever the patterns and `robots.txt` say.
`maxPages`	No	Pages to read, the start page included. An integer from 1 to 100. Defaults to 10.
`maxDepth`	No	Link hops from the start page, an integer from 0 to 5. Defaults to 2. `0` reads the start page alone.
`maxChars`	No	Caps each page’s `markdown`, an integer from 1,000 to 50,000. Defaults to 20,000.
`includePaths`, `excludePaths`	No	Path patterns, as in `WEB_MAP`: up to 20 each, starting with `/`, at most 200 characters. `` matches within one segment, `*` across segments. `excludePaths` wins.
`includeSubdomains`	No	`true` follows links to the site’s subdomains. Defaults to `false`.
`country`	No	The region pages are requested from. Defaults to `US`.

The crawl follows a link when it stays on the site (with or without www.), matches the path patterns, names a web page rather than a file (PDFs, images, archives and other files are skipped), and is allowed by the site’s robots.txt for every crawler.

Response

For example:

{
  "url": "https://www.example.com/docs",
  "pages": [
    {
      "url": "https://www.example.com/docs",
      "finalUrl": "https://www.example.com/docs",
      "depth": 0,
      "statusCode": 200,
      "title": "...",
      "markdown": "# ...",
      "images": [],
      "truncated": false
    }
  ],
  "failed": [{ "url": "https://www.example.com/docs/old", "depth": 1, "error": "the site answered 404 for this page" }],
  "stopReason": "max-pages"
}

Field	Description
`pages`	The pages read, in the order the crawl found them. Each has `WEB_SCRAPE`’s fields plus `depth`, its link hops from the start page.
`failed`	Pages that could not be read, each with `url`, `depth` and the site’s reason as `error`.
`stopReason`	Why the crawl ended. See below.

`stopReason`	Meaning
`done`	No link was left to follow.
`max-pages`	`maxPages` pages were read.
`failures`	10 pages in a row could not be read, or the total number of failed pages reached `maxPages`.
`time`	The crawl’s time ran out. New pages stop starting after about 4 minutes.

Credits

When you submit, the task holds 1 credit per page of maxPages (25 credits in the example above). When it completes, it is charged 1 credit for each page in pages, and the rest of the hold is released. Pages in failed are not charged. A crawl that returns 3 of 25 pages costs 3 credits. See Credits and limits for how holds work.

Limits and errors

When the start page cannot be read, the task fails with PAGE_UNAVAILABLE. Any other page that cannot be read goes to failed and the crawl continues.
Invalid options return 422 VALIDATION_ERROR naming the field.
WEB_CRAWL ignores markdown, rawResponse and searchQueries. It rejects html and analysis with 422 and citationAnalysis with 400; see unsupported flags.
Other request errors are listed in Errors.

Web map to list a site’s addresses without reading them
Web scrape for one page

Get started

Core concepts

Answer engines

Search and web data

Request options

Research

GEO monitoring

Account and data

Website crawler to Markdown API (WEB_CRAWL)

Request

Parameters

Response

Credits

Limits and errors

​Request

​Parameters

​Response

​Credits

​Limits and errors

​Related

Request

Parameters

Response

Credits

Limits and errors

Related