FeaturePOST /v1/crawl

Crawl, DNS & Robots

Discover pages, read DNS, check robots — discovery for inventories and infra checks.

What you can do with crawl, DNS, and robots

Discover pages from a start URL, look up DNS records, or check robots permission — the discovery family for inventories and SEO/infra checks.

Crawl with depth/pages/same-origin limits
DNS records (A, AAAA, TXT, MX, …)
robots.txt: allowed + matched_rule + sitemaps
discovered_urls / pages for downstream processing

API Call

editable

Request body

{'{'}

{'}'}

Crawl: adjust depth, pages, and same_origin — DNS/robots have their own fields.

Response

Discovered URLs

—

Details and raw data in the Response tab.

Endpoints & parameters

Choose an endpoint — below you'll see an example request and explanations for all relevant parameters.

POST /v1/crawl

Link discovery from start_url.

Example request

json

{
  "start_url": "https://example.com",
  "max_depth": 1,
  "max_pages": 10,
  "same_origin": true,
  "seed_from_sitemap": false,
  "respect_robots": true,
  "settle_ms": 250
}

Parameters

start_urlstring required
Starting point for discovery.
max_depthnumber 0–5
How deep to follow links.
max_pagesnumber 1–100
Upper limit of discovered pages.
same_originboolean
Only links from the same origin.
allow_hostsstring[]
Additionally allowed hosts (≤20).
seed_from_sitemapboolean
Use sitemap as seed.
settle_ms / goto_timeout_ms / respect_robotsvarious
Timing and robots.