Crawl, DNS & Robots

Discover pages, read DNS, check robots — discovery for inventories and infra checks.

Crawl Docs

POST /v1/crawl

Halcyon

halcyon.edu/

0/14

Autumn · 2026

Maps the coast

4.8k

students

colleges

programs

Cartographer

crawl graph · same_origin

seed · 0

Home

Admissions

Colleges

Research

Campus

Library

Apply

Tuition

Harbor

Ashwood

Labs

Open data

Housing

Athletics

0 pagesmax_depth 2

POST /v1/crawl200 · 14 urls

{'{'}
  "seed": "https://halcyon.edu/",
  "same_origin": true,
  "max_depth": 2,
  "pages": 14,
  "urls": [
    {'{'} "url": "/", "depth": 0 {'}'},
    {'{'} "url": "/admissions", "depth": 1 {'}'},
    {'{'} "url": "/colleges", "depth": 1 {'}'},
    {'{'} "url": "/research", "depth": 1 {'}'},
    {'{'} "url": "/campus", "depth": 1 {'}'},
    {'{'} "url": "/library", "depth": 1 {'}'},
    {'{'} "url": "/admissions/apply", "depth": 2 {'}'},
    {'{'} "url": "/admissions/tuition", "depth": 2 {'}'},
    {'{'} "url": "/colleges/harbor", "depth": 2 {'}'},
    {'{'} "url": "/colleges/ashwood", "depth": 2 {'}'},
    {'{'} "url": "/research/labs", "depth": 2 {'}'},
    {'{'} "url": "/research/data", "depth": 2 {'}'},
    {'{'} "url": "/campus/housing", "depth": 2 {'}'},
    {'{'} "url": "/campus/athletics", "depth": 2 {'}'}
  ]
{'}'}

What you can do with crawl, DNS, and robots

Discover pages from a start URL, look up DNS records, or check robots permission — the discovery family for inventories and SEO/infra checks.

Crawl with depth/pages/same-origin limits
DNS records (A, AAAA, TXT, MX, …)
robots.txt: allowed + matched_rule + sitemaps
discovered_urls / pages for downstream processing

API Call

editable

Request bodyJSON

{'{'}

{'}'}

Crawl: adjust depth, pages, and same_origin — DNS/robots have their own fields.

Discovered URLs

—

Run crawl, DNS, or robots to see the live payload.

Endpoints & parameters

Pick an endpoint for a sample request and a clear list of parameters.

POST /v1/crawl

Link discovery from start_url.

Example request

{
  "start_url": "https://example.com",
  "max_depth": 1,
  "max_pages": 10,
  "same_origin": true,
  "seed_from_sitemap": false,
  "respect_robots": true,
  "settle_ms": 250
}

Parameters

Name	Type	Description
`start_url`req	string	Starting point for discovery.
`max_depth`	number 0–5	How deep to follow links.
`max_pages`	number 1–100	Upper limit of discovered pages.
`same_origin`	boolean	Only links from the same origin.
`allow_hosts`	string[]	Additionally allowed hosts (≤20).
`seed_from_sitemap`	boolean	Use sitemap as seed.
`settle_ms / goto_timeout_ms / respect_robots`	various	Timing and robots.