Appearance
firecrawl.crawling.crawl-urls
Crawl multiple URLs based on options
Overview
| Property | Value |
|---|---|
| Workflow type | Atomic |
| Library | App-firecrawl |
| Version | 1.0 |
Input Schema
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
organization_uuid | uuid | No | — | Organization that owns the connection. Resolved from the authenticated context; callers never pass it |
cloud_connection_uuid | uuid | No | — | Firecrawl connection to use (API key and optional self-hosted base_url). Omit to use the organization's only active Firecrawl connection |
url | string | Yes | — | The base URL to start crawling from |
excludepaths | list | No | — | URL pathname regex patterns that exclude matching URLs from the crawl. For example, if you set "excludePaths": ["blog/.*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. |
includepaths | list | No | — | URL pathname regex patterns that include matching URLs in the crawl. Only the paths that match the specified patterns will be included in the response. For example, if you set "includePaths": ["blog/.*"] for the base URL firecrawl.dev, only results matching that pattern will be included, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. |
maxdepth | integer | No | — | Maximum depth to crawl relative to the base URL. Basically, the max number of slashes the pathname of a scraped URL may contain. |
maxdiscoverydepth | integer | No | — | Maximum depth to crawl based on discovery order. The root site and sitemapped pages has a discovery depth of 0. For example, if you set it to 1, and you set ignoreSitemap, you will only crawl the entered URL and all URLs that are linked on that page. |
ignoresitemap | boolean | No | — | Ignore the website sitemap when crawling |
ignorequeryparameters | boolean | No | — | Do not re-scrape the same path with different (or none) query parameters |
limit | integer | No | — | Maximum number of pages to crawl. Default limit is 10000. |
allowbackwardlinks | boolean | No | — | Allows the crawler to follow internal links to sibling or parent URLs, not just child paths. false: Only crawls deeper (child) URLs. → e.g. /features/feature-1 → /features/feature-1/tips ✅ → Won't follow /pricing or / ❌ true: Crawls any internal links, including siblings and parents. → e.g. /features/feature-1 → /pricing, /, etc. ✅ Use true for broader internal coverage beyond nested paths. |
allowexternallinks | boolean | No | — | Allows the crawler to follow links to external websites. |
delay | float | No | — | Delay in seconds between scrapes. This helps respect website rate limits. |
webhook | json | No | — | A webhook specification object. |
scrapeoptions | json | No | — | — |
Output Schema
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
organization_uuid | uuid | No | — | Organization that owns the connection. Resolved from the authenticated context; callers never pass it |
cloud_connection_uuid | uuid | No | — | Firecrawl connection to use (API key and optional self-hosted base_url). Omit to use the organization's only active Firecrawl connection |
url | string | Yes | — | The base URL to start crawling from |
excludepaths | list | No | — | URL pathname regex patterns that exclude matching URLs from the crawl. For example, if you set "excludePaths": ["blog/.*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. |
includepaths | list | No | — | URL pathname regex patterns that include matching URLs in the crawl. Only the paths that match the specified patterns will be included in the response. For example, if you set "includePaths": ["blog/.*"] for the base URL firecrawl.dev, only results matching that pattern will be included, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. |
maxdepth | integer | No | — | Maximum depth to crawl relative to the base URL. Basically, the max number of slashes the pathname of a scraped URL may contain. |
maxdiscoverydepth | integer | No | — | Maximum depth to crawl based on discovery order. The root site and sitemapped pages has a discovery depth of 0. For example, if you set it to 1, and you set ignoreSitemap, you will only crawl the entered URL and all URLs that are linked on that page. |
ignoresitemap | boolean | No | — | Ignore the website sitemap when crawling |
ignorequeryparameters | boolean | No | — | Do not re-scrape the same path with different (or none) query parameters |
limit | integer | No | — | Maximum number of pages to crawl. Default limit is 10000. |
allowbackwardlinks | boolean | No | — | Allows the crawler to follow internal links to sibling or parent URLs, not just child paths. false: Only crawls deeper (child) URLs. → e.g. /features/feature-1 → /features/feature-1/tips ✅ → Won't follow /pricing or / ❌ true: Crawls any internal links, including siblings and parents. → e.g. /features/feature-1 → /pricing, /, etc. ✅ Use true for broader internal coverage beyond nested paths. |
allowexternallinks | boolean | No | — | Allows the crawler to follow links to external websites. |
delay | float | No | — | Delay in seconds between scrapes. This helps respect website rate limits. |
webhook | json | No | — | A webhook specification object. |
scrapeoptions | json | No | — | — |
status_code | integer | No | — | HTTP status code of the completed call |
response | json | No | — | Parsed JSON response body |
failure_reason | string | No | — | — |
failure_type | string | No | — | — |
failed_at | string | No | — | — |
failed_step | string | No | — | — |
failed_layer | string | No | — | — |
failed_at_state | string | No | — | — |
error | string | No | — | — |
error_type | string | No | — | — |
States
| State | Initial | Terminal | Success | Auto-advance | Description |
|---|---|---|---|---|---|
pending | Yes | No | — | execute | Waiting to call POST /crawl |
completed | No | Yes | Yes | — | HTTP call succeeded |
failed | No | Yes | No | — | HTTP call failed |
State Diagram
Transitions
| From | Action | To | Description |
|---|---|---|---|
pending | execute | completed | Perform POST /crawl |
* (any state) | fail | failed | Record the failure reason |
Outcomes
| Outcome | Type | Description | State Data Keys |
|---|---|---|---|
completed | SUCCESS | The provider accepted the request | status_code, response |
failed | FAILURE | The provider rejected the request or was unreachable | failure_reason, failure_type, status_code |
Business Errors
| Code | Message Template |
|---|---|
HTTP_AUTHENTICATION_FAILED | The provider rejected the credential (HTTP {status_code}) |
HTTP_AUTHORIZATION_FAILED | The credential is not permitted to perform this operation (HTTP {status_code}) |
HTTP_NOT_FOUND | The provider has no resource at {path} (HTTP 404) |
HTTP_RATE_LIMITED | The provider rate-limited this request (HTTP 429) |
HTTP_REQUEST_FAILED | The provider returned HTTP {status_code} for {method} |
HTTP_RESPONSE_SHAPE_UNEXPECTED | The provider response for {method} {path} could not be parsed as JSON |
FIRECRAWL_CONNECTION_MISSING | firecrawl_connection_missing: this organization has no active {connection_type} connection. Create one with connection.setup (type {connection_type}). |
FIRECRAWL_CONNECTION_AMBIGUOUS | firecrawl_connection_ambiguous: this organization has more than one active {connection_type} connection. Pass cloud_connection_uuid to choose one. |
FIRECRAWL_CONNECTION_NOT_FOUND | firecrawl_connection_not_found: no {connection_type} connection {connection_uuid} in this organization. |
FIRECRAWL_CONNECTION_CREDENTIAL_MISSING | firecrawl_connection_credential_missing: the {connection_type} connection has no api_token. Update the connection with a Firecrawl API key. |
API Usage
bash
POST /api/workflows/start
Content-Type: application/json
{
"workflow_type": "firecrawl.crawling.crawl-urls",
"initial_data": {
"url": "value"
}
}