# How do I crawl an entire website and get content for every page?

Point a crawl API at a starting URL and it follows every internal link automatically, returning the content of each page as clean markdown. No need to know the site structure or list URLs in advance.

[Firecrawl's crawl endpoint](https://docs.firecrawl.dev/features/crawl) handles discovery, [JavaScript rendering](/content/glossary/web-scraping-apis/what-is-javascript-rendering-web-scraping/index.html), [rate limiting](/content/glossary/web-scraping-apis/how-web-scraping-apis-handle-rate-limiting-quotas/index.html), and [robots.txt](/content/glossary/web-crawling-apis/what-is-robots-txt-protocol/index.html) automatically:

```python
from firecrawl import Firecrawl
from firecrawl.types import ScrapeOptions

firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")

result = firecrawl.crawl(
    "https://example.com",
    limit=100,
    scrape_options=ScrapeOptions(formats=["markdown"], only_main_content=True)
)
```

Use `limit` to cap pages, `max_depth` to control link depth, and `include_paths` / `exclude_paths` to scope the crawl to specific sections. For a quick one-off crawl from the terminal, the [Firecrawl CLI](https://docs.firecrawl.dev/sdks/cli) also works: `firecrawl crawl https://example.com --wait`.

Last updated: Feb 23, 2026