Launch Week I / Day 5: Real-Time Crawling with WebSockets

Welcome to Day 5 of Firecrawl's Launch Week! We're excited to introduce an exciting new feature that will bring your web scraping projects to the next level: Real-Time Crawling with WebSockets.

Introducing Crawl URL and Watch

We're thrilled to announce our new WebSocket-based method, Crawl URL and Watch. This powerful feature enables real-time data extraction and monitoring, opening up new possibilities for immediate data processing.

How It Works

The Crawl URL and Watch method initiates a crawl job and returns a watcher object. You can then add event listeners for various events like "document" (when a new page is crawled), "error" (if an error occurs), and "done" (when the crawl is complete).

This approach allows you to process data in real-time, react to errors immediately, and know exactly when your crawl is finished.

We're excited to see how you'll use this new feature to enhance your web scraping projects and create more dynamic, responsive applications.

Check out our documentation for a detailed guide on how to implement Crawl URL and Watch in your projects: Firecrawl WebSocket Documentation

About the Author

Eric Ciarla is a co-founder of Firecrawl. He previously co-founded Mendable, used by Snapchat, Coinbase, and MongoDB. He's been building products in the AI and data space since 2022.

Eric Ciarla @ericciarla

More articles by Eric Ciarla

• Introducing /parse: Turn any document into LLM-ready data
• Firecrawl /search is now available on OpenRouter
• Introducing Firecrawl web-agent: Build and Deploy Your Own Web Research Agent
• Introducing Fire-PDF: Firecrawl's New PDF Parsing Engine
• Introducing /interact — Scrape and interact with a web page
• Partnering with Wikipedia for a More Sustainable Web
• Introducing PDF Parser v2: Faster Extraction with Auto Mode
• Browser Sandbox: Secure Environments for Agents to Interact with the Web
• Branding Format v2: Improved Logo Extraction
• Extract Web Data at Scale With Parallel Agents