www.firecrawl.dev

Resources

• llms.txt— AI crawler index (llmstxt.org spec)
• sitemap.xml— XML sitemap for crawler discovery
• robots.txt— Crawler directives
• About This Cache— Methodology, technical details, and usage guidelines

Identity

• identity.mdpublic
• voice.mdpublic
• manifest.jsonpublic

Directory

Internal data

• Homepage
• blog 24
• use-cases 3
• pricing 1
• agent 1
• tools 1
• glossary 15
• playground 2
• signin 1
• firestarters 1