Glossary
Review the definitions for terms used across Cloudflare's AI Crawl Control documentation.
| Term | Definition |
|---|---|
| AI crawler | A bot which scrapes content from websites in support of an AI model, including by scraping content for indexing, retrieval augmented generation, or training. |
| Merchant of Record | The entity who facilitates "buying and selling". For pay per crawl, Cloudflare is the merchant of record. |
| robots.txt | A text file which lists pages in your website that are off-limits for bots. Well-behaved bots respect this file, but some bots may violate it. You can enforce |
Was this helpful?
- Resources
- API
- New to Cloudflare?
- Directory
- Sponsorships
- Open Source
- Support
- Help Center
- System Status
- Compliance
- GDPR
- Company
- cloudflare.com
- Our team
- Careers
- © 2025 Cloudflare, Inc.
- Privacy Policy
- Terms of Use
- Report Security Issues
- Trademark