Robots.txt Generator

New

Generate robots.txt files to control how search engines crawl and index your website

Configure crawling rules for search engine bots

1 group

User-Agent

Rules

Sitemap URLs

Crawl Delay (seconds)

Host

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml