Robots.txt Generator 2026: Complete Syntax Guide + 12 Templates
Use Korelyy free robots.txt generator online. Write correct robots.txt syntax: User-agent, Allow, Disallow, Sitemap, Crawl-delay, Wildcard. Includes WordPress, Shopify, Next.js, Vue, Cloudflare Pages 2026 latest templates. No accidental noindex.
1. What Is a robots.txt File? Why Wrong Syntax Gets You Delisted
A robots.txt is a plain-text file at /robots.txt telling search engine crawlers which pages to crawl and which to skip. A single stray Disallow: / can remove your entire site from Google index in 72 hours. This guide uses Korelyy generator to build rules safely with the 2026 updated standard (Googlebot, Bingbot, YandexBot, ClaudeBot, GPTBot 2026 additions).
1.1 2026 New Crawlers You MUST Block
- GPTBot / ChatGPT-User — OpenAI scraping to train their next model (70% enterprise sites block as of 2026-07).
- ClaudeBot / anthropic-ai — Anthropic scrapes for Claude model training.
- Google-Extended — Google AI training crawler, separate from Googlebot search index.
- Amazonbot / Bytespider (TikTok) / PetalBot (Huawei) — 2026 fastest growing scrapers consuming bandwidth.
2. Korelyy Robots.txt Generator — 30-Second Safe Build
Open Korelyy → Tools → SEO → Robots.txt Generator. (1) Pick platform preset (WordPress/Shopify/Next.js/static). (2) Toggle AI-scraper block (recommended on). (3) Paste Sitemap URL if any. (4) Add per-folder rules (Allow wp-admin/admin-ajax.php, Disallow /search/, /tag/, /author/ for WordPress). (5) Click Generate, copy, upload to / via SFTP/Cloudflare Pages Assets. Korelyy validates wildcard syntax (No $ after /*, correct Allow/Disallow precedence 2026 rules).
2.1 12 Ready-to-Use Templates (Copy Paste)
- T1. Static Blog (Jekyll/Hugo/Cloudflare Pages): Allow all public, Disallow /private/ + /draft/ + /admin/, Block AI crawlers. Add Sitemap URL.
- T2. WordPress 2026: Allow /wp-admin/admin-ajax.php. Disallow /xmlrpc.php, /wp-json/, /?s=*, /search/, /page/*/comment-page-*, /tag/, /category/page/, /author/, /feed/. Block AI scrapers.
- T3. Shopify: Use built-in online-store robots.txt editor. Add AI crawler block lines. Sitemap auto at /sitemap.xml.
- T4. Next.js Static Export (Korelyy case): Disallow Next.js internals (/next/, /_next/), Disallow *.json, Sitemap /sitemap-index.xml, AI blocker.
3. 7 Deadly Robots.txt Mistakes 2026
- E1. Disallow: / — Typo while trying Disallow /private/. Deindexes entire site.
- E2. Using Crawl-delay for Googlebot — Google ignores Crawl-delay since 2019. Use Search Console crawl rate.
- E3. Allowed .php but forgot Allow admin-ajax (WordPress) — breaks many plugins that serve dynamic content on frontend.
- E4. Disallow /*.pdf$ — Google does not treat $ as end-of-string. Use Noindex HTTP header instead.
- E5. Robots.txt alone to hide private pages — still indexed if linked externally. Use meta robots noindex + password.
4. Verify + Monitor After Upload
Step 1: curl -I https://yoursite.com/robots.txt → expect HTTP 200 Content-Type: text/plain. Step 2: Google Search Console → Settings → Robots.txt tester. Paste problematic URLs like /search/ or /wp-admin/. Step 3: Request Google re-crawl via URL Inspection Tool (click Live Test then Request Indexing). Step 4: Check GSC Coverage report weekly for 4 weeks — watch for sudden Crawled — currently not indexed drops.