Tell crawlers where they can go.
Each starts with / and * matches anything - separate with spaces.
Blocking training doesn't affect Google or Bing search.
Tells crawlers where to find every page - separate several with spaces.
User-agent: *
Disallow:
- Search engines can crawl your whole site.
- No Sitemap line - add your sitemap address so crawlers find every page.
What to do next
Robots.txt, explained.
Where does robots.txt go?
At the root of your site, so it opens at yoursite.com/robots.txt. Crawlers only look there - a file in a subfolder is ignored. Each subdomain needs its own.
What should I block?
Only areas with nothing worth finding in search: admin and login pages, carts and checkouts, internal search results. Never block the CSS and JavaScript your pages load - Google needs them to see the page the way visitors do.
Does blocking a page remove it from Google?
No. robots.txt stops crawling, not indexing - a blocked page can still appear in results if other sites link to it, just without a description. To keep a page out of Google, leave it crawlable and add a noindex tag.
Should I block AI crawlers?
It depends on what you want. Blocking training crawlers keeps your content out of AI model training without affecting Google or Bing. Blocking AI search crawlers also stops tools like ChatGPT search and Perplexity from citing your pages - which most businesses want to keep.
Why add a sitemap line?
It tells every crawler where your sitemap is, without you submitting it to each search engine. Use the full address, starting with https://.
My site runs on Shopify, Wix or Squarespace - can I use this?
Those platforms create robots.txt for you and limit how you edit it - Shopify through a theme template, Wix and Squarespace through their SEO settings. Use the file here as a reference for the rules to add there.
Is my data stored?
No. The file is built in your browser - nothing you enter is sent to us or saved.
Read more on this.
Robots.txt and Sitemap XML: How to Check, Find and Generate Them
Learn how to check robots.txt, find a website's sitemap and generate robots.txt with Anantya TechLabs' free tools, then verify the live files.
How to Write Meta Tags That Get Clicks
Learn how to write meta tags, preview your title and description and generate HTML with Anantya’s free tool. Follow a practical example.
Why Does a Small Business Need Digital Marketing?
Learn why digital marketing for small business matters, from local SEO and online visibility to building trust, generating leads and reaching customers.
Being crawlable is only the first step.
We fix the technical SEO, content and site speed that decide whether search traffic turns into leads.