Where do I put robots.txt?

In the root of your site so it opens at yoursite.com/robots.txt.

Does robots.txt hide pages from Google?

It stops crawling, but blocked pages can still appear in results if linked. Use a noindex tag to keep a page out of search.

Should I add my sitemap?

Yes. A Sitemap line helps search engines find all your pages faster.

Yes. Tick GPTBot, CCBot or Google-Extended to ask those AI crawlers not to use your site.

Is crawl-delay supported by Google?

No, Google ignores it, but Bing and Yandex respect it.

Robots.txt Generator – Create Robots.txt File Online Free

How to Use the Robots.txt Generator

Choose whether all robots are allowed by default, add folders you want to block (one per line), pick any bots to block completely and add your sitemap URL. The robots.txt updates live; copy it or download and upload it to your website's root folder.

Robots.txt Generator— Create a robots.txt file for your website.

How to Use the Robots.txt Generator

Frequently Asked Questions

More free tools

How to Use the Robots.txt Generator

Frequently Asked Questions

Where do I put robots.txt?

Does robots.txt hide pages from Google?

Should I add my sitemap?

Can I block AI bots?

Is crawl-delay supported by Google?

More free tools