robots.txt Generator — free & private, in your browser
Create a valid robots.txt for your site — instantly, with nothing uploaded.
This tool runs entirely in your browser. Your files are never uploaded — they stay on your device.
User-agent: * Disallow:
A robots.txt file tells search engines and other crawlers which parts of your site they may visit. Getting it right matters: one stray rule can hide your whole site from Google, or expose pages you meant to keep out of the index.
This generator builds the file as you type, so you can see exactly what each rule does. Everything happens in your browser — no data is sent anywhere — and you can copy the result or download it as robots.txt when it looks right.
How to use robots.txt Generator
- Set the user-agent the rules apply to (use * for all crawlers).
- Add the paths you want to Disallow, and optionally any paths to Allow.
- Optionally add a crawl-delay and your sitemap URL.
- Review the live preview, then copy it or download it as robots.txt.
- Upload the file to the root of your site, e.g. example.com/robots.txt.
Frequently asked questions
Is anything uploaded or sent to a server?
No. The robots.txt text is built entirely in your browser as you type. Nothing you enter is uploaded, stored or shared.
Where do I put the robots.txt file?
It must live in the root of your domain, reachable at https://yourdomain.com/robots.txt. Crawlers do not look for it in subfolders.
What does “Disallow: /” do?
A Disallow of a single slash blocks the entire site for that user-agent. Leave the Disallow paths empty (or use “Allow: /”) if you want everything crawled.
Does robots.txt keep pages out of Google completely?
Not necessarily. It asks well-behaved crawlers not to fetch a path, but a blocked URL can still be indexed if linked elsewhere. Use a noindex meta tag or HTTP header to reliably keep a page out of search results.
Should I add a sitemap line?
It is optional but recommended. A Sitemap line points crawlers to your XML sitemap and helps them discover your pages faster.