Robots.txt for Bloggers: What It Does and What Not to Block

Robots.txt is a file that tells crawlers which parts of your site they may fetch, but it doesn't keep pages out of Google, and most small blogs need few or no rules in it. Google says the file is used mainly to stop crawlers overloading your site with requests, and that it is "not a mechanism for keeping a web page out of Google" (Google: Introduction to robots.txt, last updated 10 December 2025). You don't even need one: Search Console Help says "Not having a robots.txt file is fine", and that Google can then crawl every URL on your site (Search Console Help: robots.txt report, checked 8 October 2026).

Here is what the file does and can't do, the few rules a blogger might add, the ones to avoid, and how to check yours.

What robots.txt does

Robots.txt is a plain text file at the root of your site, such as https://example.com/robots.txt. Google's crawlers download it before crawling a site, to find out which parts they may crawl (Google: robots.txt specification, last updated 31 August 2026).

The file is made of groups. Each group starts with a User-agent line naming a crawler, or * for all crawlers except Google's AdsBot crawlers, which must be named. Disallow lines list paths that crawler shouldn't fetch, and Allow lines reopen a path inside a disallowed folder. Anything no rule blocks is allowed (Google: How to write and submit a robots.txt file, last updated 21 November 2025).

A typical blog's file

On a WordPress site installed at the root of its domain, WordPress's own code produces these lines (WordPress: do_robots()):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

They keep crawlers out of the admin area, then reopen one file inside it, admin-ajax.php, which handles WordPress's AJAX requests, including those from public pages (WordPress Plugin Handbook: AJAX). On a public site, WordPress 5.5 and later add a Sitemap: line for the sitemap index (WordPress: add_robots()). Nothing else is blocked, so your posts and pages stay open. Plugins and themes can change this output, so check what yours shows.

A Sitemap: line tells crawlers where your sitemap is, and XML sitemaps covers whether your blog needs one. A robots.txt file covers only the host it sits on, so rules for a blog at blog.example.com go in that subdomain's own file, as subdomain vs subfolder explains.

On a hosted platform such as Wix or Blogger, you may not need to, or be able to, edit the file directly; Google notes that the platform may offer a search settings page instead (Google: How to write and submit a robots.txt file).

How the rules match

Google's specification sets out how rules match addresses. Four details are worth knowing:

• Paths match from the start. Disallow: /photo blocks /photo/, /photos/ and /photography-tips/, while Disallow: /photo/ blocks only that folder and what's in it. In a path, * stands for any characters and $ marks the end of the address.

• Case counts. /Photo/ and /photo/ are different paths.

• The longest matching rule wins. If an Allow and a Disallow both match, Google follows the one with the longer path, and on a tie, the less restrictive one. That's how WordPress's Allow line works.

• Some lines do nothing in Google. Google supports neither crawl-delay (Google: robots.txt specification) nor noindex in robots.txt (Google: Block search indexing with noindex, last updated 10 December 2025).

What robots.txt can't do

• Remove or hide a page. If other pages link to a blocked page, Google can still index its address without visiting it, and the result then has no description (Google: Introduction to robots.txt). Search Console's Page indexing report calls this Indexed, though blocked by robots.txt (Search Console Help: Page indexing report). To keep a page out of Google, Google points to noindex or a password instead.

• Protect anything. Each crawler decides whether to obey the file: Googlebot and other respectable crawlers do, but others might not (Google: Introduction to robots.txt). The file is also public, and Google warns that a rule naming private content can reveal where it is (Google: Useful robots.txt rules, last updated 12 June 2026). Use a password for private material.

• Work the same way everywhere. Other search engines may not support every rule, and crawlers can read the syntax differently (Google: Introduction to robots.txt).

Rules a blogger might add

Google gives two reasons to block web pages: you think its crawler will overwhelm your server, or you want to keep it from crawling unimportant or similar pages (Google: Introduction to robots.txt). Its crawl budget guide, which deals with crawling load, is written for very large, frequently updated sites, and says you can skip it if your pages seem to get crawled the day you publish them (Google: Optimize your crawl budget, last updated 22 July 2026; more in does a small blog need to worry about crawl budget?). So start with no extra rules, and add a group only for a decision you've made, such as these two.

Keeping your images out of Google. This group blocks Google's image crawler from your whole site:

User-agent: Googlebot-Image
Disallow: /

Google says that keeps your images off Google wherever it shows images, Google Images and Discover included (Google: Useful robots.txt rules), and that Googlebot-Image rules also reach every Search feature showing images, logos and favicons (Google: common crawlers, last updated 14 July 2026). If Discover sends you readers, disallow only the image paths you want kept out.

Opting out of AI model training. Google and OpenAI name their own crawlers, and a rule affects only the names you list:

• Google-Extended controls whether content Google crawls from your site can train future Gemini models and ground answers in Gemini Apps and Vertex AI. Google says it "does not impact a site's inclusion in Google Search" and isn't a ranking signal (Google: common crawlers).

• GPTBot crawls content that may train OpenAI's models. OAI-SearchBot is separate: it's used to show sites in ChatGPT's search results, and a site that blocks it won't be shown in ChatGPT's search answers, though it can still appear as a navigational link. OpenAI adds that robots.txt may not apply when ChatGPT visits a page at a user's request (OpenAI: crawlers and user agents, checked 8 October 2026).

These two groups opt your site out of OpenAI's model training and Google's Gemini training and grounding, without blocking Googlebot or OAI-SearchBot:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

What never to block

• Posts and pages you want in search. If a rule covers pages you want found, remove it, as Search Console Help advises (Search Console Help: Page indexing report).

• The CSS, JavaScript and images your posts load. Google says not to block files whose absence makes your pages harder for its crawler to understand, or it won't analyze those pages well (Google: Introduction to robots.txt). Its specification uses CSS and JavaScript files as an example, noting that "Google needs them for rendering". Check that no rule in a User-agent: * or Googlebot group covers the folders your theme or plugins load files from; Googlebot ignores a Googlebot-Image group like the one above.

• Pages you've marked noindex. Google has to crawl a page to see that tag (Google: Block search indexing with noindex); why Google hasn't indexed your page covers how the two interact.

• Your sitemap. Crawlers can follow a Sitemap: line only if the sitemap itself isn't disallowed (Google: robots.txt specification).

• Your whole site. Disallow: / under User-agent: * blocks crawling of everything, and Google notes the URLs may still be indexed (Google: Useful robots.txt rules). Search Console Help names a robots.txt block as one possible cause when indexed pages drop without a matching rise in errors (Search Console Help: Page indexing report).

How to check your robots.txt

1. Open the file

In a private browsing window, go to your domain followed by /robots.txt; Google suggests this to confirm the file is publicly accessible (Google: How to write and submit a robots.txt file). If you see rules, compare them with the lists above. A page-not-found error probably means you have no file, which Google treats as no crawl restrictions (Google: robots.txt specification).

2. Read Google's copy in Search Console

Open the robots.txt report. It works only for a Domain property, or a URL-prefix property without a path such as https://example.com/ (Search Console Help: robots.txt report); what is Google Search Console? covers setting one up. A Domain property can list both the http:// and https:// address of each host. Check each one's Fetch status:

• Fetched. Google read the file. The Issues column counts parsing problems; Google skips lines it can't parse and uses the rest. Click the file to see the version Google last fetched, with problems highlighted.

• Not Fetched - Not found (404). There's no file at that address, which is fine if you never made one. An address that keeps this status for 30 days drops out of the report, so an empty list can mean the same.

• Not Fetched - Any other reason. Something else stopped Google, and the help points to its list of indexing issues for the cause.

Fix fetch errors quickly. If a robots.txt file exists but returns a server error, Google stops crawling the site for the first 12 hours while it retries, then uses its last good copy for up to 30 days; timeouts and other network failures count as server errors (Google: robots.txt specification).

3. Test one post

In Search Console, paste a post's full address into the inspection bar at the top of any screen; it must belong to the property you have open. Expand Page indexing and read Crawl allowed?, which shows whether a robots.txt rule blocks Google from the page. The first result reflects Google's last crawl, so click Test live URL to test the page as it is now. After a successful live test, click View tested page and open the Screenshot tab to see what Google rendered; Google says differences from your page can come from blocked resources (Search Console Help: URL Inspection tool).

4. After you change the file

If you edit the file yourself, start from a copy of the current one, use a plain text editor rather than a word processor, save it as UTF-8 text and upload it to your site's root; Google suggests asking your host how to upload it (Google: Update your robots.txt file, last updated 21 November 2025; Google: How to write and submit a robots.txt file). Then repeat step 1 to confirm the new file is public.

Google updates its cached copy every 24 hours (Google: Update your robots.txt file). Once the robots.txt report's Checked on time is later than your change, read the file's status and issues there again. To speed that up after unblocking important pages or fixing a fetch error, open the robots.txt report, select the more settings icon next to the file and click Request a recrawl (Search Console Help: robots.txt report). That asks Google to recrawl the file, not your pages: Google says it doesn't guarantee an immediate recrawl of the URLs you unblocked.

What to do on a small blog

• Keep your platform's default file, or none, unless you have a specific reason to change it.

• Add a group only for a decision you've made, such as opting out of AI model training.

• Never use robots.txt to hide or remove a post; use noindex or a password, as Google advises.

When posts drop out of Google unexpectedly, recheck the file with the robots.txt report and URL Inspection. For the wider picture of how your blog is doing in search, see how to check your website's SEO performance.