Robots.txt

Quick Definition

robots.txt is a public text file at the root of a website. It tells search engine crawlers which paths they may fetch and which paths they should not fetch. Its main purpose is crawl control.

The most important SEO sentence is this: robots.txt controls crawling, not secure indexing and not access protection. If a page should not be publicly accessible, you need password protection, login, or server rules. If a page should not be in the index, you often need noindex, and Google must be able to crawl the page to see it.

robots.txt is therefore useful, but risky when used for the wrong job.

Plain-English Explanation

Think of robots.txt as a sign at the entrance of a building. The sign tells crawlers: "Please do not enter this hallway" or "You can find the map here." Respectable crawlers such as Googlebot read this sign before fetching URLs.

But a sign is not a lock. Anyone can open the file in a browser. Bots that ignore rules can ignore it. And if other websites link to a blocked URL, Google can still know the URL. Under some circumstances, Google may show the URL even though it did not crawl the content.

robots.txt is good for questions like:

• Which URL patterns should crawlers not fetch?
• Which areas waste crawl attention?
• Where should Google find a sitemap?
• Which resources may be fetched for page analysis?

It is bad for questions like:

• How do we protect confidential data?
• How do we safely remove a page from Google?
• How do we consolidate duplicates?

Where robots.txt Lives

The file always lives at the root of a specific protocol and host:

~~~txt https://example.com/robots.txt ~~~

It does not automatically apply to every version of a domain. These hosts and protocols can each have their own file:

~~~txt https://example.com/robots.txt https://www.example.com/robots.txt https://shop.example.com/robots.txt http://example.com/robots.txt ~~~

This matters during migrations, subdomain setups, international sites, CDNs, and staging environments. A perfect robots.txt on www.example.com does not help if Googlebot is crawling shop.example.com.

The Core Directives

User-agent

User-agent says which crawler a rule group applies to. User-agent: * applies broadly to crawlers. User-agent: Googlebot targets Google's web crawler.

Disallow

Disallow says which paths a crawler should not fetch. Example:

~~~txt User-agent: * Disallow: /internal-search/ ~~~

This means crawlers should not fetch paths under /internal-search/.

Allow

Allow can define exceptions inside blocked areas. Example:

~~~txt User-agent: Googlebot Disallow: /downloads/ Allow: /downloads/public-guide.pdf ~~~

This blocks the broader folder but allows one specific file.

Sitemap

Sitemap lists the absolute URL of a sitemap or sitemap index:

~~~txt Sitemap: https://www.example.com/sitemap.xml ~~~

This line is a hint. It does not replace a clean sitemap or strong internal linking.

A Simple Example

~~~txt User-agent: Googlebot Disallow: /test/

User-agent: * Disallow: /internal-search/

Sitemap: https://www.example.com/sitemap.xml ~~~

In this example, Googlebot should not fetch /test/. All crawlers should avoid /internal-search/. A sitemap is also listed.

In practice, every line should have a documented reason. If nobody knows why a path is blocked, robots.txt quickly becomes a risk file.

What robots.txt Really Controls

Crawling

robots.txt tells crawlers which paths they may fetch. That is the core job.

Crawl Traffic

On large websites, robots.txt can help keep crawlers away from unimportant, similar, or endless URL patterns: internal search, parameters, filters, sorting URLs, or test areas.

Resource Fetching

robots.txt can also block images, scripts, or CSS. Use this carefully. If Google cannot fetch important resources, it may understand the page less well.

Sitemap Discovery

The sitemap line helps search engines find sitemaps. It is practical, but it is not the main reason robots.txt exists.

What robots.txt Cannot Do

It Is Not Real Security

robots.txt is public. Anyone who opens https://example.com/robots.txt can see the blocked paths. Private content belongs behind login, password protection, VPN, IP rules, or other access controls.

It Is Not Reliable Deindexing

If a page is blocked by robots.txt, Google is not allowed to crawl it. That also means Google cannot see a noindex tag on the page. If other pages link to the URL, Google may still know it.

If you want to remove a page from the index, choose the right method: noindex, removal, 404/410, access control, or combinations depending on the case.

noindex Does Not Belong in robots.txt

Google does not support noindex in robots.txt as a rule. Disallow is not noindex.

It Is Not a Canonical Replacement

Google needs to see canonical signals. If you block duplicates with robots.txt, Google may not be able to read the canonical hints on those pages.

robots.txt, noindex, and canonical

robots.txt

Use robots.txt when you want to control crawling: "Please do not fetch these paths."

noindex

Use noindex when an accessible page should not appear in the index. Important: the page must be crawlable so Google can see noindex.

canonical

Use canonicals when multiple URLs show similar content and you want to communicate a preferred version.

Access Control

Use access control when content is private. That is not an SEO trick; it is security.

Common SEO Use Cases

Internal Search Results

Internal search pages often create many thin or duplicate pages. robots.txt can help keep those patterns out of crawling.

Faceted Navigation

Ecommerce filters often create huge numbers of URL combinations. Some filter pages are valuable, many are not. robots.txt can be part of the control strategy, but only after deciding which filter pages have search value.

Parameters and Sorting

Sorting, tracking parameters, and session IDs can create unnecessary URL variants. robots.txt can block patterns, but it should not be the only solution for poor URL logic.

Staging

Staging should not only be blocked in robots.txt. It should be protected. Otherwise it is still publicly reachable and only "forbidden" to polite crawlers.

Media Files

For images, videos, or other files, robots.txt can make sense depending on the goal. For HTML pages, the distinction from noindex is especially important.

Practical Example

An ecommerce site sees Googlebot fetching many URLs like:

~~~txt /shoes?color=black&size=42&sort=price /shoes?color=black&size=43&sort=price /shoes?color=blue&size=42&sort=price ~~~

The team first blocks all parameter URLs broadly. Crawl load drops, but later the team notices that some filter combinations match real demand, such as "black running shoes men." Those pages would have been better as curated landing pages.

The better solution is selective. Valuable filter topics get landing pages, internal links, indexable content, and clear canonicals. Endless sorting and irrelevant parameters are restricted. robots.txt becomes part of a strategy, not a rough emergency switch.

Common Mistakes

• Accidentally deploying Disallow: / to production.
• Using robots.txt for privacy or security.
• Expecting Disallow to safely remove a URL from the index.
• Blocking a page while also expecting Google to see noindex.
• Blocking important CSS, JavaScript, or image resources.
• Testing rules on the wrong host, protocol, or subdomain.
• Blocking all filter pages without checking demand and value.
• Listing sitemaps with wrong or non-absolute URLs.
• Deploying robots.txt changes without review.
• Not checking changes with Search Console or URL Inspection.

Mini Workflow

1. Clarify the intent: control crawling, prevent indexing, consolidate duplicates, or protect content? 2. Use robots.txt only for crawl control. 3. Check host, protocol, and subdomain. 4. Write rules as narrowly as possible. 5. Check whether important pages, CSS, JavaScript, images, or canonical signals are affected. 6. Test the file in a browser and with Search Console. 7. Document the rule, date, reason, and expected effect. 8. Monitor crawling and indexing separately. 9. Remove old rules when their purpose no longer exists.

Contextter Perspective

For Contextter, robots.txt is not an isolated technical detail. The file affects which content search engines can fetch at all. A content hub therefore needs not only strong writing, but also clear technical boundaries: make core pages visible, reduce low-value variants, and protect private or unfinished areas properly.

Good robots.txt work always asks the same question: does this rule help search engines understand the important content better, or is it only hiding an architecture problem?

Related Terms

Good next reads:

• crawling
• crawl-budget
• xml-sitemap
• noindex
• indexing
• canonical-tag

Review Sources

• Google Search Central: Introduction to robots.txt
• Google Crawling Infrastructure: How Google interprets robots.txt
• Google Crawling Infrastructure: Create a robots.txt file
• Google Search Console Help: robots.txt report
• Google Search Central: Block indexing with noindex
• Google Search Central: Robots meta tag specifications
• Google Search Central: What is Googlebot
• Google Search Central: Troubleshoot crawling errors