←  All terms
R
SEO & AEO

robots.txt

A robots.txt file gives crawlers rules about which areas of a website they may access.

A robots.txt file helps control crawler access by specifying which parts of a website should or should not be requested. It can guide crawling behavior, but it does not reliably prevent already known URLs from being indexed. For effective technical SEO, it should work alongside clear site architecture, appropriate indexation controls, and an up-to-date sitemap.xml. The concept is closely related to Crawling and sitemap.xml.

Read more

Crawling

sitemap.xml

AI Crawler

Indexing

FAQ

robots.txt

How do robots.txt and sitemap.xml work together?
The robots.txt file can point to the sitemap.xml through a Sitemap entry, which makes discovery easier. The two files should not contradict each other: sitemap URLs blocked by a Disallow rule send conflicting signals. The sitemap lists desired pages, while robots.txt limits what may be fetched.
Can AI crawlers be blocked through robots.txt?
Partly. Many providers document their own user agents, which can be addressed with a Disallow rule. However, the file is a request, not a technical barrier, and compliance is voluntary. Binding protection requires server-side measures. It also has to be weighed whether visibility in AI answers is desired.
What typical mistakes occur with robots.txt?
The costliest mistake is an accidental Disallow for the entire site, often carried over from a staging environment. Others are blocked CSS and JavaScript files that hinder rendering, and the belief that the file protects confidential content. It is publicly readable and offers no access protection.
Where does the robots.txt file have to be located?
The file sits in the root directory of the domain, for example at /robots.txt, and applies only to that exact host including its protocol. Subdomains need their own file. A file placed in a subfolder is not evaluated by crawlers.
Does robots.txt keep pages out of the search index?
No. The robots.txt file controls crawling, not indexing. A blocked URL can still appear in the index without content if other pages link to it. Exclusion is handled with a noindex directive, which only works if the page remains crawlable.