Skip to content
Learn · Business

What is robots.txt?

robots.txt is a file that tells search engine crawlers which parts of a site they should not crawl.

Why it matters

A wrong rule can hide your whole site from Google.

Practical tip

Never block CSS or JS files that your pages need to render.

Frequently asked questions

What is robots.txt?

robots.txt is a file that tells search engine crawlers which parts of a site they should not crawl.

Why does robots.txt matter for my website?

A wrong rule can hide your whole site from Google.

Any practical tip about robots.txt?

Never block CSS or JS files that your pages need to render.

Related terms

More about business