Robots.txt Generator

Define crawling policies and control search engine access to your website

XML Sitemap URL

Helps crawlers find your content structure faster

Paths to Block

Advanced Rules

# Configure settings to generate robots.txt

Step 1

Download the robots.txt file.

Step 2

Upload it to your web server's root folder.

Step 3

Verify it at yourdomain.com/robots.txt

Crawl Delay

SEC

Related Tools

A robots.txt file provides instructions to web crawlers about which sections of your website should or should not be indexed.

It must be placed in the root directory of your website (e.g., example.com/robots.txt).