Robots.txt Generator

Builds a robots.txt file that disallows a list of paths for all crawlers. Enter the paths separated by commas, like "/admin, /private".

Ad placeholder: Resp_Horizantal_1 (horizontal)
Ad placeholder: Resp_Square_1 (square)

Builds a robots.txt file that disallows a list of paths for all crawlers. Enter the paths separated by commas, like "/admin, /private".

Use cases

  • Blocking crawlers from indexing an admin or staging area
  • Generating a starting robots.txt for a new site

Examples

Two disallowed paths

/admin, /private
{"content": "User-agent: *\nDisallow: /admin\nDisallow: /private"}

FAQ

Is my data sent to a server?

Yes — the calculation runs on the server for this tool.

Related tools