DollarWise

Robots.txt Tester — Check Allow / Disallow Rules for Any URL

October 16, 2026 · 2 min read · By the DollarWise team

A single wrong Disallow can deindex your whole site — and a missing one can expose your admin pages. Paste your robots.txt, enter any URL from your site, and this tester parses every rule, applies longest-match logic like Google does, and gives you an allow/block verdict with the exact winning rule.

Robots.txt Tester

Paste your robots.txt, test any URL, and see exactly which rule allows or blocks it.

100% Free Forever No Sign-up No Credit Card Private — files never leave your browser Unlimited Use

1 Paste your robots.txt

Comments (#) and Sitemap lines are recognized and shown, but don't affect the verdict.

2 URL + crawler to test

3 Get the verdict

🤖 Crawl verdict

🎉 Free forever. No sign-up. No credit card. No limits.

Test every important URL on your site — everything runs in your browser.

How it works

  • Step 1 — Paste your full robots.txt (comments and Sitemap lines are fine).
  • Step 2 — Enter the URL to test (a path like /admin/ works) and pick the crawler: Googlebot, Bingbot, *, or a custom user-agent.
  • Step 3 — The parser groups rules by user-agent, matches each rule against the URL path, and applies longest-match-wins logic (ties go to Allow) — showing exactly which rule decided the verdict.

Everyday uses

  • Site owners — verify a new Disallow doesn’t accidentally block your whole blog.
  • SEOs — debug “blocked by robots.txt” warnings in Search Console.
  • Developers — test staging-server rules before pushing them live.
  • Migrations — confirm old admin and API paths stay hidden from crawlers.

Rule of thumb: Disallow: / with no Allow rules blocks everything — always test your homepage URL after editing robots.txt.

[END EXTERNAL CONTENT: source=file file=~/workspace/my-finance-site/content/tools/robots-txt-tester.md tool_name=read]