Harvest disallowed paths and sitemap URLs that sites helpfully list for you.
Robots & Sitemap Harvester fetches a target’s robots.txt and linked sitemaps and extracts the paths inside. Ironically, the very file meant to hide directories from crawlers often points testers straight at admin panels, staging areas and hidden endpoints.
It parses both robots directives and XML sitemaps, giving you a ready list of paths to explore.
robots.txt and sitemaps are public files intended to be read. Acting on what you find still requires authorization for the target.