There are numerous reasons you could want to discover all of the URLs on a web site, but your specific purpose will decide Everything you’re looking for. For instance, you might want to: Recognize each indexed URL to analyze challenges like cannibalization or index bloat