Archive Crawler

This guide walks you through the process of crawling archived URLs using Archive Crawler.

Why and when to use Archive Crawler

Best for: Discovering historical content and forgotten endpoints

Archive Crawler obtains URLs from internet archives and then crawls every discovered URL to identify pages, endpoints, files, parameters, and historical attack surfaces that may no longer be linked from the live website.

Use when:

• Discovering legacy web pages
• Finding removed or hidden content
• Enumerating old API endpoints
• Expanding attack surface coverage
• Collecting historical URLs for analysis
• Identifying forgotten files and backups

Configurations

Archive Sources

Configure which archive providers should be queried for URLs.

Common Crawl

Queries Common Crawl indexes to obtain archived URLs belonging to the target domain and its subdomains.

Benefits:

• Massive web-scale dataset
• Historical coverage across many years
• Fast URL enumeration
• Useful for discovering pages not present in other archives

Wayback Machine

Queries the Wayback Machine CDX API to obtain archived URLs captured over time.

Benefits:

• Historical snapshots
• Excellent coverage of older websites=
• Useful for discovering removed content

Include Subdomains

Configures the crawler to collect URLs from all subdomains associated with the target domain.

Examples:

• api.example.com
• admin.example.com
• cdn.example.com
• old.example.com

Deduplicate URLs

Removes duplicate URLs returned by multiple archive providers before crawling begins.

This reduces unnecessary requests and improves performance.

Follow Redirects

Configures the crawler to automatically follow HTTP redirects encountered while crawling archived URLs.

Ignore URL Parameters

Normalizes URLs by removing query parameters when performing deduplication.

Example:

The following URLs would be treated as the same resource:

• https://example.com/page?id=1
• https://example.com/page?id=2
• https://example.com/page?id=3

Crawl Depth

Controls how deep the crawler should continue following links discovered from archived pages.

• Depth 0: Crawl only archive-discovered URLs
• Depth 1: Follow links found on archive-discovered URLs
• Depth 2+: Continue recursively following discovered links

URL Limit

Limits the maximum number of archive URLs that will be crawled.

Useful when archives contain millions of URLs.

Archive Collection Process

Crawling