Introduction to SpiderSuite

Overview

Let’s face it: manual recon is a time-sink. Spider Suite was built to handle the heavy lifting. It’s a powerful crawler that actually understands modern web architecture, giving you a readable map of a target’s entire footprint. If you’re hunting bugs or pentesting a complex app, it gives you the bird’s-eye view you need to spot vulnerabilities that others miss.

Requirements

Spider Suite runs on 64-bit machines only and is currently available for Windows and Linux operating systems. It does not support x32 systems.

Features

Crawlers

SpiderSuite includes five specialized crawler types:

  • Standard Crawler - Best for static pages and traditional websites
  • Headless Crawler - Perfect for JavaScript-heavy SPAs
  • Bruteforce Crawler - Directory and file discovery
  • Onion/TOR Crawler - Specialized for darkweb crawling
  • Links Crawler - Extract and crawl links from various file formats

Testing

  • Intercept - Man-in-the-middle proxy for HTTP(S) traffic
  • Request - Custom HTTP request builder
  • Fuzzer - Automated fuzzing with multiple techniques

Tools

  • Decoder Tool - A tool for encoding/decoding, encrypting/decrypting, compressing/decompressing data. Many encryption, encoding and compressor algorithms are supported.
  • Compare Tool - A tool for comparing different webpages. The comparison shows similarities and differences between the webpages.

Import

Import http(s) webpages obtained from other tools. Supported formats and Tools includes:

  • HTTP Archive (.har) - HTTP archive format that is supported by all web browsers
  • Web Archive (.warc & .warc.gz) - Web Archive format that is supported by most web archiving sites e.g. archive.org
  • Katana (.json & .txt) - Katana crawler export files
  • Caido (.csv & .json) - Caido proxy export files
  • Burpsuite (.xml) - BurpSuite tool export files
  • Fiddler (.saz) - Fiddler proxy export files
  • SpiderSuite (.json/.jsonl/.xml) - Spider Suite's own export file types

Export

Export http(s) webpages (content or links) to different storage formats. Supported formats includes:

  • JSON (.json & jsonl) - Stores the webpage's requests and responses to json or jsonl filetypes
  • XML (.xml) - Stores the webpage's requests and responses to xml filetype
  • HTML (.html) - Stores the webpage's URL, status, size, content-type and html title to a html filetype
  • Filesystem - Save the webpages to the filesystem. Each webpage is saved as a distinct file on the filesystem
  • Sitemap (.xml) - Saves the webpages into a sitemap file which appropriate location and last modification parameters
  • HTTP Archive (.har) - Saves the webpage's requests and responses into a http archive file
  • Web Archive (.warc / .warc.gz) - Saves the webpage's requests and responses into a web archive file
  • LLM Ready (.json/.jsonl) - Processes the HTML and JSON webpages into LLM-ready data then stores the data into JSON file

HTTPQL Filter

HTTPQL is a query language designed by the team at Caido for filtering and analyzing HTTP traffic. The HTTPQL syntax used by Spider Suite has been modified to suit the application's provided features.