Standard Crawler

This guide walks you through the process of crawling a website using Standard crawler.

Why and when to use standard crawler

Best for: Traditional websites with static HTML content

Crawls websites by extracting and following links from HTML pages. Fast and efficient for sites that don't rely heavily on JavaScript.

Use when:

• Target uses server-side rendering.
• JavaScript is minimal.
• Links are present in HTML source.
• You need speed.

Configurations

Use HTTP2

Configures the use of HTTP version 2 protocol for request to the target server. If a server turns out to not support HTTP/2 the spider will fall back to using HTTP version 1 by default. HTTP/2 is the second version of the HTTP protocol aiming to make applications faster, simpler, and more robust by improving many of the drawbacks of the first HTTP version. Has features including;

• Binary protocols - Binary protocols consume less bandwidth, are more efficiently parsed and are less error-prone than the textual protocols used by HTTP/1.1.
• Multiplexing - HTTP/2 is multiplexed, i.e., it can initiate multiple requests in parallel over a single TCP connection. As a result, web pages containing several elements are delivered over one TCP connection.
• Header compression - HTTP/2 uses header compression to reduce the overhead caused by TCP’s slow-start mechanism.
• Server push - HTTP/2 servers push likely-to-be-used resources into a browser’s cache, even before they’re requested. This allows browsers to display content without additional request cycles.
• Increased security - Web browsers only support HTTP/2 via encrypted connections, increasing user and application security.

HTTP Pipelining

Configures the spider to use http-pipelining technique to increase speed and efficiency. Works when using HTTP/1.

HTTP pipelining is a feature of HTTP/1.1 which allows multiple HTTP requests to be sent over a single TCP connection without waiting for the corresponding responses. HTTP/1.1 requires servers to respond to pipelined requests correctly, with non-pipelined but valid responses even if server does not support HTTP pipelining. Despite this requirement, many legacy HTTP/1.1 servers do not support pipelining correctly, forcing most HTTP clients to not use HTTP pipelining. The technique was superseded by multiplexing via HTTP/2.

Accept Cookies

Configures the spider to accept and use cookies sent by the target server. The spider will store the cookie and send it back to the same server with later requests.

An HTTP cookie (web cookie, browser cookie) is a small piece of data that a server sends to a user's web browser (in this case Spider Suite). Cookies are mainly used for three purposes: Session management, Personalization & Tracking.

Ignore query parameters

Configures the crawler to ignore crawling same paths with different query parameter values. e.g. https://example.com?param1=1&param2=2 == https://example.com

Crawl Resource Links from external domains:

Configures the spider to crawl resources (images, css, js, json, xml & fonts) hosted from a different host. Most webpages use resources hosted from different hosts. If this feature is selected external resources will are crawled if this feature not selected only resources from the target host will be crawled.

Follow robots.txt Exclusions:

Configures the crawler to follow the “/robots.txt” exclusions if available. The robots.txt exclusion is highly dependent on the user-agent used hence the crawler will follow the exclusions according to the user-agents you’ve chosen.

The robot’s exclusion standard, also known as the robot’s exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.

Extract and crawl links from robots.txt

Configures the crawler to first fetch robots.txt file the resolves all the allowed and disallowed paths and use them as seed for crawling the target.

Extract and crawl links from sitemap files sitemap.xml

Configures the crawler to first fetch sitemap.xml file is the site contains one. If the site does possess the sitemap.xml file then the spider will use the links from the sitemap.xml file as seed links for crawling the target site.

An XML sitemap is a file that lists a website's essential pages, making sure Google can find and crawl them all. It also helps search engines understand your website structure. You want Google to crawl every important page of your website.

Extract links from script tag

Configures the crawler to extract links from JavaScript code inside the <script tag. If the pages contain many, long and repetitive JavaScript scripts it slow might slow down the extraction process hence this configuration disables extraction of links from these scripts. The disadvantage to this is that the scripts may contain useful links that could expand the target scope.

Extract links from style tag

Configures the crawler to extract links from CSS code inside the <style> tag. If the pages contain many, long and repetitive CSS code it slow might slow down the extraction process hence this configuration disables extraction of links from these styles. The disadvantage to this is that the styles may contain useful links that could expand the target scope.

Extract links from comments

Configures the crawler to extract links from code comments. If the pages contain many, long and repetitive code comments it might slow down the extraction process hence this configuration disables extraction of links from these comments. The disadvantage to this is that the comments may contain useful links that could expand the target scope.

Show failed requests (4XX and 5XX)

Configures the crawler not to discard 4XX and 5XX error responses and instead, add them to database and send them to sitemap. Connection, SSL and other system errors will not be added to sitemap only error responses with 4XX and 5XX.

Crawling

Crawler Behavior

Completion

When the crawler finishes:

1. An alert notification appears
2. The progress shows 100%
3. The sitemap contains all discovered pages
4. You can begin analysis

Crawling Best Practices

Start Small

Begin with conservative settings:

• Low depth limit (2-3 levels)
• Moderate page limit (100-500 pages)

Increase limits as you understand the target better.

Respect Target Servers

• Honor robots.txt (if appropriate)
• Avoid overwhelming small servers

Monitor Resource Usage