Links Crawler

This guide walks you through the process of crawling a website using Links crawler.

Why and when to use links crawler

Best for: Processing existing link lists

Fetches and processes URLs from files like sitemaps/project_files or link lists.

Use when:

• Processing exported URLs
• Validating link lists

Configurations

Use HTTP2

Configures the use of HTTP version 2 protocol for request to the target server. If a server turns out to not support HTTP/2 the spider will fall back to using HTTP version 1 by default. HTTP/2 is the second version of the HTTP protocol aiming to make applications faster, simpler, and more robust by improving many of the drawbacks of the first HTTP version. Has features including;

• Binary protocols - Binary protocols consume less bandwidth, are more efficiently parsed and are less error-prone than the textual protocols used by HTTP/1.1.
• Multiplexing - HTTP/2 is multiplexed, i.e., it can initiate multiple requests in parallel over a single TCP connection. As a result, web pages containing several elements are delivered over one TCP connection.
• Header compression - HTTP/2 uses header compression to reduce the overhead caused by TCP’s slow-start mechanism.
• Server push - HTTP/2 servers push likely-to-be-used resources into a browser’s cache, even before they’re requested. This allows browsers to display content without additional request cycles.
• Increased security - Web browsers only support HTTP/2 via encrypted connections, increasing user and application security.

HTTP Pipelining

Configures the spider to use http-pipelining technique to increase speed and efficiency. Works when using HTTP/1.

HTTP pipelining is a feature of HTTP/1.1 which allows multiple HTTP requests to be sent over a single TCP connection without waiting for the corresponding responses. HTTP/1.1 requires servers to respond to pipelined requests correctly, with non-pipelined but valid responses even if server does not support HTTP pipelining. Despite this requirement, many legacy HTTP/1.1 servers do not support pipelining correctly, forcing most HTTP clients to not use HTTP pipelining. The technique was superseded by multiplexing via HTTP/2.

Show failed requests (4XX and 5XX)

Configures the crawler not to discard 4XX and 5XX error responses and instead, add them to database and send them to sitemap. Connection, SSL and other system errors will not be added to sitemap only error responses with 4XX and 5XX.

Crawling

Crawler Behavior