mirrors/stormcrawler

Keep the fetcher off private address space by default (#2127)

Keep the fetcher off private address space by default (#2080)

The IP address filter was the only check that looked at the address the fetcher actually connects to, and both of its keys shipped commented out, so the interceptor was never installed at library defaults. The regex exclusion list in the archetypes matched literal localhost and four dotted-quad ranges and nothing else: link-local, CGNAT, IPv6 unique-local and the abbreviated and integer IPv4 forms the resolver maps to 127.0.0.1 all passed it.

crawler-default.yaml ships http.filter.ipaddress.exclude enabled (loopback, RFC1918, link-local, CGNAT, IPv6 unique-local); a fetched page decides which hosts the fetcher connects to, and these ranges host unauthenticated services a public index must not leak into. Crawling an intranet needs an explicit opt-out, documented in the comment next to the key
the archetype default-regex-filters.txt (and its opensearch/solr copies) gains rules for 169.254.0.0/16, 100.64.0.0/10, 0.0.0.0/8, IPv6 fc00::/7 and fe80::/10, the abbreviated loopback forms (127.1) and integer IPv4 hosts (2130706433), and says in its comment that a regex list judges the bytes of the URL only, so the IP filter stays the authoritative check

Fix the CGNAT regex, close case and IPv6-literal gaps (#2080)

Review feedback on #2127:

the CGNAT alternation blocked 100.60-100.63 (public space) and missed 100.65-100.69 entirely: the second octet runs 64 to 127, so the first branch is 6[4-9]. Tests pin 100.65.0.1 as blocked and 100.63.0.1 as accepted, both ends of the mistake
RegexURLFilter compiles case-sensitively, so LOCALHOST, [FD00::1] and [FE80::1] slipped past the rules: the private-range rules carry (?i)
::ffff:127.0.0.1 and 0:0:0:0:0:0:0:1 resolve to loopback through the JVM resolver and matched nothing: a rule catches the IPv4-mapped and fully expanded loopback literals
the yaml opt-out advice was wrong: exclude rules win over include rules in IPFilterRules.accept, so an operator has to override http.filter.ipaddress.exclude explicitly; the comment says so, and the shipped exclude list covers the full ULA range (fc00::/7)
core/src/test/resources/default-regex-filters.txt is back to its upstream content: RegexFilterTest and URLFilterBoltTest load it, and overwriting it here was an unrelated change. The archetype rules ship as default-regex-filters-archetype.txt for the new test instead
the description documents that the resolver rejects hex (0x7f000001) and does not read leading zeros as octal, so the \d{1,10} rule closes the only numeric form that actually works

Normalize imports and formatting to satisfy CI

The import groups are collapsed into the single ASCII-sorted group the project checkstyle config demands (STATIC###THIRD_PARTY_PACKAGE), the constructors in URLFilters are grouped as checkstyle requires, and the touched files are reformatted with the google-java-format version the git-code-format plugin pins (1.35.0, AOSP). No code changes.

Close the 0/8 and IPv6-unspecified gaps in the shipped IP policy (#2080)

Review feedback on #2127:

the shipped http.filter.ipaddress.exclude omitted 0.0.0.0/8 and the IPv6 unspecified address: IPFilterRules.accept returned true for 0.0.0.0, 0.1.2.3 and ::, and blocking the literal URLs in the regex does not cover a hostname that resolves to them. Both are added (0.0.0.0/8 and ::/128); ::/128 rather than an ::/0 range so no real address is excluded

core/src/test/resources/default-regex-filters.txt restored to its pre-PR content; this PR’s fixture is the separate default-regex-filters-archetype.txt, so RegexFilterTest and URLFilterBoltTest are untouched as before

the shipped-default test now walks every excluded range with its boundary addresses and the public neighbours, plus the documented explicit opt-out

HttpProtocolIPFilterTest proves the policy is enforced by the actual HTTP protocol, not just the standalone predicates: a fetch of http://localhost (resolves to loopback) throws the interceptor’s Forbidden IOException under the shipped list, and the same fetch succeeds once the exclude list is overridden to empty

Do not apply the IP address filter to fetches through a proxy

Behind a proxy the socket is connected to the proxy, which resolves the target host itself, so the filter only ever saw the proxy’s address. With the new default exclude list, an HTTP proxy on a private range failed every fetch. The interceptor now lets proxied connections through, a log line at configure time says so, and the yaml and configuration.adoc document that the proxy’s own egress rules apply there.

configuration.adoc also shows the new default of http.filter.ipaddress.exclude and no longer describes the filter as optional.

Co-authored-by: Richard Zowalla rzo1@apache.org

Apache StormCrawler is an open source collection of resources for building low-latency, scalable web crawlers on Apache Storm. It is provided under Apache License and is written mostly in Java.

Quickstart

NOTE: These instructions assume that you have Apache Maven installed. You will need to install Apache Storm 3.0.0 to run the crawler.

StormCrawler requires Java 25 or above. To execute tests, it requires you to have a locally installed and working Docker environment.

Once Storm is installed, the easiest way to get started is to generate a new StormCrawler project following the instructions below:

mvn archetype:generate -DarchetypeGroupId=org.apache.stormcrawler -DarchetypeArtifactId=stormcrawler-archetype -DarchetypeVersion=<CURRENT_VERSION>

Be sure to replace with the latest released version, which you can find on search.maven.org.

You’ll be asked to enter a groupId (e.g. com.mycompany.crawler), an artefactId (e.g. stormcrawler), a version, a package name and details about the user agent to use.

This will not only create a fully formed project containing a POM with the dependency above but also the default resource files, a default CrawlTopology class and a configuration file. Enter the directory you just created (should be the same as the artefactId you specified earlier) and follow the instructions on the README file.

Alternatively if you can’t or don’t want to use the Maven archetype above, you can simply copy the files from archetype-resources.

Have a look at crawler.flux, the crawler-conf.yaml file as well as the files in src/main/resources/, they are all that is needed to run a crawl topology : all the other components come from the core module.

Getting help

The documentation is a good place to start your investigations but if you are stuck please use the tag stormcrawler on StackOverflow or ask a question in the discussions section.

The project website has a page listing companies providing commercial support for Apache StormCrawler.

Note for developers

Please format your code before submitting a PR with

mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false

You can enable pre-commit format hooks by running:

mvn clean install -Dskip.format.code=false

Building from source

The requirements for building from source are as follows

JDK 25+
Apache Maven 3
Docker (if you want to run tests)

The build itself is straightforward:

mvn clean install

Note: We use some binary files for testing advanced crawler functionality. These files are located exclusively in the src/test directories of the respective modules.

Thanks

alt tag

YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.