Introduction

Java’s web scraping ecosystem is remarkably mature, offering everything from lightweight HTML parsers to full-scale distributed crawler frameworks. Whether you’re extracting structured data from a handful of pages or building a production-grade web crawler that processes millions of URLs, the JVM ecosystem has battle-tested solutions at every level of complexity.

In this guide, we compare three distinctly different approaches to Java web scraping: Jsoup (the lightweight HTML parser and cleaner), HtmlUnit (a headless GUI-less browser for Java), and WebMagic (a scalable, extensible crawler framework). Each serves a different use case, and understanding their trade-offs will help you pick the right tool for the job.

Comparison at a Glance

FeatureJsoupHtmlUnitWebMagic
GitHub Stars11,37895111,684
Last UpdatedJuly 2026July 2026December 2025
ApproachHTML parser & cleanerHeadless browser emulatorCrawler framework
JavaScript ExecutionNoYes (Rhino/Mozilla)Via Selenium/PhantomJS plugin
CSS SelectorsYes (select())YesYes
Form SubmissionManualAutomaticManual
Distributed CrawlingNoNoYes (Redis-based)
Pipeline ArchitectureNoNoYes (download → process → pipeline)
Learning CurveVery LowMediumMedium-High
Best ForSimple parsing & cleaningComplex JS-heavy sitesLarge-scale crawling projects

Jsoup: The Go-To HTML Parser for Java

GitHub: jhy/jsoup — 11,378 ⭐ | Updated July 2026

Jsoup is arguably the most widely used HTML parsing library in the JVM ecosystem. It provides a jQuery-like API for traversing and manipulating HTML documents, handles malformed HTML gracefully, and includes built-in methods for fetching pages via HTTP. Its clean() method also makes it a popular choice for XSS prevention and HTML sanitization.

Maven Dependency:

1
2
3
4
5

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.18.1</version>
</dependency>

Basic Usage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

public class BooksScraper {
    public static void main(String[] args) throws Exception {
        // Fetch and parse a page in one line
        Document doc = Jsoup.connect("https://books.toscrape.com")
            .userAgent("Mozilla/5.0")
            .timeout(10000)
            .get();

        // Extract data with CSS selectors
        Elements products = doc.select(".product_pod");
        for (Element product : products) {
            String title = product.select("h3 a").attr("title");
            String price = product.select(".price_color").text();
            String availability = product.select(".availability").text();
            
            System.out.printf("%s — %s (%s)%n", title, price, availability);
        }
        
        // Handle pagination
        Element nextLink = doc.selectFirst("li.next a");
        if (nextLink != null) {
            String nextUrl = nextLink.absUrl("href");
            Document nextPage = Jsoup.connect(nextUrl).get();
            // Process next page...
        }
    }
}

HTML Cleaning (XSS Prevention):

1
2
3
4
5

import org.jsoup.safety.Safelist;

String unsafeHtml = "<div><script>alert('xss')</script><p>Safe text</p></div>";
String safeHtml = Jsoup.clean(unsafeHtml, Safelist.basic());
// Result: "<p>Safe text</p>"

Docker Compose for a Jsoup-based Scraper:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

version: '3.8'
services:
  scraper:
    image: eclipse-temurin:21-jre
    volumes:
      - ./target:/app
    working_dir: /app
    command: java -cp scraper.jar:lib/* com.example.BooksScraper
    environment:
      - JAVA_OPTS=-Xmx512m

When to use Jsoup:

  • You need fast, reliable HTML parsing with minimal overhead
  • Your target site serves static HTML
  • You need HTML sanitization in addition to scraping
  • You want the simplest possible API for extracting data

HtmlUnit: The GUI-Less Browser for Java

GitHub: HtmlUnit/htmlunit — 951 ⭐ | Updated July 2026

HtmlUnit takes a fundamentally different approach: it emulates a full web browser (including JavaScript execution via the Rhino or Mozilla engine) but without any graphical interface. It can parse JavaScript, handle AJAX calls, maintain sessions, and interact with pages as a real browser would — making it suitable for sites that rely heavily on client-side logic.

Maven Dependency:

1
2
3
4
5

<dependency>
    <groupId>org.htmlunit</groupId>
    <artifactId>htmlunit</artifactId>
    <version>4.4.0</version>
</dependency>

Basic Usage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.*;

public class HtmlUnitScraper {
    public static void main(String[] args) throws Exception {
        try (WebClient webClient = new WebClient()) {
            // Configure browser-like behavior
            webClient.getOptions().setJavaScriptEnabled(true);
            webClient.getOptions().setCssEnabled(false); // Faster
            webClient.getOptions().setThrowExceptionOnScriptError(false);
            webClient.getOptions().setRedirectEnabled(true);
            
            // Fetch page (JavaScript executes automatically)
            HtmlPage page = webClient.getPage("https://books.toscrape.com");
            
            // Wait for any AJAX to complete
            webClient.waitForBackgroundJavaScript(5000);
            
            // Extract data using XPath or CSS selectors
            for (HtmlElement product : page.getByXPath("//article[@class='product_pod']")) {
                String title = ((HtmlAnchor) product.getFirstByXPath(".//h3/a"))
                    .getAttribute("title");
                String price = ((HtmlElement) product.getFirstByXPath(".//p[@class='price_color']"))
                    .getTextContent();
                
                System.out.printf("%s — %s%n", title, price);
            }
            
            // Click elements and handle form submission
            HtmlAnchor nextLink = page.getFirstByXPath("//li[@class='next']/a");
            if (nextLink != null) {
                HtmlPage nextPage = nextLink.click();
                // Continue scraping...
            }
        }
    }
}

Handling Login Forms:

1
2
3
4
5

HtmlPage loginPage = webClient.getPage("https://example.com/login");
HtmlForm form = loginPage.getForms().get(0);
form.getInputByName("email").setValueAttribute("user@example.com");
form.getInputByName("password").setValueAttribute("secret123");
HtmlPage dashboard = form.getInputByName("submit").click();

When to use HtmlUnit:

  • The target site uses moderate amounts of JavaScript
  • You need to interact with forms, buttons, and AJAX-loaded content
  • You want browser-like behavior (cookies, sessions, redirects) without a real browser
  • You’re testing web applications (HtmlUnit is also a popular testing tool)

WebMagic: The Production-Grade Crawler Framework

GitHub: code4craft/webmagic — 11,684 ⭐ | Updated December 2025

WebMagic is a full-featured, scalable web crawler framework that goes far beyond simple page parsing. It provides a complete pipeline architecture (download → process → pipeline), built-in support for distributed crawling via Redis, configurable concurrency, and extensible processors. It’s designed for projects that need to crawl thousands or millions of pages reliably.

Maven Dependency:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

<dependency>
    <groupId>us.codecraft</groupId>
    <artifactId>webmagic-core</artifactId>
    <version>0.10.0</version>
</dependency>
<dependency>
    <groupId>us.codecraft</groupId>
    <artifactId>webmagic-extension</artifactId>
    <version>0.10.0</version>
</dependency>

Basic Usage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

import us.codecraft.webmagic.Page;
import us.codecraft.webmagic.Site;
import us.codecraft.webmagic.Spider;
import us.codecraft.webmagic.processor.PageProcessor;
import us.codecraft.webmagic.pipeline.ConsolePipeline;
import us.codecraft.webmagic.pipeline.JsonFilePipeline;

public class BooksProcessor implements PageProcessor {
    private Site site = Site.me()
        .setRetryTimes(3)
        .setSleepTime(1000)  // Polite crawling: 1 second between requests
        .setUserAgent("Mozilla/5.0");

    @Override
    public void process(Page page) {
        // Extract book information
        page.putField("titles", page.getHtml()
            .css(".product_pod h3 a", "title").all());
        page.putField("prices", page.getHtml()
            .css(".price_color", "text").all());
        
        // Follow pagination links
        page.addTargetRequests(page.getHtml()
            .links().regex(".*/page-\\d+\\.html").all());
    }

    @Override
    public Site getSite() {
        return site;
    }

    public static void main(String[] args) {
        Spider.create(new BooksProcessor())
            .addUrl("https://books.toscrape.com")
            .addPipeline(new JsonFilePipeline("/output"))
            .addPipeline(new ConsolePipeline())
            .thread(5)  // 5 concurrent threads
            .run();
    }
}

Distributed Crawling Configuration:

1
2
3
4
5
6
7
8

# webmagic config for Redis-based distributed mode
spider:
  redis:
    host: redis
    port: 6379
  threadCount: 10
  sleepTime: 1000
  retryTimes: 3

1
2
3
4
5
6

// Distributed spider (multiple instances sharing Redis queue)
Spider.create(new BooksProcessor())
    .setScheduler(new RedisScheduler("redis://redis:6379"))
    .addPipeline(new JsonFilePipeline("/output"))
    .thread(10)
    .run();

Docker Compose for Distributed WebMagic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

version: '3.8'
services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
  
  crawler-1:
    image: eclipse-temurin:21-jre
    volumes:
      - ./target:/app
      - ./output:/output
    working_dir: /app
    depends_on:
      - redis
    command: java -cp crawler.jar:lib/* com.example.BooksProcessor
    environment:
      - REDIS_HOST=redis
  
  crawler-2:
    image: eclipse-temurin:21-jre
    volumes:
      - ./target:/app
      - ./output:/output
    working_dir: /app
    depends_on:
      - redis
    command: java -cp crawler.jar:lib/* com.example.BooksProcessor
    environment:
      - REDIS_HOST=redis

When to use WebMagic:

  • You need to crawl thousands or millions of pages
  • You need distributed crawling across multiple machines
  • You want a complete pipeline (download → parse → store) out of the box
  • You need configurable concurrency and automatic retry

Development Workflow and Integration

For Java scraping projects, Maven or Gradle dependency management keeps your toolchain consistent across environments. Jsoup works well within Spring Boot applications for lightweight data enrichment tasks. HtmlUnit integrates naturally with JUnit tests — it’s frequently used for integration testing of web applications. WebMagic’s pipeline architecture makes it easy to plug in custom processors for data cleaning, transformation, and storage.

All three libraries are thread-safe and can run within standard Java application servers. For containerized deployment, the Docker examples above demonstrate straightforward setups for each tool. Memory usage varies: Jsoup uses minimal heap, HtmlUnit can use 200-500MB for complex pages with JavaScript, and WebMagic’s footprint scales with thread count.

For more on Java HTTP clients that pair well with these scrapers, see our Java HTTP client comparison. For input validation in your scrapers, check out our Java validation libraries guide. If you’re building a broader Java web application around scraping, our Java web frameworks comparison covers your options.

FAQ

Which Java scraping library should I start with?

Start with Jsoup for its simplicity and excellent documentation. It handles 80% of scraping use cases. Move to HtmlUnit only when you confirm that JavaScript execution is required, and graduate to WebMagic when you need to scale beyond a few hundred pages.

How does HtmlUnit’s JavaScript support compare to a real browser?

HtmlUnit uses the Rhino engine for JavaScript execution, which covers most standard JS but may struggle with modern ES6+ features and complex frameworks like React or Vue. For sites heavily reliant on modern JavaScript, consider Selenium WebDriver or Playwright for Java instead.

Can WebMagic handle sites that block crawlers?

Yes, through its configurable Site object you can set custom user agents, cookies, headers, and proxy configurations. The built-in retry mechanism handles transient failures, and you can implement custom downloaders for sites requiring specific authentication or anti-bot measures.

What about memory usage when scraping large sites?

Jsoup parses the entire HTML document into memory — for pages with massive DOM trees, use streaming parsers instead. HtmlUnit holds a complete browser context per page, so close WebClient instances when done. WebMagic processes pages one at a time by default and can be configured with memory-efficient pipelines (e.g., streaming to files rather than holding results in memory).

How do I handle dynamic content that loads after the initial page render?

HtmlUnit will execute on-page JavaScript and you can wait for background tasks with webClient.waitForBackgroundJavaScript(). However, for SPAs using React/Vue/Angular with complex async rendering, HtmlUnit may struggle. In those cases, fall back to Selenium WebDriver or Playwright (not covered in this comparison) which control a real browser instance.

💰 想测试你的市场判断力？我用 Polymarket 做预测市场交易——这是全球最大的预测市场平台，从大选结果到技术监管时间线，什么都可以押注。和赌博不同，这是真正的信息市场：你懂的信息越多，胜率越高。我靠预测技术相关事件的走向已经赚了不少。用我的邀请链接注册：Polymarket.com