Web Scraping in Java With Jsoup: A Step-By-Step Guide

For the tags, there are multiple elements. We can select them all into an Elements collection again:


Elements quoteTags = quote.select(".tag");


Then loop through to get the individual tag strings:


List tags = new ArrayList<>();

for (Element quoteTag : quoteTags) {
tags.add(quoteTag.text());
}


At this point, we‘ve successfully extracted all the data we want from a single page of quotes. But there are multiple pages! Let‘s use pagination to scrape them all.

Handling Pagination

The quotes site uses a typical pagination system with "Next" links at the bottom of each page. By following those links, we can scrape every page.

To select the Next link:


Element next = doc.select(".next a").first();


This finds the first element within an element with class "next". The next step is extracting the URL to the next page from the attribute. But first we should check if there actually is a Next link (on the last page there isn‘t):


if (next != null) {
String nextUrl = next.attr("href");
} else {
// we‘re on the last page
}


Putting it all together into a loop:


String quotesUrl = "https://quotes.toscrape.com/";

while (true) {
// scrape quotes from current page
// ...

// check for next page  
Element next = doc.select(".next a").first();

if (next != null) {  
    // there is a next page
    String nextUrl = next.attr("href");
    quotesUrl = "https://quotes.toscrape.com" + nextUrl;
    doc = Jsoup.connect(quotesUrl).get();    
} else {
    // we‘re on the last page
    break;  
}

}


We start on the base URL "https://quotes.toscrape.com/". After scraping the quotes on the current page, we look for the Next link. If it exists, we extract its href value, concatenate it onto the base URL (since it‘s a relative URL), and fetch the next page. Once we reach the last page, the loop exits.

Now we‘re able to scrape all pages of quotes from the site!

Outputting Data to CSV

Finally, let‘s save our scraped data to a CSV file for easy use in other analysis and tools. We‘ll create a basic nested class to represent each "Quote" and store the data:


private static class Quote {
String text;
String author;
List tags;

public Quote(String text, String author, List<String> tags) {
    this.text = text;  
    this.author = author;
    this.tags = tags;  
}

}


Then after each quote is scraped, create a new Quote object and add it to a List:


List allQuotes = new ArrayList<>();

// ...

Quote quote = new Quote(quoteText, quoteAuthor, tags);
allQuotes.add(quote);


Finally to output the quotes data to CSV:


try (PrintWriter pw = new PrintWriter("quotes.csv")) {
pw.println("Quote|Author|Tags");

for (Quote quote : allQuotes) {
    String tags = String.join(",", quote.tags);
    pw.println(quote.text + "|" + quote.author + "|" + tags);
}

}


This writes each Quote‘s data to a file "quotes.csv" with pipe "|" used as a delimiter between the fields (since the quote text itself may contain commas) and each tag separated by commas.

And there we have it! A fully functioning web scraper built with Jsoup. Some additional things to consider:

Avoiding Getting Blocked When Web Scraping

Some websites don‘t like being scraped and will attempt to block requests they identify as coming from a scraper. Some techniques they use:

• User-Agent detection: Blocking requests with blank, missing, or known bot User-Agents
• Rate limiting: Blocking IPs that send too many requests too quickly
• Honeypot traps: Hiding links that only bots can find in order to catch and block them

To avoid these, try to make your scraper behave more like a normal user:

• Set a common browser User-Agent like we did above
• Introduce random delays between requests to simulate reading time
• Rotate proxy IPs and user-agents if scraping large amounts from individual sites
• Respect robots.txt if present

Alternatives to DIY Web Scraping

Depending on your specific needs, writing and running your own web scraper may not be necessary:

• Many sites offer APIs providing their data in structured formats
• Pre-made datasets are increasingly available covering common scraping needs
• Third-party scraping services and tools handle the scraping and data cleaning for you

If you‘re looking for a large amount of web data without the hassle of scraping it yourself, it‘s worth checking if a pre-made dataset or API already exists. Bright Data, for example, offers datasets on many topics as well as customized data collection services.

Conclusion

Web scraping is an incredibly useful skill, allowing you to collect data otherwise locked within websites. With the Jsoup library, Java developers can scrape websites with just a few lines of code.

In this guide, we covered all the steps to building a complete web scraper in Java:

• Setting up Jsoup in your project
• Fetching web pages and parsing the HTML
• Navigating the HTML document using CSS selectors to find the desired elements
• Extracting text, attributes, and HTML from elements
• Detecting and following pagination links to scrape all pages
• Saving the scraped data into a structured format like CSV
• Tips for avoiding anti-scraping countermeasures
• Alternatives for cases where pre-made datasets or scraping services are preferable

You should now have a solid foundation for scraping websites using Java and Jsoup. Remember to always be respectful when scraping and abide by the rules in robots.txt. Happy scraping!

If you want to learn more, check out our other articles on web scraping, Java, and working with data. And feel free to leave any questions or thoughts in the comments below!