Do You Have Bad Bots? 4 Key Ways to Identify Malicious Bot Activity on Your Website

Malicious web bots are a rapidly growing threat, causing major damage to businesses across every industry. According to Imperva‘s 2021 Bad Bot Report, 25.6% of all web traffic comes from bad bots—a nearly 6% increase from the previous year. These automated attacks result in over $10 billion in losses annually from ad fraud, content scraping, inventory hoarding, and more.

As these malicious bots become more sophisticated, simply deploying a basic firewall is no longer enough. To fully protect your business, you need to be able to accurately identify bot activity and understand the techniques they use to avoid detection.

In this guide, we‘ll provide an in-depth look at the world of malicious bots and share expert tips for spotting their activity on your site. You‘ll learn:

• The key differences between good bots and bad bots
• The major types of malicious bots and how they operate
• 4 proven techniques for identifying bad bot activity
• Actionable steps to block bots and protect your site

If you suspect bots are skewing your web traffic and harming your business, read on to learn how to spot and stop them.

Understanding the Difference Between Good Bots and Bad Bots

First, it‘s important to note that not all web bots are malicious. In fact, many bots provide useful services and comprise a key part of the modern internet. Common good bots include:

• Search engine crawlers like Googlebot that index web pages for search results
• Website monitoring tools that test site speed and uptime
• Copyright bots that scan for unauthorized use of content
• Price comparison bots that help consumers find the best deals
• Chatbots that provide 24/7 customer support

Good bots identify themselves in their user agent string, respect robots.txt rules, and consume minimal server resources.

In contrast, bad bots are built to perform malicious, harmful, or deceptive activities such as:

• Web scraping to steal content and pricing data
• Click fraud and ad fraud to waste advertising spend
• Inventory hoarding to snap up limited-stock items
• Spam content posting and fake account creation
• Credential stuffing to take over user accounts
• Distributed Denial of Service (DDoS) attacks to overwhelm sites with traffic

Malicious bots are intentionally difficult to detect, using tactics like rotating IP addresses, manipulating user agent info, and mimicking human behavior. A 2021 survey found that 80% of companies have problems distinguishing bots from humans.

The High Cost of Malicious Bots

The impact of malicious bots goes far beyond wasted website traffic. Bots can cause extensive damage to businesses in the form of:

Beyond these direct costs, bots can also lead to skewed web analytics, slower site performance, and strained IT resources. In a recent survey, 54% of companies reported website downtime due to bots.

Technique 1: Analyze User Behavior for Bot Red Flags

One of the most reliable ways to identify bots is by analyzing how users interact with your site. Because bots are automated scripts, not humans, their behavior will often have unnatural patterns and red flags.

Key bot behavior indicators include:

• Unusually fast page loads and navigation, like viewing 100 pages in 10 seconds
• Consistently random or jerky mouse movements that don‘t align to page content
• Filling out forms with unrealistic speed, such as typing 500 wpm or more

To spot this suspicious activity, closely analyze user session recordings, heatmaps, and event tracking data. Compare typical human behavior metrics to the anomalies.

For example, if real users spend an average of 1 minute on your registration page, sessions that consistently complete the form in 2 seconds are almost certainly bots. Likewise, if human visitors normally pause an average of 5 seconds between page loads, users that navigate every 0.2 seconds are likely automated.

Technique 2: Monitor Traffic Patterns for Bot Clues

Analyzing your web traffic sources and patterns can also uncover clues about bot activity. Because bots are often launched in large batches from a single source, they leave distinct traffic patterns you can identify.

Bot traffic red flags to watch for:

• Unusually large spikes in traffic with no clear source like a marketing campaign
• High bounce rates and low time on site, indicating uninterested visitors
• Abnormal geolocation spread, like a surge in users from countries you don‘t serve
• Traffic spikes during off-hours for your business, like 3am on weekends

To identify these patterns, segment your traffic in your analytics by dimensions like:

• Geographic location: Country, state/region, city
• Time frame: Hour of day, day of week, month
• Device type: Desktop vs mobile vs tablet
• Traffic source: Direct, organic search, paid ads, referring sites

Look for segments and time periods that have a significantly different profile from your usual human visitors. For example, if your normal audience is 90% US desktop users arriving from Google, but you suddenly see a flood of mobile traffic from Vietnam in the middle of the night, you likely have a bot problem.

Technique 3: Inspect for Manipulated User Data

Because bots don‘t want to be detected and blocked, they often manipulate the data they send to websites to try to blend in with human visitors. Common bot cloaking techniques include:

• Using headless browsers that don‘t load JavaScript tracking code
• Rotating user agent strings to pretend to be various device/browser combos
• Connecting via proxy servers or VPNs to hide their hosting location
• Cycling through IP addresses to distribute traffic and avoid rate limits

To spot this suspicious data, compare front-end tracking data to back-end server logs. For example, if you see a high volume of traffic that doesn‘t have a normal browser fingerprint or doesn‘t trigger typical JavaScript events, it‘s likely bot traffic.

Also utilize IP reputation databases to check traffic sources against lists of known bot hosting services, anonymizing proxies, VPNs, and spoofed user agents. Over 38% of bad bots disguise themselves as Google Chrome, and the majority route through cloud hosting services like Amazon Web Services.

Technique 4: Compare Conversion Rates to Detect Bots

Conversion rate is another powerful bot indicator. Because the goal of many malicious bots is to interact with your site but not actually make a purchase, their activity will often skew your conversion funnel.

Potential signs of conversion-skewing bots include:

• Abnormally high bounce rates on product and pricing pages
• Adding items to cart but never completing checkout
• Unusually high traffic to signup/login pages vs low account activity
• Filling out lead forms with fake or low-quality contact info

To identify these discrepancies, benchmark your typical human conversion rates for each funnel stage and compare that to different traffic segments.

For example, if verified human users have a 1.5% conversion rate from visit to purchase, but certain geographies or time windows have a 0.01% rate, you likely have a bot clicking through to product pages but not actually converting.

Putting Your Bot Identification Findings Into Action

Once you‘ve uncovered your malicious bot traffic, it‘s time to take action to block their access and prevent future attacks. Effective bot-fighting techniques include:

• Blocking all traffic from known bot hosting services and proxy networks
• Requiring CAPTCHAs or other bot challenge tools on key conversion points
• Strengthening login security with multi-factor authentication and password requirements
• Limiting the number of actions (searches, form fills, etc) allowed per IP/user
• Implementing a dedicated bot management solution to monitor and deflect attacks

Above all, protecting your site from malicious bots requires constant vigilance. Because attack techniques continuously evolve, your bot-fighting strategy must evolve too.

By proactively analyzing your site‘s user behavior and server data using the techniques outlined in this guide, you can stay one step ahead of bad bots and keep them from harming your business. And if you need expert help assessing your bot traffic and improving your defenses, contact the bot protection pros at Bright Data.