<img height="1" width="1" style="display:none;" alt="" src="https://dc.ads.linkedin.com/collect/?pid=95044&amp;fmt=gif">
Back to blog

Bad Robot!

Thursday, October 08, 2015

No, I’m not talking about that tag you see at the end of any JJ Abrams production (LOST, the Star Trek reboots, countless other films & TV shows). I’m talking about those pesky automated web agents (AKA “bots”) that are constantly hitting websites every single day. But, why is that? One can’t always be sure, but we do know that they're out there, and they could very well be throwing off your site traffic and eating up bandwidth.

A couple of months ago, we noticed that some of our clients had a serious uptick in traffic. Unfortunately, we also saw a spike in bounce rates, which Google defines as the percentage of single page visits to your site. There could be many reasons for why a site has a high bounce rate, one of which being a site design that's not very user friendly, which can discourage users from sticking around.

It could be that your paid search ads are leading to landing pages that aren’t consistent with the messaging of those ads, and this can lead to an end user being unsatisfied and ditching you after they click through (we can definitely help you with that!).

Maybe you're getting a lot of bad traffic from a (ro)bot - a content scrapping bot - or a bot trying to harvest info for a number of malicious purposes. Either way, one thing is for sure - too much of this could skew your web analytics and lead you to believe that your site is underperforming. A spike in automated bot traffic can also lead you to think that there is something wrong with the aforementioned bounce rate.

Another new client of ours asked us to look into a period when their overall site conversion rate seemed to tank. After some investigation, we discovered what all of these clients had in common—an abnormal spike in traffic at around the same time. More specifically, all of this traffic was directed to the homepage, and even MORE specifically, it was direct traffic (no referrer) from the city of Ashburn, VA.

Strange, right? After doing a little more research, we found that it just so happens that some of Amazon’s servers reside there—in particular, Amazon’s Web Services, including its cloud computing service. At first we wondered why in the world Amazon would be sending out all these bots, but after a little more digging, we found that the more likely reason is NOT that Amazon itself is behind this, but it's instead entities that are using Amazon web servers in order to cover their own tracks.

So now that we have identified the bad robot traffic, what do we do next? 

If you are using Google Analytics, the first thing to do is to filter out traffic from known bots & spiders. To do this, simply go to your Google Analytics Admin-> View Settings-> then look for the Bot Filtering check box, and check it.  (See below)

Check out these traffic/bounce numbers in the 4 weeks after we filtered out bots, compared to the previous 4 weeks:

Our analytics data started normalizing the very next day, and it continues to do so. As much as I would love to say that Big Squid was responsible for a 13,490.02% increase in conversion rates of any kind, it’s pretty obvious that in addition to whatever nefarious purposes those bots originally had, it was also throwing off data that we use to help our clients get the most out of their marketing initiatives.

We were able to make sense of the bigger picture, and we were left with better usability data, but we aren’t patting ourselves on the back. Yes, we do have a clearer idea of bounce rates, but what are we going to do to improve that? We have a better grasp on overall site conversion rates, but what channels are driving that success?  How can we capitalize on this cleaner web traffic data?

We will save that for another day, but in the true spirit of JJ Abrams’ BAD ROBOT productions, we will leave you with this ending… turns out, Andrés wasn’t LOST in web analytics data. It was all just a dream.

Andrès Galdames

Written by Andrès Galdames

Recent Posts

Comment Below

Subscribe to Email Updates