In July, Yahoo announced that they had hired Google famed Marissa Mayer as their new CEO. Unlike everyone else in the tech sector, I was disappointed. My disappointment stemmed from the fact that for the last several years Yahoo has been the surefire place where things go to die. This made me sad because Mayer has always been at the center of innovation as Google’s Vice President of Search Products and User Experience during the company’s most radical growth period. Yahoo seemed like a step down. But at the same time I have been hopeful that she could turn Yahoo around and build something amazing. Then this weekend something weird happened…
Over the weekend I was checking the traffic stats for one of my less serious projects with Statcounter and saw something odd in the visitor report:
When I saw “Yahoo! Slurp 3.0″ I was shocked and confused. Slurp is the name of Yahoo’s web crawler. Version 3.0 launched in April of 2008. However, almost a year later Yahoo quit crawling the web and replaced their results with Bing. Because of this I was almost certain that someone was spoofing their user agent to hide their identity. In fact many of the commercial scrappers on the market allow you the ability to change your user agent to what ever you want. There is even a handy little browser plugin that does this as well.
I posted a status on Facebook and soon learned that my buddy Meg Geddes has been seeing the same thing in Statcounter.
A buttload?? This got me thinking….so I went back to my Statcounter login and drilled down a bit further and this is what I found:
From the above we can see that the request is absolutely coming from Yahoo, that the client has Javascript enabled, and that a screen resolution is recorded. At first glance this looks like a typical request from a browser such as Chrome or Firefox. But why wouldn’t they just disclose that in the user agent? Because maybe this is a browser/crawler hybrid?
So this morning I wake up and can’t stop thinking about Meg’s buttload…Its one thing to see this type of behavior in random intervals, its another to have a buttload. (Also, I like repeating the term “buttload”, I am so sorry.) So, to get further to the bottum of this, I sent Meg an email with some questions:
JH: When did you first start seeing Slurp in your reports this year?
MEG: I don’t know when they started, but I probably first noticed around late spring.
JH: Is there ever any Referring URLs?
MEG: No there are never any referring URLs.
JH: How many pages per domain does it typically visit?
MEG: Probably around 50-100 per day across many sites. Interestingly, I don’t think they’re showing up in Google Analytics, so it may well just be a Statcounter thing.
JH: Do your server logs show the same type of data that Statcounter does?
MEG: Yep.
JH: Peanut butter: crunchy or creamy?
MEG: Dude. Extra Crunchy Jif. Sometimes I add MORE peanuts. I’m hardcore like that.
Meg also shared with us this forum posting where Statcounter moderators point out that these might be image crawling bots which are indexing the 1 pixel image Statcounter uses to send visitor data back to their servers. However, last time I checked (just now), Yahoo’s image search still says, “Powered by Bing™” at the bottom.
Yahoo’s alliance with Bing is supposed to continue til 2019, but that doesn’t impede on Yahoo to continue developing their own internal search products. In fact, they could be developing something that integrates with a future product set that is not search related at all.
Either way its fun to think that maybe somewhere on the horizon there is the possibility of a more competitive search landscape with Mayer commanding a serious competitor. If that is ever the case, then shes already got a new fanboy!
If you thought this post was half way interesting, you might want to follow @joehall on Twitter. Hes a total nut bag over there, but worth a laugh.
