Over the last year or so, I’ve noticed some unusual traffic on my site. It’s a bot, typically, one hitting every file on my site, but not affiliated with a known search engine such as Google. Being the curious type, I tend to plug in IP addresses to this nify “whois” service in order to find out who’s playing around on my site.
The first bot I noticed was Cyveillance. This one has been around a while; at least since 2003. The company’s website claims that it “is focused on helping organizations monitor the Internet for issues such as identity theft, fraud, security risks, unauthorized product distribution and many forms of brand abuse.” It’s interesting that this bot never identifies itself as Cyveillance; you can only tell that it’s a bot by watching the rate at which it chews through your bandwidth.
The second one I observed was NameProtect, which bills itself as a company aiming to “empower our clients with proactive, filtered and actionable eMarket Intelligence supporting the protection of brand assets, recovery of diverted revenues and detection of online identity theft & fraud in today’s global economy.” Again, this bot did not identify itself.
The most recent one, which I observed last week, was MarkMonitor. This company claims to specialize “in protecting corporate intellectual property online from infringers, scam artists and online black-market activities.” The bot, unlike the other two, did identify itself as a Markmonitor bot.
I found an article at CSOOnline where someone from World Wrestling Entertainment was discussing their use of MarkMonitor to find everything from black market DVDs to misappropriated website images.
I suppose overall I don’t really blame companies for tracking this sort of thing, but it does occur to me that these companies are making money (in part) from my bandwidth and that of a million other small website owners. I imagine that the majority of website owners these days are not sitting around trying to figure out how to steal your intellectual property, and if some evil bastard does steal your property, somebody eventually will complain. Hell, your money would be better spent if you paid web-savvy teenagers to tattle on bad, wicked website people; they seem to be jumping at the chance for spare cash. You won’t generate goodwill by stealing bandwidth from people who buy your products every day.
Furthermore, Googling these various bots (and indeed, my own personal experience) reveals that the bots are not well-behaved guests. They completely ignore robots.txt instructions, crawling wherever they please on your site. Some people have gone so far as to ban blocks of IP addresses from these companies. I’m too lazy to do that.
Not completely lazy, though. Seeing these bots on my site gives me a near-irresistable urge to pull a South Park movie Cartman and scream “Coca Cola IBM Nike Armani Gap Honeywell McDonald’s Sony Hershey’s Mattel Harley Davidson Jesus H. Christ!!!!!!!”
Just in the case, you know, that the IP Police are listening.