George Hubka

Ever wonder how to find out what robots/crawlers have visited your site? If you don’t have robust log analyzer there is another way. If you are using apache there is a simple way from the command line to figure out how as been visiting.

grep robots.txt /etc/httpd/domlogs/bumideas.com|awk -F '"' '{print $6}'|sort -k 1|uniq -c

What is returned is a list of the robots and the number of visits reported in the log.

9 BlogPulseLive (support@blogpulse.com)
53 ia_archiver
18 ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
32 ia_archiver-web.archive.org
4 Mediapartners-Google
2 Mozilla/4.0 (compatible: FDSE robot)
2 Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communications Corp., Korea; +http://ws.daum.net/aboutkr.html)
6 Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
49 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
94 Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
16 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16
1 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
8 msnbot/1.1 (+http://search.msn.com/msnbot.htm)
16 msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
2 msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
4 panscient.com
3 SurveyBot/2.3 (Whois Source)