标签归档:爬虫

小小网站竟也有这么多蜘蛛(robot,spider)光顾,要不要更新个robots呢

网站程序:wordpress,自带主题

更新情况:15年后大约一年半未更新,17年7月中旬有更新,半个月更新两篇文章

来访蜘蛛汇总:

mj12bot
“Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)”

AhrefsBot
“Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)”

seznambot
“Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)”

SEOkicks-Robot
“Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)”

YandexBot
“Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)”

DomainCrawler
“DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/linxiongxiong.com)”

常见蜘蛛:
googlebot
“Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

bingbot
“Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)”

Baiduspider
“Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)”

Sogou web spider
“Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)”

360Spider
“Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider”

Yahoo! Slurp
“Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)”

继续阅读