Free-Conversant Support / List of Known Robot User-Agent Fields
 Home   About Conversant   Free Sites   Hosting   Support   XML-RPC 

Search




Subject List of Known Robot User-Agent Fields
Posted 4/29/2001; 6:59 PM by Seth Dillingham
Last Modified 8/21/2001; 10:23 AM by Seth Dillingham
In Response To Conditional Macro: userAgentIsRobot (#2529)
Label Docs Read 36540
<Previous Next> Thread: Forward chronological view Reverse chronological view Hierarchical outline view Edit Reply

Here's the list of the User-Agent fields that we know are used by robots (such as web crawlers). There are currently three entries ending with an asterisk, which indicates that the name is a prefix. This prefix is used by many User-Agent's, and all of them are robots. For example, there are many different versions of the "FAST-WebCrawler" agent.


User-Agent:  " Email
Collector
"DIIbot/1.2 http://www.findsame.com/robot.html"

User-Agent:  A Email
Collector
AcoiRobot/1.0 libwww/5.3.2
Acoon Robot v1.01 (www.acoon.de)
AgentName/0.1 libwww-perl/5.50
AlkalineBOT/1.4 (1.4.0326.0 RTM)
AltaVista Intranet*
AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au; http://faq.anzwers.com.au/anzwerscrawl.html)
appie/1.1
Arachnoidea (arachnoidea@euroseek.com)
ArchitextSpider
asterias/2.0
Autonomy Spider

User-Agent:  B Email
Collector
BaiDuSpider
beholder (www.vigiltech.com/esensedisclaim.html)
BlogBot/1.1
bumblebee@relevare.com

User-Agent:  C Email
Collector
CherryPickerElite/1.0
CherryPickerSE/1.0
cosmos/0.7_(mihai.preda@xyleme.com)
cosmos/0.8_(robot@xyleme.com)
Crawl_Application
Crescent Internet ToolPak HTTP OLE Control v.1.0

User-Agent:  D Email
Collector
daypopbot/0.2
DIIbot/1.2 http://www.findsame.com/robot.html

User-Agent:  E Email
Collector
EmailCollector/1.0
EmailSiphon
EmailWolf 1.00
Explorer www@openxxx.net
ExtractorPro

User-Agent:  F Email
Collector
FAST-WebCrawler/*
fastlwspider/1.0
fido/1.0 Harvest/1.4.pl2

User-Agent:  G Email
Collector
GAIS Robot/1.0B2
gazz/2.1 (gazz@nttrd.com)
gigabaz/3.14 (baz@gigabaz.com; http://gigabaz.com/gigabaz/)
Googlebot-for-IDG/2.1 (+http://www.googlebot.com/bot.html)
Googlebot/2.1 (+http://googlebot.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Gulliver/1.3
Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)

User-Agent:  H Email
Collector
hcat/1.0
HLoader
HomePageSearch(hpsearch.uni-trier.de)
htdig/3.1.2 (webmaster@box.sk)
htdig/3.1.5 (unconfigured@htdig.searchengine.maintainer)
http://pawel.dyndns.org/ pol76@wanadoo.fr
httpGlooton/1.0

User-Agent:  I Email
Collector
ia_archiver
IncyWincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprovements.com/robot.html)
InfoNaviRobot(F107)
Infoseek Sidewinder/0.9
Inktomi Search
InternetAmi IOR/0.2 (http://internetami.se/ior.html)
InternetLinkAgent/3.1
InternetSeer.com

User-Agent:  J Email
Collector
JennyBot/0.1
jokescan/0.08

User-Agent:  K Email
Collector
Kenjin Spider
KIT-Fireball/2.0
KIT_Fireball/2.0
kobot/5.2.8 libwww/5.2.8

User-Agent:  L Email
Collector
larbin_1.2.2 larbin1.2.2@somewhere.com
larbin_1.2.2 nobody@nowhere
larbin_2.1.0 larbin2.1.0@somewhere.com
larbin_2.1.1 larbin2.1.1@somewhere.com
larbin_devel sebastien.ailleret@inria.fr
larbinDeViennot Laurent.Viennot@inria.fr
LEIA/3.01pr (LEIAcrawler; leia@gseek.com; http://www.gseek.com)
LexiBot/1.00
libWeb/clsHTTP -- hiongun@kt.co.kr
Linkbot
LinkLint-checkonly/2.1
Links (0.95; NetBSD 1.5U i386)
LinkWalker
LNSpiderguy
lwp-trivial/1.27
Lycos_Spider_(modspider)
Lycos_Spider_(T-Rex)
Lycos_Spider_(T-Rex)/3.0

User-Agent:  M Email
Collector
MARS SV
Mata Hari/2.00
MedicalMatrix
Mercator-1.0
Mercator-1.1
Microsoft URL Control - 6.00.8169
MIIxpc/4.2
moget/2.1 (moget@goo.ne.jp)
Mozilla/2.0 (compatible; Ask Jeeves)
Mozilla/2.0 (compatible; EZResult -- Internet Search Engine)
Mozilla/2.0 (compatible; NEWT ActiveX; Win32)
Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)
Mozilla/3.0 (compatible; Indy Library)
Mozilla/3.0 (compatible; MuscatFerret/1.4.1; olly@muscat.co.uk)
Mozilla/3.0 (compatible; MuscatFerret/1.5.2; olly@muscat.co.uk)
Mozilla/3.0 (compatible; MuscatFerret/1.5.3; olly@muscat.co.uk)
Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/4.0 (compatible; FastCrawler3, support-fastcrawler3@fast.no)
Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Microsoft
Mozilla/4.0 (compatible; MuscatFerret/2.0; http://www.webtop.com/)
Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)
Mozilla/4.04 [de] (Win95; I ;Kolibri gncwebbot)
Mozilla/4.04 [de] (Win95; I ;Nav; Kolibri gncwebbot)
Mozilla/4.1
MuscatFerret

User-Agent:  N Email
Collector
NationalDirectory-WebSpider/1.3

User-Agent:  O Email
Collector
Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)
Openfind piranha,Shark/0.95h+(tjsheu@gais.cs.ccu.edu.tw;+http://www.openfind.com/)

User-Agent:  P Email
Collector
PlantyNet_WebRobot_V1.9 dhkang@plantynet.com
PM WebSpyder V1.0
polybot 1.0
psbot/0.1 (+http://www.picsearch.com/bot.html)
psbot/0.1 (+http://www.picsearch.org/bot.html)

User-Agent:  Q Email
Collector
QUOSA

User-Agent:  R Email
Collector
RaBot/1.0 Agent-admin/webmaster@krivet2.or.kr
RealNamesBot/2.7
Robozilla/1.0

User-Agent:  S Email
Collector
Scooter-3.0.d1
scooter-3.0.DY
Scooter-3.0.EU
Scooter-3.0.FS
Scooter-3.0.g36-47
Scooter-3.0.g60-71
Scooter-3.0.g72-83
Scooter-3.0.g84-95
Scooter-3.0.s1
Scooter-W3-1.0
Scooter/1.0
Scooter/1.0 scooter@pa.dec.com
Scooter/1.1 (custom)
Scooter/2.0 G.R.A.B. V1.1.0
Scooter/2.0 G.R.A.B. X2.0
search.at V1.2
sexsearcher
Slurp.so/1.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp/2.0-condor_daily_west (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp/2.0-condor_hourly (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp/2.0-Redtail (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp/cat (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp/si-emb (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
SlySearch (fitzboy@bmrc.berkeley.edu)
SlySearch (slysearch@slysearch.com)
SlySearch fitzboy@bmrc.berkeley.edu
SlySearch slysearch@slysearch.com
SOFTWING_TEAR_AGENT_1_0
SpiderMan
Sqworm/2.9.43-BETA
Surfnomore Spider v1.1
SurfWatchSpider/2.0
SwishSpider
SwissSearch V1.2

User-Agent:  T Email
Collector
The Informant
tivraSpider/1.0
tivraSpider/1.0 (crawler@tivra.com)
TridentSpider3
True_Robot/1.0 libwww/5.2.8
TV33_Mercator_1-1.0

User-Agent:  U Email
Collector
Ultraseek
URL Checker

User-Agent:  V Email
Collector
VbTcpQuery sample (http://www.hexillion.com)
vischeck_spiderBot/0.1libwww-perl/5.48
vspider
vspider/3.6

User-Agent:  W Email
Collector
WebBandit/2.1
WebBandit/3.50
Webbandit/4.00.0
WebCrawler-AddURL/2.0
WebCrawler/3.0 Robot libwww/5.0a
WebTrends Link Analyzer
WiseDoc-Robot (Google WAP Proxy/1.0)
WiseWire
WiseWire*

User-Agent:  X Email
Collector
x-spyonit/1.1
Xenu's Link Sleuth 1.1a
xyro_(xcrawler@cosmos.inria.fr)

User-Agent:  Z Email
Collector
ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)
<Previous Next> Thread: Forward chronological view Reverse chronological view Hierarchical outline view Edit Reply
ENCLOSURES

None.
REPLIES

RE: List of Known Robots
5/2/2001 by Bradley Peters
This page (http://mosa.unity.ncsu.edu/brabec/antispam.html), created by a North

RE: List of Known Robot User-Agent Fields
6/15/2001 by Seth Dillingham
Morbus Iff from disobey.com contacted me today to ask me why I'd listed AmphetaDesk

RE: List of Known Robot User-Agent Fields
2/9/2011 by EddieS
Which one is for Bing? msnbot/1.0 (+http://search.msn.com/msnbot.htm) msnbot/1.1


TRACKBACKS