User-agent: Disallow: User-agent: Ad Muncher Disallow: User-agent: Anonymized by ProxyOS Disallow: User-agent: CE-Preload Disallow: User-agent: CJB.NET Proxy Disallow: User-agent: DeleGate Disallow: User-agent: DoCoMo Disallow: User-agent: Dual Proxy Disallow: User-agent: FairAd Client Disallow: User-agent: FANGCrawl Disallow: User-agent: Finjan-prefetch Disallow: User-agent: Goldfire Server Disallow: User-agent: Hatena Mobile Gateway Disallow: User-agent: http://Anonymouse.org/ Disallow: User-agent: HTTPEyes Disallow: User-agent: J-PHONE/J-SH07 Disallow: User-agent: KDDI-SN22 UP.Browser Disallow: User-agent: MagicWML Disallow: User-agent: Microsoft_Internet_Explorer0.438 Disallow: User-agent: MIIxpc Disallow: User-agent: via Avirt Gateway Server v4.0 Disallow: User-agent: via proxy gateway CERN-HTTPD libwww7 Disallow: User-agent: Covac UPPS Cathan;) Disallow: User-agent: Crayon; snprtz|T04056566514940; Disallow: User-agent: Mozilla1 Disallow: User-agent: egothorg; +http://ego.ms.mff.cuni.cz/) Disallow: User-agent: Paros.12 Disallow: User-agent: MSProxy Disallow: User-agent: multiBlocker Disallow: User-agent: Nokia7110 Disallow: User-agent: NutchCVS Disallow: User-agent: Oracle Application Server Web Cache 10g Disallow: User-agent: OSSProxy 16:30:54) Disallow: User-agent: Privoxy Disallow: User-agent: PureSight Disallow: User-agent: RAYSPIDER/Nutch Disallow: User-agent: Rewebber Disallow: User-agent: semaforo.net Disallow: User-agent: Squid-Prefetch Disallow: User-agent: squidclam Disallow: User-agent: SquidClamAV_Redirector Disallow: User-agent: SURF Disallow: User-agent: Twotrees Reactive Filter Disallow: User-agent: Watchfire WebXM Disallow: User-agent: WebFilter Robot Disallow: User-agent: WebRACE Disallow: User-agent: WebTrafficExpress Disallow: User-agent: XRL Disallow: User-agent: Y!OASIS/TEST no-ad Disallow: User-agent: Google Wireless Transcoder;) Disallow: User-agent: Space Bison2 [fu] Disallow: User-agent: Utopia WebWasher Disallow: User-agent: BravoBrian bstop.bravobrian.it Disallow: User-agent: BrightCrawler Disallow: User-agent: BStop.BravoBrian.it Agent Detector Disallow: User-agent: EmeraldShield.com WebBot Disallow: User-agent: UnChaos Disallow: User-agent: UnChaosBot Disallow: User-agent: http://www.sygol.com Disallow: User-agent: */Nutch Disallow: User-agent: +SitiDi.net/SitiDiBot Disallow: User-agent: -DIE-KRAEHE- META-SEARCH-ENGINE Disallow: User-agent: 192.comAgent Disallow: User-agent: 4anything.com LinkChecker Disallow: User-agent: :robot Disallow: User-agent: A-Online Search Disallow: User-agent: A1 Sitemap Generator miggibot Disallow: User-agent: aardvark-crawler Disallow: User-agent: AbachoBOT Disallow: User-agent: ABCdatos BotLink Disallow: User-agent: Aberja Checkomat Disallow: User-agent: abot Disallow: User-agent: Aboutlibwww-perl7 Disallow: User-agent: Accelatech RSSCrawler Disallow: User-agent: accoona Disallow: User-agent: Accoona-AI-Agent Disallow: User-agent: Ack Disallow: User-agent: AcoiRobot Disallow: User-agent: Acoon Robot Disallow: User-agent: Acorn/Nutch Disallow: User-agent: AESOP_com_SpiderMan Disallow: User-agent: agadine Disallow: User-agent: Agent-sharewarePlazaFileCheckBot Disallow: User-agent: AIBOT Disallow: User-agent: aipbot Disallow: User-agent: Aladin Disallow: User-agent: Aleksika Disallow: User-agent: AlkalineBOT Disallow: User-agent: Allesklar Disallow: User-agent: AltaVista Intranet V2.0 AVS EVAL Disallow: User-agent: AltaVista Intranet V2.0 Compaq Altavista Eval Disallow: User-agent: AltaVista Intranet V2.0 Disallow: User-agent: AltaVista Disallow: User-agent: AmfibiBOT Disallow: User-agent: Amfibibot Disallow: User-agent: amibot Disallow: User-agent: AnnoMille Disallow: User-agent: AnswerBus Disallow: User-agent: antibot Disallow: User-agent: AnzwersCrawl Disallow: User-agent: Apexoo Disallow: User-agent: Aport Disallow: User-agent: appie Disallow: User-agent: ArabyBot Disallow: User-agent: ArachBot Disallow: User-agent: Arachnoidea Disallow: User-agent: ArchitextSpider Disallow: User-agent: archive.org_bot Disallow: User-agent: Arikus_Spider Disallow: User-agent: Arquivo-web-crawler Disallow: User-agent: ASAHA Search Engine Turkey V.001 Disallow: User-agent: Asahina-Antenna Disallow: User-agent: ask.24x.info Disallow: User-agent: AskAboutOil Disallow: User-agent: asked/Nutch Disallow: User-agent: ASPSeek Disallow: User-agent: ASSORT Disallow: User-agent: asterias Disallow: User-agent: AtlocalBot Disallow: User-agent: Atomz Disallow: User-agent: Attentio/Nutch Disallow: User-agent: augurfind Disallow: User-agent: autowebdir Disallow: User-agent: AV Fetch Disallow: User-agent: AVSearch Disallow: User-agent: axadine Disallow: User-agent: AxmoRobot Disallow: User-agent: BaboomBot Disallow: User-agent: BaiduImagespider+ Disallow: User-agent: BaiDuSpider Disallow: User-agent: Balihoo/Nutch Disallow: User-agent: BarraHomeCrawler Disallow: User-agent: bdcindexer Disallow: User-agent: BDFetch Disallow: User-agent: BDNcentral Disallow: User-agent: beautybot Disallow: User-agent: BebopBot Disallow: User-agent: BigCliqueBOT Disallow: User-agent: BIGLOTRON Disallow: User-agent: Bigsearch.ca/Nutch Disallow: User-agent: BilgiBetaBot Disallow: User-agent: BilgiBot Disallow: User-agent: Bitacle bot Disallow: User-agent: Bitacle Robot Disallow: User-agent: Blaiz-Bee Disallow: User-agent: BlitzBOT@tricus.net Disallow: User-agent: BlogBot Disallow: User-agent: BlogzIce Disallow: User-agent: BloobyBot Disallow: User-agent: Bloodhound/Nutch Disallow: User-agent: boitho.com-dc Disallow: User-agent: boitho.com-robot Disallow: User-agent: BPImageWalker Disallow: User-agent: BravoBrianEngine MarcoPolo Disallow: User-agent: BruinBot Disallow: User-agent: BSDSeek Disallow: User-agent: BTbot Disallow: User-agent: BuildCMS Disallow: User-agent: BullsEye Disallow: User-agent: bumblebee@relevare.com Disallow: User-agent: BurstFindCrawler Disallow: User-agent: Buscaplus Robi Disallow: User-agent: Cabot/Nutch Disallow: User-agent: carleson Disallow: User-agent: Carnegie_Mellon_University_Research_WebBOT Disallow: User-agent: Carnegie_Mellon_University_WebCrawler Disallow: User-agent: Catall Disallow: User-agent: CazoodleBot/CazoodleBot Disallow: User-agent: CCBot Disallow: User-agent: ccubee Disallow: User-agent: Ceramic Tile Installation Guide Disallow: User-agent: cfetch Disallow: User-agent: ChristCRAWLER Disallow: User-agent: CipinetBot Disallow: User-agent: ClariaBot Disallow: User-agent: Claymont.com Disallow: User-agent: CloakDetect Disallow: User-agent: Clushbot Disallow: User-agent: Clushbot-BinaryFury Disallow: User-agent: Clushbotx-Ajax Disallow: User-agent: Clushbotx-Hector Disallow: User-agent: Clushbotx-Peleus Disallow: User-agent: combine Disallow: User-agent: cometrics-bot Disallow: User-agent: Computer_and_Automation_Research_Institute_Crawler Disallow: User-agent: Comrite.1 Disallow: User-agent: Convera Internet Disallow: User-agent: ConveraCrawler Disallow: User-agent: ConveraCrawlerd Disallow: User-agent: ConveraMultiMediaCrawler Disallow: User-agent: CoolBot Disallow: User-agent: cosmos_ Disallow: User-agent: CougarSearch Disallow: User-agent: Covac TexAs Arachbot Disallow: User-agent: Cowbot Disallow: User-agent: CrawlConvera Disallow: User-agent:@alexa.com Disallow: User-agent:Boy Disallow: User-agent: Crawllybot Disallow: User-agent: CreativeCommons6 Disallow: User-agent: CrocCrawler Disallow: User-agent: csci_b6593 Disallow: User-agent: Cuasarbotb Disallow: User-agent: CurryGuide SiteScan Disallow: User-agent: CyberPatrol SiteCat Webbot (cyberpatrolcrawler.asp) Disallow: User-agent: CydralSpider Disallow: User-agent: DataFountains/DMOZ Downloader Disallow: User-agent: DataFountains/DMOZ Feature Vector Corpus Creator Disallow: User-agent: DataparkSearch7 Disallow: User-agent: DataparkSearchx Disallow: User-agent: DataSpear Disallow: User-agent: DataSpearSpiderBot Disallow: User-agent: DatenBot Disallow: User-agent: DaviesBot Disallow: User-agent: daypopbot Disallow: User-agent: dbDig Disallow: User-agent: dCSbot Disallow: User-agent: de.searchengine.comBot Disallow: User-agent: deepak-USC/ISI Disallow: User-agent: DeepIndex Disallow: User-agent: DeepIndexer.ca Disallow: User-agent: Denmex websearch Disallow: User-agent: dev-spider2.searchpsider.comb Disallow: User-agent: DiaGem Disallow: User-agent: Diamond Disallow: User-agent: DiamondBot Disallow: User-agent: Digger JDK.0rc3 Disallow: User-agent: DigOut4U Disallow: User-agent: DIIbot Disallow: User-agent: disco/Nutch Disallow: User-agent: DittoSpyder Disallow: User-agent: dloader Disallow: User-agent: DoCoMo/Nxxxi/c10 Disallow: User-agent: DoCoMo/Nxxxi/c10/TB Disallow: User-agent: DoCoMo P900iV Disallow: User-agent: DoCoMo SH902i Disallow: User-agent: DoCoMo/SO502i Disallow: User-agent: dodgebot/experimental Disallow: User-agent: Download-Tipp Linkcheck Disallow: User-agent: Drecombot Disallow: User-agent: dtSearchSpider Disallow: User-agent: DuckDuckBot Disallow: User-agent: Dumbot Disallow: User-agent: Dumbot) Disallow: User-agent: Dumbot Disallow: User-agent: e-sense Disallow: User-agent: e-SocietyRobot Disallow: User-agent: eApolloBot Disallow: User-agent: EARTHCOM.info Disallow: User-agent: EARTHCOM.infobeta Disallow: User-agent: EasyDLx Disallow: User-agent: EchO! Disallow: User-agent: egothora Disallow: User-agent: EgotoBot Disallow: User-agent: ejupiter.com Disallow: User-agent: elfbot Disallow: User-agent: ELI/20070402:2.0 Disallow: User-agent: EMPAS_ROBOT Disallow: User-agent: EnaBot Disallow: User-agent: Enfish Tracker Disallow: User-agent: Enterprise_Search Disallow: User-agent: Enterprise_Search Disallow: User-agent: envolk Disallow: User-agent: envolk[ITS]spider Disallow: User-agent: EroCrawler Disallow: User-agent: ES.NET_Crawler Disallow: User-agent: eseek-larbin Disallow: User-agent: ESISmartSpider Disallow: User-agent: eStyleSearch Disallow: User-agent: EuripBot GetFile Disallow: User-agent: EuripBot GetRobots Disallow: User-agent: EuripBot PreCheck Disallow: User-agent: Eurobot Disallow: User-agent: EvaalSE Disallow: User-agent: eventax Disallow: User-agent: Everest-Vulcan Disallow: User-agent: Exabot-Images Disallow: User-agent: Exabot-Test Disallow: User-agent: Exabot Disallow: User-agent: ExactSeek Disallow: User-agent: exactseek-crawler3 Disallow: User-agent: exactseek-pagereaper3 Disallow: User-agent: exactseek.com Disallow: User-agent: Exalead NG/MimeLive Client Disallow: User-agent: Excalibur Internet Disallow: User-agent: Execrawl Disallow: User-agent: exooba/exooba Disallow: User-agent: exooba/exooba Disallow: User-agent: ExperimentalHenrytheMiragoRobot Disallow: User-agent: EyeCatcher Disallow: User-agent: Factbot Disallow: User-agent: Fast Gold Edition Disallow: User-agent: FAST Enterprise Disallow: User-agent: FAST FirstPage retriever Disallow: User-agent: FAST MetaWeb Disallow: User-agent: Fast PartnerSite Disallow: User-agent: FAST-WebCrawler Disallow: User-agent: fastbot Disallow: User-agent: FastBug Disallow: User-agent: FastCrawler Disallow: User-agent: FastSearch Disallow: User-agent: Favcollector Disallow: User-agent: favo.eu Disallow: User-agent: Faxobot Disallow: User-agent: Feed Seeker Disallow: User-agent: Feed24.com Disallow: User-agent: FeedChecker1 Disallow: User-agent: Feedfetcher-Google; Disallow: User-agent: FeedHub FeedDiscovery Disallow: User-agent: FeedHub MetaDataFetcher Disallow: User-agent: Feedjit Favicon Disallow: User-agent: Feedster Disallow: User-agent: Felix - Mixcat Disallow: User-agent: FFC Trap Door Disallow: User-agent: Filtrbox Disallow: User-agent: Findexa Disallow: User-agent: findlinks/x.xxx Disallow: User-agent: FineBot Disallow: User-agent: Firefly Disallow: User-agent: Firefox Disallow: User-agent: FirstGov.gov Disallow: User-agent: firstsbot Disallow: User-agent: Flapbot Disallow: User-agent: Flexum Disallow: User-agent: FlickBot Disallow: User-agent: flunky Disallow: User-agent: FocusedSampler Disallow: User-agent: Folkd.com Disallow: User-agent: Fooky.com/ScorpionBot/ScoutOut; scorpionbots Disallow: User-agent: FreeFind.com-SiteSearchEngine Disallow: User-agent: FreshNotes Disallow: User-agent: FuseBulb.Com Disallow: User-agent: FyberSpider Disallow: User-agent: GAIS RobotB2 Disallow: User-agent: Gaisbot Disallow: User-agent: GalaxyBot Disallow: User-agent: Gallent Search Disallow: User-agent: gamekitbot Disallow: User-agent: GammaSpider Disallow: User-agent: gazz Disallow: User-agent: genieBot Disallow: User-agent: GeonaBot Disallow: User-agent: gigabazx Disallow: User-agent: Gigabot Disallow: User-agent: Gigabotatt Disallow: User-agent: Gigabot Disallow: User-agent: GigabotSiteSearch Disallow: User-agent: GNODSPIDER Disallow: User-agent: Goblin Disallow: User-agent: Goblin Disallow: User-agent: GoForIt.com Disallow: User-agent: GOFORITBOT Disallow: User-agent: gonzo1[P] Disallow: User-agent: gonzo2[P] Disallow: User-agent: Goofer Disallow: User-agent: Googlebot-Image Disallow: User-agent: Googlebot Disallow: User-agent: Googlebot/Test Disallow: User-agent: GrapeFX Disallow: User-agent: great-plains-web-spider/flatlandbot Disallow: User-agent: GrigorBot Disallow: User-agent: Gromit Disallow: User-agent: grub Disallow: User-agent: grub-client Disallow: User-agent: gsa-crawler Disallow: User-agent: Gulliver Disallow: User-agent: Gulper Web Bot Disallow: User-agent: Gungho8004 Disallow: User-agent: GurujiBot Disallow: User-agent: GurujiImageBot Disallow: User-agent: HappyFunBot Disallow: User-agent: Harvest-NG.2 Disallow: User-agent: Hatena Antenna Disallow: User-agent: Hatena Pagetitle Agent Disallow: User-agent: Hatena RSS Disallow: User-agent: hbtronix.spider.2 Disallow: User-agent: HeinrichderMiragoRobot Disallow: User-agent: Helix Disallow: User-agent: HenriLeRobotMirago Disallow: User-agent: HenryTheMiragoRobot Disallow: User-agent: CsCrawler Disallow: User-agent: Hippias Disallow: User-agent: HitList Disallow: User-agent: Hitwise Disallow: User-agent: holmes1 Disallow: User-agent: holmes Disallow: User-agent: holmesx Disallow: User-agent: HomePageSearch Disallow: User-agent: HomerbotDisallow: User-agent: Honda-Search Disallow: User-agent: HooWWWer Disallow: User-agent: HooWWWer Disallow: User-agent: HPL/Nutch Disallow: User-agent: htdig Disallow: User-agent: http://Ask.24x.Info/ Disallow: User-agent: i1searchbot Disallow: User-agent: IAArchiver Disallow: User-agent: iaskspider2 Disallow: User-agent: ia_archiver Disallow: User-agent: ICC-Crawler Disallow: User-agent: ichiro Disallow: User-agent: IconSurf favicon finder Disallow: User-agent: IconSurf favicon monitor Disallow: User-agent: ICRA_label_spider Disallow: User-agent: icsbot Disallow: User-agent: ideare - SignSite Disallow: User-agent: igdeSpyder Disallow: User-agent: IIITBOT Disallow: User-agent: ilial/Nutch Disallow: User-agent: IlseBot Disallow: User-agent: IlTrovatore-Setaccio Disallow: User-agent: ImageWalker Disallow: User-agent: IncyWincy Disallow: User-agent: IndexTheWeb.com7 Disallow: User-agent: Inet library Disallow: User-agent: info@pubblisito.com Disallow: User-agent: InfoFly Disallow: User-agent: INFOMINE Adders Disallow: User-agent: INFOMINE RemoteServices Disallow: User-agent: INFOMINE VLCrawler Disallow: User-agent: InfoNaviRobot Disallow: User-agent: InfoSeek Sidewinder Disallow: User-agent: InfoSeek SidewinderA Disallow: User-agent: Infoseek SideWinder5 Disallow: User-agent: Infoseek SideWinderB Disallow: User-agent: INGRID MT Disallow: User-agent: Inktomi Search Disallow: User-agent: InnerpriseBot Disallow: User-agent: Insitor.com Disallow: User-agent: Insitornaut Disallow: User-agent: Internet Ninja Disallow: User-agent: InternetSeer.com Disallow: User-agent: IPiumBot laurioncom Disallow: User-agent: IpselonBotx Disallow: User-agent: IRLbot Disallow: User-agent: IWAgent Disallow: User-agent: Jabot Disallow: User-agent: Jack Disallow: User-agent: Jambot Disallow: User-agent: Jayde Disallow: User-agent: Jetbot Disallow: User-agent: JobSpider_BA Disallow: User-agent: Jyxobot Disallow: User-agent: k2spider Disallow: User-agent: KAIST AITrc Disallow: User-agent: KakleBot Disallow: User-agent: kalooga/kalooga-datahouse Disallow: User-agent: Kenjin Disallow: User-agent: Kevin Disallow: User-agent: KE libwww.8 Disallow: User-agent: KFSW-Bot Disallow: User-agent: kinja-imagebot Disallow: User-agent: kinjabot Disallow: User-agent: KIT-Fireball Disallow: User-agent: KnowItAll Disallow: User-agent: Knowledge.com Disallow: User-agent: Krugle Disallow: User-agent: KSbot Disallow: User-agent: kuloko-bot Disallow: User-agent: kulokobot Disallow: User-agent: kulturarw3 Disallow: User-agent: LapozzBot Disallow: User-agent: larbin Disallow: User-agent: larbin_basileocaml Disallow: User-agent: larbin_devel Disallow: User-agent: lawinfo-crawler/Nutch Disallow: User-agent: LECodeChecker libgetdoc Disallow: User-agent: LEIA0 Disallow: User-agent: LEIA1pr Disallow: User-agent: LexiBot0 Disallow: User-agent: Libby/libwww-perl7 Disallow: User-agent: LibertyW Disallow: User-agent: libWeb/clsHTTP Disallow: User-agent: libwww-perl1 Disallow: User-agent: libwww-perl5 Disallow: User-agent: libwww-perl8 Disallow: User-agent: libwww-perl2 FP Disallow: User-agent: libwww-perl2 FP Disallow: User-agent: libwww-perl5 Disallow: User-agent: libwww-perl00 Disallow: User-agent: libwww.2 Disallow: User-agent: LijitSpider/Nutch Disallow: User-agent: linkbot Disallow: User-agent: linknzbot Disallow: User-agent: Links Disallow: User-agent: Links SQL Disallow: User-agent: LinkScan Disallow: User-agent: LiveTrans/Nutch Disallow: User-agent: Llaut Disallow: User-agent: lmspider Disallow: User-agent: LNSpiderguy Disallow: User-agent: LocalBot Disallow: User-agent: LocalcomBot.x Disallow: User-agent: Lockstep Disallow: User-agent: Look.com Disallow: User-agent: Lovel as Disallow: User-agent: LTI/LemurProject Nutch/Nutch Disallow: User-agent: lwp-trivial2 Disallow: User-agent: lwp-trivial4 Disallow: User-agent: lwp-trivial4 Disallow: User-agent: LWP::Simple2 Disallow: User-agent: LWP::Simple6 Disallow: User-agent: LWP::Simple8 Disallow: User-agent: LWP::Simple0 Disallow: User-agent: LWP::Simple1 Disallow: User-agent: LWP::Simple3 Disallow: User-agent: LWP::Simple3 Disallow: User-agent: Lycos_Spider Disallow: User-agent: Lynx Disallow: User-agent: Mackster Disallow: User-agent: Mahiti.Com/Mahiti Disallow: User-agent: Mail.Ru Disallow: User-agent: mammoth Disallow: User-agent: MantraAgent Disallow: User-agent: MapoftheInternet.com Disallow: User-agent: Marinerb Disallow: User-agent: Marketwave Hit List Disallow: User-agent: Martini Disallow: User-agent: Marvin Disallow: User-agent: MaSagool Disallow: User-agent: MasterSeek Disallow: User-agent: Mata Hari Disallow: User-agent: Matrix S.p.A. - FAST Enterprise Disallow: User-agent: maxomobot/dev-20051201 Disallow: User-agent: MDbot Disallow: User-agent: MediaCrawler Disallow: User-agent: Mediapartners-Google Disallow: User-agent: MediaSearch Disallow: User-agent: MegaSheep Disallow: User-agent: Megite2.0 Disallow: User-agent: Mercator Disallow: User-agent: Mercator Disallow: User-agent: Mercator-Scrub Disallow: User-agent: Metaeuro Web Disallow: User-agent: MetaGer-LinkChecker Disallow: User-agent: MetagerBot Disallow: User-agent: MetaGer_PreChecker Disallow: User-agent: Metaspinner Disallow: User-agent: metatagsdir Disallow: User-agent: MicroBaz Disallow: User-agent: Microsoft Small Business Indexer Disallow: User-agent: Misterbot-Nutch Disallow: User-agent: Miva Disallow: User-agent: MJ12bot Disallow: User-agent: MJBot Disallow: User-agent: MnogoSearch.xx Disallow: User-agent: moget Disallow: User-agent: mogimogi Disallow: User-agent: MojeekBot Disallow: User-agent: Morris - Mixcat Disallow: User-agent: Mouse-House Disallow: User-agent: mozDexx Disallow: User-agent: BotSeer Disallow: User-agent: Ask Jeeves/Teoma) Disallow: User-agent: Fluffy Disallow: User-agent: MuscatFerret Disallow: User-agent: Webinator Disallow: User-agent: Webinator-indexer Disallow: User-agent: INGRID MT Disallow: User-agent: Slurp.so/Goo Disallow: User-agent: Slurp/cat Disallow: User-agent: Slurp/si Disallow: User-agent: Vagabondo MT; Disallow: User-agent: Vagabondo Disallow: User-agent: VagabondoBeta Disallow: User-agent: ChristCrawler.com Disallow: User-agent: DAUMOA-video Disallow: User-agent: FastCrawler3 Disallow: User-agent: grub-client Disallow: User-agent: Site Server Robot Disallow: User-agent: NetNose-Crawler Disallow: User-agent: TrueRobot; Disallow: User-agent: VoilaBot BETA Disallow: User-agent: VoilaBot; Disallow: User-agent: www.galaxy.com www.psychedelix.com Disallow: User-agent: Asterias Disallow: User-agent: Skampy.x Disallow: User-agent: TargetSeek Disallow: User-agent: ODP entries t_st Disallow: User-agent: ODP links test Disallow: User-agent: ZoomSpider.net Disallow: User-agent: heritrix Disallow: User-agent: QihooBot Disallow: User-agent: MS Search Robot Disallow: User-agent: DAUMOA; DAUM Web Robot Disallow: User-agent: NaverBot Disallow: User-agent: Yahoo Japan Disallow: User-agent: NetFront MMP Disallow: User-agent: Googlebot Disallow: User-agent: FurlBot/Furl Search Disallow: User-agent: ZyBorg Disallow: User-agent: Dulance bot Disallow: User-agent: Mozilla__TrueRobot libwww.8 Disallow: User-agent: Mozilla__VoilaBot libwww.2 Disallow: User-agent: Intelliseek Disallow: User-agent: WhizBang Disallow: User-agent: Mammoth Disallow: User-agent: Clustered-Search-Bot Disallow: User-agent: AnsearchBot Disallow: User-agent: archive.org_bot Disallow: User-agent: Ask Jeeves/Teoma Disallow: User-agent: BecomeBot3 Disallow: User-agent: BecomeBotx Disallow: User-agent: BecomeBotbeta Disallow: User-agent: BecomeBot Disallow: User-agent: BecomeJPBot Disallow: User-agent: BlogRefsBot Disallow: User-agent: BuzzRankingBot Disallow: User-agent: Charlotteb Disallow: User-agent: DNS-Digger-Explorer Disallow: User-agent: DNS-Digger Disallow: User-agent: EARTHCOM.info1 Disallow: User-agent: Exabot Disallow: User-agent: FatBot Disallow: User-agent: heritrix Disallow: User-agent: Hermit Search Disallow: User-agent: HyperixScoop Disallow: User-agent: IDBot Disallow: User-agent: MojeekBot Disallow: User-agent: Podtech Disallow: User-agent: OnetSzukaj) Disallow: User-agent: PalmeraBot Disallow: User-agent: PWeBot Disallow: User-agent: Scrubby Disallow: User-agent: ShunixBot Disallow: User-agent: Synoobot Disallow: User-agent: Theophrastus Disallow: User-agent: Vagabondo Disallow: User-agent: worio bot heritrix) Disallow: User-agent: Yahoo! DE Slurp) Disallow: User-agent: Yahoo! Slurp China Disallow: User-agent: Yahoo! Slurp Disallow: User-agent: Zenbot Disallow: User-agent: Slurp/cat Disallow: User-agent: Slurp/si Disallow: User-agent: Twiceler Disallow: User-agent: NimbleCrawler1 Disallow: User-agent: VoilaBot Disallow: User-agent: NimbleCrawler2 Disallow: User-agent: URL-Spider Disallow: User-agent: usww.com-Spider-for-w8.net Disallow: User-agent: Gulper Web Bot Disallow: User-agent: MQbot Disallow: User-agent: MQBOT/Nutch mqbot@cs.uiuc.edu) Disallow: User-agent: msnbot-media Disallow: User-agent: msnbot-Products Disallow: User-agent: MSNBOTx Disallow: User-agent: msnbot Disallow: User-agent: MSNBOT_Mobile Disallow: User-agent: MSNPTC Disallow: User-agent: MSRBOT Disallow: User-agent: multicrawler Disallow: User-agent: MultiText Disallow: User-agent: MusicWalker Disallow: User-agent: Mylinea.com Disallow: User-agent: Naamah/Blogbot Disallow: User-agent: NABOT Disallow: User-agent: nabot Disallow: User-agent: NationalDirectory-WebSpider Disallow: User-agent: NationalDirectoryAddURL Disallow: User-agent: NaverBot Disallow: User-agent: NaverBot_dloader Disallow: User-agent: NavissoBot Disallow: User-agent: NavissoBot Disallow: User-agent: NCSA Disallow: User-agent: Nebullabot Disallow: User-agent: NEC Research Agent Disallow: User-agent: NetinfoBot Disallow: User-agent: NetLookout4 Disallow: User-agent: Netluchs Disallow: User-agent: NetNoseCrawler/v1.0 Disallow: User-agent: Netprospector JavaCrawler Disallow: User-agent: NetResearchServer Disallow: User-agent: NetResearchServer Disallow: User-agent: NetSprint Disallow: User-agent: NetWhatCrawler6 Disallow: User-agent: NetZippy Disallow: User-agent: NextGenSearchBot Disallow: User-agent: NextopiaBOT Disallow: User-agent: NG-Search0 Disallow: User-agent: NG Disallow: User-agent: NG.1229 Disallow: User-agent: NITLE Blog1 Disallow: User-agent: Noago Disallow: User-agent: Nokia-WAPToolkit Disallow: User-agent: Nokia6610 Profile/MIDP Configuration/CLDC Disallow: User-agent: NokodoBot Disallow: User-agent: Norbert the Disallow: User-agent: noxtrumbot Disallow: User-agent: noyona_0_1 Disallow: User-agent: NP Disallow: User-agent: NPBot Disallow: User-agent: NPBot-1 Disallow: User-agent: nsyght.com/Nutch Disallow: User-agent: nttdirectory_robot Disallow: User-agent: nuSearch Disallow: User-agent: obidos-bot Disallow: User-agent: ObjectsSearch Disallow: User-agent: oBot Disallow: User-agent: Ocelli Disallow: User-agent: Octora Beta Disallow: User-agent: Octora Beta Bot Disallow: User-agent: OmniExplorer_Botx Internet CategorizerOmniExplorer Disallow: User-agent: OmniExplorer_Botx Job Disallow: User-agent: OmniExplorer_Botx Torrent Disallow: User-agent: OmniExplorer_Bot WorldIndexer Disallow: User-agent: Onet.pl SA - http://szukaj.onet.pl Disallow: User-agent: OntoSpider Disallow: User-agent: OpenAcoon Disallow: User-agent: Openbot Disallow: User-agent: Openfind data gatherer Disallow: User-agent: Openfind RobotA2 Disallow: User-agent: OpenISearch Disallow: User-agent: OpenTaggerBot Disallow: User-agent: OpenTextSiteCrawler Disallow: User-agent: OpenWebSpider Disallow: User-agent: OpidooBOT Disallow: User-agent: Oracle Ultra Search Disallow: User-agent: OrangeSpider Disallow: User-agent: Orbiter/T Disallow: User-agent: Overture-WebCrawler/Fresh Disallow: User-agent: ozelot Disallow: User-agent: PADLibrary Disallow: User-agent: PageBitesHyperBot/600 Disallow: User-agent: Pagebull Disallow: User-agent: page_verifier Disallow: User-agent: parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1 Disallow: User-agent: ParaSiteb Disallow: User-agent: Patwebbot Disallow: User-agent: pd02 Disallow: User-agent: PEERbot Disallow: User-agent: PicoSearch Disallow: User-agent: Piffany_Web_Scraper_v0.x Disallow: User-agent: Piffany_Web_Spider_v0.x Disallow: User-agent: pipeLinera Disallow: User-agent: pipeLinerx Disallow: User-agent: Pita Disallow: User-agent: PJspider Disallow: User-agent: PlagiarBot Disallow: User-agent: PluckFeedCrawler Disallow: User-agent: Pluggd/Nutch Disallow: User-agent: polybot Disallow: User-agent: Pompos Disallow: User-agent: Popdexter Disallow: User-agent: PortalBSpider Disallow: User-agent: potbot Disallow: User-agent: PRCrawler/Nutch Disallow: User-agent: PrivacyFinder Cache Bot Disallow: User-agent: PrivacyFinder Disallow: User-agent: Project XP5 Disallow: User-agent: PROve AnswerBot Disallow: User-agent: ProWebGuide Link Checker Disallow: User-agent: psbot Disallow: User-agent: PubCrawl Disallow: User-agent: pulseBot Disallow: User-agent: PWeBot Inspector Disallow: User-agent: PycURL Disallow: User-agent: Qango.com Web Directory Disallow: User-agent: QEAVis Agent/Nutch Disallow: User-agent: QPCreep Test Rig Disallow: User-agent: QuepasaCreep Disallow: User-agent: QuepasaCreep v0x Disallow: User-agent: QueryN Metasearch Disallow: User-agent: QweeryBot1 Disallow: User-agent: Qweery_robot.txt_CheckBot1 Disallow: User-agent: R6_CommentReader Disallow: User-agent: R6_FeedFetcher Disallow: User-agent: rabaz Disallow: User-agent: RaBot Agent-admin/phortse@hanmail.net Disallow: User-agent: ramBot xtreme x.x Disallow: User-agent: RAMPyBot Disallow: User-agent: Rankivabot Disallow: User-agent: Rational SiteCheck Disallow: User-agent: Reaper Disallow: User-agent: Reaperx Disallow: User-agent: RedCarpet Disallow: User-agent: RedCell Disallow: User-agent: RedKernel Disallow: User-agent: rico Disallow: User-agent: RixBot Disallow: User-agent: RoboCrawl Disallow: User-agent: RoboCrawl Disallow: User-agent: RoboPal Disallow: User-agent: Robot/www.pj-search.com Disallow: User-agent: Robot: NutchCrawler- Owner: wdavies@acm.org Disallow: User-agent: Robozilla Disallow: User-agent: Rotondo libwww.1 Disallow: User-agent: RRC Disallow: User-agent: RSSMicro.com RSS/Atom Feed Robot Disallow: User-agent: RufusBot Disallow: User-agent: RufusBot Disallow: User-agent: sait/Nutch Disallow: User-agent: SandCrawler Disallow: User-agent: savvybot Disallow: User-agent: SBIder Disallow: User-agent: ScanWeb Disallow: User-agent: ScholarUniverse Disallow: User-agent: schwarzmann.biz-Spider_for_paddel.org+ Disallow: User-agent: ScollSpider Disallow: User-agent: Scooter.EU Disallow: User-agent: Scooter.FS Disallow: User-agent: Scooter.HD Disallow: User-agent: Scooter.VNS Disallow: User-agent: ScooterQI Disallow: User-agent: Scooter Disallow: User-agent: Scooter.BT Disallow: User-agent: Scooter.DIL Disallow: User-agent: Scooter.EX Disallow: User-agent: Scooter.JT Disallow: User-agent: Scooter.NIV Disallow: User-agent: Scooter.SF0 Disallow: User-agent: Scooter.snippet Disallow: User-agent: Scooterdev Disallow: User-agent: Scooter-ARS Disallow: User-agent: Scooter-ARS-ih Disallow: User-agent: scooter-venus.vns Disallow: User-agent: Scooter-W3 Disallow: User-agent: Scooter Disallow: User-agent: Scooter G.R.A.B. Disallow: User-agent: Scooter.QA.pczukor Disallow: User-agent: Scooter.vscooter Disallow: User-agent: Scooter_SF Disallow: User-agent: Scooter2_Mercator_x-x.0 Disallow: User-agent: Scooter_bh0.3 Disallow: User-agent: Scooter_trk3.3 Disallow: User-agent: ScoutAbout Disallow: User-agent: ScoutAnt Disallow: User-agent: scoutmaster Disallow: User-agent: Scrubby Disallow: User-agent: Search-Engine-Studio Disallow: User-agent: search.ch V1.4 Disallow: User-agent: search.ch V1 Disallow: User-agent: SearchByUsa Disallow: User-agent: SearchdayBot Disallow: User-agent: SearchExpress Disallow: User-agent: SearchGuild/DMOZ/Experiment Disallow: User-agent: SearchGuild_DMOZ_Experiment Disallow: User-agent: Searchit-Now Robot Disallow: User-agent: Searchmee Disallow: User-agent: SearchSight Disallow: User-agent: Searchspider Disallow: User-agent: SearchTone Disallow: User-agent: Seekbot HTTPFetcher Disallow: User-agent: Seekbot RobotsTxtFetcher Disallow: User-agent: Seekbot RobotsTxtFetcher Disallow: User-agent: Seeker.lookseek.com Disallow: User-agent: Semager Disallow: User-agent: Semager Disallow: User-agent: Sensis Web Disallow: User-agent: Sensis.com.au Web Disallow: User-agent: SeznamBot Disallow: User-agent: SeznamBot Disallow: User-agent: SeznamBot-test Disallow: User-agent: Shim Disallow: User-agent: Shim-Crawler Disallow: User-agent: ShopWiki Disallow: User-agent: Shoula.com Disallow: User-agent: SietsCrawler Disallow: User-agent: Sigram/Nutch Disallow: User-agent: Siigle Orumcex v.001 Turkey Disallow: User-agent: silk Disallow: User-agent: silk Disallow: User-agent: Sirketcebot Disallow: User-agent: SiteSpider Disallow: User-agent: SiteTruth.com site rating system Disallow: User-agent: SiteXpert Disallow: User-agent: Skampy.x Disallow: User-agent: Skimpy Disallow: User-agent: Skywalker Disallow: User-agent: Slarp Disallow: User-agent: Slider_Search_v1-de Disallow: User-agent: Slurp Disallow: User-agent: Slurp-KiteWeekly Disallow: User-agent: Slurp/si Disallow: User-agent: Slurpy Verifier Disallow: User-agent: SlySearch Disallow: User-agent: SlySearch robotinfo.html Disallow: User-agent: SlySearch http://www.slysearch.com Disallow: User-agent: smartwit.com Disallow: User-agent: SmiffyDCMetaSpider Disallow: User-agent: SnykeBot Disallow: User-agent: SocSciBot Disallow: User-agent: SoftHypermarketFileCheckBot Disallow: User-agent: sohu-search Disallow: User-agent: Sosospide+ Disallow: User-agent: speedfind ramBot xtreme Disallow: User-agent: Speedy Disallow: User-agent: Speedy Entireweb Disallow: User-agent: Speedy_Spider Disallow: User-agent: Sphere Scout Disallow: User-agent: Sphider Disallow: User-agent: Spida Disallow: User-agent:-Sleek Disallow: User-agent:.batsch.com Disallow: User-agent:.yellopet.com Disallow: User-agent:/maxbot.com Disallow: User-agent:KU Disallow: User-agent:Man Disallow: User-agent:Monkeyx Disallow: User-agent: Spinne Disallow: User-agent: Spinne med Disallow: User-agent: Spinne med_AH Disallow: User-agent: Spock Disallow: User-agent: sportsuchmaschine.de-Robot Disallow: User-agent: sproose-alpha Disallow: User-agent: Sqworm.81-BETA Disallow: User-agent: Sqworm.85-BETA Disallow: User-agent: StackRambler Disallow: User-agent: Steeler Disallow: User-agent: Steeler Disallow: User-agent: Strategic Board Bot Disallow: User-agent: suchbaer.de Disallow: User-agent: suchbot Disallow: User-agent: Suchknecht.at-Robot Disallow: User-agent: suchpadbot Disallow: User-agent: SurferF3 1/0 Disallow: User-agent: suzuran Disallow: User-agent: Swooglebot Disallow: User-agent: SWSBot Disallow: User-agent: SygolBot Disallow: User-agent: SynoBot Disallow: User-agent: Syntryx ANT Scout Chassis Pheromone; Mozilla compatible Disallow: User-agent: Szukacz Disallow: User-agent: tags2dir.com Disallow: User-agent: Tagword Disallow: User-agent: Talkro Web-Shot Disallow: User-agent: TCDBOT/Nutch Disallow: User-agent: TECOMAC-Crawler Disallow: User-agent: Tecomi Bot Disallow: User-agent: Teemer Disallow: User-agent: Teoma MP Disallow: User-agent: teomaagent-admin@teoma.com Disallow: User-agent: teomaagent1 Disallow: User-agent: teoma_agent1 Disallow: User-agent: Teradex Mapper Disallow: User-agent: terraminds-bot Disallow: User-agent: TerrawizBot Disallow: User-agent: TestCrawler/Nutch Disallow: User-agent: TheRarestParsera Disallow: User-agent: TheSuBot Disallow: User-agent: thumbshots-de-Bot Disallow: User-agent: timboBot Disallow: User-agent: TinEye Disallow: User-agent: tivraSpider Disallow: User-agent: TJG/Spider Disallow: User-agent: Tkensaku/x.x Disallow: User-agent: Topodia Disallow: User-agent: Toutatis Disallow: User-agent: Toutatis Disallow: User-agent: Toutatis Disallow: User-agent: traazibot/testengine Disallow: User-agent: Trampelpfad-Spider Disallow: User-agent: Tumblr RSS syndication Disallow: User-agent: TurnitinBot Disallow: User-agent: Turnpike Emporium LinkChecker Disallow: User-agent: TutorGig Disallow: User-agent: Tutorial Disallow: User-agent: Twiceler Disallow: User-agent: Twiceler robot.html Disallow: User-agent: Tycoon Agent/Nutch Disallow: User-agent: TygoBot Disallow: User-agent: TygoProwler Disallow: User-agent: UIowaCrawler Disallow: User-agent: UKWizz/Nutch Disallow: User-agent: Ultraseek Disallow: User-agent: UofTDB_experiment Disallow: User-agent: updated-alpha Disallow: User-agent: updatedbeta Disallow: User-agent: Uptimebot Disallow: User-agent: UptimeBot Disallow: User-agent: URL Pro Disallow: User-agent: URL_Spider_Pro Disallow: User-agent: URL_Spider_Pro Disallow: User-agent: User-Agent: SKIZZLE! Distributed Internet v1.0 - www.SKIZZLE.com) Disallow: User-agent: USyd-NLP-Spider Disallow: User-agent: Vagabondo-WAP Disallow: User-agent: Vagabondo MT Disallow: User-agent: Vagabondo Disallow: User-agent: Vakes1 Disallow: User-agent: versus Disallow: User-agent: VeryGoodSearch.com.DaddyLongLegs Disallow: User-agent: verzamelgids.nl Disallow: User-agent: Verzamelgids Disallow: User-agent: Vespa Disallow: User-agent: VisBot Disallow: User-agent: Vision Research Lab image Disallow: User-agent: VMBot Disallow: User-agent: Vortex Disallow: User-agent: voyager-hc Disallow: User-agent: voyager Disallow: User-agent: VSE Disallow: User-agent: vspider Disallow: User-agent: VWBOT/Nutch Disallow: User-agent: W3SiteSearch_v1.1 Disallow: User-agent: wadaino.jp-crawler Disallow: User-agent: Wavefire Disallow: User-agent: Waypath development Disallow: User-agent: Waypath Scout Disallow: User-agent: Web Snooper Disallow: User-agent: web2express.org/Nutch Disallow: User-agent: WebAlta Disallow: User-agent: WebarooBot Disallow: User-agent: webbanditx Disallow: User-agent: Webclipping.com Disallow: User-agent: WebCompass Disallow: User-agent: WebCorp Disallow: User-agent: webcrawl.net Disallow: User-agent: WebFindBot Disallow: User-agent: Webglimpse Disallow: User-agent: Weblog Attitude Diffusion Disallow: User-agent: webmeasurement-bot Disallow: User-agent: WebRankSpider7 Disallow: User-agent: WebSearch.COM.AU Disallow: User-agent: WebSearchBench WebCrawler Disallow: User-agent: WebsiteWorth Disallow: User-agent: Webspinne Disallow: User-agent: Websquash.com Disallow: User-agent: WebStat Disallow: User-agent: Webster Disallow: User-agent: WebVac Disallow: User-agent: Webverzeichnis.de Disallow: User-agent: WFARC Disallow: User-agent: whatUseek_winona Disallow: User-agent: WhizBang! Lab Disallow: User-agent: Willow Internet by Twotrees Disallow: User-agent: WinkBot Disallow: User-agent: WIRE Disallow: User-agent: WISEbot Disallow: User-agent: worio heritrix bot Disallow: User-agent: woriobot Disallow: User-agent: WorldLight Disallow: User-agent: Wotbox Disallow: User-agent: Wotbox Disallow: User-agent: WSB WebCrawler Disallow: User-agent: WSB Disallow: User-agent: wume_crawler Disallow: User-agent: Wwlib/Linux Disallow: User-agent: www.arianna.it Disallow: User-agent: WWWeasel Robot Disallow: User-agent: wwwster Disallow: User-agent: X-Crawler Disallow: User-agent: xirq-beta Disallow: User-agent: Y!J-BSC Disallow: User-agent: Y!J-SRD Disallow: User-agent: Y!J Disallow: User-agent: yacy Disallow: User-agent: yacybot Disallow: User-agent: Yahoo Pipes Disallow: User-agent: Yahoo! Mindset Disallow: User-agent: Yahoo-Blogs Disallow: User-agent: Yahoo-MMAudVid Disallow: User-agent: Yahoo-MMCrawler Disallow: User-agent: Yahoo-Test Disallow: User-agent: Yahoo-VerticalCrawler-FormerWebCrawler Disallow: User-agent: YahooFeedSeeker Disallow: User-agent: YahooSeeker-Testing Disallow: User-agent: YahooSeeker) Disallow: User-agent: YahooSeeker/CafeKelsa Disallow: User-agent: Yandex Disallow: User-agent: yarienavoir.net Disallow: User-agent: Yeti Disallow: User-agent: yggdrasil/Nutch Disallow: User-agent: YodaoBot Disallow: User-agent: yoofind/yoofind Disallow: User-agent: yoogliFetchAgent Disallow: User-agent: yoono web-crawler Disallow: User-agent: YottaCars_Bot Disallow: User-agent: YottaShopping_Bot Disallow: User-agent: Zao-Crawler Disallow: User-agent: Zao-Crawlerb Disallow: User-agent: Zao Disallow: User-agent: ZBot0 Disallow: User-agent: Zearchit Disallow: User-agent: ZeBot_lseek.net Disallow: User-agent: ZeBot_www.ze.bz Disallow: User-agent: zedzo.digest Disallow: User-agent: zermelo Disallow: User-agent: zerxbot/Version Disallow: User-agent: Zeus ThemeSite Viewer Disallow: User-agent: Zeus xxxxx Webster Pro Disallow: User-agent: Zeusbot7 Disallow: User-agent: ZipppBotx Disallow: User-agent: ZIPPPCVSx Disallow: User-agent: Zippy Disallow: User-agent: ZoomSpider Disallow: User-agent: zspider Disallow: User-agent: ZyBorg Disallow: User-agent: UdmSearch Disallow: User-agent: Generic Mobile Phone Googlebot-Mobile Disallow: User-agent: NICO Disallow: User-agent: PlantyNet_WebRobot Disallow: User-agent: * Disallow: /