| Unidentified Indexer/Scraper |
108,746 |
629 |
| Amazonbot/0.1 |
26,964 |
350 |
| Scrapy/2.11.2 |
22,824 |
0 |
| AhrefsBot/7.0 |
17,441 |
871 |
| Anonymous Crawler 1 |
11,981 |
0 |
| GPTBot/1.2 |
10,159 |
1,232 |
| MJ12bot/v1.4.8 |
9,342 |
463 |
| Unidentified WordPress Searcher |
8,960 |
0 |
| SemrushBot/7~bl |
8,609 |
1,571 |
| Googlebot/2.1 |
8,006 |
898 |
| bingbot/2.0 |
7,716 |
417 |
| Bytespider |
7,653 |
415 |
| Unidentified Directory Searcher |
7,456 |
0 |
| BLEXBot/1.0 |
6,001 |
273 |
| Barkrowler/0.9 |
5,712 |
71 |
| ClaudeBot/1.0 |
5,012 |
738 |
| Sogou web spider/4.0 |
4,863 |
71 |
| AwarioBot/1.0 |
3,214 |
175 |
| Go-http-client/2.0 |
2,602 |
36 |
| facebookexternalhit/1.1 |
2,597 |
2,564 |
| Unidentified Compressed Archive Searcher |
1,920 |
0 |
| meta-externalagent/1.1 |
1,909 |
0 |
| serpstatbot/2.1 |
1,671 |
9 |
| YandexBot/3.0 |
1,572 |
687 |
| Anonymous Crawler 3 |
1,554 |
0 |
| Unidentified PHP Searcher |
1,302 |
0 |
| DuckDuckBot/1.1 |
1,300 |
0 |
| Unidentified Image Searcher |
1,180 |
0 |
| OAI-SearchBot/1.0 |
1,157 |
722 |
| DotBot/1.2 |
1,111 |
700 |
| Unidentified Configuration Searcher |
1,065 |
1 |
| DreamHost Data Team |
1,064 |
0 |
| Reeder |
1,061 |
0 |
| MJ12bot/v2.0.2 |
942 |
346 |
| Go-http-client/1.1 |
915 |
7 |
| Mozlila/5.0 |
856 |
0 |
| Applebot/0.1 |
702 |
175 |
| netEstate NE Crawler |
684 |
20 |
| ImagesiftBot |
678 |
127 |
| SeznamBot/4.0 |
620 |
148 |
| ChatGPT-User/1.0 |
583 |
53 |
| MarketGoo/2.1 |
561 |
16 |
| GoogleOther |
528 |
0 |
| CensysInspect/1.1 |
518 |
3 |
| PerplexityBot/1.0 |
394 |
42 |
| Dataprovider.com |
382 |
15 |
| DataForSeoBot/1.0 |
318 |
62 |
| Googlebot-Image/1.0 |
309 |
0 |
| Managed SSL Firewall Checker |
292 |
0 |
| Turnitin |
273 |
40 |
| CCBot/2.0 |
269 |
48 |
| MJ12bot/v2.0.4 |
262 |
101 |
| SeekportBot |
256 |
3 |
| SemrushBot-BA |
252 |
123 |
| Thinkbot/0.5.8 |
237 |
0 |
| BitSightBot/1.0 |
221 |
0 |
| Unidentified Demandware Searcher |
200 |
0 |
| TikTokSpider |
197 |
0 |
| Timpibot/0.9 |
182 |
0 |
| Tumblr/14.0.835.186 |
181 |
0 |
| Anonymous Crawler 2 |
179 |
0 |
| YandexRenderResourcesBot/1.0 |
173 |
0 |
| CCBot |
158 |
154 |
| SEOkicks |
155 |
3 |
| ALittle Client |
153 |
0 |
| InternetMeasurement/1.0 |
148 |
0 |
| MJ12bot/v2.0.0 |
139 |
49 |
| CheckMarkNetwork/1.0 |
126 |
10 |
| Expanse |
125 |
0 |
| PetalBot |
123 |
106 |
| panscient.com |
108 |
8 |
| ZoominfoBot |
107 |
70 |
| Qwantbot/1.0 |
99 |
33 |
| VelenPublicWebCrawler/1.0 |
90 |
6 |
| AliyunSecBot/Aliyun |
79 |
30 |
| AdsBot-Google-Mobile |
76 |
0 |
| Baiduspider-render/2.0 |
76 |
0 |
| BW/1.3; rb.gy/qyzae5 |
72 |
5 |
| wpbot/1.3 |
64 |
22 |
| CMS-Checker/1.0 |
62 |
0 |
| AppEngine-Google |
56 |
0 |
| FeedBurner/1.0 |
56 |
0 |
| Baiduspider/2.0 |
55 |
0 |
| YandexFavicons/1.0 |
53 |
0 |
| Unidentified SQL File Searcher |
47 |
0 |
| zgrab/0.x |
45 |
0 |
| MojeekBot/0.11 |
44 |
22 |
| 2ip bot/1.1 |
42 |
0 |
| HawaiiBot |
42 |
42 |
| fluid/0.0 |
41 |
41 |
| AdsBot-Google |
39 |
0 |
| GPTBot/1.0 |
39 |
36 |
| ev-crawler/1.0 |
37 |
2 |
| NetcraftSurveyAgent/1.0 |
37 |
0 |
| MixrankBot |
36 |
0 |
| Scrapy/2.12.0 |
33 |
0 |
| MJ12bot/v2.0.1 |
30 |
10 |
| ThinkChaos/0.3.0 |
30 |
0 |
| Photon/1.0 |
29 |
0 |
| Chrome Privacy Preserving Prefetch Proxy |
28 |
0 |
| ct‑git‑scanner |
28 |
0 |
| DomainStatsBot/1.0 |
26 |
13 |
| ips-agent |
24 |
7 |
| researchscan.comsys.rwth-aachen.de |
24 |
0 |
| Let's Encrypt validation server |
22 |
0 |
| SurdotlyBot/1.0 |
22 |
3 |
| YisouSpider |
19 |
9 |
| SemanticScholarBot |
18 |
12 |
| WPCheckBot |
17 |
0 |
| DnBCrawler |
16 |
1 |
| DotBot/1.1 |
16 |
6 |
| Embarcadero URI Client/1.0 |
16 |
0 |
| GeedoProductSearch |
15 |
5 |
| ShellBot 2.0 |
15 |
0 |
| Website-info.net-Robot |
15 |
9 |
| Feedly/1.0 |
14 |
0 |
| Linkbot 1.0 |
14 |
0 |
| Pandalytics/2.0 |
14 |
5 |
| YandexImages/3.0 |
14 |
0 |
| Grammarly/1.0 |
12 |
0 |
| AGU/0.8.6 |
11 |
0 |
| SEMrushBot |
11 |
0 |
| ORT/v1.0.0 |
10 |
10 |
| ScannerBot/1.0 |
10 |
0 |
| Yeti/1.1 |
10 |
6 |
| BacklinksExtendedBot |
8 |
4 |
| linkbot 1.0 |
8 |
8 |
| mithril-crawler/1.0 |
8 |
6 |
| Timpibot/0.8 |
8 |
0 |
| HatenaBookmark/4.0 |
7 |
0 |
| Pinterestbot/1.0 |
7 |
3 |
| Screaming Frog SEO Spider/20.2 |
7 |
0 |
| DomainChecker/1.0 |
6 |
0 |
| StartmeBot/1.0 |
6 |
0 |
| DNSScanner/1.0 |
5 |
0 |
| Screaming Frog SEO Spider/19.8 |
5 |
0 |
| Nicecrawler/1.1 |
4 |
0 |
| SemrushBot |
4 |
2 |
| URLCheckerBot |
4 |
0 |
| WanScannerBot/1.1 |
4 |
0 |
| archive.org_bot |
3 |
0 |
| DuckAssistBot/1.2 |
3 |
0 |
| KStandBot/1.0 |
3 |
0 |
| PriEcoBot/1.0 |
3 |
0 |
| VsuSearchSpider/1.0 |
3 |
1 |
| Xenu Link Sleuth/1.3.8 |
3 |
0 |
| Claude-SearchBot/1.0 |
2 |
1 |
| Clearbit Logo API |
2 |
0 |
| Google AppsViewer |
2 |
0 |
| Google-Firebase |
2 |
0 |
| MakeMerryBot/1.0 |
2 |
0 |
| PDF Drive Crawler 1.05 |
2 |
0 |
| SuperFastScraper/2.0 |
2 |
0 |
| WireReaderBot/1.0 |
2 |
1 |
| yacybot |
2 |
2 |
| YoohooSearch |
2 |
0 |
| DARPResearchBot |
1 |
1 |
| Googlebot-Mobile/2.1 |
1 |
0 |
| IABot/2.0 |
1 |
0 |
| trafilatura/2.0.0 |
1 |
0 |
| Twitterbot/1.0 |
1 |
0 |