About Crawler Activity

Introduction

The Crawler Activity web application provides a user interface for exploring the activity of web crawlers at my two websites, sphaerula.com and conradhalling.com. Since I suspected that the bulk of traffic at both of these sites was from web crawlers, I decided to analyze the access log files produced by the Apache httpd web server to identify the crawlers that were accessing my web pages.

As presented on the Summary page, more than 96% of the traffic on my websites comes from web crawlers.