Automated Network Monitoring

Text from presentation made at the M.I.T. Workshop on Internet Survey Methodology and Web Demographics, 29-30 January 1996, Cambridge, MA, USA 30 January 1996

Marc Abrams and Stephen Williams

Computer Science Department
Virginia Tech, Blacksburg, VA 24061-0106,


[Link to postscript version of powerpoint slides.]


The Four Logging Methods

Automated Network Logging

Computer network diagram showing what components we log

What Network Monitoring Can Do

Dark Side of Network Monitoring

Employer could monitor what employees are doing.

(But did you realize this is could already be happening without network monitoring?)

Up Side of Network Monitoring

Log File Record

Our log records contain the following fields; the first six are in the common log format generated by popular Web servers.
Client machine (optional)
Timestamp of GET packet
Command (containing URL)
HTTP version
Return code
Document size

User identity (optional)
Browser information
Gateway or proxy server version

MIME type of document returned
URL linked from (i.e., the parent's URL) Additional resolution of GET timestamp, giving milliseconds Connect time, from GET packet to first response packet (in seconds) 

How Network Monitoring Works

What Could You Do With Client Logs?

Application of Automated Monitoring

Privacy Issues in Our Monitoring Activities

HTTP Protocol Suggestions

Tool Set Available


Appendix -- More Details

Details of Method

Volume of Collected Data

Department backbone traffic workload:

Graph of Daily WWW Traffic on CS Backbone vs. time omitted

Error Sources in Resultant Log