A Constraint-Guided Web Walker
for Specialized Activities

John C. Mallery, Andrew J. Blumberg & Christopher R. Vincent
Intelligent Information Infrastructure Project
Artificial Intelligence Laboratory
Massachusetts Institute of Technology

Proceedings of the Dynamic Objects Workshop at Object World East, Boston, Massachusetts, May 5-6, 1996

Abstract: A Web walker for the Hypertext Transfer Protocol (HTTP) implements a constraint posting control architecture. The Web walker uses a declarative and extensible vocabulary of constraints to characterize traversals of Web structures. Starting from a root resource, the walker recursively follows all hyperlinks whose associated resource satisfies the constraints guiding the walk. Constraints are sorted according to efficiency class before application to candidate resources. This constraint ordering conserves computational and network resources. As the walker traverses the structure it performs operations that are specified in a declarative and extensible action vocabulary. Taken together, a set of constraints and a set of actions comprise an activity, which can be named and reused.

Several Web-accessible applications use this constraint-guided framework:

The Web walker was implemented in Common Lisp as a facility for the Common Lisp HTTP Server.

Keywords: Actions, Common Lisp, Constraints, Constraint Posting, HTML, HTTP, Intelligent Agents, Internet, Resource Discovery, Technology, Servers, Web Robot, Web Walker, World Wide Web.

Hypertext: http://www.ai.mit.edu/projects/iiip/doc/cl-http/w4/w4.html
Postscript: http://www.ai.mit.edu/projects/iiip/doc/cl-http/w4/w4.ps