A Constraint-Guided Web Walker
for Specialized Activities

Abstract: A Web walker for the Hypertext Transfer Protocol (HTTP) implements a constraint posting control architecture. The Web walker uses a declarative and extensible vocabulary of constraints to characterize traversals of Web structures. Starting from a root resource, the walker recursively follows all hyperlinks whose associated resource satisfies the constraints guiding the walk. Constraints are sorted according to efficiency class before application to candidate resources. This constraint ordering conserves computational and network resources. As the walker traverses the structure it performs operations that are specified in a declarative and extensible action vocabulary. Taken together, a set of constraints and a set of actions comprise an activity, which can be named and reused.

Several Web-accessible applications use this constraint-guided framework:

• Web Mapper: Hyperlinks from a root uniform resource indicator (URI) are followed and displayed in Hypertext Markup Language (HTML).

• Web Document Locator: A Salton-style statistical retrieval algorithm is applied to TEXT or HTML documents, which are reported to the user when their score exceeds a threshold.

• Web Archiver: Regions of Web structure are traversed and transferred to a local disk drive. Hyperlinks within the structure are remapped to preserve link structure in the new location.