A Forest of Sensors:
Using adaptive tracking to classify and monitor activities

MIT AI Lab

Eric Grimson, Paul Viola,

Chris Stauffer, Raquel Romano,

Lily Lee, Gideon Stein

A Forest of Sensors

Given autonomous vision modules (AVMs): low power, low cost, can compute wide range of visual routines

Create a forest of disposable AVMs:

attached to trees, buildings, vehicles

Question: Can the forest bootstrap itself to monitor sites for activities?

Slide 4

Capabilities and components

Self calibration

building rough site models

detecting visibility

primitive detection

moving object modeling

activity detection

activity calibration

Working hypothesis

Can achieve all of these capabilities simply by accurately tracking moving objects in the scene.

A robust, real-time tracker

Model each pixel as an independent process

Model previous n samples with weighted mixture of Gaussians, using exponentially decaying time window

Background defined as set of dominant models that account for T percent of data

Update weights and parameters

Pixel > 2 sigma from background is moving

Select significant blobs as objects

Adaptive tracking

Dynamic calibration

Track objects in multiple cameras

use correspondences to find a homography, assuming planar motion

modify homography by fitting image features

use non-planar features to solve for epipolar geometry, and refine

Dynamic calibration

Site modeling

N camera stereo from calibration

dynamic tracking updates

visibility detection and placement update

reconstruction from multiple moving views

Site models by tracking

Site modeling from stereo

Multi-camera stereo reconstruction

Stereo reconstruction

Activity detection & calibration

Need to find common patterns in track data

Pattern tracks

Detect Regularities & Anomalies?

Example track patterns

Running continuously for almost 1 year

during snow, wind, rain, ...

one can observe patterns over space and over time

need a system to detect automatically

Classifying objects

Simple classifiers based on feature selection

Flexible template querying

Example

Detection

Classifying objects

Using flexible templates to detect vehicles

Classifying activities

Quantize state space of continuous observations by fitting Gaussians

map sequence of observables to sequence of labels

compute co-occurrence of labels over sequences -- defines joint probability

cluster by E/M methods to find underlying probability distributions

Automatic activity classification

Mapping patterns to maps

Multi-camera coordination

Finding unusual events

Finding similar events

Future plans

integration of pieces

site models from dynamic tracking

activity classification from multiple views

variations on classification methods

vocabulary of activities and interactions

interactions of activity tracks

fine scale actions


	MIT AI Lab
	Eric Grimson, Paul Viola,
	Chris Stauffer, Raquel Romano,
	Lily Lee, Gideon Stein


	Given autonomous vision modules (AVMs): low power, low cost, can compute wide range of visual routines
	Create a forest of disposable AVMs:
		attached to trees, buildings, vehicles
	Question: Can the forest bootstrap itself to monitor sites for activities?


	Self calibration
	building rough site models
	detecting visibility
	primitive detection
	moving object modeling
	activity detection
	activity calibration


	Can achieve all of these capabilities simply by accurately tracking moving objects in the scene.


	Model each pixel as an independent process
	Model previous n samples with weighted mixture of Gaussians, using exponentially decaying time window
	Background defined as set of dominant models that account for T percent of data
	Update weights and parameters
	Pixel > 2 sigma from background is moving
	Select significant blobs as objects


	Track objects in multiple cameras
	use correspondences to find a homography, assuming planar motion
	modify homography by fitting image features
	use non-planar features to solve for epipolar geometry, and refine


	N camera stereo from calibration
	dynamic tracking updates
	visibility detection and placement update
	reconstruction from multiple moving views


	Running continuously for almost 1 year
		during snow, wind, rain, ...
	one can observe patterns over space and over time
	need a system to detect automatically


	Quantize state space of continuous observations by fitting Gaussians
	map sequence of observables to sequence of labels
	compute co-occurrence of labels over sequences -- defines joint probability
	cluster by E/M methods to find underlying probability distributions


	integration of pieces
	site models from dynamic tracking
	activity classification from multiple views
	variations on classification methods
	vocabulary of activities and interactions
	interactions of activity tracks
	fine scale actions