The objects of interest in the billiards game (the table, the pockets and the balls) possess some properties which make it particularly well-suited for computer vision. An important feature that comes to mind is color. The green felt of the table and the distinctive contrasting colors of the balls and pockets allow robust color segmentation. In addition, the structures we are tracking have simple geometric shapes. The balls and pockets have a standard circular structure so radial symmetry can be used.
Figure: The Algorithm's Main Loop
Figure 
depicts the algorithm loop that iterates
repeatedly over video data. The system begins by searching the image
for a region that contains green felt colors. The largest green felt
patch corresponds to the top of the table and its contour is
determined. The interior of the table is then analyzed to find balls
by computing symmetry. The centers of circular objects are detected in
this region. Similarly, the periphery of the table is also processed
to compute radial symmetry and generates symmetry peaks at the centers
of the pockets. Each radial symmetry peak is then classified to
determine which object generated it (ball, pocket, table). This is
done by analyzing the colors of the symmetric objects and checking
them against a database of color models for each ball, pocket and
table. Once all symmetry peaks are classified, we can compute linear
trajectories from the cue ball to every other solid or striped ball
(depending on what the user selects). Then, we compute trajectories
from each of these to a destination pocket. Each of these trajectories
is evaluated to see if it is feasible and rated in difficulty
depending on how accurate the shot must be to sink the ball. The
easiest shot is then displayed on-screen using linear
trajectories. These trace out the paths of the cue ball and the ball to sink.