Validation level: 4. Other peer-reviewed paper publication

ShadowGuides is a system for in-situ learning of multi-touch and whole-hand gestures on interactive surfaces. ShadowGuides provides on-demand assistance to the user by combining visualizations of the user’s current hand posture as interpreted by the system (feedback) and available postures and completion paths necessary to finish the gesture (feedforward).

Copy Bibtex Freeman, D.; Benko, H.; Morris, M. R. and Wigdor, D. ShadowGuides: Visualizations for In-situ Learning of Multi-touch and Whole-hand Gestures. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, pages 165-172, ACM, New York, NY, USA, ITS '09 , 2009.
Also featured in
Storyboard of Functions

ShadowGuides consists of two parts: a registration pose guide for teaching various registration poses and user shadow annotations that guide the user from the current hand pose through completion of the gesture.

The registration pose guide is a pop-up panel that shows diagrams of available registration poses, each accompanied by a list of gestures that can be completed from that pose.

Once the user begins a gesture with a specific registration pose, we show user shadow annotations to guide her to completion of the desired gesture. The annotation consists of three parts: arrows, shape deformation keyframes, and dynamic markers

  • arrows
  • shape deformation keyframes
  • dynamic markers

Users of ShadowGuides are able to learn what aspects of the gesture were most important (due to the emphasized information in the system’s feedforward annotations), as well as see how their changing posture affected recognition (due to the feedback of the user shadow)


Using video to teach gestures seemed more effective at conveying the general idea, or gist, of the motion.