When faced with complex design, analysis, or engineering tasks, novices and professionals alike attempt to better understand problems through diagrams, and a natural first step in this process is working on a whiteboard. Through their drawings, people can gain valuable insights into subtleties of design and analysis tasks, but once a diagram gains sufficient complexity, further progress becomes tedious (or even intractable) without the aid of a computer. Sketch recognition interfaces over the last few decades have sought to ease this barrier to entry through pen-based interaction, enabling users to directly sketch the structures they want to analyze, leveraging their previous experience with drawing diagrams. From circuit design, chemical analysis, and even 3D modeling, these applications have allowed people to more effectively utilize the power of computation in their everyday work. Unfortunately, despite providing a more familiar interaction style, interface hardware requirements mean that sketch recognition interfaces still go largely unused; desktop-scale pen capture displays presently remain largely relegated to CAD firms or art studios, with the whiteboard-scale equivalents, necessary for collaborative design tasks, being even more exotic. The goal of this work is to utilize common consumer hardware (webcams, smartphones, and projectors when available) to enable sketch recognition where people are already drawing: whiteboards, chalkboards, and even on loose paper. In service of this goal, we have created SPARK, the Sketch Practically Anywhere Recognition Kit. Our system enables a person to interact with real world drawings by recognizing meaning from images of hand drawn diagrams that are captured via a smartphone or a webcam, and by providing an interface through augmenting projectors or the phone's own display. The system is constructed in three parts: a stand-alone stroke-based sketch recognition framework, a module for extracting stroke data from static images, and finally a component to extract key frames from a video stream of an active whiteboard for interactive recognition. As evidence of our methods, we have created a series of prototype applications that exercise each module: SketchVis applies traditional, virtual stroke sketch recognition techniques to data exploration through charting on a whiteboard-scale interface. Our Turing machine app enables simulation of Turing machine diagrams drawn with physical ink through a mobile, explicit capture interface. Finally, the equation graphing application serves as a proof of concept exercise of the continuous sketch recognition of--and interaction with--physical ink captured with a webcam.
展开▼
