A demonstrational interface for recording technical procedures by annotation of videotaped examples

Journal/Book: Int J Hum-Comput Studies. 1995; 43: 24-28 Oval Rd, London, England NW1 7DX. Academic Press (London) Ltd. 383-417.

Abstract: In conventional knowledge acquisition, a domain expert interacts with a knowledge engineer, who interviews the expert, and codes knowledge about the domain objects and procedures in a rule-based language, or other textual representation language. This indirect methodology can be tedious and error-prone, since the domain expert's verbal descriptions can be inaccurate or incomplete, and the knowledge engineer may not correctly interpret the expert's intent. We describe a user interface that allows a domain expert who is not a programmer to construct representations of objects and procedures directly from a video of a human performing an example procedure. The domain expert need not be fluent in the underlying representation language, since an interaction is through direct manipulation. Starting from digitized video, the user selects significant frames that illustrate before- and after- states of important operations. Then the user graphically annotates the contents of each selected frame, selecting portions of the image to represent each part, labeling the parts, and indicating part/whole relationships. The actions that represent the transition between frames are described using the technique of programming by demonstration (also called programming by example). The user performs operations on concrete visual objects in the graphical interface, and the system records the user's actions. Explanation-based learning techniques are used to synthesize a generalized program that can be used on subsequent examples. The knowledge acquisition and video annotation facilities are implemented as part of the graphical editor Mondrian, which incorporates a programming by demonstration facility. We explain the operation of Mondrian's interface in its base domain of graphical editing as well as for the video annotation and knowledge acquisition application. The result of the knowledge acquisition process is object descriptions for each object in the domain, generalized procedural descriptions, and visual and natural language documentation of the procedure. We illustrate the system in the domain of documentation of operational and maintenance procedures for electrical devices.

Note: Article H Lieberman, Mit, Media Lab, Cambridge, MA 02139 USA

Zurück | Weiter