THE CHALLENGE OF AUTOMATICALLY ANNOTATING SOLUTION DOCUMENTS: Comparing Manual and Automatic Annotation of Solution Documents in the Field of Mechanical Engineering

摘要

This paper contains part of the actual research in the use case PROCESSUS of the German research program THESEUS. A case study about comparing manual and automatic annotation of solution documents in the field of mechanical engineering is described. A set of six solution documents was annotated manually by four users. Then, the same set of documents was annotated automatically by an ontology-based system. The two annotations are compared considering proposed ranking numbers. These ranking numbers give the weighting of annotations according to the overall and merged manual annotations. Therewith, they serve as a reference for the expected result of the automatic annotation. Comparing the automated and the manual annotation can not only reveal limitations of the automatic annotation process but also raise interesting questions to what extent domain specific knowledge has to be represented in the ontology.