Reference-less Measure of Faithfulness for Grammatical Error Correction

摘要

We propose USlM. a semantic measure for Grammatical Error Correction (GEC) that measures the semantic faithfulness of the output to the source, thereby complementing existing reference-less measures (RLMs) for measuring the output's grammatically. USlM operates by comparing the semantic symbolic structure of the source and the correction, without relying on manually-curated references. Our experiments establish the validity of USlM. by showing that (1) semantic annotation can he consistently applied to ungram-matical text; (2) valid corrections obtain a high USlM similarity score to the source; and (3) invalid corrections obtain a lower score.