This paper proposes tree-based decomposition and sub-expression interchange for generating new syntactically valid handwritten mathematical expressions (HMEs) from a given set of HMEs to train an HME recognition model and a mathematical language model (LM). The recognition model is dual trained using weakly supervised learning and encoder-decoder attention loss on the generated samples. Recognition experiments indicate that the proposed data generation method is superior to other such methods for offline HMEs. The HME recognition model increases the expression recognition rates by 1.47, 2.88, and 2.67 points on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014, 2016, and 2019 testing sets, respectively. The LM also increases them by 8.92, 6.88, and 2.59 points on the testing sets. Further adding extra LaTeX sequences is cost effective in strengthening the LM for the expression recognition rates being 2.54, 2.8, and 1.25 points higher than without them on the CROHME testing sets, respectively. Among academic systems, the trained HME recognition system achieves the best performance with 64.60 and 66.08 expression recognition rates on the CROHME 2014 and 2016 testing sets and a comparable expression recognition rate of 58.72 on the CROHME 2019 testing set, respectively. Comparison with top systems from companies on CROHME 2019 suggests that more real and/or generated HME patterns will improve the performance of HME recognition models as well as mathematical language models. (c) 2021 Elsevier B.V. All rights reserved.
展开▼
