基于改进主题分布特征的神经网络语言模型

摘要

在递归神经网络(RNN)语言模型输入中增加表示当前词所对应主题的特征向量是一种有效利用长时间跨度历史信息的方法.由于在不同文档中各主题的概率分布通常差别很大,该文提出一种使用文档主题概率改进当前词主题特征的方法,并将改进后的特征应用于基于长短时记忆(LSTM)单元的递归神经网络语言模型中.实验表明,在PTB数据集上该文提出的方法使语言模型的困惑度相对于基线系统下降11.8％.在SWBD数据集多候选重估实验中,该文提出的特征使LSTM模型相对于基线模型词错误率(WER)相对下降6.0％;在WSJ数据集上的实验中,该特征使LSTM模型相对于基线模型词错误率(WER)相对下降6.8％,并且在eval92测试集上,改进隐含狄利克雷分布(LDA)特征使RNN效果与LSTM相当.%Attaching topic features to the input of Recurrent Neural Network (RNN) models is an efficient method to leverage distant contextual information. To cope with the problem that the topic distributions may vary greatly among different documents, this paper proposes an improved topic feature using the topic distributions of documents and applies it to a recurrent Long Short-Term Memory (LSTM) language model. Experiments show that the proposed feature achieved an 11.8％relatively perplexity reduction on the Penn TreeBank (PTB) dataset, and reached 6.0％and 6.8％relative Word Error Rate (WER) reduction on the SWitch BoarD (SWBD) and Wall Street Journal (WSJ) speech recognition task respectively. On WSJ speech recognition task, RNN with this feature can reach the effect of LSTM on eval92 testset.