Lexico-syntactic normalization model for noisy SMS text

摘要

Today, digital mediated interactions and communications being an important constituent. The expeditious growth of electronic communications such as Emails, micro blogs, SMS and chats etc has fabricated extensively noisy forms of text. It predominantly in young urbanites. The tremendous growth of noises in text are due to a variety of factors, such as the small number of characters allowed per text messages (160 characters is allowed per SMS and 140 characters allowed per tweets), inventing new abbreviations, using non standard orthographic forms, phonetic substitution etc. In this paper we introduce a lexico-syntactic normalization model for cleaning the noisy texts. The normalization is based on the channelized database and a user feedback system. The syntactic analysis of sentences is based on a bottom up parser. The model will capture the user interaction for improving the model accuracy. Precursory evaluation shows that the channel model will normalize the noisy word to their standard peer with better accuracy. The sentence validation achieved 95.7% accuracy.