An Efficient Informal Data Processing Method by Removing Duplicated Data

摘要

Useful information is extracted from many social networking services (SNS). The data uploaded by many users include the preferences or comments for the specific topics, which can be used for sentiment analysis. This sentiment information might be various social information, personal services and so on. However, SNS data are included a lot of duplicated data and spam data, which make slow down the processing time and the accuracy of the sentiment analysis. This paper presents an effective informal big data processing method by filtering out duplicated data or spam data. Hadoop Distributed File System (HDFS) and MapReduce method are used to extract sentiment information through machine learning. Experiment results increase not only the processing performance but also but also the accuracy of sentiment analysis. When duplication and spam are filtered out, the upload time is reduced about 4 seconds for 133,500 data. The analysis time after duplication and spam data are removed, is reduced by 36 percent and 41.26 percent respectively for 133,500 data. The incorrect spam detection is 20.53 percent for 35,320 data.