Dice Similarity Based Ensemble Clustering for Sparsely Distributed High Dimensional Data

摘要

Data mining is the process of extracting the valuable patterns from large volume of data. Clustering in data mining is the process of dividing the data points depending on their similarity level. Clustering techniques for managing the high dimensional data is more complicated because of intrinsic sparsity nature of high dimensional data. However, the clustering accuracy and similarity measurement time was not improved using existing clustering techniques such as fuzzy c-means and spectral clustering. In order to overcome these limitations, Dice Similarity Threshold based Ensemble Clustering (DST-EC) Technique is introduced. DST-EC technique clusters the sparsely distributed high dimensional data points based on the similarity value. Initially in DST-EC technique, Dice Similarity Coefficient Measurement Algorithm is introduced to measure the similarity between two high dimensional data points with minimum similarity measurement time consumption and higher true positive rate. After finding the similarity, the different similarity threshold range is set for clustering the data points. Finally based on the similarity threshold value, Similarity Threshold Ensemble Clustering Algorithm clusters the similar data points to form number of clusters with higher clustering accuracy. The performance of DST-EC technique is measured in terms of true positive rate, similarity measurement time and clustering accuracy with El Nino weather data sets from UCI Machine Learning Repository. The experimental result explains that the DST-EC technique improves the clustering accuracy by 15% and reduces the similarity measurement time by 21% when compared to state-of-the-art-works.