Skip to content

Research at St Andrews

Large-scale automatic k-means clustering for heterogeneous many-core supercomputer

Research output: Contribution to journalArticlepeer-review

Author(s)

Teng Yu, Wenlai Zhao, Pan Liu, Vladimir Janjic, Xiaohan Yan, Shicai Wang, Haohuan Fu, Guangwen Yang, John Donald Thomson

School/Research organisations

Abstract

This article presents an automatic k-means clustering solution targeting the Sunway TaihuLight supercomputer. We first introduce a multilevel parallel partition approach that not only partitions by dataflow and centroid, but also by dimension, which unlocks the potential of the hierarchical parallelism in the heterogeneous many-core processor and the system architecture of the supercomputer. The parallel design is able to process large-scale clustering problems with up to 196,608 dimensions and over 160,000 targeting centroids, while maintaining high performance and high scalability. Furthermore, we propose an automatic hyper-parameter determination process for k-means clustering, by automatically generating and executing the clustering tasks with a set of candidate hyper-parameter, and then determining the optimal hyper-parameter using a proposed evaluation method. The proposed auto-clustering solution can not only achieve high performance and scalability for problems with massive high-dimensional data, but also support clustering without sufficient prior knowledge for the number of targeted clusters, which can potentially increase the scope of k-means algorithm to new application areas.
Close

Details

Original languageEnglish
Pages (from-to)997-1008
Number of pages12
JournalIEEE Transactions on Parallel and Distributed Systems
Volume31
Issue number5
Early online date27 Nov 2019
DOIs
Publication statusPublished - May 2020

    Research areas

  • Supercomputer, Heterogeneous many-core processor, Data partitioning, Clustering, Scheduling, AutoML

Discover related content
Find related publications, people, projects and more using interactive charts.

View graph of relations

Related by journal

  1. Collaborative heterogeneity-aware OS scheduler for asymmetric multicore processors

    Yu, T., Zhong, R., Janjic, V., Petoumenos, P., Zhai, J., Leather, H. & Thomson, J. D., 1 May 2021, In: IEEE Transactions on Parallel and Distributed Systems. 32, 5, p. 1224-1237 14 p.

    Research output: Contribution to journalArticlepeer-review

  2. Parallel and streaming truth discovery in large-scale quantitative crowdsourcing

    Ouyang, R. W., Kaplan, L. M., Toniolo, A., Srivastava, M. & Norman, T. J., 1 Oct 2016, In: IEEE Transactions on Parallel and Distributed Systems. 27, 10, p. 2984-2997 14 p.

    Research output: Contribution to journalArticlepeer-review

ID: 264077454

Top