Investigation of Mixed Mode Programming on GPU Clusters for Matrix-Matrix Multiplication

Abstract

With the advancement of the high performance computing and recent developments in GPU computing, GPU accelerated multicore systems are becoming ubiquitous. At present, research is carried out with the objective of exploring the possibility of exploiting the tremendous parallelism, aggregate compute power and hierarchy of memory that GPU clusters offer. But the architectural heterogeneity of a GPU accelerated cluster poses new challenges like increased complexity, communication overhead and task management issues etc. Coupling a parallel programming model with a GPGPU programming platform by mixed mode programming is a viable solution for the aforementioned problems. Objective of this research is to investigate mixed mode programming on a GPU cluster. We have considered two possible mixed mode programming models: distributed-memory model with MPI+CUDA and partitioned global address space model (PGAS) with UPC+CUDA. As a case study, we designed and implemented the Strassen algorithm using MPI+CUDA and UPC+CUDA optimized to exploit the overlying memory model. Then we evaluated the strengths and weaknesses of suggested two hybrid models in terms of the performance, memory efficiency and programmability using several experiments. Experimental results and observations indicate that with the proper hardware support, our implementations of the Strassen algorithm outperform the existing matrix-matrix multiplication methods in terms of performance and scalability with reduced tradeoffs in communication and programmability.