MPI

"Message Passing Interface (MPI) is a standardized and portable message-passing system designed by a group of researchers from academia and industry to function on a wide variety of parallel computers. The standard defines the syntax and semantics of a core of library routines useful to a wide range of users writing portable message-passing programs in C, C++, and Fortran. There are several well-tested and efficient implementations of MPI, many of which are open-source or in the public domain. These fostered the development of a parallel software industry, and encouraged development of portable and scalable large-scale parallel applications."

https://en.wikipedia.org/wiki/Message_Passing_Interface


Tutorials

Multi-GPU Programming with MPI - http://on-demand.gputechconf.com/gtc/2015/presentation/S5117-Jiri-Kraus.pdf

The latest OpenMPI includes these capabilities.