Mesos

"Apache Mesos is an open-source cluster manager that was developed at the University of California, Berkeley. It "provides efficient resource isolation and sharing across distributed applications, or frameworks". The software enables resource sharing in a fine-grained manner, improving cluster utilization.

Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.

Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elasticsearch) with API’s for resource management and scheduling across entire datacenter and cloud environments.

Mesos consists of a master daemon that manages agent daemons running on each cluster node, andMesos frameworks that run tasks on these agents.
The master enables fine-grained sharing of resources (CPU, RAM, …) across frameworks by making them resource offers.  The master decides how many resources to offer to each framework according to a given organizational policy, such as fair sharing or strict priority. To support a diverse set of policies, the master employs a modular architecture that makes it easy to add new allocation modules via a plugin mechanism.

A framework running on top of Mesos consists of two components: a scheduler that registers with the master to be offered resources, and an executor process that is launched on agent nodes to run the framework’s tasks (see the App/Framework development guide for more details about framework schedulers and executors). While the master determines how many resources are offered to each framework, the frameworks' schedulers select which of the offered resources to use. When a framework accepts offered resources, it passes to Mesos a description of the tasks it wants to run on them. In turn, Mesos launches the tasks on the corresponding agents."

https://en.wikipedia.org/wiki/Apache_Mesos

http://mesos.apache.org/

http://mesos.apache.org/documentation/latest/frameworks/

https://github.com/mesosphere/marathon

Software that will run on Mesos includes:

• Cray Chapel is a productive parallel programming language. The Chapel Mesos scheduler lets you run Chapel programs on Mesos.
• Dpark is a Python clone of Spark, a MapReduce-like framework written in Python, running on Mesos.
• Exelixi is a distributed framework for running genetic algorithms at scale.
• Hadoop Running Hadoop on Mesos distributes MapReduce jobs efficiently across an entire cluster.
• Hama is a distributed computing framework based on Bulk Synchronous Parallel computing techniques for massive scientific computations e.g., matrix, graph and network algorithms.
• MPI is a message-passing system designed to function on a wide variety of parallel computers.
• Spark is a fast and general-purpose cluster computing system which makes parallel jobs easy to write.
• Storm is a distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

MPI

" We have also run MPICH2 directly on top of mesos. See MESOS_HOME/frameworks/mpi for this port. Basically it sets up the MPICH2 MPD ring for you when you use nmpiexec."

http://mesos.readthedocs.io/en/0.21.1/running-torque-or-mpi-on-mesos/

https://github.com/apache/mesos/tree/master/mpi

https://github.com/mesosphere/mesos-hydra