Introduction

Danger

This project is ENTIRELY EXPERIMENTAL at the moment. Use at your own will or if you want to contribute to it

Mario is an easy python DSL for describing steps of execution of a pipeline.

Even more, the steps can be scaled up independently and spread in a network.

This is NOT a DATA pipeline framework

_images/icon-fireball.png

This framework allows you to describe workers using a simple DSL. Each worker produces a given job_type.

A pipeline is a sequence of job types that will be coordinate with any idle workers that announce their availability.

This gives you the advantage of scaling your infrastructure horizontally and vertically with very little effort.

Features

  • Describe step workers in python classes and get them running within minutes
  • Describe pipelines that can juggle with any available steps
  • Easily scale steps individually in a pipeline
  • Easily scale pipelines onto clusters
  • Empower sys-admins to take quick action and increase the number of steps on demand, be it with new machines, new docker instances or even one-off spawned processes.
  • On-demand live web interface with live pipeline cluster information
  • Redis queue and metrics persistence