Cloud Agonistic Automated Cluster Provisioning

Abhijit Patharkar
Cluster Provisioning AWS Azure Google Cloud Terraform

Cloud Agnostic Automated Cluster Provisioning

Problem Statement

Cloud agonstic automated cluster provisioning using Cloud SDKs and an UI management layer to run tools, consume capacity and then destroy it.

User Inputs

At a high level, the user needed to input the following:

  • Cluster Size
  • Compute per container/instance
  • Total Memory
  • Memory per container/instance
  • Storage
  • Tool to run
  • Input Data

Cloud SDKs

The ask here was to create a UI management layer for the cluster setup, pricing forecast and destroying with a history maintanance for billing purposes. Each of the cloud providers intended (AWS, Azure and Google Cloud) either provided SDKs or APIs to communicate to the Cloud to create and manage the infrastructure.

This being cloud agnostic, it had to have a separate management interface which would decide on to which cloud to communicate with. At a high-lever following was the development cum deployment architecture involved:

Technologies used

Based on the SDKs availability the backend development was distributed along:

  • JEE
  • Python
  • Shell Scripting