App and Data Migration on the Cloud

Snehal Ghosh
App Migration Data Migration AWS

App and Data Migration on the Cloud

Problem Statement

Migrate a on-prem legacy app along with its data on the cloud to leverage Cloud specific features with a very thin monthly budget constraint.

Need

Following aspects were considered as the need for moving to the Cloud:

  • Easy Scale-up and Scale-down
  • App Failover (Active - Passive)
  • Real time data replication
  • Monitoring and Alerting
  • Storage Backups
  • CI/CD Pipelines

Aspects Considered

Followibg aspects were considered in this whole activity:

  • Infrastrtuture & Architecture
  • Security
  • Cookbook: CI/CD Pipelines
  • Switchover Plan
  • Monitoring & Alerting

Infrastrtuture & Architecture

Following are the top level infra bits to be considered while setting up the infrastructure:

  • Network
  • App and DB Server (Active and Passive)
  • GIT Server
  • Images
  • Storage Backups
  • Static IPs

Network

Need to setup all the servers / instances inside a AWS Public Network. Earlier the thought process was to have a private network and expose the app via a load balancer mapped to a domain, but since that will introduce a need for having a NAT server which introduced a risk of going over-budget.

With public networks too we get a good level of external security from AWS hence for simplicity we will go with Public Network and solid Security groups which will provide the same security as that of a Private Network. Security Groups will be discussed later in this document.

App & DB Server

Both of these servers resided in the Network created above, however master and slave were placed in different availability zones since we need separate physical locations for the master and slave server.

GIT Server

Complete code for the legacy app was migrated to an GIT Server which was used by the CI/CD pipelines. A appropriate GIT branching strategy was suggested to the cliene based on their development cycle.

Images

We saw the need to have only one image created that of the legacy app. This helped us re-create the App Server in a quick and effortless mode. This image will be used for the following scenarios

  • Master and Slave failure
  • Re-creating the app node in case of future clustering.

Storage Backup

Even though we have setup the database as a Master and Slave setup, there are chances that both may go down. In such cases it was a good to have daily / weekly backups taken of the databases and stored. At a given point of time we stored past 5 backups and clear the rest of them. We opted for AWS S3 for this as it is very cost effective and reliable.

Static IPs

It was better to have a static IP for the slave server too, but during failover we will switch the IP of master to the slave since changing the domain mapping of the lumus domains won’t be quick.

Security

Covered 2 aspects:

  • External
  • Internal
External

External Security covered the external attacks aspects such as brute force attack etc. Security Groups is a concept introduced by AWS which basically tells us which server and port is accessible from where. For example, in normal situations apart from port 8080 (http) and 443 (https) no other port should be open to the outside world. Ports like 22 (SSH), 21 (FTP) and other ports should only be opened up on demand and that too only for a selected set of IPs. For Storage Backups, AWS provides role based access controls for each of the backup folders.

Internal

Internal Security deals with the internal server security. Scriptuit suggested Red Hat Enterprise Linux (RHEL) as the OS option. RHEL provides a very solid firewall / IP tables. Along with that we uninstalled / stopped any vulnerable services running. Each server had a password-less login which is vulnerable to dictionary attacks. The SSH access was made available only using keys and that too for specific IPs. Root access for each of the servers were disabled permanently.

Database access was restricted to only the apps. AWS Accounts access was limited and sharing the master AWS login was barred and AWS IAMs were used instead.

Cookbook

CI/CD pipelines were built for the following flows:

  • Setting up a fresh App Server
  • Creating a image from a App Server
  • Keeping the image of App Server updated
  • Deploying apps on App Server
  • Spawning off a new server from a App Server image
  • Switching from Master to Slave Server in case of Master failover
  • DB Backup to storage
  • Restoring App Server from a storage backup
  • Software / Firewall Updates

Architecture Diagram

Considering all the above scenarios, following was the deployment architecture diagram:

Switch Over Plan

Switch over plan to consisted of the following aspects:

  • Setup new server on temporary sub domains
  • Database Migrations
  • App Migration
  • Minimum downtime of the app
  • Live Domain migration pointing to the new Cloud IP

Monitoring & Alerting

One of the core reasons to migrate to the cloud was to use it's full Monitoring and Alerting Capability. Following metrics were monitored and alerts were setup:

  • Master Server Down
  • Slave Server Down
  • App Server(s) CPU reaching 70% threshold
  • DB Storage at 70%
  • DB Storage at 80%
  • DB Storage at 90%
  • App Server(s) Memory Utilization reaching 50% thhreshold