DevOps Blog: What is DevOps?

What is DevOps?
DevOps culture is described by strengthened collaboration, shared responsibility, improving quality from valued feedback and increasing automation.

DevOps is a blend between Developers and Operations teams, they follow a set of processes and tools that help them to create faster and more stable applications.

The role of a Developer is to create applications that are cutting edge and easy to use. The Operations team is tasked with keeping the application as stable as possible.

What goals do DevOps hope to achieve?
Their goal is to vastly improve the workflow to satisfy the following:

  • Deployment frequency
  • Achieve quicker release times
  • Lower failure rate/ bugs of new releases
  • Shorten downtime between fixes

Background
Before DevOps, the process used was called  Waterfall and it was very different to how things are done now. Fast forward to today and it the process is known as CICD.

What is Waterfall?
Waterfall was a process where the applications used to be fully developed before being released. When the application was released, there was no solutions for fixing the bugs!

What is CICD?
CICD stands for continuous integrations and continuous development. This allows the applications to be released before being fully developed. Developers upload their code to a CI server where the code is checked to see if it compatible with the current code and ensure there will not be any clashes.

Why is DevOps important?

Build
Building is when the developers are building the desired application for the target audience.

Test
Testing is when the application goes through a rigorous testing process to check it is suitable to release to end users.

Release
The application is released after it has gone through the testing phase and it will be perceived to be suitable for end user usage.

Monitor
Here you should monitor how your application is performing and also when receiving feedback from target audience, list out what the issues are and discuss the plan on how to fix them.

Plan
With all issues gathered from the feedback, now you can address the issues, recode and test again before releasing. The diagram below shows the CICD workflow.

DevOps Blog: Infrastructure as code

One of the first things about Cloud computing that really fired up my imagination was the concept of infrastructure as code. The phrase instantly conjured up the merging of two worlds in a very exciting way. As I looked into it more I realised that this concept was clearly one of the cornerstones of the public  Cloud (and of DevOps as well but that is probably best left for another blog post).

This all sounds wonderful but what does it actually mean? To understand this it is easiest to consider the two parts of the term separately. In a traditional IT environment the infrastructure consists of the servers and storage that runs the applications a business uses on a daily basis as well as the networking components that plumb everything together. The code is the software that actually makes up these applications written in a programming language. The infrastructure would be physical but the code would be stored digitally and importantly could be backed up, copied and different versions could be maintained.

Virtual Infrastructure 

In the world of the cloud however the hardware infrastructure components are (from the user’s point of view anyway) virtual. If you need a new server you just log into your account and with a few clicks of your mouse you can have it up and running. The concept of infrastructure as code takes this a step further however and allows the virtual, cloud based infrastructure to be described textually in a way very reminiscent of computer code. This textual description or template can then be used to request the infrastructure.

This has many remarkable and powerful implications. It means that your infrastructure is now reproducible at the click of a button which is great for producing test and development environments that are identical to a given production environment. It removed a large degree of human error from the process. Your infrastructure is also documented in a single location and the templates can be stored, backed up and version controlled in the same way that computer code can be.

How to Achieve It 

To give a concrete example, Amazon Web Services (AWS) <https://aws.amazon.com/> has a service called Cloudformation <https://aws.amazon.com/cloudformation/> which allows you to describe your infrastructure using a language called JSON <https://en.wikipedia.org/wiki/JSON> (JavaScript Object Notation). You then upload this template and the AWS Cloudformation service works out the dependencies between the various infrastructure components specified and requests that they are created. In AWS terms, this set of infrastructure is referred to as a stack. If you need to modify the stack then you can just make changes in the template and then Cloudformation can work out what has changed and apply them for you.

Infrastructure as code really is a concept that sounds simple and innocuous at first but the more you think about the more you realise how much potential it has to transform the way you work.

Cloud is about how you do computing, not where you do computing.

DevOps Blog: Cross Region Stack Management

Using StackSet For Cross Region Stack Management

AWS CloudFormation StackSets allows you to create, update, or delete CloudFormation stacks across multiple accounts and regions with a single operation.

This article will focus on how to deploy the same CloudFormation stack in multiple regions using AWS StackSets.

Scenario

Create S3 buckets in two different regions regions using StackSets.

Steps

Note: Use you account id when it asks for AdministratorAccountId

Note: Please be aware that this template grants Administrator access and so you might want to modify it to be more restrictive

  • Once the two roles are created we can begin working on StackSets
  • First we have to create and save the yaml file, we can call it s3.yaml

CloudF1

  • Go to the CloudFormation page and click on StackSets in the left tab
  • Select Create Stackset then Upload a template file of s3.yaml and click next
  • Put in the StackSet name and description and click next
  • Select self service permissions and select AWSCloudFormationStackSetAdministrationRole for IAM admin role ARN the IAM execution role name should be AWSCloudFormationStackSetExecutionRole
  • Under Account numbers put in your account ID and under Specify regions put in the regions you’d like the StackSets to be run in and then submit
  • Once the Stackset is created, select Stack Instance and the status should say OUTDATED but the status reason should say User Initiated, this means that the stack instance is getting configured. After a couple of minutes the status should change to Current and you can go to the Cloudformation pages in the regions you specified and you see that a new cloudformation stack has been created in those regions.

 

FF
Written by
Fayomi Fashanu: Senior AWS Solutions Architect at Ubertas Consulting

DevOps Blog: DMS: An Introduction (Part 1)

DMS: An Introduction (Part 1)

Anatomy and how it works

The Database Migration Service (DMS) is an AWS service that enables us to migrate vast amounts of data from our source databases either as a one-time load or without ever incurring downtime via continuous replication.

Over the years DMS has continuously evolved to support a wide range of engines and also the capability to undertake migrations where the source and destination engine are different (heterogeneous).

DMS Supports the following source/destination engines

Source:

  • Oracle
  • MySQL
  • Microsoft SQL Server
  • MariaDB
  • MongoDB
  • Db2 LUW
  • SAP
  • PostgreSQL
  • Amazon Aurora (MySQL & PostgreSQL)

Target:

  • Oracle
  • MySQL
  • Microsoft SQL Server
  • Amazon Aurora
  • Amazon Redshift
  • Amazon S3
  • Amazon DynamoDB
  • SAP ACE

 

Anatomy

Endpoints

Below are the key structural components of DMS. Naturally we start off with our endpoints, which when created must be defined as either source or target.

Along with this we simply configure our authentication information and optional connection attributes.

We can modify connection attributes in order to override particular settings within the DMS agent’s session, depending on your source/target database.

By default, DMS loads the tables in alphabetical order which isn’t always desirable depending on your database structure.

At Ubertas Consulting, we regularly encounter relationships between relational database tables which contain foreign key constraints and this can cause errors due to the way that DMS agents loads the tables. Due to this, we often update the connection attributes within the target endpoint to include the following:

 

stmt=SET FOREIGN_KEY_CHECKS=0

This ensures that we don’t received false-positive errors during the full-load.

Either side of our endpoints we have our source and target databases. Your source and destination databases don’t specifically need to be hosted within AWS, but must be supported by the DMS agent which runs on the replication instance.

Replication Instances

Next up we have our replication instances, which represent how we are actually charged for using DMS. AWS don’t charge for tasks or endpoints. The only costs that should be associated with carrying out a migration project outside of your replication instances would be the following services:

  • CloudWatch — keep an eye on excessive storage charges if you ever need to enable severe logging within your tasks. This modification will result in SQL statements and other verbose information being sent to your CloudWatch log groups/streams.
  • Data Transfer — depending on the location of your source/target databases in relation to your replication instances, charges for the ingress/egress of data can result in charges per GB.

Our replication instances are backed by AWS EC2 and much of the configuration is abstracted away from us.

The configuration options are limited to a subset of instance types (listed below), disk size (limited to gp2 volume type), VPC/Subnets and whether the instance publicly accessible. We can also make our replication instance support Multi-AZ, which means that it is deployed in a highly-available state so that in the event of an outage it can failover and prevent your critical migrations from being disrupted.

DMS Replication Instance Types:

  • dms.t2.micro ~ dms.t2.large
  • dms.c4.large ~ dms.c4.4xlarge
  • dms.r4.large ~ dms.r4.8xlarge

Migration Tasks

Finally we have DMS Migration Tasks, which run on our replication instances and represent what exactly it is we are migrating and how we’re doing it. AWS recommend that we break our migration into multiple Migration Tasks, and this is also evident in the service limits which are imposed upon us.

We highly recommend that you spend plenty of time analysing your source database and planning Migration Tasks before diving into your migration. For example, if you have particularly large tables with large-object columns then we recommend creating separate tasks for these. This will allow your other tasks to progress faster without being blocked.

It’s especially important to split out your migration into multiple tasks so that should something go wrong, you are able to respond to failures with more agility.

Service limits:

  • Replication Instances: 20
  • Migration Tasks: 200
  • Endpoints: 100

Console UI

AWS recently updated the design of the DMS in March 2020, here are some screenshots (April 2020). The main difference is that the previous sections listed below are now better organised into tabs.

Overview

Within this section we can view the metadata of our Migration Task, and there’s a helpful link to our CloudWatch logs. Logging within your DMS Migration Tasks is optional, but very useful for debugging if you ever experience connection or permission issues.

We can also view our task settings as JSON, which is intentionally there for you to view within the console because this is how we update our Migration Tasks. As of April 2020, we are still significantly limited to update we can make within the console and must the AWS CLI and pass JSON when the Migration Task is not in a running state.

 

Table statistics

The ability to view the status and progress of each table is in our opinion the most useful components within the Migration Task console view.

Below we can see there are 2 tables within the same schema named “auditing” — both have carried out an initial full-load, row validation has been completed and ongoing changes are now being captured using the source database’s binary logs.

 

CloudWatch metrics

Within the CloudWatch metrics section we are able to easily monitor a particular Migration Task metrics without having to go to CloudWatch directly.

However, we can still create custom dashboards within CloudWatch, for example if we would like to created aggregated views in order to summarise the overall progress of our database migration.

For a detailed explanation of the different CloudWatch metrics to DMS Migration Tasks, you can check out this AWS documentation page:

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Monitoring.html

 

Mapping rules

This is an area that we eluded to in an earlier section of this article. Within DMS, we are able to use a strategy that involves breaking up our migration project into multiple Migration Tasks. We can update our table mapping either by using the Console UI, or via JSON.

As well as specifying particular schemas within the database to exclude/include, we can also apply filters to tables if for example we have large tables and want to migrate our data in smaller chunks.

For more information on Mapping rules, you can visit the relevant AWS documentation page here:

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.html

Within this article, we explained the basic anatomy of DMS and how it works.

In part 2, we will dive deeper into how to optimise your Migration Tasks for speed and agility.

Andy Hammond
Written by
Andy Hammond: Tech Lead at Ubertas Consulting

Blog: What are Containers?

What are Containers?

Containers are a method of operating system virtualization that allow you to run an application and its dependencies in resource-isolated processes. Containers allow you to easily package an application’s code, configurations, and dependencies into easy to use building blocks that deliver environmental consistency, operational efficiency, developer productivity, and version control. Containers can help ensure that applications deploy quickly, reliably, and consistently regardless of deployment environment. Containers also give you more granular control over resources giving your infrastructure improved efficiency. Running containers in the AWS Cloud allows you to build robust, scalable applications and services by leveraging the benefits of the AWS Cloud such as elasticity, availability, security, and economies of scale. You also pay for only as much resources as you use.

Containers and Virtual Machines

monolith_2-VM-vs-Containers

Benefits of Containers

 

Containers_Benefit_EnvironmentConsistency

Environment Consistency

Containers enable portability and help reduce the organizational and technical frictions of moving an application through the development, testing, and production lifecycle. Containers encapsulate all the necessary application files and software dependencies and serve as a building block that can be deployed on any compute resource regardless of software, operating system, or hardware configurations (e.g., you can run the same container on your Ubuntu laptop and on your Red Hat Enterprise Linux production servers). Whatever you package as a container locally will deploy and run the same way whether in testing or production. This is beneficial for you and your organization because you can deploy an application reliably and consistently regardless of environment. This helps you avoid manually configuring each server and allows you to release new features faster.

 

Containers_Benefit_OperationalEfficiency

Operational Efficiency

Containers can help you get more from your computing resources by allowing to you easily run multiple applications on the same instance. With containers, you can specify the exact amount of memory, disk space, and CPU to be used by a container on an instance. Containers have fast boot times because each container is only a process on the operating system running an application and its dependencies. This reduced footprint enables you to quickly create and terminate applications or tasks encapsulated in a container allowing you to rapidly scale applications up and down. You can use blue-green deployment patterns to rollout new application versions (e.g., using Amazon Elastic Container Service) because the entire application and all its dependencies are contained in an image. Containers also provide process isolation, which allows you to put each application and its dependencies into a separate container and run them on the same instance. There are no shared dependencies or incompatibilities because each container is isolated from the other (e.g., you can run two containers that use different library versions on the same Amazon EC2 instance).

You can also create container images that serve as the base for other images. Operations teams can create a base image composed of the operating system, configurations, and the various utilities they want. Development teams can then build their application on top of the base image. This allows you to avoid the complexities of server configuration.

 

Containers_Benefit_DeveloperProductivity

Developer Productivity

Containers increase developer productivity by removing cross-service dependencies and conflicts. Each application component can be broken into different containers running a different microservice. Containers are isolated from one another, so you don’t have to worry about libraries or dependencies being in sync for each service. Developers can independently upgrade each service because there are no library conflicts.

 

Containers_Benefit_VersionControl

Version Control

Containers allow you to track versions of your application code and their dependencies. Docker container images have a manifest file (Dockerfile) that allows you to easily maintain and track versions of a container, inspect differences between versions, and roll-back to previous versions.

 

Cloud is about how you do computing, not where you do computing.