Aspect Capital

Migration to Amazon Managed Workflows for Apache Airflow (MWAA)

About Aspect Capital

Aspect Capital is a global systematic investment manager, founded in 1997 by pioneers in quantitative investment techniques. Aspect has since grown into a team of over 130 professionals, managing over $7.5bn. Research and technology are at the core of what Aspect does, ensuring the innovative evolution of its quantitative models.

Aspect has traditionally maintained most of its Microsoft stack applications and databases on-premises. Ubertas Consulting is assisting Aspect Capital to migrate several key workloads to AWS.

The Challenge

Aspect Capital previously relied on workload automation solutions to perform routine tasks, such as Extract Transform Load (ETL), data analysis and forecasting, reporting, and signalling to other workloads.

The large number of tasks performed by these workflows was highly compute-intensive and ran on expensive static machines; often overprovisioned and underutilised between workflow executions. Scaling out when workloads spiked unexpectedly was difficult, given the inelastic nature of the static hosts.

The environments were all managed by in-house staff, which carried a large amount of unwanted overhead. This was a concern for business continuity and disaster recovery also.

The challenge for Ubertas was to migrate a workload designed to make use of a more elastic compute solution, include orchestrated infrastructure deployments, and provide more control and flexibility of costs.

The Solution

As part of the AWS Migration Acceleration Program (MAP), Ubertas Consulting worked closely with members of Aspect Capital’s infrastructure and development teams to perform several exploratory proof of concept (PoC) exercises, before augmenting both teams on a day-to-day basis to deliver a working end-to-end solution.

Amazon Managed Workflows for Apache Airflow (MWAA) was wrapped in a reusable and highly configurable bespoke Terraform module. This module not only conforms to Aspect’s own high security requirements, but also to AWS Well-Architected Framework best practices.

AWS Elastic Container Service (ECS) was adopted to host containerised tasks and sub-tasks from the wider workflows. ECS containers are currently hosted on autoscaling AWS Elastic Compute Cloud (EC2) clusters, due to the client’s high floating- point precision requirements having ruled out the immediate use of Fargate (AWS serverless compute for containers).

Further migration to Fargate containers can now be easily achieved when the Fargate service can provide the precision that Aspect requires. Baseline EC2 capacity can be purchased upfront, using Reserved Instances, to realise a significant cost discount.

Logging and monitoring of the entire solution is now provided by use of:

AWS CloudWatch Logs (covering MWAA, ECS, EC2, and app logs)
AWS CloudWatch Alarms
AWS CloudWatch Log Metrics
AWS CloudWatch Metrics
AWS Simple Notification Service (SNS)
OpsGenie integrations
Searchable audit logs – written by the MWAA service

Finally, Ubertas and Aspect team members collaborated on testing of Direct Acyclic Graphs (DAG) within Airflow, to help prepare Aspect Capital for future workload migrations and automated Continuous Integration (CI) pipelines. These pipelines are used for building and deploying DAGs to different MWAA environments and were constructed on top of Aspect’s existing Jenkins CI solution.

The Benefits

60%

cost saving through use of Reserved Instances

50%

reduction in related maintenance overheads

40%

faster speed-to-market for new environments

Ease-of-deployment

The bespoke Terraform module, built to configure and manage MWAA environments, provides a repeatable, predictable orchestration of infrastructure resources. The module is highly configurable, to suit the individual requirements of each account or environment, but also to ensure strong idempotency and parity between environments; this is particularly important where enforcing internal security standards and infrastructure patterns is concerned.

Improved DR and BC

Airflow environments are now easily redeployed, such as would be required in a disaster scenario. Version-controlled deployments contribute also.

Improved scaling and resiliency

EC2 Autoscaling of the ECS clusters is easily configured via the Terraform module. Autoscaling includes self-healing across multiple availability zones (AZs). Scale-out will occur when demand spikes, with automatic scale-in to a targeted minimum node count when demand reduces. The ECS containers, which are created dynamically by the AWS MWAA DAGs, have similar autoscaling & self-healing abilities.

Better Control of Costs

Improved options for rightsizing of EC2 hosts, use of task-derived compute configurations, and the flexibility of autoscaling provided a far reduced amount of underutilised capacity. Further cost savings are achieved through upfront purchasing of baseline compute, through EC2 Reserved Instances (circa 60% savings over on-demand).