Scaling DevOps for Amazon EBS — Series#2
Read the previous article of this series.
Leveraging AWS Service Catalog to build a self-service platform
AWS Service Catalog provides the capability to offer a self-service experience to developers who are constantly looking for agility and faster time to market. At the same time, it helps organizations to bring in control, standardization and apply governance to all of the Cloud resources deployed within their purview. This self-service capability helps developers to move at their own pace where they don’t have to wait for long ticket and approval workflows for provisioning resources they want.
The architecture shown above shows how DevOps teams may use AWS Service Catalog to create a self-service platform that allows customers to consume Amazon EBS. The first step would be to develop CloudFormation templates to define a product. In the context of AWS Service Catalog, a product consist of one or more AWS resources or packaged AWS Marketplace products made available to users for deployment on AWS.
With Service Catalog DevOps teams can organize, govern and distribute curated catalog of such products which the users (who would mostly be developers) can browse through and launch their choice of products. This set up enables users to provision approved Cloud resources without needing direct access to the AWS console. Every product launch action creates an actual CloudFormation stack which represents an instance of the product.
Added to the CloudFormation resources which are needed to deploy the EBS volumes DevOps engineers can also add CloudWatch alarms and metrics which would help administrators to have a notification and monitoring capability for the EBS and its related resources deployed.
Over time as the AWS Cloud platform matures, there would be a need to update the organization standards and policies for which DevOps teams can create different versions of a product. Service Catalog helps users to upgrade their product from one version to another thus ensuring the application stacks and configurations are up to date. While changes are being made to the stacks DevOps engineers can leverage AWS Config to track them and alert configuration drifts if any.
To enrich this self-service platform with visibility capability, AWS CloudTrail can be integrated that logs all API calls made to individual AWS services. The logs can be collected in Amazon S3 buckets for long-term storage needed for audit retrieval. Athena can be brought in to analyze and query these logs for getting an insight into the resources accessed by users at a given point of time.
Developing the CloudFormation template
The above building blocks of the self-service platform can be programmed as code with AWS CloudFormation templates. These template define the required AWS resources, the relationships among them, the input parameters that the user can give in when they launch the product. Below are few use cases for building products in AWS Service Catalog that a DevOps team would need in order to adopt and scale Amazon EBS across an organization.
Add Amazon EBS volume
Below sample CloudFormation template adds additional volume to an existing EC2 instance whose id can be passed in as an input parameter.
AWSTemplateFormatVersion: 2010–09–09
Description: >-
AWS CloudFormation sample template.
Attach a new EBS volume to an existing EC2 instance
Parameters:
inputEC2InstanceID:
Type: String
Resources:
myVolume:
Type: ‘AWS::EC2::Volume’
Properties:
Size: 8 # 8 GiB
AvailabilityZone: ap-southeast-2b
volumeAttach: # attach an Amazon EBS volume to an EC2 instance
Type: ‘AWS::EC2::VolumeAttachment’
Properties:
InstanceId: !Ref inputEC2InstanceID
VolumeId: !Ref myVolume # ref to a new EBS volume
Device: /dev/sdh
DevOps engineer can build a product for users to add additional volume to a particular EC2 instance using the above CloudFormation template.
Encrypt Amazon EBS volume
The template can also have the security and compliance policies defined hence the user need not worry if the provisioned stack of resources are meeting their organization’s required set of security standards.
For instance, according to CIS AWS Foundation Benchmark, EBS volume encryption should be enabled. Currently CloudFormation does not have the capability to encrypt EBS encryption at the time of this writing. One can create a Lambda backed custom resource wherein a Lambda function can be provisioned in the CloudFormation template that would enable EBS encryption by default in a region.
Another option would also be to create a service action in AWS Service Catalog using AWS Systems Manager automation runbook — “AWSConfigRemediation-EnableEbsEncryptionByDefault. Service actions would be available as actions that a user could perform once a product is launched/provisioned. These actions could be operational tasks or running approved commands in the resources.
Manage Amazon EBS Snapshots
CloudFormation templates can be written to create and delete snapshots using Amazon Data Lifecycle Manager. Amazon DLM supports Amazon EBS volumes and snapshots. The template can create lifecycle policies for multiple retention periods with an IAM policy defined that would allow action on ec2:CreateSnapshot, ec2:CreateSnapshots, ec2:DeleteSnapshot, ec2:DescribeVolumes, ec2:DescribeInstances and ec2:DescribeSnapshots to automate operations on the specified resources.
Users consuming Amazon EBS require to provision and manage Amazon EC2 instances as well, teams can expand their AWS Service Catalog product portfolio continuously to embrace the related AWS services. Storage is shrinking as well as expanding, depending on the application, use case, and workload. There is no perfect solution to the AWS Service Catalog products that teams must build to do DevOps for storage, teams should evolve their process, culture and tool stack from time to time.
