aws

Provisioning an RDS Database with CloudFormation (part 2)

Provisioning an RDS Database with CloudFormation (part 2)

In part 1, we automated the provisioning of your Amazon EC2 instance using AWS CloudFormation. When you built the EC2 instance manually in the past, you were seeing inconsistencies between environments, had to manually test your infrastructure setup, manually deploy your team's web app, which happens all to infrequently. All of this has been error prone and time consuming. Bugs introduced into production code are more difficult and expensive to fix and your customers were ultimately the ones who suffered.

In addition to the EC2 instance, you also need a Postgresql database. In the past, it was running on the same instance as your webapp and sometimes web app traffic impacted the database and vice versa. In this post, part 2, you'll add an Amazon RDS Postgresql database to the CloudFormation template you built in part 1 so that both the EC2 instance and the database can be provisioned together as a set of resources. And since RDS is a managed service, you won’t have to do operating system patches or database patches — it’s all managed for you. And eventually, you want to introduce a continuous integration/continuous deployment (CI/CD) pipeline to automate the build, test, and deploy phases of your release process. Both part 1 and part 2 set you up to do that.

As a reminder, here's what we covered and where we're going:

  • automate the provisioning of your Amazon EC2 instance using AWS CloudFormation (part 1),
  • add an Amazon RDS Postgresql database to your stack with CloudFormation (this post, part 2), and
  • create an AWS CodePipeline with CloudFormation (part 3).

Prerequisites

To work through the examples in this post, you’ll need:

  • an AWS account (you can create your account here if you don’t already have one),
  • the AWS CLI installed (you can find instructions for installing the AWS CLI here), and
  • a key-pair to use for SSH (you can create a key-pair following these instructions).

Unfamiliar with CloudFormation or feeling a little rusty? Check out part 1 or my Intro to CloudFormation post before getting started.

Updating the CloudFormation Template

First we'll add the RDS database and then we'll add a security group to allow inbound traffic on port 5432. At the end of this post, you’ll delete the stack you’ve created and any snapshots so that you don’t incur any charges and then you can (quickly) customize and recreate the stack in the future.

Just want the code? Grab it here.

Let’s get started!

1. Add RDS Postgresql Database

First, we'll add an RDS database resource with the type AWS::RDS::DBInstance to the CloudFormation template. We set the Engine to the database engine we want to use, in this case postgres.

# 05_rds.yaml
AWSTemplateFormatVersion: 2010-09-09
Description: Part 2 - Add a database with CloudFormation

Parameters:
  AvailabilityZone:
    Type: AWS::EC2::AvailabilityZone::Name
  EnvironmentType:
    Description: "Specify the Environment type of the stack."
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - test
      - prod
  AmiID:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Description: "The ID of the AMI."
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
  KeyPairName:
    Type: String
    Description: The name of an existing Amazon EC2 key pair in this region to use to SSH into the Amazon EC2 instances.

  DBInstanceIdentifier:
    Type: String
    Default: "webapp-db"
  DBUsername:
    NoEcho: "true"
    Description: Username for Postgresql database access
    Type: String
    MinLength: "1"
    MaxLength: "16"
    AllowedPattern: "[a-zA-Z][a-zA-Z0-9]*"
    ConstraintDescription: Must begin with a letter and contain only alphanumeric characters.
    Default: "postgres"
  DBPassword:
    NoEcho: "true"
    Description: Password Postgresql database access
    Type: String
    MinLength: "8"
    MaxLength: "41"
    AllowedPattern: "[a-zA-Z0-9]*"
    ConstraintDescription: Must contain only alphanumeric characters.

Mappings:
  EnvironmentToInstanceType:
    dev:
      InstanceType: t2.nano
    test:
      InstanceType: t2.micro
    prod:
      InstanceType: t2.small

Resources:
  WebAppInstance:
    Type: AWS::EC2::Instance
    Properties:
      AvailabilityZone: !Ref AvailabilityZone
      ImageId: !Ref AmiID
      InstanceType:
        !FindInMap [
          EnvironmentToInstanceType,
          !Ref EnvironmentType,
          InstanceType,
        ]
      KeyName: !Ref KeyPairName
      SecurityGroupIds:
        - !Ref WebAppSecurityGroup

  WebAppSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Join ["-", [webapp-security-group, !Ref EnvironmentType]]
      GroupDescription: "Allow HTTP/HTTPS and SSH inbound and outbound traffic"
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          CidrIp: 0.0.0.0/0

  WebAppEIP:
    Type: AWS::EC2::EIP
    Properties:
      Domain: vpc
      InstanceId: !Ref WebAppInstance
      Tags:
        - Key: Name
          Value: !Join ["-", [webapp-eip, !Ref EnvironmentType]]

  WebAppDatabase:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceIdentifier: !Ref DBInstanceIdentifier
      AllocatedStorage: "5"
      DBInstanceClass: db.t3.micro
      Engine: postgres
      MasterUsername: !Ref DBUsername
      MasterUserPassword: !Ref DBPassword
      Tags:
        - Key: Name
          Value: !Join ["-", [webapp-rds, !Ref EnvironmentType]]
    DeletionPolicy: Snapshot
    UpdateReplacePolicy: Snapshot

Outputs:
  WebsiteURL:
    Value: !Sub http://${WebAppEIP}
    Description: WebApp URL

  WebServerPublicDNS:
    Description: "Public DNS of EC2 instance"
    Value: !GetAtt WebAppInstance.PublicDnsName

  WebAppDatabaseEndpoint:
    Description: "Connection endpoint for the database"
    Value: !GetAtt WebAppDatabase.Endpoint.Address

In the template above, we've also added new parameters to customize the name of the database, the username, and password, as well as two new outputs to make our work easier: WebServerPublicDNS and WebAppDatabasePublicDNS. We'll use both of these outputs in step 2.

If you're creating your stack from scratch, you can run the create-stack command:

$ aws cloudformation create-stack --stack-name rds-example --template-body file://05_rds.yaml \
--parameters ParameterKey=AvailabilityZone,ParameterValue=us-east-1a \
ParameterKey=EnvironmentType,ParameterValue=dev \
ParameterKey=KeyPairName,ParameterValue=jenna \
ParameterKey=DBPassword,ParameterValue=Abcd1234

Or, if you're updating the stack you created in part 1, you can use the update-stack command instead.

Notice that we're only specifying the new DBPassword parameter and relying on the default values for DBInstanceIdentifier and DBUsername that are specified in the template. If you'd like to customize these values, you can add them to the command in the same ParameterKey,ParameterValue format shown above.

DeletionPolicy and UpdateReplacePolicy

You may have also noticed the DeletionPolicy and UpdateReplacePolicy properties set to Snapshot. If and when you ever need to re-provision the database, you'll want to make sure you don't lose your precious data. Some template changes will require the resource to be recreated (as opposed to updated). When this happens, you'll want to be in control of how that happens and what happens to your data. These two properties give you that control. In the case of Snapshot, CloudFormation will create a snapshot of the database when the stack is updated or deleted. You can read more about update and delete behaviors of stack resources here.

After creating or updating the stack, you'll now have your EC2 instance (and supporting resources from part 1) and an RDS database. Right now, no one can access that database instance from the outside world, so next we'll enable inbound traffic to the Postgresql port.

2. Enable Inbound Traffic on Port 5432

To allow inbound traffic on port 5432 so that our EC2 instance can talk to the RDS database, we'll add a security group with type AWS::EC2::SecurityGroup.

# 06_rds.yaml
AWSTemplateFormatVersion: 2010-09-09
Description: Part 2 - Add a database with CloudFormation

Parameters:
  AvailabilityZone:
    Type: AWS::EC2::AvailabilityZone::Name
  EnvironmentType:
    Description: "Specify the Environment type of the stack."
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - test
      - prod
  AmiID:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Description: "The ID of the AMI."
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
  KeyPairName:
    Type: String
    Description: The name of an existing Amazon EC2 key pair in this region to use to SSH into the Amazon EC2 instances.

  DBInstanceIdentifier:
    Type: String
    Default: "webapp-db"
  DBUsername:
    NoEcho: "true"
    Description: Username for Postgresql database access
    Type: String
    MinLength: "1"
    MaxLength: "16"
    AllowedPattern: "[a-zA-Z][a-zA-Z0-9]*"
    ConstraintDescription: Must begin with a letter and contain only alphanumeric characters.
    Default: "postgres"
  DBPassword:
    NoEcho: "true"
    Description: Password Postgresql database access
    Type: String
    MinLength: "8"
    MaxLength: "41"
    AllowedPattern: "[a-zA-Z0-9]*"
    ConstraintDescription: Must contain only alphanumeric characters.

Mappings:
  EnvironmentToInstanceType:
    dev:
      InstanceType: t2.nano
    test:
      InstanceType: t2.micro
    prod:
      InstanceType: t2.small

Resources:
  WebAppInstance:
    Type: AWS::EC2::Instance
    Properties:
      AvailabilityZone: !Ref AvailabilityZone
      ImageId: !Ref AmiID
      InstanceType:
        !FindInMap [
          EnvironmentToInstanceType,
          !Ref EnvironmentType,
          InstanceType,
        ]
      KeyName: !Ref KeyPairName
      SecurityGroupIds:
        - !Ref WebAppSecurityGroup

  WebAppSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Join ["-", [webapp-security-group, !Ref EnvironmentType]]
      GroupDescription: "Allow HTTP/HTTPS and SSH inbound and outbound traffic"
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          CidrIp: 0.0.0.0/0

  WebAppEIP:
    Type: AWS::EC2::EIP
    Properties:
      Domain: vpc
      InstanceId: !Ref WebAppInstance
      Tags:
        - Key: Name
          Value: !Join ["-", [webapp-eip, !Ref EnvironmentType]]

  WebAppDatabase:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceIdentifier: !Ref DBInstanceIdentifier
      VPCSecurityGroups:
        - !GetAtt DBEC2SecurityGroup.GroupId
      AllocatedStorage: "5"
      DBInstanceClass: db.t3.micro
      Engine: postgres
      MasterUsername: !Ref DBUsername
      MasterUserPassword: !Ref DBPassword
      Tags:
        - Key: Name
          Value: !Join ["-", [webapp-rds, !Ref EnvironmentType]]
    DeletionPolicy: Snapshot
    UpdateReplacePolicy: Snapshot

  DBEC2SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Join ["-", [webapp-db-security-group, !Ref EnvironmentType]]
      GroupDescription: Allow postgres inbound traffic
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 5432
          ToPort: 5432
          SourceSecurityGroupName:
            Ref: WebAppSecurityGroup
      Tags:
        - Key: Name
          Value: !Join ["-", [webapp-db-security-group, !Ref EnvironmentType]]

Outputs:
  WebsiteURL:
    Value: !Sub http://${WebAppEIP}
    Description: WebApp URL

  WebServerPublicDNS:
    Description: "Public DNS of EC2 instance"
    Value: !GetAtt WebAppInstance.PublicDnsName

  WebAppDatabaseEndpoint:
    Description: "Connection endpoint for the database"
    Value: !GetAtt WebAppDatabase.Endpoint.Address

In the template above, we've added the database security group and added it to the VPCSecurityGroups property. We've also set SourceSecurityGroupName to the WebAppSecurityGroup we created earlier, to restrict the inbound traffic to that coming from the WebAppSecurityGroup, or our EC2 instance.

To update your stack, you'll use the update-stack command with the same parameters and values as before:

$ aws cloudformation update-stack --stack-name rds-example --template-body file://06_rds.yaml \
--parameters ParameterKey=AvailabilityZone,ParameterValue=us-east-1a \
ParameterKey=EnvironmentType,ParameterValue=dev \
ParameterKey=KeyPairName,ParameterValue=jenna \
ParameterKey=DBPassword,ParameterValue=Abcd1234

Now, you should have your EC2 instance (and supporting resources from part 1), an RDS database, and another security group exposing the database port. But how do we know the EC2 instance can talk to the database?

Testing the Security Group

To test your security group to make sure your EC2 instance can talk to the RDS database you provisioned in the last step, you can SSH into the instance and use the psql client to connect to the database.

If you don't remember how to SSH into the instance, you can read more about that in part 1 or grab the WebServerPublicDNS from the Outputs. Using the command below, replace the WebServerPublicDNS and YOUR_KEY_PAIR_NAME parts and SSH into the intstance:

$ ssh -i "YOUR_KEY_PAIR_NAME.pem" ec2-user@WebServerPublicDNS

Once you're in the EC2 instance, you'll need to enable the postgresql library from the Amazon Linux Extras repository and then install it. We could do this all in our CloudFormation template, but this is only for demonstration purposes, so we'll do it manually this one time. You can run these commands to do that:

sudo amazon-linux-extras list # Find the postgresql library and version to enable in the next command
sudo amazon-linux-extras enable postgresql13 -y
sudo yum clean metadata
sudo yum install postgresql

Now that psql is available, you can test the connection to the database like this, replacing WebAppDatabaseEndpoint with the corresponding value in the Outputs:

$ psql -h WebAppDatabaseEndpoint -U postgres -d postgres
Password for user postgres:
psql (13.2, server 12.5)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=>

Congrats! You've successfully allowed your EC2 instance to talk to your RDS database.

If the connection isn't successful, check the CloudFormation console to make sure the RDS database and security group resources were created successfully. Additionally, you can check to make sure the database has the security group attached and the inbound rule opens up port 5432 (the default postgres port).

Wrapping Up

Delete Your Stack & Snapshots

Don’t forget to delete your stack so you don’t accrue charges. You can do that with the delete-stack command:

$ aws cloudformation delete-stack --stack-name rds-example

If you left the DeletionPolicy and UpdateReplacePolicy properties set to snapshot and you no longer need those snapshots, then you can also delete those snapshots using the AWS Console so you don't accrue charges for those either.

Navigate to the RDS Management Console. From there, go to the Snapshots menu option. Select the snapshots created from your stack (hint: they will have a snapshot name that starts with your stack name) and select Delete snapshot from the Actions menu.

RDS Management Console showing selected snapshots to be deleted

What You Learned

In this post, we updated the CloudFormation template from part 1 to provision an RDS database and enabled inbound traffic for the database. Now that both your EC2 instance and RDS database (and supporting resources) are all managed with code, you can setup and teardown the stack of resources together. This sets you up for part 3, where we'll create an AWS CodePipeline with CloudFormation (part 3) so we can build, test, and deploy our web app to the through each environment to production.

You can grab the final CloudFormation template we created here.

Like what you read? Follow me over on the Dev.to community or give me a follow on Twitter to stay updated!

Get the goods. In your inbox. On the regular.