Resources in an infrastructure based on AWS can be either created manually or programmatically with AWS CloudFormation. Our preference is to describe resources and infrastructure as code with CloudFormation templates: check out David’s blog post to find out why. I like CloudFormation because I think having a consistent, repeatable, code-described infrastructure across multiple environments saves a lot of time and leaves less room for errors. Another thing I like about CloudFormation is that it has a resource initialization mechanism called Init that allows to perform actions like automated package installation, command execution or passing configuration variables that can be read e.g. from Puppet.
CloudFormation templates are written in JSON format; editing templates is not difficult, but it is a process that could require some time. I thought it might have been convenient to interactively generate skeleton templates with a script, which I wrote in Python.
An AWS Virtual Private Cloud (VPC) is composed of different elements: gateways, route tables, routes, subnets and so forth. Some resources also need to be associated to other ones, like routes and subnets to route tables for instance. I think when it comes to have something ready in a little amount of time, or if the architecture for a project has a lot of resources to be described, some automation help might be good. Thus, I think the script I wrote is valuable in this sense: help facilitate performing common, basic tasks.
The template generator script is available at this URL.
This script generates a template that describes a basic configuration for a VPC, and the user that runs the script just has to answer a few input questions, like the AWS Region where to create the CloudFormation stack, how many AWS Availability Zones to use in the selected region, which Amazon NAT AMI to use, how many public/private subnets and CIDR blocks for the VPC and for the subnets. The script also offers to create a Bastion Host to access the VPC. User input validation for CIDR blocks has been kept to a minimum: the script just validates input CIDRs against a regular expression, to make sure IP addresses belong to a local space, and the subnet mask value is between the /16 and /28 range. The script is not checking for overlapping CIDR blocks across subnets, or if subnet IP addresses are extending outside of the specified VPC subnet.
Here is an overview of what the script does:
- Boto library functions are used to display a list of AWS Regions the user can choose from; the GovCloud AWS Region is excluded by default in the script with a configuration variable.
- The script displays Availability Zones (retrieved dynamically with Boto function calls) in the selected region, so the user can input a valid number of AZs to use.
- The scripts creates a VPC resource with a user-specified CIDR block; an “InternetGateway” resource is also created and attached to the VPC.
- Private EC2 instances (not described by the script) that will need to get to the Internet will use NAT instances as gateways: therefore, the scripts also generates one NAT instance resource on each Availability Zone.
- Boto library functions are also used to retrieve Amazon NAT AMI information, that will be shown in a compiled list so the user can choose the NAT AMI for the template. After the user has completed the AMI NAT selection, the script retrieves AMI Ids of the chosen NAT AMI from the other AWS Regions and maps those values in the template.
- A NAT Security Group for the NAT instance(s) will be created with Ingress and Egress access rules referencing the specified VPC CIDR block and, if present, the Bastion Host (see below).
- If the user elects to add a Bastion Host, a Bastion Host Security Group will also be created. This security group will contain a source-lock rule referencing a CIDR input parameter on the template the user will fill in when instantiating the stack, and a TCP port referencing another input parameter on the template. The port value provided by the user will be set as the default value for the input parameter on the template.
- The total number of Availability Zones is needed to compute routes and route table distributions across subnets. For instance, if the user specifies 2 AZs, 2 public subnets and 2 private subnets, then the distribution logic will:
- Create two public route tables
- Create two 0.0.0.0/0 public route resources with “InternetGateway” as a target
- Place the first public route on the first public route table
- Place the second public route on the second public route table
- Associate the first public subnet to the first public route table
- Associate the second public subnet to the second public route table
- Create two NAT instances with one Elastic IP associated to each one of them
- Place the first NAT instance in the first public subnet
- Place the second NAT instance in the second public subnet
- The routes/route tables distribution across subnets is the same for the two private subnets, with the exception that route targets for the 0.0.0.0/0 default destination will be references to the first NAT instance for the first private route’s target, and to the second NAT instance for the second private route’s target.
Besides stack resources, the script also handles Mappings, Parameters and Outputs:
- Template Mappings will be created for NAT AMIs based on user selection
- The following Parameters will be added to the template:
- CIDR block to source-lock access to the Bastion Host, if present
- KeyPair name to launch instances with
- NAT instances type, e.g. m1.small
- If there is a Bastion Host defined, the Bastion Host instance type
- Availability Zone parameters: if the user specifies two AZs, there will be two AvailabilityZone parameters, whose default values will be the first two AZs in the user-specified AWS Region.
- The following Outputs will be added to the template:
- Elastic IP of the Bastion Host, if present
- Elastic IP(s) of the NAT instance(s).
When all the items above are defined, the script assembles the template. The script uses the “json.dumps” function in the json module to pretty-print template contents and to sort all the JSON keys alphabetically. The script then connects to the AWS CloudFormation service on the AWS Region the user specified at the beginning, and validates the newly generated template. If validation is successful, the template is then saved to disk with a filename specified by the user.
If you take a look at the script, you will see I have chosen an interactive method to collect user input: I thought it would have been more intuitive. The script just allows to quickly build a basic VPC skeleton template, but it can be expanded and generalized to describe other AWS resources. I would love to hear your comments!