AWS CloudFormation - Tips for the Novice (That would be me)

Lot's of talks this week at AWS re:Invent around automation, specifically using the CloudFormation tool, so of course I need to do a deep dive.  First some essential links:

• Home page - http://aws.amazon.com/cloudformation/
• User's Guide - http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html

Having four hours to kill between Las Vegas and Philadelphia I decided to start my CloudFormation journey.  I downloaded the CloudFormation docs to my Kindle and started to figure out how to use this product (not an easy task on Spirit Airlines, which has the most uncomfortable seats in the air!  I swear they must steal these seats from planes that have been junked in the desert).  My butt still hurts.



One of the things I realized is that reading technical documentation on a plane is frustrating and not just because the seats are uncomfortable, the lighting is poor and there is the constant drone of an engine in your head.  A typical way to learn about new things (tech related that is) is to read as little as possible and then see if you can make something work with that minimal knowledge.  At least that's my MO.  Your mileage may vary.  I then tend to iterate on making things work, breaking things, figuring things out and then reading some more.  Lather, rinse, repeat.

So reading technical documentation on a plane, at least for me, raises more questions than are answered.  While the basic premise of CloudFormation makes sense to me - create, then process a template to create an infrastructure out of AWS resources - my ignorance around the mechanics of how this thing actually works created a lot of confusion regarding understanding the syntax and the relationships between the components and the template I was creating.  In other words, I found it nearly impossible to figure out CloudFormation unless I understood precisely the lifecycle of the stack creation process.  I'm sorry to tell you that you won't find the documentation very helpful there.  I had to spend about 6 hours figuring it out by trial and error.  I'm still figuring it out. :-(

Here's a diagram that describes pictorially and in words what I think is going on.

Hopefully as I go through this explanation the diagram will start to make some sense to me and you.

First problem for me was, the ami I wanted to use was not the pre-built Amazon amis (they do have the software included that is necessary to make automation work) and was not CloudFormation enabled so to speak.  I imagine most folks, like I do, tend to start with an ami they are more familiar with.  In my case CentOS 6.4 because it included packages and repositories I needed as a Perl developer.  So first tip is...

1. Make sure you have the necessary tools included in your ami, i.e. it's CloudFormation enabled!

Specifically you're going to want to make sure that the ami has the cloud-init package and the Amazon CloudFormation helper scripts.  I finally ended up using this ami as my base:

Centos 6.3 x86_64 - Minimal with cloud-init aws-cfn-bootstrap ec2-api-tools. - ami-642bba0d

Centos 6.3 x86_64 - Minimal with cloud-init, aws-cfn-bootstrap and ec2-api-tools. Similar setup as Amazon Linux but base is CentOS.

Tip: I'm in the us-east region, this ami may not be available depending on where you are.  Search for centos cloud-init bootstrap.

Once I knew I had the requisite software, I could test a basic CloudFormation template and start modifying it for my purposes.  I can easily update this AMI to CentOS 6.4 with yum if necessary, so this one will do quite nicely thank you!

As it turns out you can build your own ami image if you include the right tools and your so inclined to waste a few hours.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-helper-scripts-reference.html

If you must, try to install the CloudFormation helper scripts on CentOS...

$ sudo yum install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.amzn1.noarch.rpm

And then this to install the cloud-init package.

$ sudo yum install cloud-init

Warning: You'll still need to make sure the cloud-init package is setup correctly for CloudFormation to work properly. The config file that was already set up in the AMI I chose above worked for me since it was configured specifically for CloudFormation.  Installing cloud-init from the RPM (depending on the repo you pull it from) is going to give you a generic config file and that's probably not going to work.  While you could bypass CloudFormation altogether, read a lot of documentation and become a cloud-init expert while you amaze your friends and neighbors, the AWS thing to do is to use CloudFormation.

Update: It's now Sunday night and I've been hacking on CloudFormation all weekend.  I'm now dangerous, so I've decided to go back and clean up some of the mess.  First order of business is to revisit the choice of amis.  I went back and grabbed a CentOS 6.4 image with cloud-init created by Bashton Ltd.  They seem to have created a few CentOS images, however the 6.4 image still did not have the bootstrap tools or the AWS command line tools.  I created an ami that is working out pretty well.  Here's what I did to their base image:

Added epel repo.  I added exclude=cloud-init to yum.conf to prevent yum from pulling in the epel version of cloud-init on a yum update, as per the instructions on the Bashton site.  I'll make this ami public if that is acceptable to the folks at Bashton, but stuff in this blog entry is still relevant.

Installed the AWS command line tools. 

$ wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
$ unzip awscli-bundle.zip
$ sudo ./awscli-bundle/install-i /usr/local/aws -b /usr/local/bin/aws

Installed the CloudFormation helper scripts

$ sudo yum intstall -y https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.amzn1.noarch.rpm

Let's move on...

Stuff is going to go wrong and the formation process is going to fail. Grab a cup (or pot) of coffee.  It's a tedious process until you stumble upon this realization...the software that bootstraps your instance is part of a grander process that is bootstrapping all of the resources from the template file, so your instance is not the only resource being created during the provisioning sequence.  See the picture above.

However, once you are setting your instance (usually one of the more time consuming and tedious parts of the stack) with your custom configuration, you can make it a lot easier on yourself if you walk through the process manually first, noting everything you need to do to prepare your instance.  So tip #2 is...

2. Manually walk through the process of creating your instance and write down each step.

Sounds obvious? Well, that's only half the battle (or more like 1/3).  Now you need to figure out how to get the CloudFormation mojo to do the things you know how to do in your sleep.   So here's tip number 3 taken right out of the Perl book.  Don't be scared, 'cause...

3. TMTOWTDI - There's More Than One Way To Do It

Most likely the place you're going to spend the most time is in the part of the template that prepares your instance from the resources you've provisioned.  Remember this is not a complete tutorial, just some tips and tricks.  You do need to RTFM regarding how to configure your instance using the AWS::CloudFormation::Init object.  That said, the brains of the process is the part of the template that provides the helper script with the information it needs to configure your instance rather than the Resources section which is more or less the heart of the process as it provisions resources.  The provisioning is done by CloudFormation by reading your Resources object and finding the AWS resources that it needs to spark up or create.

Specifically within the template's Resources object you'll probably have one or more objects of type AWS::EC2::Instance and the Metadata object associated with the EC2 instance object will have an AWS::CloudFormation::Init object where the magic is going to take place once your instance is alive and it begins to configure itself.   

Before I go on, I'll explain the relationship between some of these components by displaying a JSON representation of one resource (an EC2 instance) minus most of the guts that describe the resource.

"Resources" : {

  "MyWebServer" : {

    "Type" : "AWS::EC2::Instance",

    "Metadata" : {

      "AWS::CloudFormation::Init" : {

        "config" : {

          ...

        }

      },

    },  

    "Properties" : {

      "ImageId" : { ... },

      "InstanceType" : { ... },

      "SecurityGroups" : [ { ... },

      "KeyName": { ...},

      "UserData" : { ... }

    }

  ]

  ...

}

The object MyWebServer is of type AWS::EC2::Instance and has metadata and properties associated with it.  Hence, the resource has Type, Metadata and Properties objects, some of which may have their own child objects.  Some of this information is used by the CloudFormation process on AWS when your resources are being provisioned, some by the cloud-init package when bootstrapping your instance and some is used by the helper scripts.  The cloud-init package is passed the UserData object which is then written temporarily to /var/lib/cloud/instance/scripts/part-001 and eventually executed as a script.  Here's what a template of a script might (sort of) look like when not written inside of the template file and before values are substituted.

#!/bin/bash -v

# Helper function

function error_exit

{

  /usr/bin/cfn-signal -e 1 -r $1 { Ref : WaitHandle }

  exit 1

}

# opportunity for some pre-processing

# process AWS::CloudFormation::Init object

/usr/bin/cfn-init -s { Ref : AWS::StackId } -r WebServer  

    --region { Ref : AWS::Region }  || error_exit 'Failed to run cfn-init'

# opportunity for some post-processing

# All is well so signal success

/usr/bin/cfn-signal -e 0 -r \Stack setup complete\  { Ref : WaitHandle }

The { Ref : AWS::StackId } type constructs are replaced by CloudFormation when they are passed as UserData to the instance to be launched.  The cloud-init script writes the script to disk and then executes the script during the boot process.

Actually as it turns out, this process is fairly convoluted as well.  The user data is passed as a base 64 object to one of the Amazon EC2 APIs.  The raw data (not the base 64 object) should be <16K.  User data is then made available to the instance at the URL:

http://169.254.169.254/latest/user-data


Give this a try:

$ sudo wget http://169.254.169.254/latest/user-data -O - | less

Via a route that snakes through cloud-init, the data is then turned into a multi-part mime file by some python scripts.  The resulting bash script is then executed by cloud-init.  hehe...snakes through cloud-init...python script...see what I did there? Okay, anyway...

The UserData object, when formatted as a JSON object for the CloudFormation process to pass to the cloud-init scripts for execution, would look like this...

"UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [

    "#!/bin/bash -v\n",

    "# Helper function\n",

    "function error_exit\n",

    "{\n",

    "  /usr/bin/cfn-signal -e 1 -r \"$1\" '", { "Ref" : "WaitHandle" }, "'\n",

    "  exit 1\n",

    "}\n",

    "# opportunity for some pre-processing\n",

    "\n",

    "# process AWS::CloudFormation::Init object\n",

    "/usr/bin/cfn-init -s ", { "Ref" : "AWS::StackId" }, " -r WebServer ",

    "    --region ", { "Ref" : "AWS::Region" }, " || error_exit 'Failed to run cfn-init'\n",

    "# opportunity for some pre-processing\n",

    "\n",

    "# All is well so signal success\n",

    "/usr/bin/cfn-signal -e 0 -r \"Stack setup complete\" '", { "Ref" : "WaitHandle" }, "'\n"

]]}}

Hmmm...isn't there a better way to do this?  I mean, isn't there at least a better way to create the templates that makes this all seem a bit more obvious?

So to recap, we have essentially three things going here as far as I can see:

1. CloudFormation reads your template and begins to provision the resources you've requested via the somewhat verbose and convoluted JSON object you constructed.  It may prompt the user for data if you are running the template from the AWS CloudFormation console.  You can setup a myriad of dereferencing inside the template.  It's maddening.
2. Your EC2 instances starts booting, cloud-init startup scripts runs, and grab the EC2  UserData which is then converted to a script and executed by cloud-init
3. Finally, in the bash script encapsulated as UserData, you provide whatever other setup you need to do for the EC2 instance and possibly execute the cfn-init helper script that interprets the AWS::CloudFormation::Init object, performs additional setup, followed by any post set up actions you may need to take in your script.  The cfn-signal helper is used to signal CloudFormation that your script has completed successfully.

Back to tip #3.  Since there is more than 1 way to do it, clearly we could opt to create a more elaborate script embedded in the UserData rather than do anything at all with cfn-init and it's data driven package provisioning.  In fact, if you aren't using the AWS stock amis, you're probably going to have to take some custom actions anyway.  Just keep in mind your script should be <16k.

Putting a lot of this together here's my UserData script for creating a Bedrock enabled server.   I've highlighted the lines I've added to get this incantation working.

"UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [

    "#!/bin/bash -v\n",

[1] "yum-config-manager --enable epel\n",

    "# Helper function\n",

    "function error_exit\n",

    "{\n",

    "  /usr/bin/cfn-signal -e 1 -r \"$1\" '", { "Ref" : "WaitHandle" }, "'\n",

    "  exit 1\n",

    "}\n",

    "# Install LAMB packages\n",

[2] "/usr/bin/cfn-init -s ", { "Ref" : "AWS::StackId" }, " -r WebServer ", "-c bedrockConfig ",

    "    --region ", { "Ref" : "AWS::Region" }, " || error_exit 'Failed to run cfn-init'\n",

    "# configure Bedrock IDE\n",

[3] "echo 'Include /usr/lib/bedrock-ide/config/perl_bedrock-ide.conf' >> /etc/httpd/conf.d/perl_bedrock.conf\n",

[4] "/sbin/service httpd restart\n",

    "# create session database\n",

[5] "mysqladmin create bedrock\n",

[6] "cat /usr/share/bedrock/create-session.sql | mysql -u root bedrock\n",

    "# All is well so signal success\n",

    "/usr/bin/cfn-signal -e 0 -r \"LAMB Stack setup complete\" '", { "Ref" : "WaitHandle" }, "'\n"

]]}}

[1] Enable the EPEL yum repository since we need some Perl modules found there.

[2] Specify a configSet named bedrockConfig.  This forces cfn-init to process my configuration in two passes, in the order I specified - rpmPrep, config.  Thus,  I can add a new repository first, then specify the packages I need from that repository in the second configuration.  Here's what that mess looks like...

"configSets" : { 

    "bedrockConfig" : ["rpmPrep", "config"]

},

"rpmPrep" : {

    "files" : {

"/etc/yum.repos.d/bedrock.repo" : {

   "source" : "https://s3.amazonaws.com/openbedrock/bedrock.repo",

   "mode"   : "000644",

   "owner"  : "root",

   "group"  : "root"

}

    }

},

"config" : {

    "packages" : {

"yum" : {

   "mysql"        : [],

   "mysql-server" : [],

   "mysql-libs"   : [],

   "bedrock"      : [],

   "bedrock-ide"  : []

}

    }

}

[3] Add the Bedrock IDE configuration information to the Bedrock's Apache configuration file.

[4] Restart the web server to have the Bedrock IDE configuration take effect.

[5] Create a database to hold Bedrock's persistent session storage.

[6] Create the table (session) within the bedrock database to hold session state.

Is all this necessary?  Maybe I'm missing something? Probably, since this seems too hard, however from what I've seen on the internet this seems to be the way to bootstrap your servers using cloud-init and CloudFormation.  I failed several times, and burned lots of minutes until it dawned on me to actually try to run the script that I was trying to construct - once I understood how (and where) cloud-init was invoking the script. Umm...duh...tip #4...

4.  Try running the script dummy!

$ sudo /var/lib/cloud/instance/scripts/part-001

Yeah, go ahead and push the Easy Button!  And while your at it, take a look at the log file (although all it will probably tell you is that your script failed).

Update: If you log in to the instance while it's still booting (yes that is possible) you can run top, view the logs in real time and generally get a sense of what is happening.

$ sudo less /var/log/cloud-init.log

When you startup the CloudFormation template, make sure you check Show Advanced Options and click No for Rollback on Failure. 

Okay, now I can see why that script was failing! Here were some of the reasons.

[1] My ami had the helper scripts installed at /usr/bin instead of /opt/aws/bin which was the path shown in all of the examples that assumed I was using the Amazon ami. 

[2] I was unaware of the fact that the repo I thought I was installing by cleverly moving the files section of the config object before the yum command was not actually happening before the yum command.  Turns out the order in which the config object is processed is fixed:

packages -> groups -> users-> sources -> files -> commands -> services

https://forums.aws.amazon.com/message.jspa?messageID=414670#

So in order for me to get my repo installed in /etc/yum.repos.d, I needed to use configSets and set the order of the configuration objects such that my config with the files section only was processed first (see above).

I'm guessing the order above makes some sense...since

• services need to be started last after everything is installed.
• commands are executed before starting up services
• files override installed sources
• sources are installed after users source files may need to be owned by specific users
• groups are installed before users, since users have to belong to groups
• packages are installed from repos to create a base version of the system

[3] My rpms were not signed properly. I needed to properly sign the Bedrock and IPC::Shareable packages...I'll blog on that adventure too.

[4] The EPEL repo was disabled on the ami.  I added the line to the script to enable the repo.

yum-config-manager --enable epel

[5] I forgot the -y option to the yum command to make sure there was no prompt.

Actually if I stare at the working script long enough, I'm sure there were other things I did wrong...certainly a few typos along the way. 

And here's the final CloudFormation template:

https://s3.amazonaws.com/openbedrock/bedrock.tmpl

Why Not Use the commands Section?

"commands" : {
  "test" : {
    "command" : "echo \"$MAGIC\" >> test.txt",
    "env" : { "MAGIC" : "I am test 2!" },
    "cwd" : "~"
}

Ummm, yeah, this looks enticing.  As if, writing a bash script as a series of JSON strings wasn't fun enough?  I think I'll try

 my darnedest not to use the command section.

What's Next?

CloudFormation allows you to do a lot more than create a single instance of a web server, so what you've seen in the template I've produced is clearly the tip of the iceberg.   Creating an entire architecture and auto scaling groups is next!  Oh boy....more reading to do:

http://awsdocs.s3.amazonaws.com/AWSCloudFormation/latest/cfn-ug.pdf

Lot's of good examples there - they have snippets you can just copy & paste into your templates.  Here's to provision an RDS instance.

"MyDB" : {

 "Type" : "AWS::RDS::DBInstance",

 "Properties" : {

 "DBSecurityGroups" : [

 {"Ref" : "MyDbSecurityByEC2SecurityGroup"}, {"Ref" : "MyDbSecurityByCID 

RIPGroup"} ],

 "AllocatedStorage" : "5",

 "DBInstanceClass" : "db.m1.small",

 "Engine" : "MySQL",

 "MasterUsername" : "MyName",

 "MasterUserPassword" : "MyPassword"

 },

 "DeletionPolicy" : "Snapshot"

}

You get the idea...some work to do with these snippets, but a nice starting place.  Good luck!