In AWS, delivering credentials to an instance has traditionally been problematic. You can pass them in via instance user data at instance creation time, but that's somewhat inconvenient, and instance data cannot be changed after the instance has started. You can pass them in by pre-loading on an AMI, but that's even more hassle and static. Or you can have the instance fetch them from somewhere else at boot time, for example through CloudInit, but that then only moves the problem because you then need to secure that fetch. And of course you can copy credentials to the instance after it has booted, but then you have to wait until the ssh sever has regenerated its keys, and you must have security groups and VPC routes to allow access, and it precludes you running an instance manually from the AWS Console.

I've ignored that problem for a while, simply relying on Chef to distribute credentials. But for a current project I'm trying to do without custom AMIs and a Chef server: I want to run an instance, and have it do some setup and secure configuration at creation time. For example, it could setup an OpenVPN server in a VPC, or could create a shared secret for Chef Encrypted Data Bags prior to registering as a Chef client.

In June 2012, AWS announced IAM roles for EC2 instances – Simplified Secure Access to AWS service APIs from EC2 which addresses this very use case: it allows you to pass IAM credentials to instances.

So my plan was:

  1. make credentials available to the instance, using IAM roles
  2. create a script to download further scripts and credentials from a private S3 bucket
  3. fetch and execute this script at boot time, using CloudInit

Creating the IAM Role

For the first step, the user guide has a section Granting Applications that Run on Amazon EC2 Instances Access to AWS Resources which explains the approach, and illustrates it with a video that uses the AWS Console. I prefer doing IAM configuration from the command-line, so that I can document and script it more easily, and track changes in a git repo. What is not obvious is that when you use the command line, you need to explicitly create a instance profile as a separate step.

iam-rolecreate -r myrole -p / -s -v
iam-roleuploadpolicy -r myrole -p mypolicy -f ./mypolicy.txt
iam-instanceprofilecreate -p / -r mypolicy -s myrole

See the AWS Identity and Access Management CLI Reference for details.

I first ran an instance from the AWS Console by hand, and then determined how to do this from my ruby scripts that use Fog:

    server = @compute.servers.create(
            :iam_instance_profile_name => 'myrole'

With the instance running, I verified that the metadata service made the credentials available:

wget -O - -q ''
wget -O - -q ''

and verified that they worked by downloading the latest EC2 API Tools and Java, and running ec2-describe-instances, which was one of the permissions granted in the policy file.

Temporary credentials in s3cmd

I tried using the AccessKeyId and SecretAccessKey from the metadata in by s3cmd .s3cfg file, with a simple s3cmd ls s3://mybucket/ command, but got:

ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you provided does not exist in our records.

The reason is that for IAM temporary credentials to work, you need to provide the token from the metadata, in a separate HTTP header when talking to the AWS S3 API (see this FAQ). Some googling found someone using security token for s3cmd put (with this change) but just using --add-header=x-amz-security-token didn't work; it looks like s3cmd only uses that for put/sync/cp/mv.

There is a popular open feature request to Support IAM roles / instance profiles; it'd be great if you could just tell s3cmd to disengage its config file and "use the metadata service". But, I found this post and patch which at least adds minimal support. For convenience I pulled it into my fork.

So then it's just a matter of copying the credentials from the metadata to the s3 config file. Bear in mind that they expire; you'll need to re-generate them, and the simplest is to do that for every use. Here is an example with bash and jq:

wget -O mycreds -q ''
SECRET_KEY=`jq -r '.SecretAccessKey' <mycreds`
ACCESS_KEY=`jq -r '.AccessKeyId' <mycreds`
TOKEN=`jq -r '.Token' <mycreds`
cat >s3cfg <<EOM
access_key = $ACCESS_KEY
secret_key = $SECRET_KEY
security_token = $TOKEN
s3cmd --config s3cfg ls s3://mybucket/

I'd like to see this (or better) support in s3cmd, but the last release was nearly a year ago, and it sounds like the project has a lack of leadership problem (link expired). I think it'd be great it Amazon could take this project over, or if they could provide similar functionality on top of its AWS SDK for Java.

As for my project, all the pieces seem to be in place; doing the whole end-to-end CloudInit bootstrap is for another day.