Knife EC2: Manage Amazon EC2 Instances With Chef

August 3, 2015October 15, 2018 ~ Matthias Marschall

Chef enables you to automate your infrastructure. It provides a command line tool called knife to help you manage your configurations. Using the knife EC2 plugin you can manage your Amazon EC2 instances with Chef. knife EC2 makes it possible to create and bootstrap Amazon EC2 instances in just one line – if you go through a few setup steps. In this article, I want to show you how to setup your Chef installation and AWS configuration so that you can easily bootstrap new Amazon EC2 instances with Chef’s knife.

Prepare SSH Access To Your Amazon EC2 Instances

Configure Your Amazon Security Group
As Amazon blocks all incoming traffic to your EC2 instances by default. you’ll need to open the SSH port for knife to access a newly created instance. This is pretty easy. Just login to the AWS management console and navigate to EC2 under Services > Compute. Go to Security Groups and select the default group.

Open the Actions drop-down and choose Edit inbound rules:
AWS Security Group: Choose menu - Edit inbound rules

Then add a rule for Type SSH with Source Anywhere and save the new inbound rule:
AWS Security Group: Edit inbound rules screen

This enables SSH connections from anywhere (0.0.0.0/0) in the world. If you want to limit access to just your home or work network, choose Custom IP instead of Anywhere and enter the corresponding net mask.

When you’re done, your default security group should look like this:
AWS security setting SSH

Generate Key Pair in AWS Console
To enable SSH access to your Amazon EC2 instances you need to create a key pair. Amazon will install the public key of that key pair on every EC2 instance. knife will use the private key of that key pair to connect to your Amazon EC2 instances.

Just select Key Pairs under NETWORK & SECURITY in your AWS management console and press Create Key Pair. Give it a name (e.g. knife so you know that this key pair will be used by knife) and press Create. This will create the key pair and download the private key to your local workstation.

AWS management - create key pair

Store the downloaded private key knife.pem in ~/.ssh/knife.pem

Prepare your SSH configuration to avoid host key mismatch errors
Open your ~/.ssh/config and add:

Host ec2*compute-1.amazonaws.com
  StrictHostKeyChecking no
  User ubuntu
  IdentityFile  /Users/mm/.ssh/knife.pem

(make sure you fix the path to your home dir)

Now, SSH access to your Amazon EC2 instances will work. Time to move on to the next step.

Configure knife Enabling it to Manage EC2 Instances

Install the knife EC2 plugin
To enable knife to manage Amazon EC2 instances you need to install the knife EC2 plugin.

Either add it to your Gemfile or install it using gem:

$ gem install knife-ec2

The knife EC2 plugin adds the ec2 sub-command to knife, which we’ll use to manage our Amazon EC2 instances.

Tell knife about your AWS credentials
Create a new user for knife in your AWS management console under Services > Administration & Security. You’ll use this user’s AWS credentials (access key ID and secret access key) to manage your Amazon EC2 instances with knife EC2.

Add the AWS credentials of your knife user to your knife configuration file ~/.chef/knife.rb:

knife[:aws_access_key_id] = "..."
knife[:aws_secret_access_key] = "........."

Now, knife should be able to use the Amazon AWS API to manage Amazon EC2 instances.

Choose an AMI for your Amazon EC2 instances
We’re going to instantiate an Amazon EC2 instance running Ubuntu 14.04 LTS.

Look here for Ubuntu 14.04 LTS AMI IDs which you can use to instantiate Amazon EC2 instances with knife.

You need to choose the right AMI for your region, architecture and root storage. Note down the AMI ID (ami-XXXXXXXX) to use it with knife.

Create an EC2 instance using Chef knife
Now, it’s time to use knife to fire up and configure a new Amazon EC2 instance.

$ knife ec2 server create -r "role[ubuntu]" -I  ami-1ed88f69 -f t2.small 
  -S knife -i ~/.ssh/knife.pem --ssh-user ubuntu --region eu-west-1 -Z eu-west-1a

"role[ubuntu]" is the run_list I want to associate with the newly created node. You can put any roles and recipes you like here
-I is the AMI ID you selected earlier
-f is the Amazon EC2 instance type (see Model)
-S is the name you gave to the SSH key pair generated in the AWS management console
-i points to the private key file of that SSH key pair as downloaded when the key pair was created in the AWS management console
--ssh-user the official Ubuntu EC2 AMIs use ubuntu as the default user
--region eu-west-1 If you want your instances to be deployed in any specific Amazon AWS region, add this parameter and the desired region
-Z eu-west-1a is the availability zone within your region

ATTENTION: make sure to kill the instance again if not needed anymore 😉

Managing Amazon EC2 Instances With knife

Once you’ve started up at least one Amazon EC2 instance with knife, you can use knife to find running EC2 instances like this:

$ knife ec2 server list --region eu-west-1

(make sure you use the correct --region parameter)

And, if you want to get rid of an instance (terminate instance and delete the corresponding Chef node), it’s as easy as:

$ knife ec2 server delete i-XXXXXXXX --region eu-west-1
$ knife node delete i-XXXXXXXX

(i-XXXXXXXX is the ID of the instance as found in the AWS management console or a knife ec2 server list call)

After getting the initial setup right it’s a breeze to start, list, and stop your EC2 instances at Amazon with Chef’s knife ec2. With just a single command you can instantiate a new server, bootstrap it as a Chef client and run all Chef recipes defined in the run_list. Pretty sweet. What are your experiences with knife and Amazon EC2 (or other cloud providers)? Let us know in the comments…

Published by Matthias Marschall

The opinions stated here are my own, not those of my company. View all posts by Matthias Marschall

27 thoughts on “Knife EC2: Manage Amazon EC2 Instances With Chef”

Sean Gallagher says:

July 15, 2011 at 1:44 am

Thanks for the post!
Check your command for:
“Create the EC2 instance using opscode chef knife”
>> should be -I (capital) for ami and -i (lowercase) for pem file

LikeLike

Reply
1. Matthias Marschall says:
  
  July 15, 2011 at 1:49 pm
  
  Hi Sean, thanks for pointing this out. Actually, I wrote this post for Chef 0.9.x. The parameters changed with Chef 0.10. I updated the post for Chef 0.10.
  
  LikeLike
  
  Reply
Rajat says:

July 26, 2012 at 8:02 pm

Hi Matthias

Thanks for the post. I have a ques. I want to use my own custom ami to launch the ec2 machine. How to do that?

Thanks in advance
Rajat

LikeLike

Reply
1. Matthias Marschall says:
  
  July 27, 2012 at 8:53 am
  
  You need to create your own AMI and upload it to AWS. Then you just use the ID of your own AMI with the knife call.
  
  LikeLike
  
  Reply
anoldguy says:

August 3, 2012 at 1:38 pm

I think you’re missing a ‘-r’ before “role[ubuntu]” in your knife command. It should look like this:

knife ec2 server create -r “role[ubuntu]” -I ami-399ca94d -f m1.small -S knife -i ~/.ssh/knife.pem –ssh-user ubuntu –region eu-west-1 -Z eu-west-1a

LikeLike

Reply
1. Matthias Marschall says:
  
  August 3, 2012 at 2:38 pm
  
  Good catch. Fixed.
  
  Thanks a lot!
  
  LikeLike
  
  Reply
EJC says:

August 9, 2012 at 9:31 pm

No matter what I do, I keep getting the message:

ERROR: You have not provided a valid image (AMI) value. Please note the short option for this value recently changed from ‘-i’ to ‘-I’.

I even tried the exact example you guys gave as well as with some of the ami’s I’ve created with no luck. Any suggestions? Adding -VV doesn’t help either…

LikeLike

Reply
1. Matthias Marschall says:
  
  August 10, 2012 at 9:26 am
  
  I had issues with this as well (see: knife ec2 server create not possible for eu-west-1 AMIs.
  
  For me, when copying the stuff the double-dashes collapsed to become single dashes. Make sure you use double-dashes for all long options.
  
  Hope this helps…
  
  LikeLike
  
  Reply
2. Cassiano Leal says:
  
  August 21, 2012 at 10:40 pm
  
  Also, you might try to specify on the command line parameters such as region and availability_zone, maybe others.
  Currently there’s a bug in knife-ec2 that overrides some parameters set in knife.rb.
  
  LikeLike
  
  Reply
EJC says:

August 13, 2012 at 4:59 pm

hmmmm- interesting – but I’m not using any double dashes anywhere. The help output suggests -I = the AMI for the server and the -i = the identitiy file – so I’m NOT mixing those up either.

Any other suggestions?

LikeLike

Reply
1. EJC says:
  
  August 13, 2012 at 6:24 pm
  
  Ok, started over, created a new ami and left it running. From there I took note of zone and region. Then I could run the command with the region and zone options and now it’s sitting waiting for sshd…
  
  I get the feeling it’s going to crash out…
  
  LikeLike
  
  Reply
EJC says:

August 13, 2012 at 7:01 pm

Yeah, not sure what else to do, but the ssh portion never seems to connect/get set up properly.

Any final suggestions?

LikeLike

Reply
EJC says:

August 13, 2012 at 7:17 pm

ok, got it – enabled the ssh rule for the default security group (doh!).

Next issue is we’re hosting our own chef server – this writeup is ONLY good for people NOT hosting their own chef server?

LikeLike

Reply
Ran says:

September 13, 2012 at 9:42 am

Hi, thanks for the post, this was very useful!

But I’m having trouble, and google didn’t help…
I get the following error:
ERROR: Fog::Compute::AWS::Error: SignatureDoesNotMatch => The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

I get this error for all knife ec2 commands (server create, server list etc)
But when using the same key ID and secret access key with the EC2 command line tools provided by amazon, all is well! So I’m a bit confused…
I tried two different amz accounts and the same problem.
I tried using -A and -K explicitly, didn’t work. I tried putting them in ~/.chef/knife.rb file and still no dice.

I haven’t found anyone else with the same error on google. Any idea how to proceed?

$ knife -v
Chef: 10.14.2

thank you!

LikeLike

Reply
1. Matthias Marschall says:
  
  September 13, 2012 at 2:48 pm
  
  Hm, I only ran into this issue when trying to use Fog with hosteurope.de cloud storage. (see: https://github.com/fog/fog/issues/583). But I cannot imagine what might be wrong at your end, I’m sorry 😦
  
  I assume you’re using the latest version of Fog on your box…
  
  LikeLike
  
  Reply
2. dflower says:
  
  October 31, 2012 at 6:32 pm
  
  Did you ever find the solution to this? I have the same issue, but it only happens when I run knife/chef from behind my companies firewall. At home the exact same command/scripts work fine.
  
  LikeLike
  
  Reply
  1. Matthias Marschall says:
    
    November 2, 2012 at 3:35 pm
    
    It worked for me by adding the “/” to the endpoint URL, but it should be fixed in latest versions of fog.
    
    LikeLike
3. Dimitri says:
  
  December 1, 2012 at 5:47 pm
  
  I also have the same issue. Would love a resolution. Thanks!
  
  LikeLike
  
  Reply
Michael Minella says:

October 29, 2012 at 5:16 pm

Just the tutorial I needed. Thanks for the great writeup! One small issue I run into (it doesn’t effect the creation/deletion of instances but was curious if you had a solution) is that I get errors as chef tries to create a node in the chef server. I’m using chef solo so I was wondering if there was a setting I’m missing to prevent knife from trying to create/remove the nodes on the server. Any insight is appreciated. Thanks again for the good writeup!

LikeLike

Reply
jumping says:

January 29, 2013 at 3:49 am

Now you could run 64bit type system on small type instance on AWS EC2.

LikeLike

Reply
Chris Olido says:

March 1, 2013 at 11:51 am

Ih having problems with the validation.pem. Technically my validation.pem is correct and identical to the chef server and to the client when i compare it.

When im issuing:

knife ec2 server create -r “role[wordpress]” -E staging -S www-sg -I ami-b2a2**** –flavor t1.micro –region ap-southeast-1 -G public_www -x ubuntu -N testme

im having an error like this:

Chef::Exceptions::InvalidPrivateKey: The file /etc/chef/validation.pem does not contain a correctly formatted private key.
The key file should begin with ‘—–BEGIN RSA PRIVATE KEY—–‘ and end with ‘—–END RSA PRIVATE KEY—–‘
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/rest/auth_credentials.rb:68:in `rescue in load_signing_key’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/rest/auth_credentials.rb:60:in `load_signing_key’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/rest/auth_credentials.rb:34:in `initialize’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/rest.rb:64:in `new’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/rest.rb:64:in `initialize’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/client.rb:325:in `new’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/client.rb:325:in `register’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/client.rb:407:in `do_run’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/client.rb:176:in `run’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/application.rb:140:in `run_chef_client’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/application/client.rb:274:in `block in run_application’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/application/client.rb:267:in `loop’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/application/client.rb:267:in `run_application’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib/chef/application.rb:72:in `run’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/bin/chef-client:26:in `’
/usr/bin/chef-client:23:in `load’

any recommendation would be appreciated.

By the way Thanks again for the good writeup…

LikeLike

Reply
1. Matthias Marschall says:
  
  March 1, 2013 at 11:55 am
  
  Have you tried to re-authenticate your client as described in http://wiki.opscode.com/display/chef/Common+Errors
  
  LikeLike
  
  Reply
Sarfraz says:

January 20, 2016 at 3:14 pm

Hi ,
Can we delete an AWS instance using IP or Node name rather doing it through node ID.

THanks & Regards,
Sarfraz.

LikeLike

Reply
1. Matthias Marschall says:
  
  January 29, 2016 at 9:25 pm
  
  Hi Sarfraz,
  
  unfortunately I’m not aware of a way to do that.
  
  LikeLike
  
  Reply
Pingback: Using Ec2 For Web Hosting | Great at WebHosting
Kiran says:

March 10, 2017 at 9:29 am

Hi,

I didn’t understand how to get the pem file(key pair) before creating the instance. As you said we can create a new Key pair under Network & Security but can we use the same key to create many instances and use to login all the instances?

LikeLike

Reply
1. Dan Ackerson says:
  
  March 28, 2017 at 9:20 am
  
  Hi Kiran,
  
  yes, you usually create/download the key pair once to your laptop and then use this key to pre-seed all the subsequent instances you setup. An alternative is to create a base AMI of an OS that you know has your key on it and then you can launch new instances from this AMI without worrying about adding another/the right key pair.
  
  Good luck,
  
  Dan
  
  LikeLike
  
  Reply