Public Ubuntu & Ruby on Rails Image for Amazon EC2
UPDATE: I have created a new improved version of this image. The new image uses mongrel_cluster, and is more Capistrano-friendly. Skip this article and check out the new one!
I have created a ready-to run public Ubuntu and Ruby on Rails image for Amazon EC2. (You can easily remove the Ruby on Rails stuff and use this as a general-purpose Ubuntu image.) My goal was to make it as easy to use as possible, so it includes a script to automate the process of re-bundling it as your own after you make configuration changes.
This means if you have an Amazon EC2 account you can run this image, log in and customize it as desired, and then with one command save a copy of it as your own.
Features include:
- Ubuntu 7.04 Feisty with Xen versions of standard libs (libc6-xen package) for better performance.
- All EC2 command-line tools installed
- Custom script to re-bundle, save and register your own copy of this image in one step
- MySQL 5
- Ruby 1.8.5
- Ruby on Rails 1.2.3
- Mongrel behind Apache 2.2, all pre-configured including /etc/init.d startup script
- MySQL and Apache configured to write logs to /mnt/log so you don’t fill up the tiny root filesystem
- Hostname set correctly to public hostname
- NTP
Soon to be added:
- Automatic MySQL backup to S3
- Mongrel Cluster
- Capistrano and elastic_rails support. I’d like it to be so that you can just start up an instance of this image and deploy to it.
The AMI id is currently ami-5d967334. UPDATE: I have created a new improved version of this image. Please try that one instead, this image will be removed soon.
Instructions to use this public AMI:
1. Sign up for an EC2 account
There is currently a waiting list, as of this writing in June 2007.
2. Install the command-line tools
The tools are installed on the image, but you need a way to start an instance of the image. See the getting started guide for details. The tools will need your Amazon access key identifiers.
3. Run the instance and log in via ssh
Again, the getting started guide describes this in detail. Obviously you can skip the “Finding a Suitable AMI” section.
NOTE: at boot time it retrieves your private key via HTTP using curl, and occasionally this fails. If you are prompted for a password and you’re sure that you’ve started the image with the right keypair and that you are providing the right private key try rebooting the image using “ec2reboot <image-id>“. UPDATE: this is fixed in the new version.
4. Get security updates
I recommend using aptitude for package management:
[server]# aptitude update [server]# aptitude upgrade
5. (Optional) Remove unwanted packages
Again, use aptitude, this time run it with no arguments on the command-line. To completely remove a package including all it’s configuration and data select it and then press underscore (’_'). If you don’t want rails just remove the package “rubygems”. But don’t remove ruby itself because it’s needed by the Amazon tools.
6. Install your Rails app
Edit /etc/mongrel and set APP_DIR to the root of your rails directory. If you want multiple mongrel containers edit /etc/init.d/mongrel and add as many as you like to the start_cmd() and stop_cmd() sections. I assume this part will need some tweaking.
Be mindful of where your app writes log files and other data, it should go into /mnt because the root partition doesn’t have much space and you don’t want to fill it up.
7. (Optional) Create a new admin user (with sudo ability) and disable the root user
Ubuntu normally installs without a root user, by default you create a user with sudo access. This is standard best practice anyway so I recommend it.
NOTE: This new user will log in the normal way, i.e. with password rather than using the public key.
First, create the new user and add it to the admin group:
[server]# adduser myuserid [server]# adduser myuserid admin
You now have a new user that can run sudo. You might want to add the following line to the end of the new user’s $HOME/.profile
source /usr/local/ec2/config
This will set up environment variables needed for the EC2 command-line tools, and add them to the path.
Now disable root login completely. Edit /etc/ssh/sshd_config and change the line
PermitRootLogin without-password
to the following:
PermitRootLogin no
Then edit /etc/passwd and change the first line to:
root:x:0:0:root:/root:/bin/false
(i.e. change “/bin/bash” to “/bin/false”). Now you can only log in using your newly created account.
8. Rebundle the image
I have included a script, rebundle.sh, that runs all the commands to bundle the new image, upload it to S3 and register it. It expects a directory, /mnt/ec2-config, that contains a config file and your AWS access identifiers. The contents of the directory are the following three files:
a) cert-XXXX.pem and pk-XXXX.pem. These are the X.509 certificate and private key files from your Amazon access key identifiers.
b) A config file named, strangely enough, config. It’s contents should look like the following:
AWS_ACCOUNT_ID=1234-1234-1234 KEY_FILE_NAME=pk-XXXX.pem CERT_FILE_NAME=cert-XXXX.pem AWS_ACCESS_KEY_ID=ABC0123 AWS_SECRET_ACCESS_KEY=abc0123abc0123abc0123 BUCKET_BASE_NAME=a-string-identifier
The values for those should be pretty self-explanatory except perhaps BUCKET_BASE_NAME. It’s value is used when generating a new S3 bucket to save your bundled image. The bucket name will be this string with a time/date stamp appended.
Once you’ve created /mnt/ec2-config and it’s contents one command will rebundle the current instance as a new AMI, save it to a new S3 bucket, and register it:
[server]# /usr/local/ec2/rebundle.sh
It takes a long time and there are long periods with no output so you might want to hit a key once in a while (or set “ServerAliveInterval 60″ in your ssh_config file) to avoid being disconnected while it’s running.
At the end you should see the id of your new AMI and the name of the S3 bucket that it’s stored in.
Phew. You’re done! :-)
A note on security
As is standard for public AMI’s, password-based logins for root are disabled. This is so that neither I nor anyone else (except you) can log into your running instance. You log in with your own public/private keypair (see step #3), otherwise there would be a window of opportunity between system startup and the moment you change the password. I have taken care to ensure that my rebundle script excludes sensitive data like your AWS access identifiers and root’s .ssh/authorized_keys file.
Acknowledgements
I used the instructions on the Atlantis Technology blog to get started with a working Ubuntu image and then many helpful posts on the Amazon Web Services developer forum to get the Amazon EC2 tools working.
award tour » ubuntu + rails ec2 image:
[…] Paul Dowman » Public Ubuntu & Ruby on Rails Image for Amazon EC2. not that i have an ec2 account, but handy. […]
10 June 2007, 11:36 pmJohn Ward:
Any reason you went with Mongrel rather than Mongrel Cluster?
John
11 June 2007, 4:41 amJohn Ward:
Scratch that, I see what you’ve done.
I was comparing to how mongrel was setup from this post.
http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/
11 June 2007, 7:13 amPaul Dowman:
Thanks for the link, I’m going to change the setup to use mongrel_cluster as described. The gem is installed but the setup isn’t using it at the moment.
This is my first attempt at a production-ready Ruby on Rails setup so I appreciate any other tips & suggestions…
11 June 2007, 1:24 pmJohn Ward:
I was looking at doing an Ubuntu image and seen you had already done it.
Thanks for this, it has saved me time. If you need any assistance on the image I’d be happy to help.
11 June 2007, 2:15 pmMike:
Hey, this is great. How does one use this? Do I need to have at least one non-ec2 machine to run my DNS etc and then loadbalance off to the EC2? Do you have any pointers on running rails setups (or architecture diagrams?)
thanks again for the tutorial on rebundling the AMI.
12 June 2007, 7:33 pmPaul Dowman:
I recommend just using hosted DNS. I have used EasyDNS for several years and I’ve been very happy with them.
Unfortunately you need to set a short TTL for your DNS records though, because EC2 instances have a dynamically assigned IP address. As long as the instance is running, including across reboots, it will keep the same external IP address, but if you replace it with a new instance for any reason you will need to change the address for that record.
13 June 2007, 7:59 amMark Thomas:
I found it interesting that ruby is required for the Amazon tools. I’d like a little more information on that. Is Amazon embracing Ruby in a big way?
14 June 2007, 5:51 amAntonio:
Great work! How would one go about using postgres instead of mySQL.. any hint on where can I make this changes.
Thanks
14 June 2007, 10:28 amAntonio:
Have you consider this .. an alternative to Elastic Rail? or maybe both ..
http://capazon.rubyforge.org/
Cheers
14 June 2007, 10:34 amPaul Dowman:
Mark: I don’t know. Some of the tools are Java-based. I’d say Ruby is just a good choice to build those types of tools. Or if you want a more interesting take read this. :-)
Antonio: To use use postgres instead of MySQL just run aptitude, deselect mysql-client and mysql-server (which will force removal of other packages like libmysql-ruby and php5-mysql), and then install the appropriate postresql packages.
As for capazon, I think that’s more of a client side thing for starting and stopping instances, I’m pretty sure you could use it with this image (perhaps instead of installing Amazon’s java-based tools? I’m not sure if it uses them or not). I could pre-install it on the next version of this image though if that would be useful.
14 June 2007, 11:49 amNeil Wilson:
Nicely done.
It would be better if you attached the uploaded public/private keypair to a user ‘deploy’ with sudo capability. That would be more in keeping with the Ubuntu way of doing things (and it makes it more in keeping with the way Capistrano works).
Have you open-sourced the script that builds this image?
Rather than using Mongrel Cluster directly, you may want to consider Rubyworks - which takes mongrel up to the next level. That would allow you to dispense with Apache and its internal balancer and move to the ‘latest thing’ in web servers which is ‘nginx’.
16 June 2007, 9:29 amPaul Dowman:
Neil,
Thanks for the suggestions.
Regarding no root login: The reason I didn’t attach the public key to a user with sudo ability instead of root is that by default you need a password to invoke sudo, and I didn’t want to create a user with a known password. The solution to this might be setting “PasswordAuthentication no” in sshd_config, and then maybe allowing sudo without authentication for that user (what’s the point in requiring the password at the sudo prompt if it’s known publicly anyway), and then perhaps also randomizing the user password just to be paranoid. But really how is a user with unauthenticated sudo access better than just allowing root login?
I have a new version almost ready to go with a regular user (who can only log in with the private key) to be more capistrano-friendly. But I don’t think the user that the rails app runs as should have sudo access at all, especially if sudo without authentication is allowed (you might as well just run the app as root!). I’ll think about this one some more, comments and suggestions are welcome!
Regarding RubyWorks and nginx: I don’t know what’s in RubyWorks, at the moment their web page just says that it’s a “production stack of application and environment tools”, which to me means a re-packaging of existing open-source tools. I’m definitely interested in nginx but there’s no debian package yet which makes it a lot more effort to maintain. I’ll get around to trying it sooner or later regardless of that though.
Regarding the licencing of the script: I’d like the whole image to be available under some sort of open source license, I guess it needs to be GPL because it’s a repackaging of GNU/Linux, but I don’t know much about licencing and haven’t looked into that yet. I’ll think about that!
16 June 2007, 11:02 amSteve:
Awsome, thanks for the image!
Every single time I start it though it doesnt get the login credentials and I have to reboot it. Any ideas?
17 June 2007, 5:55 amSteve:
Also.. .wondering if you know how to fix the locale problem? I get this all the time:
perl: warning: Falling back to the standard locale (”C”).
17 June 2007, 6:25 amlocale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
Paul Dowman:
Steve,
The login problem is fixed in the version that I’m going to make public very soon, perhaps today if I have time. I’m not sure why but fetching the public key by http was often failing (anyone know why?). The public key is also made available in /mnt and it’s there reliably so the script now falls back to that if the http get fails.
As for the locale problem I haven’t seen that myself but make sure you still have the package “locales” installed, and you could also install the package “localeconf”. If localeconf is installed you can type “sudo dpkg-reconfigure localeconf” and it will prompt you to configure the localization settings. For US English choose “en_US.utf-8″ as your locale.
17 June 2007, 9:08 amSteve:
Damn… any chance you can list your changes? I’ve started customising my image already :-)
Thanks for the help. Re locale, what you do is export LC_CTYPE=C etc for all of them then dpkg-reconfigure locales and it all seems to magically work. :-)
17 June 2007, 9:34 amNeil Wilson:
Rubyworks is created by Thoughtworks and is essentially a package wrapped around mongrel that creates a managed cluster - in a way that is somewhat superior to mongrel_cluster. And it is supported!
You need to be able to ’sudo’ to make certain changes to the web-server in Capistrano. It is a right royal pain, and I wonder whether it could be abstracted away with a set of setuid scripts. That would get rid of the need for root completely.
If you haven’t downloaded the ‘xen-tools’ package you might want to do so. I contributed the Fiesty adaption scripts in there, and you may find them useful for your image. To be honest if you could get your setup to work as a ‘xen-tools’ role script that would make auto-creation a doddle.
If you haven’t joined the Google Group ‘rubyonrails-stack’, you might want to do so. That and ‘rubyonrails-deployment’.
http://groups.google.com/group/rubyonrails-stack
http://groups.google.com/group/rubyonrails-deployment
Rgs
NeilW
18 June 2007, 9:36 amPaul Dowman:
Neil,
I have been using “set :use_sudo, false” in my capistrano deploy.rb and deploying as the user that owns the mongrel processes. But I won’t be at all surprised when you point out the case where this won’t work! :-)
Thanks for the great tips, I joined rubyonrails-stack and rubyonrails-deployment, and I’m going to check out the xen-tools package, it looks useful. Also, I’m looking forward to rubyworks being released so I can try it out.
18 June 2007, 10:30 amNeil Wilson:
It’s when you want to restart or suspend the web-server and it is sat on a privileged port. You need root to do that.
I don’t know if you’ve noticed but the spec sheet for ‘Gutsy’ included a new version of the ‘init’ replacement Upstart that is supposed to provide monitoring and process management. That may mean Rubyworks is obsolete before it gets started!
NeilW
21 June 2007, 5:51 am