EC2 on Rails version 0.9.8 is now available (or will be in a few hours when the RubyForge servers are synced). This is a recommended update for everyone.

It includes some major new features:

• monit monitoring daemon: monitors mysqld, apache, memcached, mongrels, system load and free drive space
• incremental MySQL backup (important for large databases)
• Apache SSL support
• a local Postfix SMTP server enabled by default

And most importantly this fixes the problem with broken Ubuntu package updates which was caused by a missing repository in the list of repositories.

As I mentioned yesterday, the base image is now built using Eric Hammond’s EC2 Ubuntu script.

Also, there are major new features such as incremental MySQL backup (important for large databases), Apache SSL support, and a local Postfix SMTP server enabled by default.

My priorities now are:

1. Release an update based on Ubuntu 8.04 Hardy (this version is still using Ubuntu 7.10 Gutsy because I wanted to provide a reliable update as quickly as possible due to bug #20040. But now that the base image is built with Eric Hammond’s script it should be easy to update to Hardy.)
2. Create complete documentation.
3. Release a 100% bug-free version 1.0 with the current feature-set. Please help by reporting any bugs you find, either using the RubyForge bug tracker or by email.

But in the meantime Eric Hammond has created an excellent Ubuntu AMI base install project, it has all the necessary features like installing the AMI tools, getting the public keys on first boot so that you can log in with your EC2 account’s private key, regenerating the ssh host keys on first boot, etc. And it has some cool new features like the ability to give an instance an arbitrary script on startup using EC2’s “user-supplied instance data“. And, it has an active community.

So why would I duplicate that effort?! I’d rather concentrate on improving the Rails-specific features, so I’ve adapted my build script to be run from Eric’s. And by “adapted”, I mean “took out tons of stuff that I no longer need to care about”! And when I want to make any improvements that aren’t Rails-specific, I can contribute them to Eric’s project where they’ll benefit more people.

If you’re building your own custom AMI, I highly recommend using Eric’s base install script. You’ll end up with a solid base for your image with way less work, and it has a hook to easily include your own build script to customize the build.

Version 0.9.8 of EC2 on Rails uses Eric’s script for the base install, and it will be released in a couple of days (I’m using it in production already).

Seriously, why reinvent the wheel? You gotta love open source!

A frequent topic of discussion on the EC2 forums is how to send email reliably, efficiently, and especially without it being marked as spam. I found that even with a valid SPF record most mail sent from an EC2 instance was marked as spam or silently discarded.

This is probably partly because of the lack of matching reverse DNS records. But spam filters can be a bit arbitrary and the easiest way is to relay outgoing mail through a good smtp provider. (I don’t recommend relaying outbound mail through Google Apps, they supposedly have a 500 messages/day limit according to many people on their forums, although I couldn’t find that published anywhere. UPDATE: The info is here, thanks John Ward.)

I have tried a couple of SMTP providers, and I recommend AuthSMTP. They are reliable, have good service, and our mail that’s delivered through them almost never gets marked as spam. Also, they have monthly quotas rather than daily, so you have a chance to increase it before you hit the limit.

Rather than deliver directly to the AuthSMTP mail server from your web app it’s a good idea to deliver to a local queueing mail server, which will forward via the AuthSMTP gateway. Your web app will deliver mail to localhost (or perhaps a dedicated instance if you prefer), port 25.

This has several advantages:

1. Your web server can finish the request more quickly.
2. There’s less chance that the mail server will be unavailable. At least the mail will be queued locally until the remote server becomes available again. AuthSMTP has proven to be quite reliable, but it has been unavailable on a couple of occasions.
3. AuthSMTP limits the number of concurrent connections that you can make. You can easily configure your local mail server to limit the number of outgoing connections to the gateway.

Configuration

I recommend using Postfix, it’s fast, reliable and most importantly, easy to configure. Your Linux distribution will definitely have a Postfix package available (it comes pre-installed on EC2 on Rails). On Debian or Ubuntu install with:

sudo aptitude install postfix

Here’s the config file, /etc/postfix/main.cf:

myhostname = www.YOURDOMAIN.com
mydomain = YOURDOMAIN.com
myorigin = $mydomain

smtpd_banner = $myhostname ESMTP $mail_name
biff = no
append_dot_mydomain = no

alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = localdomain, localhost, localhost.localdomain, localhost
mynetworks = 127.0.0.0/8
mailbox_size_limit = 0
recipient_delimiter = +

# SECURITY NOTE: Listening on all interfaces. Make sure your firewall is
# configured correctly
inet_interfaces = all

relayhost = [mail.authsmtp.com]
smtp_connection_cache_destinations = mail.authsmtp.com
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = static:YOUR_AUTHSMPT_USER_ID:YOUR_AUTHSMTP_PW
smtp_sasl_security_options = noanonymous

default_destination_concurrency_limit = 4

soft_bounce = yes

How simple is that?! Have you ever seen a sendmail config file?

soft_bounce is important because it means that postfix will queue the messages if they’re bounced by the remote gateway for any reason (this is only if it’s bounced by the gateway, not if it’s bounced by the destination server). This would usually be caused by some configuration problem like an authentication failure. If the message is bounced by the eventual destination server (e.g. the mailbox doesn’t exist or is full), or if the destination server can’t be contacted, your local server won’t know about it because the message has already been accepted by the gateway. (It’s probably a good idea to keep track of bounced messages returned by the eventual destination server, see “Don’t spoof the From field” below.)

default_destination_concurrency_limit is so you stay within AuthSMTP’s concurrent connection limit. If you have Postfix running on multiple instances you’ll need to adjust this accordingly.

To see mail that’s stuck in the queue:

mailq

Postfix will automatically try to resend it, but you can force it to be sent immediately using:

sudo postqueue -f

Monitoring

Of course you need to know if anything goes wrong with the mail delivery and it won’t be in your web app’s log. I use scripts in /etc/cron.hourly to check logs and mail me the output if there are errors. But when it comes you mail delivery failure you might have a bit of a chicken-and-egg problem: you can’t use postfix to send the mail if postfix is having problems. Here’s a simple ruby script to send emergency mail via a different mail server. It’s configured to use Google Apps (you’ll need to create a new account to send the mail from), if you don’t use Google Apps you can easily change this to use a different mail server.

Save this as /usr/local/bin/emergency_mail_sender:

#!/usr/bin/env ruby

# This is a simple script to send mail via an alternate server when there are
# errors with the normal queueing mail sender
# The subject is the first command-line arg and the body is received on stdin

#################################

from_address             = "admin_mail_sender@YOURDOMAIN.com"
to_address               = "admin@YOURDOMAIN.com"
smtp_server              = "smtp.gmail.com"
smtp_port                = 587
smtp_mail_from_domain    = "YOURDOMAIN.com"
smtp_account_name        = "admin_mail_sender@YOURDOMAIN.com"
smtp_password            = "YOUR_PASSWORD"
smtp_authentication_type = :plain
debug                    = false

#################################

subject = ARGV[0]
body = $stdin.read

require 'rubygems'
require 'net/smtp'
require 'tlsmail'

exit if body.nil? || body == ""

msgstr = <<END_OF_MESSAGE
Subject: #{subject}

#{body}
END_OF_MESSAGE

Net::SMTP.enable_tls(OpenSSL::SSL::VERIFY_NONE)
  smtp = Net::SMTP.new(smtp_server, smtp_port)
  smtp.set_debug_output $stderr if debug
  smtp.start(smtp_mail_from_domain, smtp_account_name, smtp_password, smtp_authentication_type) do |s|
    s.send_message msgstr, from_address, to_address
end

Here’s a script that can be run by cron every hour to check for mail delivery problems, it uses the emergency_mail_sender script to notify you of the problem. It works on Ubuntu (but it needs the logtail package installed), it might not work on other systems. Save this as /etc/cron.hourly/check_mail_logs

#!/bin/sh
hostname=`hostname -s`
mailer = /usr/local/bin/emergency_mail_sender
/usr/sbin/logtail -f/var/log/mail.warn | $mailer "$hostname: mail warnings"
/usr/sbin/logtail -f/var/log/mail.err | $mailer "$hostname: mail errors"
/usr/sbin/logtail -f/var/log/syslog | grep 'status=' | egrep -v 'status=sent' | $mailer "$hostname: undelivered mail"

SPF

Here’s your SPF record:

v=spf1 include:authsmtp.com include:aspmx.googlemail.com ~all

If you’re not using Google Apps to send mail for your domain remove include:aspmx.googlemail.com. If you want to create your own SPF record there’s a good SPF record generator at spfwizard.com.

Don’t spoof the From field

You should only send mail from somebody@yourdomain.com. If you try to send mail from somebody@pauldowman.com, for example, the receiver will see that pauldowman.com has an SPF record, and that it doesn’t authorize your mail server. Then into the spam folder you go.

To get around this you can send from something like noreply@yourdomain.com, and set the Reply-To header to somebody@pauldowman.com. You can even set the name in the from field, for example: “Paul Dowman via yoursite” <noreply@pauldowman.com>. The Reply-To header will make sure that most people’s replies go to the correct address, but a few will inevitably end up at noreply@yourdomain.com so it’s probably a good idea to set up an autoresponder at that address, or at least make sure the message bounces so the user eventually realizes the mistake.

RubyFringe is a single-track indie conference with no paid technical sponsors and a target attendance of 150.

It will run for two days (plus an opening night), with roughly 10 speakers or panels each day.

There’s a kick-ass list of speakers, including out-of-towners like Ezra Zygmuntowicz, Zed Shaw (now’s your chance to fight him! ;-), Obie Fernandez, and local Torontonians like Reg Braithwaite and Leila Boujnane.

Registration opens tomorrow.

See you there!

The site is built with Ruby on Rails and hosted on EC2.

If this sounds exciting to you, or if you know someone who might be a good fit, please see our jobs page for details.

UPDATE January 2008: we’ve hired two kick-ass developers!

Capistrano tasks

There is now a rubygem available that provides Capistrano tasks to manage the instance. There are tasks to set the server’s timezone, install packages and rubygems, backup, restore, create and delete the database, set the MySQL root password, and more. To use these in your Rails project type:

> sudo gem install ec2onrails

Put Capfile in the root of your rails folder, and put deploy.rb in the config folder.

Then, from the root of your project type:

> cap ec2onrails:setup

This automatically sets your server’s timezone, installs any custom rubygems and Ubuntu packages, and creates your database for you. You can now deploy your rails app as you normally would:

> cap deploy:migrations

Another useful task for testing is:

> cap ec2onrails:restore_db_and_deploy

This recreates the database, restores data from an S3 bucket (specified in your deploy.rb), and deploys the app. I use this to prepare a staging server with the current production data and current production version of the app. After running this task I have an exact copy of my production server. I then deploy the latest version to this server before deploying it to production. This is a good way to be really sure your production deployment won’t fail (especially your migrations).

To see a list of all available Capistrano tasks:

> cap -T

New Ubuntu version

It’s now built with Ubuntu 7.10 “Gutsy”.

Support for new instance types

There are both i386 and x86_64 versions available to support the new EC2 instance types. So you can now use large and extra-large instances.

Multiple instances

The earlier versions only worked if your rails app was running on a single server. That was lame! Now you can have multiple instances using any combination of these roles: web server, app server, primary database. I’m working on adding a MySQL slave role and eventually a Memcache role.

For full instructions and details see the project web site.

New version

There is a new server image with some minor changes and bug fixes. For details see the change log. I’ll keep the old image around for a while.

Mailing lists

I have created two Google groups, one for announcements only (new versions, etc.), and one for general discussion.

Ongoing work

I’m working on a build script so the image can be built from source. Then I will provide access to a subversion repository with the source. Some people have expressed an interest in contributing, which I’m really happy about! If you want to get involved, please join the mailing list!

Check it out.

The answer is a Rails server virtual appliance, and I’ve made one for Amazon’s EC2 hosting service.

A server is no longer a physical thing, and EC2 takes it one step further: they completely separate the concept of server images and instances. An image is like a snapshot of a server hard drive, stored in an unchangeable state in their S3 data storage service. An instance is a running server that was booted from a copy of this image. But the instance is transient; changes to it’s own working copy of the image don’t affect the original image that it was booted from, and because of this multiple server instances can be started from the same image. (Hence the “elastic” property, new instances can be created on demand, your server pool can grow and shrink dynamically.)

Now even the OS can be configured and tested once, and then simply deployed as part of the application. So how is that different from any of the third-party libraries and frameworks that we use as part of our applications? I assemble applications out of many pieces of third-party software. All of it is continually improved by other people, none of it requires painstaking building or configuration, and the OS can be used the same way.

So I’m going to continue working on my Ruby on Rails EC2 server image with the goal that you will be able to simply deploy your Rails app directly to it with Capistrano with little or no configuration.

Try it out, and tell me what you think!

I have created a ready-to run public Ubuntu and Ruby on Rails image for Amazon EC2. (You can easily remove the Ruby on Rails stuff and use this as a general-purpose Ubuntu image.) My goal was to make it as easy to use as possible, so it includes a script to automate the process of re-bundling it as your own after you make configuration changes.

This means if you have an Amazon EC2 account you can run this image, log in and customize it as desired, and then with one command save a copy of it as your own.

Features include:

• Ubuntu 7.04 Feisty with Xen versions of standard libs (libc6-xen package) for better performance.
• All EC2 command-line tools installed
• Custom script to re-bundle, save and register your own copy of this image in one step
• MySQL 5
• Ruby 1.8.5
• Ruby on Rails 1.2.3
• Mongrel behind Apache 2.2, all pre-configured including /etc/init.d startup script
• MySQL and Apache configured to write logs to /mnt/log so you don’t fill up the tiny root filesystem
• Hostname set correctly to public hostname
• NTP

Soon to be added:

• Automatic MySQL backup to S3
• Mongrel Cluster
• Capistrano and elastic_rails support. I’d like it to be so that you can just start up an instance of this image and deploy to it.

The AMI id is currently ami-5d967334. UPDATE: I have created a new improved version of this image. Please try that one instead, this image will be removed soon.

Instructions to use this public AMI:

1. Sign up for an EC2 account

There is currently a waiting list, as of this writing in June 2007.

2. Install the command-line tools

The tools are installed on the image, but you need a way to start an instance of the image. See the getting started guide for details. The tools will need your Amazon access key identifiers.

3. Run the instance and log in via ssh

Again, the getting started guide describes this in detail. Obviously you can skip the “Finding a Suitable AMI” section.

NOTE: at boot time it retrieves your private key via HTTP using curl, and occasionally this fails. If you are prompted for a password and you’re sure that you’ve started the image with the right keypair and that you are providing the right private key try rebooting the image using “ec2reboot <image-id>“. UPDATE: this is fixed in the new version.

4. Get security updates

I recommend using aptitude for package management:

[server]# aptitude update
[server]# aptitude upgrade

5. (Optional) Remove unwanted packages
Again, use aptitude, this time run it with no arguments on the command-line. To completely remove a package including all it’s configuration and data select it and then press underscore (’_'). If you don’t want rails just remove the package “rubygems”. But don’t remove ruby itself because it’s needed by the Amazon tools.

6. Install your Rails app

Edit /etc/mongrel and set APP_DIR to the root of your rails directory. If you want multiple mongrel containers edit /etc/init.d/mongrel and add as many as you like to the start_cmd() and stop_cmd() sections. I assume this part will need some tweaking.

Be mindful of where your app writes log files and other data, it should go into /mnt because the root partition doesn’t have much space and you don’t want to fill it up.

7. (Optional) Create a new admin user (with sudo ability) and disable the root user

Ubuntu normally installs without a root user, by default you create a user with sudo access. This is standard best practice anyway so I recommend it.

NOTE: This new user will log in the normal way, i.e. with password rather than using the public key.

First, create the new user and add it to the admin group:

[server]# adduser myuserid
[server]# adduser myuserid admin

You now have a new user that can run sudo. You might want to add the following line to the end of the new user’s $HOME/.profile

source /usr/local/ec2/config

This will set up environment variables needed for the EC2 command-line tools, and add them to the path.

Now disable root login completely. Edit /etc/ssh/sshd_config and change the line

PermitRootLogin without-password

to the following:

PermitRootLogin no

Then edit /etc/passwd and change the first line to:

root:x:0:0:root:/root:/bin/false

(i.e. change “/bin/bash” to “/bin/false”). Now you can only log in using your newly created account.

8. Rebundle the image

I have included a script, rebundle.sh, that runs all the commands to bundle the new image, upload it to S3 and register it. It expects a directory, /mnt/ec2-config, that contains a config file and your AWS access identifiers. The contents of the directory are the following three files:

a) cert-XXXX.pem and pk-XXXX.pem. These are the X.509 certificate and private key files from your Amazon access key identifiers.

b) A config file named, strangely enough, config. It’s contents should look like the following:

AWS_ACCOUNT_ID=1234-1234-1234
KEY_FILE_NAME=pk-XXXX.pem
CERT_FILE_NAME=cert-XXXX.pem
AWS_ACCESS_KEY_ID=ABC0123
AWS_SECRET_ACCESS_KEY=abc0123abc0123abc0123
BUCKET_BASE_NAME=a-string-identifier

The values for those should be pretty self-explanatory except perhaps BUCKET_BASE_NAME. It’s value is used when generating a new S3 bucket to save your bundled image. The bucket name will be this string with a time/date stamp appended.

Once you’ve created /mnt/ec2-config and it’s contents one command will rebundle the current instance as a new AMI, save it to a new S3 bucket, and register it:

[server]# /usr/local/ec2/rebundle.sh

It takes a long time and there are long periods with no output so you might want to hit a key once in a while (or set “ServerAliveInterval 60″ in your ssh_config file) to avoid being disconnected while it’s running.

At the end you should see the id of your new AMI and the name of the S3 bucket that it’s stored in.

Phew. You’re done! :-)

A note on security

As is standard for public AMI’s, password-based logins for root are disabled. This is so that neither I nor anyone else (except you) can log into your running instance. You log in with your own public/private keypair (see step #3), otherwise there would be a window of opportunity between system startup and the moment you change the password. I have taken care to ensure that my rebundle script excludes sensitive data like your AWS access identifiers and root’s .ssh/authorized_keys file.

Acknowledgements

I used the instructions on the Atlantis Technology blog to get started with a working Ubuntu image and then many helpful posts on the Amazon Web Services developer forum to get the Amazon EC2 tools working.

• Taking Rails Tests to the Next Level
• Doing REST Right
• JRuby on Rails

I’m especially looking forward to the scaling-related (including Amazon EC2) sessions, and LetIt::Rest. I can’t wait!

I make do with a wi-fi-enabled mobile phone, but unfortunately wi-fi is not really a viable mobile alternative. Open hot-spots are few and far between (or offered by the same three amigos and priced so as not to compete with their mobile data). I’d happily pay the flat $29/month for Toronto Hydro’s OneZone, but I’m rarely inside the coverage area and it’s (by many accounts and in my own personal experience) barely usable anyway.

Canada is a great place to live for many reasons, most of which are more important than wireless data access. But the rest of the world is at the beginning of a mobile internet boom that Canadians are generally not taking part in. Who knows how many more would-be entrepreneurs from our flourishing local tech scene might be developing innovative mobile apps if they only had the opportunity to be inspired? It’s no wonder all the innovation in the mobile space comes from elsewhere.

But what I’m really impressed with is how well they’re handling a massive growth in traffic. As of Feb 23 they’ve gone from zero to almost 18 million users in just over three years, an increase of 240% since last July alone.

From the Facebook blog five weeks ago:

…almost ten million different users sign into the site every day, or more than half the user base. During our biggest peaks - Sunday & Monday night around 10EST - more than one million people will be simultaneously logged into the site.

Friendster is famous for missing a huge opportunity in part because of poor performance (40 second page load times), and MySpace’s well-known and frequent reliability problems have continually frustrated their users.

According to Alexa (as of today, April 11) Facebook’s traffic is only slightly higher than Friendster’s (who have since redesigned their architecture), and still far behind Myspace’s. (Alexa’s stats may not be that reliable however.)

Regardless of the actual numbers, they’re obviously doing the right things to keep on top of the massive growth so far. Unfortunately though, I haven’t been able to find out many technical details except that they have two terabytes of RAM used by memcached.

UPDATE April 16: Apparently their traffic increased by another 50% during the month of March! (Link via Matthew Burpee on Facebook).

GigPark is a new service that will help you recommend your favorite service providers to friends and family, instantly get service recommendations from people you trust, and build your service business through word-of-mouth. Excited yet?! Me too!!

Please check out the GigPark blog for more information.

XML gives Java developers something to do
I think I see Sun’s master plan: the more work Java developers must do to solve a given problem, the more of them we’ll need to have. The more Java developers we have, the more popular Java appears. (The only problem left to solve is how to profit from that, but I digress). The key to this plan’s success is lack of syntactic support for XML, and the added bonus is that even more developers will occupy their time building endless numbers of libraries! “Why make trillions when we can make… BILLIONS?!” :-)

There are some decent libraries for handling XML in Java, but most are quite heavyweight, requiring you to create Java classes (in the case of the XML binding variety) or at least a config file (XML-based, naturally!).

There is another way
I’ve written before about using other languages with Java, and Groovy has some really nice features for handling XML. Groovy compiles to Java bytecode so you can write an XML handling class in Groovy and use it within your Java app. Here’s an example that uses Groovy’s GPath to read in an XML file from Sun’s XML DOM tutorial, and prints some of the elements. Just for fun, we’re loading the XML file directly by http:

def slideShow = new XmlSlurper().parse("http://java.sun.com/webservices/jaxp/dist/1.1/docs/tutorial/dom/samples/slideSample01.xml")
println("slideshow author attribute: ${slideShow['@author']}")

def slides = slideShow.slide
for (slide in slides) {
    println("slide title element: " + slide.title)
    for (item in slide.item) {
        println("  item text: ${item}")
    }
}

Here’s the Java version as a comparison. Honestly, nobody really uses the DOM API because it’s so painful (hence the existence of so many libraries to do the job), but this is the out-of-the-box Java way:

DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
URL url = new URL("http://java.sun.com/webservices/jaxp/dist/1.1/docs/tutorial/dom/samples/slideSample01.xml");
InputStream stream = url.openStream();
DocumentBuilder builder = factory.newDocumentBuilder();
Document document = builder.parse(stream);

Element rootElement = document.getDocumentElement();
String author = rootElement.getAttribute("author");
System.out.println("slideshow author attribute: " + author);

NodeList slides = rootElement.getChildNodes();

for (int i = 0; i < slides.getLength(); i++) {
    if (slides.item(i).getNodeType() == Node.ELEMENT_NODE) {
        Node slide = slides.item(i);
        NodeList slideChildren = slide.getChildNodes();
        for (int j = 0; j < slideChildren.getLength(); j++) {
            Node child = slideChildren.item(j);
            if (child.getNodeType() == Node.ELEMENT_NODE) {
                if (child.getNodeName().equals("title")) {
                    System.out.println("slide title element: "
                            + child.getTextContent());
                } else if (child.getNodeName().equals("item")) {
                    System.out.println("  item text: "
                            + child.getTextContent());
                }
            }
        }
    }
}

I have omitted error handling.

Using XPath in Java can make our job easier when it comes to searching a tree for a node (you wouldn’t want to see this done with pure DOM manipulation!), but we’ll still still have to deal with an org.w3c.dom.Node object that is the result:

XPath xpath = XPathFactory.newInstance().newXPath();
String expression = "/slideshow/slide[title = 'Overview']";
Node overview = (Node) xpath.evaluate(expression, document, XPathConstants.NODE);

Here’s the same thing done in Groovy, a simple one-liner:

def overview = slideShow.slide.find { it.title == "Overview" }

Two interesting things here are the use of a closure, and the use of == which might look wrong to a sharp-eyed Java programmer. Groovy uses == for equality and the method is() to test for identity. This is the opposite of Java but leads to more readable code because typically we compare for equality much more often than identity. And == works with null in Groovy, so your typical Java comparison (x != null && x.equals(y)) becomes x == y in Groovy, which looks more like what we mean.

Creating XML is also easy in Groovy using MarkupBuilder, here is the code to create our sample XML file:

def doc = new StreamingMarkupBuilder().bind {
    slideshow(title:"Sample Slide Show", date:"Date of publication", author:"Yours Truly") {
        slide(type:"all") {
            title("Wake up to WonderWidgets!")
        }
        slide(type:"all") {
            title("Overview")
            item("item") {
                mkp.yield("Why ")
                em("WonderWidgets")
                mkp.yield(" are great")
            }
            item()
            item("item") {
                mkp.yield("Who ")
                em("buys")
                mkp.yield(" WonderWidgets")
            }
        }

        mkp.yield("normal text here")
    }
}

System.out << doc

It’s easy to read because it has a visually similar structure to the XML that it’s creating, and the syntax is quite terse. Groovy has similar builders for other things besides XML, for example to create SWT and Swing UI’s. You can even create your own. More on how this is implemented coming soon…

There are other choices besides Groovy, of course, I haven’t tried it yet but Scala looks like it might be a good choice. (For that matter there are other choices besides XML!)

Since then many dialects have been implemented on many different platforms, including the ECMA-262 standard, but historically it’s success has mainly been on the client side, as a scripting language for UI elements. There were some early attempts at using it on the server, including Netscape’s LiveWire scripting and Microsoft’s (non-default) option for server-side JScript in ASP pages, but I don’t believe either of those were widely used.

This seems to be changing lately though, and the release of Java 6 makes it official: it includes a scripting API, and though the API aims to stay relatively language neutral the default scripting engine is a slightly modified version of Rhino, the JavaScript engine.

So Sun is finally beginning to position Java the platform as being about the JVM rather than the language, the approach that Microsoft took with .NET from the start. Sun is simply trying to stay relevant amidst the growing shift to dynamic languages. Phobos is a JavaScript-based server that is part of Sun’s GlassFish project, and one of its stated goals is to “appeal to developers who, for a variety of reasons, are hard for Java EE to reach”. But it’s a welcome change for Java developers who now have more language options while still developing for the Java platform. “Don’t worry boss, it’s JEE”. ;-)

Java developers now have options for building DSL’s using metaprogramming, and for more efficient declaration of complex literals. Just imagine how much cleaner and shorter your jUnit tests would be if you set up your test fixtures like this:

var expected = {firstName:"John", lastname:"Smith", salary: 80000}

That’s JavaScript object literal syntax. Try it.

Of course there are a variety of “scripting” languages available that can run on the JVM, most have been around for a few years already. Jython (née JPython) has been around since 1997(!). Groovy (though suffering slightly from infighting and controversy) has been around since at least 2003. And Sun just hired the two core JRuby developers. (There are also pre-compiled languages like Scala).

But JavaScript seems to be the one growing most quickly in popularity, partly because it’s shipped by default with Java 6, and partly because there is appeal in using only one language; since we’re forced to use JavaScript on the client side we might as well use it on the server too. I think the latter is a pretty weak reason though, it’s good to have a working knowledge of several languages and to use the right tool for the job. (When all you have is a hammer…) Also, though it still lacks modularity, new features like array comprehensions and destructuring assignment are bringing JavaScript more in line with other scripting languages.

Or perhaps Java developers just like the name.

But I happened to pick it up in a book store recently and actually flip through it a bit, and it looked interesting so I bought it. I’m glad I did, it was a surprisingly good read. It reminds me a bit of The Pragmatic Programmer: most of the material is fairly basic, but it’s worth reading even for experts because it’s well written, reasonably short, yet quite thorough.

It focuses mainly on PHP and MySQL (on Linux), but really this book is about general principles not implementation details, and those principles are equally applicable to other languages and platforms.

For a list of other technical books that I recommend, take a look at my reading page.

To give you an idea of the type of things I find interesting, here are a few of the links that I’ve posted to del.icio.us and other places in the past:

• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
• The trouble with rounding floating point numbers
• Continuous Integration
• The New Methodology
• Strong Typing vs. Strong Testing
• The Joel Test: 12 Steps to Better Code
• TDD Pattern: Do not cross boundaries
• 13 Exceptional Exception Handling Techniques
• Monitoring network traffic with Ruby and Pcap
• Comet: Low Latency Data for the Browser
• Functional Relational Programming
• MapReduce: Simplified Data Processing on Large Clusters
• The IBM PC’s 25 Year Legacy
• Kill All Screensavers
• Debunking the Myth of High-level Languages
• Errant Architectures
• History’s Worst Software Bugs
• How to Start a Startup
• The Guerrilla Guide to Interviewing
• FixedScopeMirage
• The Difference Between Values and Strategies
• The End of Corporate Computing
• How to Do What You Love
• Taste for Makers