« RubyFringe, Toronto, July 18-20, 2008
Open Source made my life easier today »

A rock-solid setup for sending SMTP mail from your EC2 web server

17th February 2008, 03:53 pm

(None of this is EC2-centric, but it’s particularly needed on EC2.)

A frequent topic of discussion on the EC2 forums is how to send email reliably, efficiently, and especially without it being marked as spam. I found that even with a valid SPF record most mail sent from an EC2 instance was marked as spam or silently discarded.

This is probably partly because of the lack of matching reverse DNS records. But spam filters can be a bit arbitrary and the easiest way is to relay outgoing mail through a good smtp provider. (I don’t recommend relaying outbound mail through Google Apps, they supposedly have a 500 messages/day limit according to many people on their forums, although I couldn’t find that published anywhere. UPDATE: The info is here, thanks John Ward.)

I have tried a couple of SMTP providers, and I recommend AuthSMTP. They are reliable, have good service, and our mail that’s delivered through them almost never gets marked as spam. Also, they have monthly quotas rather than daily, so you have a chance to increase it before you hit the limit.

Rather than deliver directly to the AuthSMTP mail server from your web app it’s a good idea to deliver to a local queueing mail server, which will forward via the AuthSMTP gateway. Your web app will deliver mail to localhost (or perhaps a dedicated instance if you prefer), port 25.

This has several advantages:

  1. Your web server can finish the request more quickly.
  2. There’s less chance that the mail server will be unavailable. At least the mail will be queued locally until the remote server becomes available again. AuthSMTP has proven to be quite reliable, but it has been unavailable on a couple of occasions.
  3. AuthSMTP limits the number of concurrent connections that you can make. You can easily configure your local mail server to limit the number of outgoing connections to the gateway.

Configuration

I recommend using Postfix, it’s fast, reliable and most importantly, easy to configure. Your Linux distribution will definitely have a Postfix package available (it comes pre-installed on EC2 on Rails). On Debian or Ubuntu install with:

sudo aptitude install postfix

Here’s the config file, /etc/postfix/main.cf:

myhostname = www.YOURDOMAIN.com
mydomain = YOURDOMAIN.com
myorigin = $mydomain

smtpd_banner = $myhostname ESMTP $mail_name
biff = no
append_dot_mydomain = no

alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = localdomain, localhost, localhost.localdomain, localhost
mynetworks = 127.0.0.0/8
mailbox_size_limit = 0
recipient_delimiter = +

# SECURITY NOTE: Listening on all interfaces. Make sure your firewall is
# configured correctly
inet_interfaces = all

relayhost = [mail.authsmtp.com]
smtp_connection_cache_destinations = mail.authsmtp.com
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = static:YOUR_AUTHSMPT_USER_ID:YOUR_AUTHSMTP_PW
smtp_sasl_security_options = noanonymous

default_destination_concurrency_limit = 4

soft_bounce = yes

How simple is that?! Have you ever seen a sendmail config file?

soft_bounce is important because it means that postfix will queue the messages if they’re bounced by the remote gateway for any reason (this is only if it’s bounced by the gateway, not if it’s bounced by the destination server). This would usually be caused by some configuration problem like an authentication failure. If the message is bounced by the eventual destination server (e.g. the mailbox doesn’t exist or is full), or if the destination server can’t be contacted, your local server won’t know about it because the message has already been accepted by the gateway. (It’s probably a good idea to keep track of bounced messages returned by the eventual destination server, see “Don’t spoof the From field” below.)

default_destination_concurrency_limit is so you stay within AuthSMTP’s concurrent connection limit. If you have Postfix running on multiple instances you’ll need to adjust this accordingly.

To see mail that’s stuck in the queue:

mailq

Postfix will automatically try to resend it, but you can force it to be sent immediately using:

sudo postqueue -f

Monitoring

Of course you need to know if anything goes wrong with the mail delivery and it won’t be in your web app’s log. I use scripts in /etc/cron.hourly to check logs and mail me the output if there are errors. But when it comes you mail delivery failure you might have a bit of a chicken-and-egg problem: you can’t use postfix to send the mail if postfix is having problems. Here’s a simple ruby script to send emergency mail via a different mail server. It’s configured to use Google Apps (you’ll need to create a new account to send the mail from), if you don’t use Google Apps you can easily change this to use a different mail server.

Save this as /usr/local/bin/emergency_mail_sender:

#!/usr/bin/env ruby

# This is a simple script to send mail via an alternate server when there are
# errors with the normal queueing mail sender
# The subject is the first command-line arg and the body is received on stdin

#################################

from_address             = "admin_mail_sender@YOURDOMAIN.com"
to_address               = "admin@YOURDOMAIN.com"
smtp_server              = "smtp.gmail.com"
smtp_port                = 587
smtp_mail_from_domain    = "YOURDOMAIN.com"
smtp_account_name        = "admin_mail_sender@YOURDOMAIN.com"
smtp_password            = "YOUR_PASSWORD"
smtp_authentication_type = :plain
debug                    = false

#################################

subject = ARGV[0]
body = $stdin.read

require 'rubygems'
require 'net/smtp'
require 'tlsmail'

exit if body.nil? || body == ""

msgstr = <<END_OF_MESSAGE
Subject: #{subject}

#{body}
END_OF_MESSAGE

Net::SMTP.enable_tls(OpenSSL::SSL::VERIFY_NONE)
  smtp = Net::SMTP.new(smtp_server, smtp_port)
  smtp.set_debug_output $stderr if debug
  smtp.start(smtp_mail_from_domain, smtp_account_name, smtp_password, smtp_authentication_type) do |s|
    s.send_message msgstr, from_address, to_address
end

Here’s a script that can be run by cron every hour to check for mail delivery problems, it uses the emergency_mail_sender script to notify you of the problem. It works on Ubuntu (but it needs the logtail package installed), it might not work on other systems. Save this as /etc/cron.hourly/check_mail_logs

#!/bin/sh
hostname=`hostname -s`
mailer = /usr/local/bin/emergency_mail_sender
/usr/sbin/logtail -f/var/log/mail.warn | $mailer "$hostname: mail warnings"
/usr/sbin/logtail -f/var/log/mail.err | $mailer "$hostname: mail errors"
/usr/sbin/logtail -f/var/log/syslog | grep 'status=' | egrep -v 'status=sent' | $mailer "$hostname: undelivered mail"

SPF

Here’s your SPF record:

v=spf1 include:authsmtp.com include:aspmx.googlemail.com ~all

If you’re not using Google Apps to send mail for your domain remove include:aspmx.googlemail.com. If you want to create your own SPF record there’s a good SPF record generator at spfwizard.com.

Don’t spoof the From field

You should only send mail from somebody@yourdomain.com. If you try to send mail from somebody@pauldowman.com, for example, the receiver will see that pauldowman.com has an SPF record, and that it doesn’t authorize your mail server. Then into the spam folder you go.

To get around this you can send from something like noreply@yourdomain.com, and set the Reply-To header to somebody@pauldowman.com. You can even set the name in the from field, for example: “Paul Dowman via yoursite” <noreply@pauldowman.com>. The Reply-To header will make sure that most people’s replies go to the correct address, but a few will inevitably end up at noreply@yourdomain.com so it’s probably a good idea to set up an autoresponder at that address, or at least make sure the message bounces so the user eventually realizes the mistake.

Category: EC2, Linux, Mail, Software Development, Systems  |  Comment (RSS)  |  Trackback

52 Comments

  1. Ilya Grigorik:

    Awesome writeup Paul, much appreciated! This will save us at least a couple of days of debugging.

    18 February 2008, 10:30 am

  2. Ryan Duffield:

    Very timely! I was just wondering what the best approach would be for some of my EC2 apps. :-)

    23 February 2008, 10:01 pm

  3. gilltots:

    The only downside to AuthSMTP is the cost. for example, to send 15,000 emails a month costs $248/year, which comes out to .14 cents per email. compare that with 1 google apps account which allows 500 emails/day (15,000/month) for $50/year, or .028 cents per email. that means authsmtp is 5 times as expensive! and don’t forget to factor in the outgoing bandwidth costs that you’ll be charged for sending data from EC2. maybe amazon is working on Simple SMTP or something, or google will up their daily limit. we can hope!

    3 March 2008, 11:00 pm

  4. John Ward:

    The limit for Google Apps is documented here

    http://www.google.com/support/a/bin/answer.py?answer=59797

    4 March 2008, 4:00 am

  5. Paul Dowman:

    John: Thanks for the link.

    gilltots: A hard daily limit of 500 messages is too low for me. AuthSMTP has monthly limits and they send you a warning when you hit 80% of your limit so you can increase it (they charge a pro-rated amount).

    Hey, I don’t mean to shill for any one vendor here, if it makes sense for you to save money and use Google Apps then go for it! I still recommend having a backup provider though, or some other way to receive warnings when there are problems with your normal SMTP server.

    4 March 2008, 8:15 am

  6. frederic sidler:

    I also found in the amazon forum that fastmail.fm was a good solution and reliable.
    I don’t know yet if we can use it as mail relay, but they seem to be serious and pricing is very competitive
    http://fastmail.fm/pages/fastmail/docs/pricingtbl.html

    21 March 2008, 11:54 am

  7. Neil Gunder:

    We’ve been using http://triggermail.net which has been great. They helped us with the setup and monitor all our email delivery. Also works great with google analytics. They also offer it free to smaller sites. If you go to them please tell them I sent you.

    4 April 2008, 2:47 pm

  8. Tom:

    You. Rock.

    5 May 2008, 10:31 am

  9. Bill:

    Terrific post. Just what I was looking for. We recently setup a postfix server on Slicehost, which delivers to Gmail fine, but emails to Yahoo disappear into the ether. I’m not sure what happens with Hotmail since Hotmail won’t let me log in (good grief… how big are they and they still have login bugs?), but I’m going to sign up for AuthSMTP and combined with your config file, I bet that fixes the Yahoo problems.

    27 May 2008, 8:45 pm

  10. BipinDas:

    Paul
    It was great. This will help a lot.

    30 May 2008, 3:11 am

  11. Alan:

    Paul, absolutely brilliant.

    This was indeed a problem we faced at Amazon EC2, and after doing a little digging your post shone through like a guiding light in the storm.

    Thanks for pulling this together.

    Quick Question: How are you authorising Postfix to only accept email coming from your other EC2 instances?

    26 June 2008, 5:34 am

  12. Paul Dowman:

    Alan,

    In my case I run postfix on each app server, and the app delivers mail via postfix on localhost because it’s just easier that way (one reason not to might be if you have lots of hosts and you’re exceeding the AuthSMTP’s concurrent connection limit.

    But you could use the EC2 firewall rules to allow connections only from your own instances (you should block port 25 to others anyway). You might need to change some settings like mynetworks if you want to relay from other hosts.

    26 June 2008, 8:28 am

  13. Zubair Khan:

    Hi Paul,
    I created the AuthSMTP account like you mentioned in your writeup and installed postfix using yum/remi on my fedora 4 EC2. But how do I point my php application to use this setup? My application has the option of a. smtp( server/user/password) b. sendmail (path to sendmail /usr/sbin/sendmail) or phpmail.

    Also the default /etc/postfix/main.cf file did not have the below user/password variables like you mentioned.

    smtp_connection_cache_destinations = mail.authsmtp.com
    smtp_sasl_auth_enable = yes
    smtp_sasl_password_maps = static:YOUR_AUTHSMPT_USER_ID:YOUR_AUTHSMTP_PW
    smtp_sasl_security_options = noanonymous

    Any input will be greatly appreciated.

    Thanks,

    Zubair

    4 July 2008, 11:45 pm

  14. Paul Dowman:

    Zubair, in your PHP code mail configuration you should use smtp with “localhost” as the server, “25″ as the port (it’s the default so you probably don’t need to specify that) and no username and password. And you don’t need to keep the default main.cf file, you can replace it with what I’ve suggested above or edit it as you like.

    5 July 2008, 7:28 am

  15. Zubair Khan:

    Thanks Paul for the reply, I did like you said, but it seems my script is still trying to use the local mail server. Do I need to start postfix or something? I don’t get an error but I get mail back saying ..
    The original message was received at Sat, 5 Jul 2008 15:01:25 -0400
    from localhost.localdomain [127.0.0.1]
    —– The following addresses had permanent fatal errors —–

    (reason: 553 http://www.spamhaus.org/query/bl?ip=75.101.231.205)
    —– Transcript of session follows —–
    … while talking to smtp.secureserver.net.:
    >>> RCPT To:
    <<< 553 http://www.spamhaus.org/query/bl?ip=75.101.231.205
    550 5.1.1 … User unknown
    Unknown command: “<<<”
    & 550: Invalid message number

    My apologies for sending you error messages. Just really desperate. The funny thing is, I keep getting these same reply messages even if I use sendmail or smtp.

    Thanks again in advance.

    Zubair

    5 July 2008, 2:12 pm

  16. Zubair Khan:

    Hi again Paul, FYI, I pointed my application directly to authsmtp.com and was able to send the email. Only that it reached the destination marked [SPAM].
    Thanks for your help.

    Zubair

    5 July 2008, 2:48 pm

  17. Missaka Wijekoon:

    Many thanks for the excellent write up on setting up the postfix config. You saved me many hours of trouble!

    20 July 2008, 3:24 am

  18. James Blair:

    You just saved me a bunch of time (again). Thanks!!

    22 July 2008, 9:16 pm

  19. Ezra:

    Paul,
    Thanks for the excellent article. I’d like to add that AuthSMTP does not allow arbitrary you to set arbitrary “From:” addresses, aka the “header sender address.” Many web applications rely on this, for example, Google Maps. (Get directions, and then click “Send”). AuthSMTP only allows users to set up a limited number of domains that can be in the “From:” field. (http://www.authsmtp.com/auth-smtp/service.html)
    According to this chart: http://www.birds-eye.net/article_archive/smtp_mail_relay_services.htm there are other providers that don’t limit the “From:” field.

    23 July 2008, 4:47 pm

  20. Paul Dowman:

    Ezra: See my “Don’t spoof the From field” section.

    23 July 2008, 5:28 pm

  21. frederic sidler:

    Using Authsmtp for a while, email are not delivered to Yahoo account anymore. I contact AuthSMTP regarding this problem. From the web app emails are sent. I can see them in AuthSMTP control panel, but email are not delivered at all to Yahoo customer ! We should I look at. How can I investigate ?

    20 August 2008, 4:46 am

  22. Paul Dowman:

    Frederic: Sorry, I don’t know. Maybe they’re going into the spam folder, perhaps some of the people who received it clicked the spam button? Make sure you have an unsubscribe link that’s easy to find in the email so they’re less likely to mark it as spam.

    20 August 2008, 7:43 am

  23. Oleg Shaldybin:

    Thanks for this article, it helped me to set up email infrastructure with our EC2-deployed Rails app. One question: how do you handle local mail deliveries (like cron output)? I wanted to use ~root/.forward but letters to root@ec2hostname are rejected by Google (we use Google Apps), so they don’t even hit root mailbox.

    10 September 2008, 6:55 am

  24. Paul Dowman:

    Oleg: There’s probably a much better way, I’m really don’t know postfix configuration that well, but I have the following:

    append_dot_mydomain = yes
    masquerade_domains = $mydomain

    Then I have a mail account (I also use Google Apps) named root@

    I can’t remember at the moment if there’s anything more than that required, I’ll try to look into it if I get a chance.

    12 September 2008, 2:20 pm

  25. Oleg Shaldybin:

    Thanks for the advice, Paul!

    18 September 2008, 1:47 pm

  26. shrikant:

    I am facing this issue. Please help.

    Net::SMTPAuthenticationError (435 4.7.0 Error: authentication failed: generic failure
    ):
    /usr/local/lib/ruby/1.8/net/smtp.rb:586:in `auth_login’
    /usr/local/lib/ruby/1.8/net/smtp.rb:571:in `__send__’
    /usr/local/lib/ruby/1.8/net/smtp.rb:571:in `authenticate’
    /usr/local/lib/ruby/1.8/net/smtp.rb:411:in `do_start’
    /usr/local/lib/ruby/1.8/net/smtp.rb:378:in `start’
    /usr/local/lib/ruby/1.8/net/smtp.rb:316:in `start’
    /app/newqlubb/vendor/rails/actionmailer/lib/action_mailer/base.rb:565:in `perform_delivery_smtp’
    /app/newql

    4 October 2008, 7:25 am

  27. Omar:

    Hi Paul,

    Thanks for the post.

    Doesn’t the elastic ip address offering from amazon solve the reverse dns issue associated with ec2?

    17 October 2008, 8:11 am

  28. Julien:

    Thanks Paul for this great article… I atcually set it up and it’s working great.
    We have however a problem since all the emails that Postfix sees are actually transfered to AuthSMTP even though the other “From” have not been added.
    Amonf them is MAILER-DAEMON … and we really don’t want to add to the From in AuthSMTP since we don’t want to “pay” for it. Do you know if there is any way to configure Postfix so that only a certain number of ‘from’ emails are redirected to AuthSMTP, and not all of them?

    5 December 2008, 11:10 am

  29. Daniel L:

    Thanks for the information! This is a much better approach than coding each website to connect directly to the SMTP service. This way we can change the SMTP connection from a single place in the proxy server.

    Does anyone know of a good Windows equivalent of postfix? I’m having a hard time finding something.

    12 December 2008, 3:33 pm

  30. bruno:

    Hey Paul, thanks for the great article. Just curious how I’d go about configuring things if I have two different domains running on the same server. Do I need to do something differently in my postfix config?

    16 December 2008, 9:26 pm

  31. bruno:

    Following up on my own question: you have to do some postfix configuration using the ‘transport_maps’ variable. This allows you to use different transports for different sending conditions. See:

    http://www.usenet-forums.com/linux-networking/59961-postfix-smarthost-relayhost-particular-domain.html

    and

    http://superfluo.org/blojsom/blog/pic/devel/?permalink=Relay-mail-via-Google-SMTP-or-AuthSMTP-with-Postfix.html#smtp-mix-2

    19 December 2008, 12:02 pm

  32. Alberto:

    Do you get your Hard-Bounce backs? I find that when I use AuthSMPT I don’t receive hard bounces?

    for example if I email crazyhorse@ubs.com I dont receive a error letting be know that the email did not go thru.

    Can you send a test email at that address to check if you receive the bouncebacks.

    Thanks

    2 January 2009, 3:53 pm

  33. Paul Dowman:

    Alberto: I think the destination mail server should send a bounce message to the sender address, if that’s a valid address I’d expect that you should be able to receive it. If not, sorry, I don’t know!

    To everyone else:

    I apologize for not responding to some of the recent questions, mail server configuration is really (really!!) not my forte so I would need to investigate a bit to answer all but the simplest questions and I just don’t have the time to do that at the moment.

    Feel free to post questions for others with more time and knowledge than myself, but I probably won’t be able to answer most of them, sorry!

    2 January 2009, 4:32 pm

  34. AuthSMTP:

    Alberto - just to confirm we do send standard mailer-daemon style bounce messages - if you are not receiving them then most likely your incoming server is blocking them.

    19 January 2009, 9:51 am

  35. AuthSMTP:

    Zubair - there are many reasons why an email might be marked as [spam] by the recipient - typically the spam filter settings of the recipient and the content of the actual messages - it can also be marked that way your domain uses something like SPF but you have not updated the record to include our servers (very quick and easy to do). However, we can confirm that from a technical point of view all our servers have correct DNS / reverse DNS etc. so it is unlikely to be an issue with AuthSMTP. If you want us to check specifically please contact us via our web site.

    19 January 2009, 9:56 am

  36. AuthSMTP:

    Ezra - we check the ‘from’ addresses declared by your email application for very good reason - it stops people spoofing / faking email addresses and therefore stops abuse of the AuthSMTP service. If we were not to check the ‘from’ addresses it would get abused and people would blacklist our IP’s which would result in poor deliverability. Obviously servers that do not do checking potentially leave themselves more open to abuse and blocking / poorer deliverability!

    19 January 2009, 10:01 am

  37. Luc:

    FYI, for those who want to selectively choose the emails that you want postfix to relay via authsmtp, take a look at the sender_dependent_relayhost_maps option for postfix.

    19 January 2009, 2:15 pm

  38. Michael:

    This is a great resource (the post and the comments). Has anyone implemented this on EC2 Windows instances? Any other tricks or tips for that platform?

    12 February 2009, 6:06 pm

  39. John Clancy:

    I have successfully used this recipe to send email (thank you Paul!). Now I’d like to use this recipe to receive email –
    http://blog.craigambrose.com/past/2008/2/9/respond_toemail_or_how_to_handle/

    I’ve made all of the changes suggested by Craig Ambrose to main.cf, etc.

    And, I’ve enabled port 25 using AWS default settings so:
    netstat -an | grep LISTEN
    gives me
    tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN

    However, I can’t receive email and I can’t telnet to port 25.

    Any ideas? Is there anything in Paul’s main.cf that blocking the setup suggested by Craig?

    I’ve researched each line of the main.cf and as I’ve been able to understand it there don’t seem to be any conflicts.

    25 February 2009, 9:18 pm

  40. Robert Crowe:

    I’ve set up this configuration on an Ubuntu EC2 image, but the mail keeps going to AuthSMTP with:

    MAIL FROM: www-data@mydomain.com

    and then also a header:

    From: my-authsmtp-authorized-address

    I’m sending from PHP. AuthSMTP refuses to send the mail because of the first header, so the second header isn’t good enough for them for some reason, even though it’s a confirmed an authorized address. ARGH!

    19 March 2009, 4:33 pm

  41. Robert Crowe:

    Fixed it, here’s what I had to do. If you’re coming from Amazon EC2 you might need this.

    smtp_generic_maps = hash:/etc/postfix/generic in main.cf allowed me to remap the MAIL FROM envelope to a validated email address:

    /etc/postfix/generic:

    (username)@(server domainame) (email from address validated at AuthSMTP)
    (username) (email from address validated at AuthSMTP)

    then:

    % postmap generic

    then:

    % /etc/init.d/postfix restart

    19 March 2009, 8:27 pm

  42. Sascha Konietzke:

    Very good stuff Paul. I was slowly figuring out bits and peaces of the problem until I found your blog post which basically covers everything.

    To create a summary out of the postfix logs I can recommend pflogsumm together with cron.

    16 April 2009, 9:51 am

  43. Paul Dowman:

    Sascha: Thanks for the pflogsumm tip. Another option is logwatch, it’s a bit more general though so I don’t know how it compares in the detail of the report. It gives daily summaries of system activity, it reports on lots of things like disk space, apache logs, login activity, etc.

    16 April 2009, 11:47 am

  44. Josh White:

    Hello Paul, and thanks for the excellent post. I think I may be missing the mark on one thing though -

    “Rather than deliver directly to the AuthSMTP mail server from your web app it’s a good idea to deliver to a local queueing mail server, which will forward via the AuthSMTP gateway. Your web app will deliver mail to localhost (or perhaps a dedicated instance if you prefer), port 25.”
    What’s the best strategy for doing this in a dynamic environment? Multiple web app servers will be going up and down, so how can I configure postfix to forward to a queueing server?

    17 April 2009, 2:55 pm

  45. Josh White:

    Never mind. I figured it out, sorry for that!
    -josh

    17 April 2009, 2:57 pm

  46. Paul Dowman:

    Josh: As you probably guessed, by “local” I mean on the same instance. My app delivers to the mail server on localhost, which (almost instantly) relays it to the real mail server. That helps my app get it’s pages served up quickly.

    17 April 2009, 4:14 pm

  47. Subho:

    Hello Paul,

    Thanks for your gr8 post on authsmtp configuration. I am using an EC2 server for sending mails through authsmtp. I am not using any mail queue system; which I don’t think will be an issue for mails going to SPAM folder of hotmail / yahoo boxes. With yahoo is the main issue.

    Kindly suggest what more can I do to fix the issue.

    Thanks in advance.

    28 April 2009, 1:25 am

  48. Brian Ploetz:

    Thanks for this article Paul (and the awesome ec2onrails!)

    Question: I had been planning on using Amazon SQS to queue emails that need to be sent (not the actual email, but a small data structure of what email needs to be sent to whom). Then, using the activemessaging plugin (or similiar), I would pull things off the queue and actually send the email. The goal of all of this machinery is to keep the actual sending of the email out of the HTTP request/response cycle so Rails throughput doesn’t get bogged down with potentially long running actions. I have this all running locally and it works fine.

    However, I’m now wondering if this is all overkill if mail is delivered to a local Postfix server on an ec2 instance. Do you find the response times of the local Postfix adequate to keep Rails throughput acceptable? How do you deal with failures if Postfix goes down for some reason, such that you don’t lose emails?

    Thanks again.

    22 May 2009, 5:28 pm

  49. Paul Dowman:

    Brian,

    In general I’d say that queueing a small job which causes the email to be sent from another process is a good idea, especially if you’re queueing other things to speed up your page loads, but I’d recommend not using SQS for this because SQS is *extremely* slow (i.e. high latency). I find that queuing SQS messages frequently takes well over 100ms which is way too slow to do while responding to a web request.

    If you’re using a queue to speed up page loads, use a fast local queue. If the only thing you’re queueing is email sending, and you’re sending just one or two emails per web request, you might consider just using a local postfix instead for simplicity.

    To address your two questions: yes, I find the response time adequate to keep Rails throughput acceptable (though some queues would definitely be faster) and yes there’s a small chance that a message could be lost while in transit in the local postfix (e.g. if the instance dies), but the messages don’t stay in there for long, postfix sends them out quickly enough that it doesn’t build up a queue.

    22 May 2009, 9:35 pm

  50. Brian Ploetz:

    Hey Paul,

    Regarding the Postfix failures, I was referring to the case where Postfix has fallen over for some reason and your attempt to queue an outbound email thus fails…in other words, the message never gets queued to begin with. I suppose you would have to persist that message somewhere (db/local queue) to be sent again at a later time. But at that point you’re in the situation where you need an async process to pull things off of this failure queue, and then you’re kind of back at square one again, and why not just put things into a queue to begin with? Assuming your assertion of SQS latency is accurate, perhaps that’s a local fast queue instead.

    I realize these are edge cases, and these aren’t exactly financial transactions we’re talking about here either, so a lost email here and there isn’t exactly the end of the world. I’m just curious how other people out there have approached this problem in the context of Rails where throughput depends on getting in and out of the HTTP request/response cycle as quickly as possible.

    Thanks.

    22 May 2009, 10:40 pm

  51. Paul Dowman:

    Oh, you mean the postfix daemon has died and isn’t accepting SMTP connections. You don’t need to use SMTP to send mail from your app if the mail server on the same instance. You can use “sendmail style” delivery, which I’m pretty sure will work even if the postfix daemon is dead or isn’t accepting SMTP connections. The mail will be queued locally until postfix is running again.

    I haven’t tested that though, it’s just what I think will happen.

    23 May 2009, 8:17 am

  52. Rob:

    Awesome. Came in handy in a pinch. Thanks for the contribution.

    9 June 2009, 11:00 pm

Leave a comment