Blueprint for Outbound Messaging Enhancements

Outbound Queue Performance

We would like to ship out new messages in a timely manner, and also handle retrying failed messages, but avoid unnecessary overhead, and without doing time-consuming sends in request context where they will slow down response to the user.

Right now, inserting a message in the queue (send_by_pe_id) is unconditionally followed by kicking off the outbound sending task (which runs asynchronously, not in request context). That isn't just making an attempt to send the single new message, but rather processes the entire queue. Messages that are in the queue in order to be retried due to an earlier send failure will be retried along with the one new message.

To avoid a time hit to users during requests that currently include a message send (currently the known cases are all registration email), we can instead queue up those messages. If we do that, we need to deal with the consequences of sharing the queue between new and retry messages.

Consider these cases:

• Email sending is not a common activity for the site -- sends occur less frequently than the periodic task. In that case, kicking off an immediate run of the entire outbound task is not a significant overhead. If the periodic task is set to run infrequently compared to the frequency of new sends, then an immediate run makes entire sense. If new sends are not frequent, this isn't a lot of overhead.
• However, let's say that the site actively uses messaging -- that it's a common event.
  • Say new sends are significantly more frequent than the periodic task. Now the async runs will dominate, and the periodic task and the site's chosen periodic rate will be irrelevant. The conditions that cause a message to fail and be retained in the queue for retry are unlikely to clear up in a short time, especially if the periodic time was chosen to be appropriate for retries. There will be a lot of overhead just from the task processing and futile retries.
  • If the periodic task is scheduled frequently, there would, in fact, be little benefit from launching the queue processing on each send.

(Dominic points out that if each outbound send processes the entire queue including retries, then causing outbound sends could be used as a DoS attack. That works especially "well" if the sends are sure to fail and become retries.)

Email Error Handling

Web2py's Mail.send raises exceptions, contradicting its docstring. In particular, a common configuration error, failing to edit the example email sender, will cause a ticket when a user attempts to register. (Registration includes an email send even if no verification or approval is needed, as the user will get a welcome message in that case.) Since this is not a user error, but rather a misconfiguration that the site admin needs to deal with, it should not block registration nor show an error to the user. (See also ticket #439 -- we currently delete the user's temporary registration if there is an email error, but don't report anything to the site admin.)

We would prefer to have Web2py's Mail.send behavior fixed in Web2py, but this could be a backward-compatibility issue, and might take some time to resolve. So for now we'll subclass Mail (as MailS3) and show what we'd like done in Web2py. We can store our MailS3 instance in current.mail so it will be used by Web2py's Auth for those actions we don't override -- password change and username recovery.

Since we're subclassing Mail anyway, we can introduce other changes. In particular, we might want to use the outbound message queue rather than having the user wait for a direct send to be performed. This solves the issue #439 with configuration errors versus registration-related email. The user will not see any email errors, we can leave in their temporary registration, and the actual send during queue processing will log an error for the site admin to deal with. When the configuration is repaired, the message should be retried and sent without further intervention.

These enhancements need to be in MailS3.send so they're used by Web2py's Auth, but they're not what we want pushed as a fix to Web2py's send and we still need access to the direct, non-queued send for outbound queue processing, so the send fix will be included in MailS3 as send_direct_email.

A new user does not have a pentity while going through email verification, and the approver is denoted only by an email address, so in order to queue up those messages we would like to extend outbound queuing for...

Sending to addresses without pentities