In this article we discuss the deficiencies of Jira's outgoing email handling, and how you can avoid problems by configure Jira to delegate mail sending to a local Postfix server (part 2). While we use Jira for concreteness, everything said here applies equally to Confluence. |
Per the documentation, Jira can be configured to send notification emails:
In this example, you'll see my 'Host' is smtp.gmail.com
, to which I authenticate as a dedicated jira
account. This means all outgoing email will be relayed through gmail.
This article will try to convince you not to do that. Instead you should install Postfix on your Jira server, and use Postfix as Jira's relay:
By all means, configure Postfix to relay through gmail.com if you like – just don't let Jira itself do the delivery.
There are three broad reasons for this:
The benefits of Postfix will become more apparent in part 2 – for now we just focus on the problems with Jira's status quo.
Notifications get batched in a queue and sent once a minute. Normally Jira's outgoing email works fine. But every now and then something breaks, and Jira stops sending notifications.
You might first notice this as general brightening of the office mood, and increase in productivity. But eventually some manager will complain that they're not getting updates, and you are tasked to investigate.
You will probably find Jira's mail queue is an unhealthy red, and the Error queue is full:
What causes this? Perhaps:
jira@
)Whatever the immediate cause, let's take a moment to reflect on the general awfulness of Jira's mail queue.
If some over-eager administrator (or just you in a hurry) had restarted Jira while the mail queue was red, all those blocked notifications would be gone forever.
Do lost notifications really matter? I'd say yes, because many people treat their inbox as a todo list. Jira notifications are how they know a task needs doing.
To give credit to Atlassian, only failing emails are lost. The mail queue is flushed as part of Jira's shutdown process (a good reason not to kill -9
your Java process).
Users sometimes complain that they never received a notification email from Jira. Perhaps a project's notification scheme was wrong? Perhaps Jira did send the email, but it was lost or marked as spam? How are you to tell?
What you need is an audit trail, showing what emails Jira actually sent. Sadly Jira does not provide one.
Jira's logging & profiling page does have a outgoing mail log turned ON by default:
If you thought this meant outgoing emails are logged, you'd be wrong: only errors are logged. See
Other problems become apparent over time. On we have the original lead developers of Confluence and Jira calling the mail error queue "useless":
The tickets were created in 2005 but fear not! they are "Gathering Interest".
Jira assumes you will authenticate to your relay with a username and password:
That is (sadly) no longer a good assumption. For instance, GSuite will stop supporting password-based authentication from :
Starting February 15, 2021, G Suite accounts will only allow access to apps using OAuth. Password-based access will no longer be supported. - GSuite
By then you will need to have upgraded to Jira 8.10.0, (released - release notes), which supports incoming OAuth. If you don't plan to upgrade, switching to Postfix or another MTA with OAUTH2 support is your only option.
When you configure Jira's Outgoing Mail Server, your connection details are stored in the database:
jira=> select id, name, mailfrom, server_type, protocol, mailusername, mailpassword from mailserver; ┌───────┬───────────────────┬──────────────────┬─────────────┬──────────┬──────────────────┬──────────────┐ │ id │ name │ mailfrom │ server_type │ protocol │ mailusername │ mailpassword │ ├───────┼───────────────────┼──────────────────┼─────────────┼──────────┼──────────────────┼──────────────┤ │ 10000 │ localhost │ jira@example.com │ smtp │ smtp │ jira │ hunter2 │ │ 10100 │ jira@ Mail Server │ ␀ │ pop │ imaps │ jira@example.com │ hunter2 │ └───────┴───────────────────┴──────────────────┴─────────────┴──────────┴──────────────────┴──────────────┘ (2 rows) |
First, the obvious problem: these credentials are in plaintext, and can be seen by:
In serious installations, one generally has a 'sandbox' (or 'staging') Jira for experiments and testing. The sandbox Jira data is periodically refreshed from production.
One requirement of sandbox Jira is that it must not be allowed to email real users. People get really confused if they receive notifications "from Jira" that were actually just experiments on sandbox.
However, when we restore our production data to sandbox, our SMTP details come with it:
What is to stop our sandbox Jira sending emails through smtp.gmail.com just like production?
The usual answer is: you set the -Datlassian.mail.senddisabled=true
flag to prevent emails being sent, and/or by blocking outgoing connects to 25/465/587 at the firewall (since plugins might send email directly).
But how much safer and cleaner is it to just not embed those details in the database in the first place.
In part 2 we will discuss a further advantage: on sandbox we can configure Postfix to 'blackhole' outgoing emails, i.e. send them to a local mailbox (regardless of true destination). This lets you inspect JIRA's mail output, without the risk of spamming real users.
When configuring the outgoing mail server, instead of configuring details directly one has the option of entering a JNDI Location:
What is a JNDI Location?A JNDI Location is a URL-like address specifying a configuration object, typically a database (e.g. This requires a tiny bit of background knowledge. You may be aware that JIRA runs inside 'Tomcat'. When you start JIRA (e.g. via
JIRA doesn't have to know or care about HTTP details, it just gets given a nice object. This is the Inversion of Control principle in software design, and Tomcat applies it to other things too:
Since there might be multiple databases and email servers configured in Tomcat (e.g. two mail servers, It helps to see this in practice. Here is a Tomcat
|
Using JNDI also avoids the two problems discussed above. With the JNDI method, your email credentials are kept a single file ( conf/server.xml
) on the server. not in the JIRA database. It is much easier to secure a single file than dozens of backups spread across multiple servers. Technically, you're configuring the appserver, Tomcat, with the password, not JIRA: JIRA just gets to use authenticated connections that Tomcat provides it.
With no passwords in the JIRA database, you also don't have to worry that someone will restore your data in a staging or dev JIRA, which then starts sending notification emails and filter subcriptions to people with stale data.
The downside of keeping SMTP details in conf/server.xml
is that you must remember to transfer these details across (plus copy some jar files) every time you upgrade JIRA. Also, the initial configuration and any subsequent changes (e.g. password resets) require a JIRA restart.
In general, storing SMTP credentials in Postfix has all the advantages of JNDI, and none of the downsides.
Jira's outgoing mail handling is a combination of bad design (storing SMTP credentials in Jira) and bad execution (Jira's half-baked mail queue). Read on in part 2, where we discuss a better alternative.