This page is for anyone who has been tempted to add files (e.g. js, css) into JIRA's
atlassian-jira/ directory. It may come as a surprise, but JIRA suffers from the most basic of bugs: it doesn't serve static files correctly. More specifically, if Tomcat serves the file with
gzip compression, the
Content-Length: header is incorrectly set.
Let's pick on
jira.jboss.org for our demo, but you can use any public, non-Cloud JIRA instance fronted by Apache (not nginx, which seems to mask the problem). Try hitting https://jira.jboss.org/robots.txt. This loads, but hangs for 20 seconds.
If you are using Chrome, now view the cached response by viewing chrome://view-http-cache/https://jira.jboss.org/robots.txt:
The RESPONSE_INFO_TRUNCATED status indicates that the server stopped responding before the full
Content-Length number of bytes was received.
This can also be replicated on the command-line:
The problem is that
Content-Length is being set to the size of the file (669 bytes), not the gzip-compressed size.
JIRA's static files can be seen in the
atlassian-jira/ directory. For instance, there are a bunch in
/atlassian-jira/static/. Requesting any of these with compression results in the same hang:
Consequences of the bug
Naturally, any JIRA page that refers to a static file will be affected. The page will load, but then hang for 20 seconds or so as the browser waits for the last resource to load, before giving up.
The thing is, there are actually remarkably few direct references in JIRA. All the files in
/atlassian-jira/static/, for instance, are actually served through JIRA's file batching mechanism as
batch.js resources. That is why this bug has gone undetected for so many years.
So far I know of only three situations in which the bug / hang is triggered:
/robots.txt, as illustrated above.
- The JIRA 404 error page. Try it in your JIRA: add any rubbish to the end of the URL to trigger a 404 error. The 404 error page loads but then spins for 20 seconds. For instance, on https://jira.jboss.org/invalid (please don't click, I've abused them enough), the waterfall shows the 20s hang:
The 404 page hangs because it directly pulls in the static
/atlassian-jira/static/custom-file.js, then include it in a <script> tag on JIRA pages. Your JIRA will now suffer from mysterious 20 second hangs on pages that used to be fast. This is unfortunately how I encountered the bug.
Effects on browsers
The effect on browsers deserves a bit more unpacking. We'll use https://jira.jboss.org/robots.txt again as our example.
- In Firefox, the page loads, but hangs for 20s. Another hit, another 20 second hang.
- In Chrome (64.0 tested), the page loads, but hangs for 20s. On the second hit things get interesting:
Corruption! On a third hit, more corruption, but now there is no hang:
On the second and third hit, if you view to see the state of Chrome's cache, you'll see that the page is flagged as truncated:
After the third hit, Chrome has marked the corrupt cache entry as valid:
and henceforth all requests for the URL will serve binary gibberish at the end.
This unfortunate behaviour appears to be a Chrome bug (edit: now filed as Issue 423318). Searching for RESPONSE_INFO_TRUNCATED will bring up relevant hits, including a stale bug report (https://bugs.chromium.org/p/chromium/issues/detail?id=423318) and people complaining of the same behaviour (https://stackoverflow.com/questions/47311027/response-info-truncated-file-in-chrome-cache). I am not sure what effect this binary corruption has on browsers. Possibly none, as browsers are built to handle any rubbish thrown at them.
There are two takeaways:
- Now you know why JIRA's 404 page always seems slow.
I would file a bug, but Atlassian no longer accept bug reports on https://jira.atlassian.com, filed as bug - JRASERVER-66932Getting issue details... STATUS indirectly via support request
atlassian-jira, you can't rely on Tomcat to serve them. Better to serve the static files directly via your frontend webserver.