This weekend I upgraded a Jira instance from one LTS release to the next (8.13.1 to 8.20.1).
Everything appeared to go well, but shortly afterwards users started complaining: Jira is not emailing certain users.
I checked, and sure enough: there's a situation where Jira 8.13.x and all prior releases would email, but Jira 8.20.x will not.
Specifically, this affects users like this one, who are not in a group like
jira-users blessed with 'Application access':
The 'helpdesk' user here is, per Crowdsourcing triage: A useful pattern for Jira issue ownership, an account that never logs in but does get emailed.
What should happen if a new issue is created, assigned to 'helpdesk', or with 'helpdesk' as a watcher?
- In Jira 8.13.x, a notification would go out to
- In Jira 8.20.x no notification goes out.
The 8.20.x behaviour is actually correct. An account without Browse permission shouldn't get emails. Yet this important bug fix is not one of the
Getting issues... fixed between 8.13.1 and 8.20.1 - at least, that I can see.
Anyhow, to know whether your Jira has any user accounts like this, run the following SQL (Postgres-flavoured):
If you get output, that's bad - those users will stop getting emails in 8.20.x+. Time to fix your group memberships, or bless the group your users are in (here 'MyCompany') with application access.
This page constitutes random notes from my work day as an Atlassian product consultant, put up in the vague hope they might benefit others. Expect rambling, reference to unsolved problems, and plenty of stacktraces. Check the date as any information given is likely to be stale.
Sometimes Confluence users will report that their attachments have disappeared, with an 'Unknown Attachment' placeholder image showing:
The weird thing is that the attachment(s) do actually exist, attached to the page. It's just the image reference in the page that's broken.
If you view the Confluence page source through the Source Editor plugin, you'll see something weird: the 'Unknown Attachment' isn't Confluence saying "hey, there's no attachment by this name any more". It is in fact an embedded image of the 'unknown-attachment' image:
How does this happen?
This is a result of a bug in the Confluence collaborative editor:
CONFSERVER-55928 - Getting issue details... STATUS
Am I affected?
To see if any of your Confluence's pages are affected by this, run this SQL on the database (Postgres-flavoured):
How to fix?
Since the image attachments are actually there, it's just a matter of editing the page and replacing the 'Unknown Attachment' images with the correct images.
Easier said than done sometimes, especially if the page has been edited many times after the images were corrupted. You'll need to loop tediously through the page history, finding the point at which the image(s) were deleted, then fix the current page's images.
This is exceptionally tedious, but some SQL can help ease the pain:
- First, find the
contentidIDs of affected pages, using the SQL above. Store the
contentidlist in a text file.
Now, for each
contentid(for example, 22432553) run this SQL:
This will give you three URLs: the current page, the first 'gone bad' revision, and the 'just before we went bad' revision:
- Open all three URLs in three browser tabs
- In the 'bad' tab, locate the 'Unknown Attachment' image and find some identifying text just above it. Perhaps the word Advanced in my example:
- On the 'good' tab search for the marker word. Below it you will see the correct image. Click on the image and note the end part of the URL, which will be the filename. Store the filename in your copy/paste buffer.
- On the 'current' tab:
- edit the page
- Search for your marker word
- Delete the 'Unknown Attachment' image
- Press '!' to begin inserting an image, and paste the image filename from your copy/paste buffer.
- Repeat process for however many broken images you find.
- Save the page.
Still tedious, but not so bad.
The takeaway from this article is that, if you run Jira Server, there is a config tweak you really ought to make, to allow the use of GCViewer to debug any future garbage collection problems:
For Jira, edit
/opt/atlassian/jira/bin/set-gc-params.sh and add make this change:
What's this all about?
Jira is written in Java, which is a "garbage-collected" language. This means that Java programmers don't need to worry about de-allocating memory; the Java virtual machine (JVM) periodically "garbage collects" any memory that is unused.
Garbage collection (GC) happens all the time in a running JVM, and typically takes only milliseconds. However if your JVM is under memory pressure, a full GC can't actually free up that much memory, and soon after another GC is needed, and another.. and you get a situation where the JVM is spending more time GC'ing than doing anything useful. This is called GC thrashing.
If you're lucky you'll see an
OutOfMemoryError in your
/opt/atlassian/jira/logs/catalina.out log file, making the diagnosis easy. But sometimes Jira will just be really slow, high CPU usage, and thread dumps showing a variety of threads doing normal CPU-bound things. but not quite enough to trigger an OutOfMemoryError) on garbage collector threads trying to clear up memory.
The best way to diagnose GC thrashing, or any "what the hell is Jira so busy with?" situation, is to use
atop ) with threads enabled, so you can see which Java thread(s) are the CPU consumers. I like
atop for its ability to rewind and view past states. Here is
atop after pressing 'y' to enable threads:
See in the CPU column: Jira is consuming all CPU (1092%), and specifically the only threads active (79% each) are ParGC Thread threads.
Out the box, Jira (8.7.1+) is configured with these GC configuration parameters:
If you are using Java 9 or above, this means the G1GC algorithm is in use (see Disruptive Changes to GC Logging in Java 9). Earlier versions of Jira used the older Parallel GC (see JRASERVER-64331 - Getting issue details... STATUS ), and look like this:
Either way, you should have a log file,
/opt/atlassian/jira/logs/atlassian-jira-log-gc-*.log. GC logs looking something like this:
The sample log above is from a thrashing server using the older ParallelGC algorithm. The lines to note are marked Pause Full. e.g.:
By subtracting the timestamps we see a 4.26s interval between full GCs (2465745.724 - 2465741.462), of which the GC occcupied 4.11s. This is what thrashing looks like.
Interpreting GC log files is exceptionally tedious - finding relevant lines, doing timestamp math, etc. If you did want to further analyze this GC log file, what tools could you use?
My tool of choice is GCViewer, because it has been around forever and does the job.
The problem alluded to in the introduction is that GCViewer cannot read the default format of GC logs generated by Jira and Confluence, when using Java 9+. You'll just get an error for each line parsed:
After making the
-Xlog:gc tweak suggested in the intro, and restarting Jira, GCViewer should be able to load the
(example from a perfectly healthy server, sorry)
Discovered after reading these GC tuning tips; https://gceasy.io is likely worth the $10/mo. Unlike GCViewer, no
-Xlog:gc tweaks are needed. The free tier is enough to give a general idea of GC activity: