...
Code Block |
---|
TSTAMP=$(date '+%Y-%m-%d-%H:%M:%S') mkdir -p ~/hack/typescripts/$TSTAMP script -q -t -a ~/hack/typescripts/$TSTAMP/typescript 2>~/hack/typescripts/$TSTAMP/timing |
...
On the server, as root, run:
Code Block |
---|
mkdir ~/hack/tcpdumps cd ~/hack/tcpdumps nohup tcpdump -i any -w %H%M -s 1500 -G $[60*60] & |
...
/var/log/apache2
or/var/log/nginx
/var/log/{secure*,syslog,auth.log*,kern.log}
/opt/atlassian/*/logs
/var/log/atop_*
/tmp
/var/log/journal
(if systemd journaling is enabled)~/hack/
(your terminal output and network captures so far)
Using rsync, this can be done with a command like:
...
Consider locking down other affected systems
You now have a locked down, backed up server. It is time to consider whether other systems might have been hacked too:now have a locked down, backed up server. It is time to consider whether other systems might have been hacked too:
Does Confluence store its user passwords on an external system, like AD, LDAP or Jira? Did Confluence have permission to instigate password resets? If yes and yes, that is bad news: your hacker may have reset passwords reused on other systems (e.g. Jira), and thereby accessed those other systems.
Expand title How do I tell if Confluence has read-write access to LDAP/AD/Jira? To tell:
- log in to Confluence as an administrator
go to the User Management admin section (type 'gg' then 'user management')
Confluence doesn't make our lives easy here. You'll probably see something like:
This doesn't tell us if our access to the 'JIRA Server' user directory is read-only of read/write. So click on the 'Directory Configuration Summary':
If you see a full set of 'Allowed Operations', as above, that means Confluence has permission to modify user passwords in Jira. A read-only Jira would have a much shorter list of allowed operations:Code Block Allowed operations: [UPDATE_USER_ATTRIBUTE, UPDATE_GROUP_ATTRIBUTE]
If your user directory is read-write for Confluence, then check if that system if any user passwords were reset, e.g. in Jira's audit log:
What could a malicious
confluence
user see in the system? Check the permissions of user directories in/home
. Are they world-readable/executable? If so, anything sensitive in those home directories may have been exposed.Tip Assuming you use systemd to launch Jira/Confluence, you should be running with ProtectSystem and ProtectHome parameters:
Code Block # Make /usr, /boot and /etc read-only. ProtectSystem=full # /home, /root and /run/usr should be inaccessible/empty. ProtectHome=yes
This ensures Jira/Confluence cannot see directories they don't need to.
Do not set
PrivateTmp=no
because that preventsjconsole
and friends from communicating with the Java process.- How are external backups done? Are credentials to the backup system compromised? If so, move to protect your backups before anything else.
In our case, external backups are stored on tarsnap. The backup process runs as root and the tarsnap key is only root accessible. I was fairly confidentroot
has not been compromised. - Are there usernames and passwords stored as plaintext in Confluence? If so, consider those systems breached too.
...
sha1 | 9a6ae3e9bca3e5c24961abf337bc839048d094ed |
md5 | b39d9cbe6c63d7a621469bf13f3ea466 |
Attack vector 1: Application-level vulnerabilities
Most of the time, breaches will be due to known security vulnerabilities, of which Jira and Confluence have a steady stream.
...
In my example, there are some hits, but fortunately all with 404 or 301 responses, indicating the JSPs do not exist:
Attack vector 2: User account compromises
Perhaps a user has had their password 's password has been guessed (e.g. by reusing it on other services - see https://haveibeenpwned.com), or succumbed or the user succumbed to a phishing attack and clicked on an XSRFed resource. If the account had administrator-level privileges, the attacker has full Confluence access, and possibly OS-level access (through Groovy scripts or a custom plugin).
...
- Check the audit log for suspicious admin activity, but be aware that the audit log is not trustable at this point.
Identify accounts whose password has recently changed, by comparing password hashes with that from a recent backup.
This command compares thecwd_user
table from a monthly backup to that from the currentconfluence
database:Code Block # vim -d <(pg_restore -t cwd_user --data-only /var/atlassian/application-data/confluence/backups/monthly.0/database/confluence) \ <(sudo -u postgres PGDATABASE=confluence pg_dump -t cwd_user --data-only)
(diffing database dumps like this is a generally useful technique, described here)
Check for users logging in from strange IPs, e.g. foreign countries or VPSes.
This lnav command prints a summary of Confluence access counts grouped by username and originating IP hostnameCode Block jturner@jturner-desktop:~/redradishtech.com.au/clients/$client/hack$ lnav var/log/apache2/confluence.$client.com.au/access.log* -c ";select count, cs_username, gethostbyaddr(c_ip) from (select distinct cs_username, c_ip, count(*) AS count from access_log group by 1,2 order by 3 desc limit 15) x;"
The originating IPs do not look suspicious for a small Australian business:
Attack vector 3: Lower-level vulnerabilities
It is possible the hack was doing through SSH account compromise, webserver vulnerability, Java vulnerability or something more exotic. Check w
and last
for suspicious logins, as well as dmesg
and /var/log/*.log
(with lnav) for errors.
...