Page History

...

I know I shouldn't, but for some servers I have ForwardAgent yes  in SSH so I can easily jump between servers. Agent forwarding to a hacked server is a really bad idea, as the matrix.org experience illustrates. Turn agent forwarding off in your ~/.ssh/config before continuing.

SSH in and become root

ssh in and  sudo su -  if necessary.

Record your session

If we have to go tramping through a crime scene, let's at least record what we see. As soon as you SSH in to the server, run:

...

On the server, as root, run:

mkdir -p ~/hack/tcpdumps
cd ~/hack/tcpdumps
nohup tcpdump -i any -w %H%M -s 1500 -G $[60*60] &

This records all network activity on the server. This takes a few seconds to do, and may provide valuable evidence of e.g. data exfiltration.

...

Snapshot system

...

activity

Run:

mkdir -p ~/hack/
pstree -alp > ~/hack/pstree

cd ~/hack
curl https://busybox.net/downloads/binaries/1.21.1/busybox-x86_64 -o busybox
chmod +x ./busybox

Lock down the system

Do not shut down the server. Doing so would lose potentially critical information. In my case, the malicious scripts are running from /tmp/ , Do not shut down the server. Doing so would lose potentially critical information. In my case, the malicious scripts are running from /tmp/ , so restarting the server would lose them.

...

• /var/log/apache2 or /var/log/nginx 
• /var/log/{secure*,audit*,syslog,auth.log*,kern.log}
• /opt/atlassian/*/logs 
• /var/log/atop_* 
• /tmp
• /var/log/journal  (if systemd journaling is enabled)
• /var/log/journal   (if systemd journaling is enabled)/spool (crontabs)
• /var/mail (root@ emails)
• ~confluence/{.bash*,.profile,.pam_environment,.config,.local}  (assuming confluence  is the account running Confluence.
• ~/hack/  (your terminal output and network captures so far)

...

rsync -raR --numeric-ids root@hackedserver:--rsync-path='sudo rsync' ec2-user@hackedserver:{/tmp,/var/log/{secure*,syslog,auth.log*,kern.log},.... hackedserver-contents/

...

apache2,nginx,secure,audit,syslog,auth.log,kern.log}*,/var/spool*,/var/mail*,~/hack,~confluence/{.bash*,.profile,.pam_environment,.config,.local},/opt/atlassian/*/logs} hackedserver-contents/

Now if the server spontaneously combusts, you have at least salvaged what you could.

...