An eventful day. Besides upgrading Jira and Confluence, I also do upgrades of the underlying Ubuntu operating system. Today I had 8 AWS EC2 instances to upgrade, all running Ubuntu 16.04.6 LTS.
I did the usual steps, upgrading the sandboxes first:
3 of the 4 sandboxes failed to boot! They were all stuck on Grub rescue screen, showing:
Booting from Hard Disk 0...
error: symbol `grub_calloc' not found. Entering rescue mode...
grub rescue> _
Googling yielded this askubuntu.com post, which provides a general way forward: we need to reinstall grub with
grub-install <disk> .
How to reinstall grub on AWS
The joyous thing about AWS is you only get a screenshot of the
grub rescue> prompt. You can't actually rescue anything. (Edit: this wasn't a MacGuyver-rescue'able situation anyway.)
For AWS the process is:
- Launch a recovery t2.micro in the same AZ/subnet as your borked instance(s).
- Stop the broken instance and detach the root volume (the one containing the OS).
- Attach the root volume to the recovery instance as /dev/sdf:
Now run these commands:
It looked like this:
(Edit: added fs umounts per this LP comment)
- Detach the volume from the recovery instance
- Attach the volume back on the original instance.
- Reboot original instance
Then you're golden.
What caused the problem?
Edit: rewritten 31/July based on lp#1889509 comments)
On the 3 servers that broke,
apt-get update did not prompt me for anything grub-related. Checking afterwards with
debconf-get-selections I see that debconf was pre-configured to install grub on one device:
That device symlink is wrong. It was either pre-seeded by cloud-init or by the AMI creator (I'm not sure). My servers need grub on
/dev/xvdf1 . The grub postinst script would have encountered the same failure I saw while recovering:
I must not have noticed this error in the wall of text scrolling past on the upgrade.
Grub should have failed hard, but didn't (now been filed as https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1889556/ ).
The result: grub is upgraded in my
/boot partition, but the grub loader in
/dev/xvdb is still old, and the mismatch causes the failure (hat tip to ~juliank on lp#1889509).
To reinforce this theory, we turn to 1 of the 4 sandboxes did not break. When I did
apt-get upgrade the surviving server had previously demanded I give it some 'GRUB install devices':
I had decided to take debconf's advice, and installed grub on all devices:
and that saved me, at least on this one server.
But why did this update go wrong?
I upgrade OS packages on these servers every month, but looking at my
/var/log/dpkg.log history, grub is very rarely updated. The last update was a full year ago:
So I posit that there is nothing directly wrong with this grub update specifically, but rather I (and a lot of other people) are hitting a problem in the general Debian grub update process; specifically, when grub fails to update with an error:
grub-install: warning: File system `ext2` doesn't support embedding.
grub-install: warning: Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and their use is discouraged..
grub-install: error: will not proceed with blockists
it should fail hard, but instead proceeds.
How do I know if my server will break?
If your system boots with UEFI, you're fine.
If you're on Linode with default settings, you're fine.
For BIOS users, including AWS and other VPS hosters, run the following:
This script checks if your grub version includes a fix for lp #1889556 and if not, checks if you are likely to experience boot problems.
I just upgraded a 20.04 Ubuntu laptop without drama, but those all use BIOS, not EFI. So far I've only seen this affect BIOS users.
Edit: that is correct. Per ~juliank on lp #1889509:
This is not a problem on UEFI systems fwiw, as they do not use a small image in MBR and load the rest from /boot, but a single monolithic grub image in the ESP.
So here's my litmus test: run:
If you see nothing, you're fine: either:
- you're on EFI and unaffected
- you're on BIOS, but will be forcefully prompted to pick a device, as I was.
If you get a single device back:
then you may be in trouble (not always - see below re. Linode). You'd best run
dpkg-reconfigure grub-pc before rebooting, and install Grub on some other devices to be safe.
~mnordhoff on lp #1889509 says:
On Linode, I believe you should be safe unless the kernel is set to "Direct Disk". It doesn't matter if the GRUB installation works if it's not being used.
This is correct in my experience. My own servers are on linode, where the OS disk is /dev/sda and the above command prints:
When I run
dpkg-reconfigure grub-pc and pick
/dev/sda I get a scary warning, replicable from the command-line:
However despite all the warning signs, my VM rebooted fine.
Edit: I filed a launchpad bug: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1889509
Edit: The essential problem is that the
grub-pc package doesn't fail in the presence of bad debconf data. That has been fixed per https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1889556.
Edit2: Updated 'What caused the problem' section with info from comments on the bug.