When Early Is Too Early

This will be a rant rather than an educational, well thought article. Sorry for that. I write this chiefly as a reminder to myself; a public, shameful mental note that I don’t want to do things this way anymore. The question you should ask yourself if you decide to join me in the next few lines is: when is upgrading early too early?

When Upgrading Early Is Too Early

Fedora is my favourite distro and I use it as my daily driver both at work and at home because it’s sensible, bleeding edge but reasonably stable too. Its 26th revision this past July brought a number of good improvements upon its predecessor among which we could count the adoption of Wayland as the default backend for Fedora’s GNOME installation and an additional disk partition wizard for intermediate users (that is, more experienced than a beginner but not quite advanced either) built into Anaconda, Fedora’s LiveCD installation wizard. In addition to this, some smaller changes worth mentioning would be newer releases of Go, Ruby, GCC, glibc, etc., as well as the second major release of Fedora’s new package manager frontend, DNF.

The table above lists the key dates of the Fedora 26 development lifecycle. North of 4 months of work where spent solidifying this release. Now, was this enough time to cement it and allow Fedora 25 users to enjoy a seamless, smooth transition to the future? Not for me.

I am an eager folk when it comes to getting new stuff. You could label me as an updates enthusiast. I was excited about this release so I decided to install it on the same day it officially launched thinking that, now the beta period was over, things would be very usable and stable by now. Oh boy was I wrong…

According to the Fedora wiki performing a system upgrade via DNF is almost as simple as counting 1, 2, 3:

1. Make sure the DNF cache is up-to-date: # dnf upgrade --refresh
2. Install DNF’s system upgrade plugin: # dnf install dnf-plugin-system-upgrade
3. Download the new release: # dnf system-upgrade download --refresh --releasever=26
4. Flip on the “upgrade” flag and reboot: # dnf system-upgrade reboot
5. ???
6. Profit!

The installer (upgrader?) took a while to finish its deed but nothing out of the ordinary occurred whilst the names of all the packages installed on my system took turns to appear on the dark, sober background of the Fedora installer screen. When I was finally allowed back in my computer, it didn’t take long to realise something was array. I decided to check how DNF was doing (a bit of a ritual after a system upgrade)

# I don't have any actual logs/stdout to illustrate this so we'll have to make do with
# this conservative reconstruction of the events
sudo dnf upgrade --refresh
ERROR

Error! Alright, what is it then? Something to do with some Python system libraries it says? That looks unfortunate… perhaps the upgrade didn’t go so well after all. It’s been a while since I had this problem and therefore I don’t remember all the details but I do remember DNF was not the only thing that was failing, but quite a lot more. This drove me to a state of uneasiness. Dumbfounded, I decided to browse Fedora’s bug tracker and even Red Hat’s, looking for any recent bugs related to Anaconda, Python or the F26 upgrade process. Nada. Either no-one had seen this problem before or I was paying the price for my hastiness and became the first one to come across them. What now?

The only pragmatic thing to do at this point was a system re-install. No need to do a full one as I’ve got my root directory mounted on its own partition precisely to allow for partial filesystem formats like the one I was about to do. The plan was simple: get a fresh copy of the Fedora 26 ISO, boot into it and crush the botched system install.

Presto, I pointed my browser to spins.fedoraproject.org and frantically searched for the KDE flavour of the distro, my version of choice - bam! I downloaded it, perhaps 1GB and a half, perhaps a bit less, can’t remember; grabbed my trusty USB stick, slammed a definitive dd of=/dev/sdSticky if=~/Downloads/kde-fedora-best-fedora.iso bs=4M status=progress onto it and a few minutes later I was ready… To have a bad dream.

When I rebooted my computer into the newly burned Fedora 26 LiveUSB, everything seemed smooth and familiar. The beautiful, white Plasma workspace, the installer, the endearing slowness of a SquashFS image loading via a USB port. After a few minutes of messing around with my disks’ layout, making every possible effort not to accidentally trigger a full disk reformat, I instructed the installer to proceed with the changes. Everything seemed to be progressing until the last few steps, where the initramfs image is created and the GRUB entries are written to /boot. It was at this point that Anaconda announced it had encountered a serious error in its Python libraries and that it had failed to complete the installation. I OK’d the warning and decided to reboot, hoping it was just pulling my leg, but the ominous black screen that greeted me right after made it clear Anaconda was not being funny. Somehow I or the installer managed to corrupt the /boot partition and GRUB had no idea what was going on. Right, let’s reinstall again. I’ll be extra careful this time!

Honestly, I can’t even remember how many times I repeated this cycle and its many possible variations. Perhaps downgrading to F25 would help? No, try again? Still seems like a ’no’. Before I realised it, I’d lost track of time and entered a frenzy of mindless iteration of trials and errors until I was lucky enough to eventually hit the right combination and get my system back. The upgrade was now complete, right? Not quite. Now my Windows boot entry in GRUB was gone. Rolled up sleeves again, let’s figure this out… Ah, I know what’s happened! I’ve wiped the /boot/efi partition so Windows won’t be able to boot anymore. So, how do you restore Windows’ bootloader if you’re using UEFI, then? In case you’re in the same situation as I was and can’t find the answer anywhere else, read on.

Burn a Windows ISO on a USB stick (e.g. with unetbootin), boot into it and get a command prompt open by selecting the “Advanced Diagnostics and Troubleshooting” option from the “Repair” menu. Now run in the following commands:

diskpart
list disk
sel disk <disk_num> # Select the disk containing your boot partition, i.e. /boot/efi
list vol
sel vol <vol_num> # Select the vol containing /boot/efi
# In my case I assigned the UEFI letter to be B:
assign letter=<System_drive_letter>:
exit
# Windows drive letter was D:\ in my case. Note this is installed Windows drive letter, not   the recovery USB's
bcdboot <Windows_drive_letter>:\windows /s <System_drive_letter>: /f UEFI /v # /f sets firmware to UEFI
exit
pray_to_the_norse_gods_old_and_new

That was it for Windows. But what about Fedora? How did it turn out after the upgrade? Well, not that bad, to be frank.

The main problem I experienced on Fedora during the aftermath originated from Konsole, KDE’s default terminal emulator, and my terminal of choice. Konsole was taking inordinate amounts of time to autocomplete folder contents. Ten seconds would be a best-case scenario for it. After a couple of days of investigation, the answer came to me while I was toying around with perf trace, as lines and lines of output scrolled past the screen. It was then when I caught a glimpse of a few weird calls to libnotify from Konsole. That seemed weird since after the reformat I no longer had any desktop notification daemon installed (I use i3, which doesn’t come with one, you see). On a hunch, I decided to try a different terminal emulator. So I tried urxvt and lo and behold, the issue disappeared. I also tried with gnome-terminal and no problems at all! Could it be then that Konsole was trying to fire off a desktop notification via dbus/systemd but timing out because there was no libnotify daemon installed to respond to the request? Sounded weird but worth a shot. So I installed dunst, which is the desktop notifications daemon I was using before the upgrade and, as I suspected, autocomplete slowness in Konsole was gone! But why? Beats me. I promised I’d take this problem to the KDE team so someone could get to the bottom of it but still haven’t managed to do so.

Furthermore, my Spotify Linux client has taken a massive hit in startup performance since the upgrade, as though it had acquired the necessity to “warm up” before being able to function correctly. Neither clearing the app cache nor re-installing helped a bit so I’ve given up on it.

Several package reinstalls later I managed to get my system back in shape and running on the latest version of Fedora, and here I am now, writing this from the protagonist of my tale. So… Was Fedora 26 worth the wait? Yes. Was it worth the rush? No.

I guess the morale to this story, the only useful piece of knowledge you may take away from this rambling, is that stability is good and you don’t need to rush things. It’s best to let them settle for a while before committing to them. You’ll save yourself a lot of grief.

P.S. I scanned through the current list of common bugs reported for Fedora 26 so far during the creation of this article (it would certainly not be as dramatic an account if my woes could’ve easily be remedied by me bothering to take a good look at the resources available to assist me) but even though I found a couple of good candidates, nothing quite like what I suffered that night. Again, my point is not that my troubles were rare and unfixable; rather, that being an early adopter certainly comes at a cost, as sometimes you end up being the first person experiencing them, so there’ll be nobody there to help you.