Setting up a local caching proxy for Fedora YUM repositories

Posted: December 9th, 2015 | Filed under: Coding Tips, Fedora, OpenStack, Virt Tools | Tags: , , , , | 3 Comments »

For my day-to-day development work I currently have four separate physical servers, one old x86_64 server for file storage, two new x86_64 servers and one new aarch64 server. Even with a fast fibre internet connection, downloading the never ending stream of Fedora RPM updates takes non-negligible time. I also have cause to install distro chroots on a reasonably frequent basis for testing various things related to containers & virtualization, which involves yet more RPM downloads. So I decided it was time to investigate the setup of a local caching proxy for Fedora YUM repositories. I could have figured this out myself, but I fortunately knew that Matthew Booth had already setup exactly the kind of system I wanted, and he shared the necessary config steps that are outlined below.

The general idea is that we will reconfigure the YUM repository location on each machine needing updates to point to a local apache server, instead of the Fedora mirror manager metalink locations. This apache server will be setup using mod_proxy to rewrite requests to point to the offsite upstream download location, but will also be told to use a local squid server to access the remote site, thereby caching the downloads.

Apache setup

Apache needs to be installed, if not already present:

# dnf install httpd

A new drop-in config file addition for apache is created with two mod_proxy directives. The ProxyPass directive tells apache that any requests for http://<our-ip>/fedora/* should be translated into requests to the remote site http://dl.fedoraproject.org/pub/fedora/linux/*. The ProxyRemote directive tells apache that it should not make direct connections to the remote site, but instead use the local proxy server running on port 3128. IOW, requests that would go to dl.fedoraproject.org will instead get sent to the local squid server.

# cat > /etc/httpd/conf.d/yumcache.conf <<EOF
ProxyPass /fedora/ http://dl.fedoraproject.org/pub/fedora/linux/
ProxyPass /fedora-secondary/ http://dl.fedoraproject.org/pub/fedora-secondary/
ProxyRemote * http://localhost:3128/
EOF

The ‘fedora-secondary’ ProxyPass is just there for my aarch64 machine – not required if you are x86_64 only

The out of the box SELinux configuration prevents apache from making network requests, so it is necessary to toggle a SELinux boolean flag before starting apache

# setsebool httpd_can_network_relay=1

With that done, we can start apache and set it to run on future boots too

# systemctl start httpd.service
# systemctl enable httpd.service

Squid setup

Squid needs to be installed, if not already present:

# dnf install squid

The out of the box configuration for squid needs a few small tweaks to optimize it for YUM repo mirroring. The default cache replacement policy purges the least recently used objects from the cache. This is not ideal for YUM repositories – if the YUM update needs 100 RPMS downloading and only 95 of the fit in cache, by the time the last package is downloaded we’ll be pushing the first package out of cache again, which means the next machine will have cache miss. The LFUDA policy keeps popular objects in the cache regardless of size and optimizes the byte hit rate at expense of object hit rate. Some RPMS can be really rather large, so the default maximum object size of 4 MB is totally inadequate, increasing it to 8 GB is probably overkill but will ensure we always attempt to cache any RPM regardless of its size. The cache_dir directive is there to tell squid to use threads for accessing objects to give greater concurrency. The last two directives are critical telling squid not to cache the repomd.xml files whose contents change frequently – without this you’ll often YUM trying to fetch outdated repo data files which no longer exist

# cat >> /etc/squid/squid.conf <<EOF
cache_replacement_policy heap LFUDA
maximum_object_size 8192 MB
cache_dir aufs /var/spool/squid 16000 16 256 max-size=8589934592
acl repomd url_regex /repomd\.xml$
cache deny repomd
EOF

With that configured, squid can be started and set to run on future boots

# systemctl start squid.service
# systemctl enable squid.service

Firewall setup

If a firewall is present on the cache machine, it is necessary to allow remote access to apache. This can be enabled with a simple firewall-cmd instruction

# firewall-cmd --add-service=http --permanent

Client setup

With the cache server setup of the way, all that remains is to update the Fedora YUM config files on each client machine to point to the local server. There is a convenient tool called ‘fedrepos’ which can do this, avoiding the need to open an editor and change the files manually.

# dnf install fedrepos
# fedrepos baseurl http://yumcache.mydomain/fedora --no-metalink

NB on the aarch64 machine, we need to point to fedora-secondary instead

# fedrepos baseurl http://yumcache.mydomain/fedora-secondary --no-metalink

Replace ‘yumcache.mydomain’ with the hostname or IP address of the server running the apache+squid cache of course. If the cache is working as expected you should see YUM achieve 100 MB/s download speed when it gets a cache hit.

What DevStack does to your host when setting up OpenStack on Fedora 17

Posted: November 20th, 2012 | Filed under: Fedora, libvirt, OpenStack, Virt Tools | Tags: , , , , , , , | 2 Comments »

As I mentioned in my previous post, I’m not really a fan of giant shell scripts which ask for unrestricted sudo access without telling you what they’re going todo. Unfortunately DevStack is one such script :-( So I decided to investigate just what it does to a Fedora 17 host when it is run. The general idea I had was

  • Install a generic Fedora 17 guest
  • Create a QCow2 image using the installed image as its backing file
  • Reconfigure the guest to use the QCow2 image as its disk
  • Run DevStack in the guest
  • Compare the contents of the original installed image and the DevStack processed image

It sounded like libguestfs ought to be able to help out with the last step, and after a few words with Rich, I learnt about use of virt-ls for exactly this purpose. After trying this once, it quickly became apparent that just comparing the lists of files is quite difficult because DevStack installs a load of extra RPMs with many 1000’s of files. So to take this out of the equation, I grabbed the /var/log/yum.log file to get a list of all RPMs that DevStack had installed, and manually added them into the generic Fedora 17 guest base image. Now I could re-run DevStack again and do a file comparison which excluded all the stuff installed by RPM.

RPM packages installed (with YUM)

  • apr-1.4.6-1.fc17.x86_64
  • apr-util-1.4.1-2.fc17.x86_64
  • apr-util-ldap-1.4.1-2.fc17.x86_64
  • augeas-libs-0.10.0-3.fc17.x86_64
  • binutils-2.22.52.0.1-10.fc17.x86_64
  • boost-1.48.0-13.fc17.x86_64
  • boost-chrono-1.48.0-13.fc17.x86_64
  • boost-date-time-1.48.0-13.fc17.x86_64
  • boost-filesystem-1.48.0-13.fc17.x86_64
  • boost-graph-1.48.0-13.fc17.x86_64
  • boost-iostreams-1.48.0-13.fc17.x86_64
  • boost-locale-1.48.0-13.fc17.x86_64
  • boost-program-options-1.48.0-13.fc17.x86_64
  • boost-python-1.48.0-13.fc17.x86_64
  • boost-random-1.48.0-13.fc17.x86_64
  • boost-regex-1.48.0-13.fc17.x86_64
  • boost-serialization-1.48.0-13.fc17.x86_64
  • boost-signals-1.48.0-13.fc17.x86_64
  • boost-system-1.48.0-13.fc17.x86_64
  • boost-test-1.48.0-13.fc17.x86_64
  • boost-thread-1.48.0-13.fc17.x86_64
  • boost-timer-1.48.0-13.fc17.x86_64
  • boost-wave-1.48.0-13.fc17.x86_64
  • ceph-0.44-5.fc17.x86_64
  • check-0.9.8-5.fc17.x86_64
  • cloog-ppl-0.15.11-3.fc17.1.x86_64
  • cpp-4.7.2-2.fc17.x86_64
  • curl-7.24.0-5.fc17.x86_64
  • Django-1.4.2-1.fc17.noarch
  • django-registration-0.7-3.fc17.noarch
  • dmidecode-2.11-8.fc17.x86_64
  • dnsmasq-utils-2.63-1.fc17.x86_64
  • ebtables-2.0.10-5.fc17.x86_64
  • euca2ools-2.1.1-2.fc17.noarch
  • gawk-4.0.1-1.fc17.x86_64
  • gcc-4.7.2-2.fc17.x86_64
  • genisoimage-1.1.11-14.fc17.x86_64
  • git-1.7.11.7-1.fc17.x86_64
  • glusterfs-3.2.7-2.fc17.x86_64
  • glusterfs-fuse-3.2.7-2.fc17.x86_64
  • gnutls-utils-2.12.17-1.fc17.x86_64
  • gperftools-libs-2.0-5.fc17.x86_64
  • httpd-2.2.22-4.fc17.x86_64
  • httpd-tools-2.2.22-4.fc17.x86_64
  • iptables-1.4.14-2.fc17.x86_64
  • ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch
  • iscsi-initiator-utils-6.2.0.872-18.fc17.x86_64
  • kernel-headers-3.6.6-1.fc17.x86_64
  • kpartx-0.4.9-26.fc17.x86_64
  • libaio-0.3.109-5.fc17.x86_64
  • libcurl-7.24.0-5.fc17.x86_64
  • libmpc-0.9-2.fc17.2.x86_64
  • libunwind-1.0.1-3.fc17.x86_64
  • libusal-1.1.11-14.fc17.x86_64
  • libvirt-0.9.11.7-1.fc17.x86_64
  • libvirt-client-0.9.11.7-1.fc17.x86_64
  • libvirt-daemon-0.9.11.7-1.fc17.x86_64
  • libvirt-daemon-config-network-0.9.11.7-1.fc17.x86_64
  • libvirt-daemon-config-nwfilter-0.9.11.7-1.fc17.x86_64
  • libvirt-python-0.9.11.7-1.fc17.x86_64
  • libwsman1-2.2.7-5.fc17.x86_64
  • lzop-1.03-4.fc17.x86_64
  • m2crypto-0.21.1-8.fc17.x86_64
  • mod_wsgi-3.3-2.fc17.x86_64
  • mx-3.2.3-1.fc17.x86_64
  • mysql-5.5.28-1.fc17.x86_64
  • mysql-libs-5.5.28-1.fc17.x86_64
  • MySQL-python-1.2.3-5.fc17.x86_64
  • mysql-server-5.5.28-1.fc17.x86_64
  • netcf-libs-0.2.2-1.fc17.x86_64
  • numad-0.5-4.20120522git.fc17.x86_64
  • numpy-1.6.2-1.fc17.x86_64
  • parted-3.0-10.fc17.x86_64
  • perl-AnyEvent-5.27-7.fc17.noarch
  • perl-AnyEvent-AIO-1.1-8.fc17.noarch
  • perl-AnyEvent-BDB-1.1-7.fc17.noarch
  • perl-Async-MergePoint-0.03-7.fc17.noarch
  • perl-BDB-1.88-5.fc17.x86_64
  • perl-common-sense-3.5-1.fc17.noarch
  • perl-Compress-Raw-Bzip2-2.052-1.fc17.x86_64
  • perl-Compress-Raw-Zlib-2.052-1.fc17.x86_64
  • perl-Config-General-2.50-6.fc17.noarch
  • perl-Coro-6.07-3.fc17.x86_64
  • perl-Curses-1.28-5.fc17.x86_64
  • perl-DBD-MySQL-4.020-2.fc17.x86_64
  • perl-DBI-1.617-1.fc17.x86_64
  • perl-Encode-Locale-1.02-5.fc17.noarch
  • perl-Error-0.17016-7.fc17.noarch
  • perl-EV-4.03-8.fc17.x86_64
  • perl-Event-1.20-1.fc17.x86_64
  • perl-Event-Lib-1.03-16.fc17.x86_64
  • perl-Git-1.7.11.7-1.fc17.noarch
  • perl-Glib-1.241-2.fc17.x86_64
  • perl-Guard-1.022-1.fc17.x86_64
  • perl-Heap-0.80-10.fc17.noarch
  • perl-HTML-Parser-3.69-3.fc17.x86_64
  • perl-HTML-Tagset-3.20-10.fc17.noarch
  • perl-HTTP-Date-6.00-3.fc17.noarch
  • perl-HTTP-Message-6.03-1.fc17.noarch
  • perl-IO-AIO-4.15-1.fc17.x86_64
  • perl-IO-Async-0.29-7.fc17.noarch
  • perl-IO-Compress-2.052-1.fc17.noarch
  • perl-IO-Socket-SSL-1.66-1.fc17.noarch
  • perl-IO-Tty-1.10-5.fc17.x86_64
  • perl-LWP-MediaTypes-6.01-4.fc17.noarch
  • perl-Net-HTTP-6.02-2.fc17.noarch
  • perl-Net-LibIDN-0.12-8.fc17.x86_64
  • perl-Net-SSLeay-1.48-1.fc17.x86_64
  • perl-POE-1.350-2.fc17.noarch
  • perl-Socket6-0.23-8.fc17.x86_64
  • perl-Socket-GetAddrInfo-0.19-1.fc17.x86_64
  • perl-TermReadKey-2.30-14.fc17.x86_64
  • perl-TimeDate-1.20-6.fc17.noarch
  • perl-URI-1.60-1.fc17.noarch
  • ppl-0.11.2-8.fc17.x86_64
  • ppl-pwl-0.11.2-8.fc17.x86_64
  • pylint-0.25.1-1.fc17.noarch
  • python-amqplib-1.0.2-3.fc17.noarch
  • python-anyjson-0.3.1-3.fc17.noarch
  • python-babel-0.9.6-3.fc17.noarch
  • python-BeautifulSoup-3.2.1-3.fc17.noarch
  • python-boto-2.5.2-1.fc17.noarch
  • python-carrot-0.10.7-4.fc17.noarch
  • python-cheetah-2.4.4-2.fc17.x86_64
  • python-cherrypy-3.2.2-1.fc17.noarch
  • python-coverage-3.5.1-0.3.b1.fc17.x86_64
  • python-crypto-2.6-1.fc17.x86_64
  • python-dateutil-1.5-3.fc17.noarch
  • python-devel-2.7.3-7.2.fc17.x86_64
  • python-docutils-0.8.1-3.fc17.noarch
  • python-eventlet-0.9.17-1.fc17.noarch
  • python-feedparser-5.1.2-2.fc17.noarch
  • python-gflags-1.5.1-2.fc17.noarch
  • python-greenlet-0.3.1-11.fc17.x86_64
  • python-httplib2-0.7.4-6.fc17.noarch
  • python-iso8601-0.1.4-4.fc17.noarch
  • python-jinja2-2.6-2.fc17.noarch
  • python-kombu-1.1.3-2.fc17.noarch
  • python-lockfile-0.9.1-2.fc17.noarch
  • python-logilab-astng-0.23.1-1.fc17.noarch
  • python-logilab-common-0.57.1-2.fc17.noarch
  • python-lxml-2.3.5-1.fc17.x86_64
  • python-markdown-2.1.1-1.fc17.noarch
  • python-migrate-0.7.2-2.fc17.noarch
  • python-mox-0.5.3-4.fc17.noarch
  • python-netaddr-0.7.5-4.fc17.noarch
  • python-nose-1.1.2-2.fc17.noarch
  • python-paramiko-1.7.7.1-2.fc17.noarch
  • python-paste-deploy-1.5.0-4.fc17.noarch
  • python-paste-script-1.7.5-4.fc17.noarch
  • python-pep8-1.0.1-1.fc17.noarch
  • python-pip-1.0.2-2.fc17.noarch
  • python-pygments-1.4-4.fc17.noarch
  • python-qpid-0.18-1.fc17.noarch
  • python-routes-1.12.3-3.fc17.noarch
  • python-setuptools-0.6.27-2.fc17.noarch
  • python-sphinx-1.1.3-1.fc17.noarch
  • python-sqlalchemy-0.7.9-1.fc17.x86_64
  • python-suds-0.4.1-2.fc17.noarch
  • python-tempita-0.5.1-1.fc17.noarch
  • python-unittest2-0.5.1-3.fc17.noarch
  • python-virtualenv-1.7.1.2-2.fc17.noarch
  • python-webob-1.1.1-2.fc17.noarch
  • python-wsgiref-0.1.2-8.fc17.noarch
  • pyxattr-0.5.1-1.fc17.x86_64
  • PyYAML-3.10-3.fc17.x86_64
  • qemu-common-1.0.1-2.fc17.x86_64
  • qemu-img-1.0.1-2.fc17.x86_64
  • qemu-system-x86-1.0.1-2.fc17.x86_64
  • qpid-cpp-client-0.18-5.fc17.x86_64
  • qpid-cpp-server-0.18-5.fc17.x86_64
  • radvd-1.8.5-3.fc17.x86_64
  • screen-4.1.0-0.9.20120314git3c2946.fc17.x86_64
  • scsi-target-utils-1.0.24-6.fc17.x86_64
  • seabios-bin-1.7.1-1.fc17.noarch
  • sg3_utils-1.31-2.fc17.x86_64
  • sgabios-bin-0-0.20110622SVN.fc17.noarch
  • spice-server-0.10.1-5.fc17.x86_64
  • sqlite-3.7.11-3.fc17.x86_64
  • tcpdump-4.2.1-3.fc17.x86_64
  • vgabios-0.6c-4.fc17.noarch
  • wget-1.13.4-7.fc17.x86_64
  • xen-libs-4.1.3-5.fc17.x86_64
  • xen-licenses-4.1.3-5.fc17.x86_64

Python packages installed (with PIP)

These all ended up in /usr/lib/python2.7/site-packages :-(

  • WebOp
  • amqplib
  • boto (splattering over existing boto RPM package with older version)
  • cinderclient
  • cliff
  • cmd2
  • compressor
  • django_appconf
  • django_compressor
  • django_openstack_auth
  • glance
  • horizon
  • jsonschema
  • keyring
  • keystoneclient
  • kombu
  • lockfile
  • nova
  • openstack_auth
  • pam
  • passlib
  • prettytable
  • pyparsing
  • python-cinderclient
  • python-glanceclient
  • python-novaclient
  • python-openstackclient
  • python-quantumclient
  • python-swiftclient
  • pytz
  • quantumclient
  • suds
  • swiftclient
  • warlock
  • webob

Files changed

  • /etc/group (added $USER to ‘libvirtd’ group)
  • /etc/gshadow (as above)
  • /etc/httpd/conf/httpd.conf (changes Listen 80 to Listen 0.0.0.0:80)
  • /usr/lib/python2.7/site-packages/boto (due to overwriting RPM provided boto)
  • /usr/bin/cq (as above)
  • /usr/bin/elbadmin (as above)
  • /usr/bin/list_instances (as above)
  • /usr/bin/lss3 (as above)
  • /usr/bin/route53 (as above)
  • /usr/bin/s3multiput (as above)
  • /usr/bin/s3put (as above)
  • /usr/bin/sdbadmin (as above)

Files created

  • /etc/cinder/*
  • /etc/glance/*
  • /etc/keystone/*
  • /etc/nova/*
  • /etc/httpd/conf.d/horizon.conf
  • /etc/polkit-1/localauthority/50-local.d/50-libvirt-reomte-access.pkla
  • /etc/sudoers.d/50_stack_sh
  • /etc/sudoers.d/cinder-rootwrap
  • /etc/sudoers.d/nova-rootwrap
  • $DEST/cinder/*
  • $DEST/data/*
  • $DEST/glance/*
  • $DEST/horizon/*
  • $DEST/keystone/*
  • $DEST/noVNC/*
  • $DEST/nova/*
  • $DEST/python-cinderclient/*
  • $DEST/python-glanceclient/*
  • $DEST/python-keystoneclient/*
  • $DEST/python-novaclient/*
  • $DEST/python-openstackclient/*
  • /usr/bin/cinder*
  • /usr/bin/glance*
  • /usr/bin/keystone*
  • /usr/bin/nova*
  • /usr/bin/openstack
  • /usr/bin/quantum
  • /usr/bin/swift
  • /var/cache/cinder/*
  • /var/cache/glance/*
  • /var/cache/keystone/*
  • /var/cache/nova/*
  • /var/lib/mysql/cinder/*
  • /var/lib/mysql/glance/*
  • /var/lib/mysql/keystone/*
  • /var/lib/mysql/mysql/*
  • /var/lib/mysql/nova/*
  • /var/lib/mysql/performance_schema/*

Thoughts on installation

As we can see from the details above, DevStack does a very significant amount of work as root using sudo. I had fully expected that it was installing RPMs as root, but I had not counted on it adding extra python modules into /usr/lib/python-2.7, nor the addition of files in /etc/, /var or /usr/bin. I had set the $DEST environment variable for DevStack naively assuming that it would cause it to install everything possible under that location. In fact the $DEST variable was only used to control where the GIT checkouts of each openstack component went, along with a few misc files in $DEST/files/

IMHO a development environment setup tool should do as little as humanely possible as root. From the above list of changes, the only things that I believe justify use of sudo privileges are:

  • Installation of RPMs from standard YUM repositories
  • Installation of /etc/sudoers.d/ config files
  • Installation of /etc/polkit file to grant access to libvirtd

Everything else is capable of being 100% isolated from the rest of the OS, under the $DEST directory location. Taking that into account my preferred development setup would be

$DEST
 +- nova (GIT checkout)
 +- ...etc... (GIT checkout)
 +- vroot
     +- bin
     |   +- nova
     |   +- ...etc...
     +- etc
     |   +- nova
     |   +- ...etc...
     +- lib
     |   +- python2.7
     |       +- site-packages
     |           +- boto
     |           +- ...etc...
     +- var
         +- cache
         |   +- nova
         |   +- ...etc...
         +- lib
             +- mysql

This would imply running a private copy of qpid, mysql and httpd, ideally all inside the same screen session as the rest of the OpenStack services, using unprivileged ports. Even if we relied on the system instances of qpid, mysql, httpd and did a little bit more privileged config, 95% of the rest of the stuff DevStack does as root, could still be kept unprivileged. I am also aware that current OpenStack code may not be amenable to installation in locations outside of / by default, but the code is all there to be modified to cope with arbitrary install locations if desired/required.

My other wishlist item for DevStack would be for it to print output that is meaningful to the end user running it. Simply printing a verbose list of every single shell command executed is one of the most unkind things you can do to a user. I’d like to see

 # ./devstack.sh
 * Cloning GIT repositories
     - 1/10 nova
     - 2/10 cinder
     - 3/10 quantum
 * Installing RPMs using YUM
     - 1/30 python-pip
     - 2/30 libvirt
     - 3/30 libvirt-client
 * Installing Python packages to $DEST/vroot/lib/python-2.7/site-packages using PIP
     - 1/24 WebOb
     - 2/24 ampqlib
     - 3/24 boto
 * Creating database schemas
     - 1/10 nova
     - 2/10 cinder
     - 3/10 quantum

By all means still save the full list of every shell command and their output to a ‘devstack.log’ file for troubleshooting when things go wrong.

Live yum upgrading from Fedora 15 to Fedora 16

Posted: November 11th, 2011 | Filed under: Fedora | Tags: , , , , , | 5 Comments »

With Fedora 16 out, I have to perform upgrades on my Fedora machines, two laptops, one mac mini and five servers. Officially the only supported way to upgrade Fedora is to use the regular Anaconda installer, or use pre-upgrade. In the years I’ve been using Fedora (since Fedora Core 5), I have never used Anaconda for upgrades because I rarely have direct access to the machine to boot CDs/DVDs, and whenever I have tried pre-upgrade it always fails. This time it failed on one server because /boot was on software-RAID, and failed on the other two servers because /boot was too small. So instead I’ve always ended up doing live upgrades using yum directly. The first few times I did this, there were some surprises, but in the end I’ve settled into a recipe that has a high success rate.

I’m NOT encouraging people follow my approach, but if you have ended up in a situation where a live yum upgrade is your only option, these notes might help you avoid some pitfalls.

  • Tip 1: Only attempt to upgrade 1 release at a time, don’t try to skip over releases. This reduces the chances of problems with yum calculating a suitable upgrade path
  • Tip 2: Avoid having any 3rd party repositories configured with yum, unless you know they already support both the current and target distro release. In practice this means you want to avoid anything except official Fedora, RPM Fusion and Livna repositories.

The actual upgrade process I follow is this:

  • Step 1: Ensure the current install is fully updated. 
    # yum -y update
  • Step 2: Remove any orphaned packages, ie locally installed packages which are not present in any of your active YUM repositories. Orphaned packages are the most common cause for unresolvable RPM dependencies during upgrade. If you choose to skip this step be aware that you will almost certainly need to revisit it, if yum fails to resolve an upgrade path. 
    # package-cleanup --orphans
# rpm -e ...for each orphan you want to remove...
  • Step 3: Purge all YUM cached data about current repos. This just frees up space by removing cached data files that won’t be needed anymore 
    # yum clean all
  • Step 4: Install the fedora-release, fedora-release-rawhide and fedora-release-notes RPMs for the new release. 
    # wget http://mirror.bytemark.co.uk/fedora/linux/releases/16/Everything/x86_64/os/Packages/fedora-release-16-1.noarch.rpm
# wget http://mirror.bytemark.co.uk/fedora/linux/releases/16/Everything/x86_64/os/Packages/fedora-release-notes-16.1.0-1.fc16.noarch.rpm
# wget http://mirror.bytemark.co.uk/fedora/linux/releases/16/Everything/x86_64/os/Packages/fedora-release-rawhide-16-1.noarch.rpm
# rpm -Uvh fedora-release*16*.rpm
  • Step 5: Perform the actual upgrade. 
     # yum update

Sometimes even after removing all orphan packages and 3rd party packages, the last step will still fail with unresolvable dependencies. This might be the case if an RPM was purged from latest Fedora and thus not having an upgrade path. When this happens, it is usually sufficient to just remove the obsolete package and retry the upgrade.

One significant change between Fedora 15 and 16 that is important for the upgrade process, is the switch from Grub1 to Grub2. While the live YUM upgrade will result in grub2 getting installed, it does not update your bootsector. So there are two post-upgrade tasks that must be done before reboot.

  • Generate the master grub2 config file 
    # grub2-mkconfig > /etc/grub2.cfg
  • Install grub2 into the boot sector 
    # grub2-install /dev/sda

    As mentioned earlier, on one of the machines I had software RAID configured, so I wanted grub2 installed on both disks

    # grub2-install /dev/sda
# grub2-install /dev/sdb

After rebooting into the new kernel, I once again run ‘package-cleanup –orphans’ to ensure find any obsolete packages which can be killed off.

That’s all there was to it. Where I have followed these instructions I have a 100% success rate in live yum upgrades. The only upgrade which failed was the one where I forgot to generate the grub2 config file before rebooting :-)