Upgrading TrueNAS/FreeNAS Plugin Jail to Match System Release

When I update a TrueNAS/FreeNAS plugin jail, I encounter the following error,

Updating Plugin

Error: Major ABI change detected, please run 'upgrade' instead. 

The screen shot is below

After some research on the truenas forum, I have gotten a sense what's going on. 

1. First, let's be sure what TrueNAS/FreeNAS release that we are on by issuing the uname -r command, e.g.,
   

   
   TrueNAS $ uanme -r
   12.2-RELEASE-p10
   TrueNAS $
       

2. Second, let's list the plugins. For this, we need to open a TrueNAS/FreeNAS shell, and issue iocage list, e.g.,
   

   
   TrueNAS $ iocage list
   +------+-----------------+-------+--------------+--------------+
   | JID  |      NAME       | STATE |   RELEASE    |    IP4       |
   +======+=================+=======+==============+==============+
   | 4    | calibre         | up    | 11.2-RELEASE | DHCP         |
   +------+-----------------+-------+--------------+--------------+
   | 5    | nextcloud       | up    | 12.2-RELEASE | 198.51.100.1 |
   +------+-----------------+-------+--------------+--------------+
   | 7    | plexmediaserver | up    | 12.2-RELEASE | DHCP         |
   +------+-----------------+-------+--------------+--------------+
   
   TrueNAS $
     

   where we can see that the jail of plugin calibrehas a different system release number.
3. Then, upgrade the jail's release using the iocage upgrade command, e.g.,
   

   
   TrueNAS $ iocage upgrade calibre -r 12.2-RELEASE
       ...
   TrueNAS $
   

Systemd Failed to Enable Unit?

When I attempt to enable the service using sytemd, I encounter an error where systemd complains that the server does not exists. For example, this is what I observe when I try to enable the OpenVPN service.

$ systemctl enable openvpn@server
Failed to enable unit: Unit file openvpn@server does not exist

This means that I miss the service configuration file. But which one and where should the file be? Although I don't know much about systemd, I resolve this issue by using strace. Using the OpenVPN service as an example, I do the following,

1. Run strace on the systemd process, i.e.,
   

   
       sudo sh -c 'strace -p 1 2>&1 | tee strace_systemd.log'
       

2. Next, run systemctl enable for the OpenVPN service,
   

   
   	sudo systemctl enable openvpn@server
   	

3. Examine strace_systemd.log. For this, we search the service name, e.g., openvpn, in it. In this example, we see,
   

   
   ......
   recvmsg(25, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="/org/freedesktop/systemd1\0\0\0\0\0\0\0"..., iov_len=200}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 200 
   getuid()                                = 0   
   openat(AT_FDCWD, "/etc/systemd/system.control/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/system.control/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/transient/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/generator.early/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/etc/systemd/system/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/system/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/generator/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/usr/local/lib/systemd/system/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/usr/lib/systemd/system/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/generator.late/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/etc/systemd/system.control/openvpn@.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/system.control/openvpn@.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/transient/openvpn@.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/generator.early/openvpn@.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/etc/systemd/system/openvpn@.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   ...
   
   sendmsg(25, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\3\1\0015\0\0\0\1\0\0\0g\0\0\0\5\1u\0\1\0\0\0\4\1s\0#\0\0\0"..., iov_len=120}, {iov_base="0\0\0\0Unit file openvpn@server.ser"..., iov_len=53}], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 173
       

   where systemd complains that it cannot find the configuration file of the service at these locations.
4. Luckily, I do locate a configuration file in the system, e.g.,
   $ sudo find /etc -name openvpn\* /etc/systemd/system/multi-user.target.wants/openvpn@server.service /etc/openvpn $
5. Thus, to fix this issue is to create a link to the OpenVPN service configuration file in the /etc/systemd/system/ directory where systemd is searching for the file based on the output of the strace command,
   sudo ln -s /usr/lib/systemd/system/openvpn-server\@.service /etc/systemd/system/openvpn@server.service

The systemctl enable is successful after the above.

Upgrading Linux Packages to Specific Versions On CentOS/Redhat/Fedora Linux Systems

I want to upgrade a Linux package to a specific version on a CentOS system. 

The Linux package I am trying to upgrade is giblab-ce. The gitlab-ce supports a list of upgrade paths and if the installed gitlab-ce is too old, we cannot upgrade it via dnf upgrade or yum upgrade because the said commands always attempt to upgrade the application to the latest one, which may not be permitted because the upgrade path is not in the list of the supported ones. The following is an example where such a scenario arises,

$ yum upgrade gitlab-ce
......
Running transaction
gitlab preinstall: It seems you are upgrading from major version 13 to major version 14.
gitlab preinstall: It is required to upgrade to the latest 14.0.x version first before proceeding.
gitlab preinstall: Please follow the upgrade documentation at https://docs.gitlab.com/ee/update/index.html#upgrade-paths
error: %pre(gitlab-ce-14.4.1-ce.0.el7.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package gitlab-ce-14.4.1-ce.0.el7.x86_64
  Verifying  : gitlab-ce-14.4.1-ce.0.el7.x86_64                             1/2
gitlab-ce-13.9.4-ce.0.el7.x86_64 was supposed to be removed but is not!
  Verifying  : gitlab-ce-13.9.4-ce.0.el7.x86_64                             2/2

Failed:
  gitlab-ce.x86_64 0:13.9.4-ce.0.el7     gitlab-ce.x86_64 0:14.4.1-ce.0.el7

To resolve this issue, we need to upgrade it to the most recent version we can upgrade according to the upgrade paths. The question becomes, how?

First, we need to find out the available versions of the pacakge. The following commands can answer this question,

yum list PACKAGE_NAME --showduplicates

or

dnf list PACKAGE_NAME --showduplicates

For instance, we can know the list of the available versions of the gitlab-ce

 dnf list gitlab-ce --showduplicates
......
Available Packages
gitlab-ce.x86_64               12.8.1-ce.0.el8                 gitlab_gitlab-ce
......
gitlab-ce.x86_64               13.9.4-ce.0.el8                 gitlab_gitlab-ce
gitlab-ce.x86_64               13.9.5-ce.0.el8                 gitlab_gitlab-ce
......
gitlab-ce.x86_64               14.0.12-ce.0.el8                gitlab_gitlab-ce
......
gitlab-ce.x86_64               14.4.1-ce.0.el8                 gitlab_gitlab-ce

Since gitlab-ce has the following upgrade path 13.8.8 -> latest 13.12.Z -> latest 14.0.Z -> latest 14.Y.Z, we will upgrade gitlab-ce to 13.12.5, then to 14.0.12, and finally to 14.4.1. To upgrade the specific version, we use either a yum or a dnf command in the following format,

dnf upgrade PACKAGE_NAME-PACKAGE_VERSION

or

yum upgrade PACKAGE_NAME-PACKAGE_VERSION

For instance, to upgrade gitlab-ce to version 13.12.5, we run

dnf upgrade gitlab-ce-14.0.12-ce.0.el8

We repeat the dnf upgrade command on more recent versions until we reach the latest one. If we don't know the list of the permitted upgrade paths, we can always take the trial and error approach to try to upgrade to a newer version until we success and finally reach the latest version.

Determining Dependent Packages in CentOS/Redhat/Fedora Linux Systems

I have a need to identify dependencies of on a CentOS Linux systems. Since I am using yum and dnf, I think that the method presented here also applies to other Linux distributions, like Redhat and Fedora Linux systems. 

Why do I have this need? It actually arises from my upgrading a CentOS 7 system to CentOS 8. I follow the steps in the following tecmint article,

How to Upgrade CentOS 7 to CentOS 8 Linux

There are numerous broken dependencies problems  when I invoke the command,

dnf -y --releasever=8 --allowerasing --setopt=deltarpm=false distro-sync

I address this problem by removing the packages that has the unresolved dependencies problem. Here, I only want to remove a package without removing its dependencies, otherwise, the system may be left in a completely unusable state. To remove a package without removing its dependencies, I run,

rpm -Uvh --nodeps PACKAGE_NAME  

After I removed several packages, I can finally successfully run the dnf distro-sync command. After that, I want to install the removed packages. However, I hypothesize that these packages dependencies may not be installed properly. Therefore, I need not only install these removed pacakges but also their dependent pacakges. For this, I need to figure out what the dependent packages are. This turns out to be a one-liner, i.e.,

repoquery --requires --resolve PACKAGE_NAME
  

I take notes about what pacakges I removed during the upgrade process and reinstall/install all the dependent packages in a shell script, like

#!/bin/bash

packages=(libpq-devel apr annobin)

[ -f install.txt ] && rm install.txt
[ -f reinstall.txt ] && rm reinstall.txt
[ -f pacakge_dep.txt ] && rm package_dep.txt

touch install.txt
touch reinstall.txt

for pkg in ${packages[@]}; do
    repoquery -y --requires --resolve ${pkg} > pacakge_dep.txt
    cat pacakge_dep.txt | \
        while read p; do
            rpm -q $p
            if [ $? -ne 0 ]; then
                echo $p >> install.txt
            else
                echo $p >> reinstall.txt
            fi
        done
done

for pkg in ${packages[@]}; do
    rpm -q $pkg
    if [ $? -ne 0 ]; then
        echo $pkg >> install.txt
    else
        echo $pkg >> reinstall.txt
    fi
done

if [ -s install.txt ]; then
    dnf install -y $(sort install.txt | uniq)
fi

if [ -s reinstall.txt ]; then
    dnf reinstall -y $(sort reinstall.txt | uniq)
fi

You may replace packages by your list of packages.