Upgrading TrueNAS/FreeNAS Plugin Jail to Match System Release

When I update a TrueNAS/FreeNAS plugin jail, I encounter the following error,

Updating Plugin

Error: Major ABI change detected, please run 'upgrade' instead. 

The screen shot is below

After some research on the truenas forum, I have gotten a sense what's going on. 

1. First, let's be sure what TrueNAS/FreeNAS release that we are on by issuing the uname -r command, e.g.,
   

   
   TrueNAS $ uanme -r
   12.2-RELEASE-p10
   TrueNAS $
       

2. Second, let's list the plugins. For this, we need to open a TrueNAS/FreeNAS shell, and issue iocage list, e.g.,
   

   
   TrueNAS $ iocage list
   +------+-----------------+-------+--------------+--------------+
   | JID  |      NAME       | STATE |   RELEASE    |    IP4       |
   +======+=================+=======+==============+==============+
   | 4    | calibre         | up    | 11.2-RELEASE | DHCP         |
   +------+-----------------+-------+--------------+--------------+
   | 5    | nextcloud       | up    | 12.2-RELEASE | 198.51.100.1 |
   +------+-----------------+-------+--------------+--------------+
   | 7    | plexmediaserver | up    | 12.2-RELEASE | DHCP         |
   +------+-----------------+-------+--------------+--------------+
   
   TrueNAS $
     

   where we can see that the jail of plugin calibrehas a different system release number.
3. Then, upgrade the jail's release using the iocage upgrade command, e.g.,
   

   
   TrueNAS $ iocage upgrade calibre -r 12.2-RELEASE
       ...
   TrueNAS $
   

Systemd Failed to Enable Unit?

When I attempt to enable the service using sytemd, I encounter an error where systemd complains that the server does not exists. For example, this is what I observe when I try to enable the OpenVPN service.

$ systemctl enable openvpn@server
Failed to enable unit: Unit file openvpn@server does not exist

This means that I miss the service configuration file. But which one and where should the file be? Although I don't know much about systemd, I resolve this issue by using strace. Using the OpenVPN service as an example, I do the following,

1. Run strace on the systemd process, i.e.,
   

   
       sudo sh -c 'strace -p 1 2>&1 | tee strace_systemd.log'
       

2. Next, run systemctl enable for the OpenVPN service,
   

   
   	sudo systemctl enable openvpn@server
   	

3. Examine strace_systemd.log. For this, we search the service name, e.g., openvpn, in it. In this example, we see,
   

   
   ......
   recvmsg(25, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="/org/freedesktop/systemd1\0\0\0\0\0\0\0"..., iov_len=200}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 200 
   getuid()                                = 0   
   openat(AT_FDCWD, "/etc/systemd/system.control/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/system.control/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/transient/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/generator.early/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/etc/systemd/system/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/system/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/generator/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/usr/local/lib/systemd/system/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/usr/lib/systemd/system/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/generator.late/openvpn@server.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/etc/systemd/system.control/openvpn@.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/system.control/openvpn@.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/transient/openvpn@.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/run/systemd/generator.early/openvpn@.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   openat(AT_FDCWD, "/etc/systemd/system/openvpn@.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
   ...
   
   sendmsg(25, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\3\1\0015\0\0\0\1\0\0\0g\0\0\0\5\1u\0\1\0\0\0\4\1s\0#\0\0\0"..., iov_len=120}, {iov_base="0\0\0\0Unit file openvpn@server.ser"..., iov_len=53}], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 173
       

   where systemd complains that it cannot find the configuration file of the service at these locations.
4. Luckily, I do locate a configuration file in the system, e.g.,
   $ sudo find /etc -name openvpn\* /etc/systemd/system/multi-user.target.wants/openvpn@server.service /etc/openvpn $
5. Thus, to fix this issue is to create a link to the OpenVPN service configuration file in the /etc/systemd/system/ directory where systemd is searching for the file based on the output of the strace command,
   sudo ln -s /usr/lib/systemd/system/openvpn-server\@.service /etc/systemd/system/openvpn@server.service

The systemctl enable is successful after the above.

Upgrading Linux Packages to Specific Versions On CentOS/Redhat/Fedora Linux Systems

I want to upgrade a Linux package to a specific version on a CentOS system. 

The Linux package I am trying to upgrade is giblab-ce. The gitlab-ce supports a list of upgrade paths and if the installed gitlab-ce is too old, we cannot upgrade it via dnf upgrade or yum upgrade because the said commands always attempt to upgrade the application to the latest one, which may not be permitted because the upgrade path is not in the list of the supported ones. The following is an example where such a scenario arises,

$ yum upgrade gitlab-ce
......
Running transaction
gitlab preinstall: It seems you are upgrading from major version 13 to major version 14.
gitlab preinstall: It is required to upgrade to the latest 14.0.x version first before proceeding.
gitlab preinstall: Please follow the upgrade documentation at https://docs.gitlab.com/ee/update/index.html#upgrade-paths
error: %pre(gitlab-ce-14.4.1-ce.0.el7.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package gitlab-ce-14.4.1-ce.0.el7.x86_64
  Verifying  : gitlab-ce-14.4.1-ce.0.el7.x86_64                             1/2
gitlab-ce-13.9.4-ce.0.el7.x86_64 was supposed to be removed but is not!
  Verifying  : gitlab-ce-13.9.4-ce.0.el7.x86_64                             2/2

Failed:
  gitlab-ce.x86_64 0:13.9.4-ce.0.el7     gitlab-ce.x86_64 0:14.4.1-ce.0.el7

To resolve this issue, we need to upgrade it to the most recent version we can upgrade according to the upgrade paths. The question becomes, how?

First, we need to find out the available versions of the pacakge. The following commands can answer this question,

yum list PACKAGE_NAME --showduplicates

or

dnf list PACKAGE_NAME --showduplicates

For instance, we can know the list of the available versions of the gitlab-ce

 dnf list gitlab-ce --showduplicates
......
Available Packages
gitlab-ce.x86_64               12.8.1-ce.0.el8                 gitlab_gitlab-ce
......
gitlab-ce.x86_64               13.9.4-ce.0.el8                 gitlab_gitlab-ce
gitlab-ce.x86_64               13.9.5-ce.0.el8                 gitlab_gitlab-ce
......
gitlab-ce.x86_64               14.0.12-ce.0.el8                gitlab_gitlab-ce
......
gitlab-ce.x86_64               14.4.1-ce.0.el8                 gitlab_gitlab-ce

Since gitlab-ce has the following upgrade path 13.8.8 -> latest 13.12.Z -> latest 14.0.Z -> latest 14.Y.Z, we will upgrade gitlab-ce to 13.12.5, then to 14.0.12, and finally to 14.4.1. To upgrade the specific version, we use either a yum or a dnf command in the following format,

dnf upgrade PACKAGE_NAME-PACKAGE_VERSION

or

yum upgrade PACKAGE_NAME-PACKAGE_VERSION

For instance, to upgrade gitlab-ce to version 13.12.5, we run

dnf upgrade gitlab-ce-14.0.12-ce.0.el8

We repeat the dnf upgrade command on more recent versions until we reach the latest one. If we don't know the list of the permitted upgrade paths, we can always take the trial and error approach to try to upgrade to a newer version until we success and finally reach the latest version.

Determining Dependent Packages in CentOS/Redhat/Fedora Linux Systems

I have a need to identify dependencies of on a CentOS Linux systems. Since I am using yum and dnf, I think that the method presented here also applies to other Linux distributions, like Redhat and Fedora Linux systems. 

Why do I have this need? It actually arises from my upgrading a CentOS 7 system to CentOS 8. I follow the steps in the following tecmint article,

How to Upgrade CentOS 7 to CentOS 8 Linux

There are numerous broken dependencies problems  when I invoke the command,

dnf -y --releasever=8 --allowerasing --setopt=deltarpm=false distro-sync

I address this problem by removing the packages that has the unresolved dependencies problem. Here, I only want to remove a package without removing its dependencies, otherwise, the system may be left in a completely unusable state. To remove a package without removing its dependencies, I run,

rpm -Uvh --nodeps PACKAGE_NAME  

After I removed several packages, I can finally successfully run the dnf distro-sync command. After that, I want to install the removed packages. However, I hypothesize that these packages dependencies may not be installed properly. Therefore, I need not only install these removed pacakges but also their dependent pacakges. For this, I need to figure out what the dependent packages are. This turns out to be a one-liner, i.e.,

repoquery --requires --resolve PACKAGE_NAME
  

I take notes about what pacakges I removed during the upgrade process and reinstall/install all the dependent packages in a shell script, like

#!/bin/bash

packages=(libpq-devel apr annobin)

[ -f install.txt ] && rm install.txt
[ -f reinstall.txt ] && rm reinstall.txt
[ -f pacakge_dep.txt ] && rm package_dep.txt

touch install.txt
touch reinstall.txt

for pkg in ${packages[@]}; do
    repoquery -y --requires --resolve ${pkg} > pacakge_dep.txt
    cat pacakge_dep.txt | \
        while read p; do
            rpm -q $p
            if [ $? -ne 0 ]; then
                echo $p >> install.txt
            else
                echo $p >> reinstall.txt
            fi
        done
done

for pkg in ${packages[@]}; do
    rpm -q $pkg
    if [ $? -ne 0 ]; then
        echo $pkg >> install.txt
    else
        echo $pkg >> reinstall.txt
    fi
done

if [ -s install.txt ]; then
    dnf install -y $(sort install.txt | uniq)
fi

if [ -s reinstall.txt ]; then
    dnf reinstall -y $(sort reinstall.txt | uniq)
fi

You may replace packages by your list of packages.

Mapping NFS User ID between Linux and FreeNAS/TrueNAS/FreeBSD

I want to mount a NSF share exported on a FreeNAS/TrueNAS/FreeBSD system  on a Fedora Linux system. These two systems have their own independent user  IDs and domains. To display the user name of the NSF share as a user name on the client, I figure that I need to use nfsidmap. For this, I need to solve the following problems.

1. Map the Linux client's NSF domain to that of the FreeNAS/TrueNAS/FreeBSD server. For this, I need to find out the domain name of the FreeNAS/TrueNAS/FreeBSD server. On the server, open a console/terminal, issue the following command

   hostname -d        

   Assume the output is local. We can check whether this matches the NSF domain on the Linux client side, by running the following command on the client side,

   sudo nfsidmap -d 

2. Map the Linux client's user to that of the FreeNAS/TrueNAS/FreeBSD server. For this, I update /etc/idmapd.conf

Finally, I update the Linux client's /etc/idmapd.conf configuration file. Below is an example.

[General]
Domain = local # use the output of "hostname -d" on the server side

[Mapping]
Nobody-User = nobody
Nobody-Group = nobody
        
[Translation]

[Static]
server_user1@local=client_user1

Dealing with "error: unpacking of archive failed" when upgrading Fedora Linux

I am upgrading a Fedora Linux system. A necessary step is to bring the system up-to-date. For this, I issue the command,

sudo dnf upgrade --refresh

However, the upgrade fails with an error message that complains about "unpacking of archive failed ... cpio: File from package already exists as a directory in system". Below is an exerpt of the error message,

$ sudo dnf upgrade --refresh
Docker CE Stable - x86_64                       110 kB/s | 3.5 kB     00:00
Fedora 32 openh264 (From Cisco) - x86_64        3.4 kB/s | 990  B     00:00
Fedora Modular 32 - x86_64                      146 kB/s |  12 kB     00:00
Fedora Modular 32 - x86_64 - Updates             69 kB/s |  12 kB     00:00
Fedora 32 - x86_64 - Updates                    122 kB/s |  12 kB     00:00
Fedora 32 - x86_64                              120 kB/s |  12 kB     00:00
Dependencies resolved.
================================================================================
 Package                 Arch     Version                       Repo       Size
================================================================================
Upgrading:
 filesystem              x86_64   3.14-2.fc32                   fedora    1.1 M
 gawk                    x86_64   5.0.1-7.fc32                  fedora    1.2 M
 npm                     x86_64   1:6.14.12-1.12.22.1.1.fc32    updates   3.3 M
 python3-numpy           x86_64   1:1.18.4-2.fc32               updates   4.4 M
 texlive-texlive.infra   noarch   7:20200327-21.fc32            updates   279 k

Transaction Summary
================================================================================
Upgrade  5 Packages

Total download size: 10 M
Is this ok [y/N]: y
Downloading Packages:
(1/5): texlive-texlive.infra-20200327-21.fc32.n 630 kB/s | 279 kB     00:00
(2/5): npm-6.14.12-1.12.22.1.1.fc32.x86_64.rpm  4.7 MB/s | 3.3 MB     00:00
(3/5): filesystem-3.14-2.fc32.x86_64.rpm        3.1 MB/s | 1.1 MB     00:00
(4/5): python3-numpy-1.18.4-2.fc32.x86_64.rpm   3.6 MB/s | 4.4 MB     00:01
(5/5): gawk-5.0.1-7.fc32.x86_64.rpm             2.0 MB/s | 1.2 MB     00:00
--------------------------------------------------------------------------------
Total                                           6.6 MB/s |  10 MB     00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: filesystem-3.14-2.fc32.x86_64                          1/1
  Preparing        :                                                        1/1
  Upgrading        : filesystem-3.14-2.fc32.x86_64                         1/10
Error unpacking rpm package filesystem-3.14-2.fc32.x86_64
  Upgrading        : gawk-5.0.1-7.fc32.x86_64                              2/10
error: unpacking of archive failed on file /usr/tmp: cpio: File from package already exists as a directory in system
error: filesystem-3.14-2.fc32.x86_64: install failed

Error unpacking rpm package gawk-5.0.1-7.fc32.x86_64
  Upgrading        : texlive-texlive.infra-7:20200327-21.fc32.noarch       3/10
error: unpacking of archive failed on file /usr/libexec/gawk: cpio: File from package already exists as a directory in system
error: gawk-5.0.1-7.fc32.x86_64: install failed

Error unpacking rpm package texlive-texlive.infra-7:20200327-21.fc32.noarch
  Upgrading        : python3-numpy-1:1.18.4-2.fc32.x86_64                  4/10
error: unpacking of archive failed on file /usr/share/perl5/TeXLive: cpio: File from package already exists as a directory in system
error: texlive-texlive.infra-7:20200327-21.fc32.noarch: install failed

Error unpacking rpm package python3-numpy-1:1.18.4-2.fc32.x86_64
  Upgrading        : npm-1:6.14.12-1.12.22.1.1.fc32.x86_64                 5/10
error: unpacking of archive failed on file /usr/include/numpy: cpio: File from package already exists as a directory in system
error: python3-numpy-1:1.18.4-2.fc32.x86_64: install failed

Error unpacking rpm package npm-1:6.14.12-1.12.22.1.1.fc32.x86_64
  Verifying        : npm-1:6.14.12-1.12.22.1.1.fc32.x86_64                 1/10
  Verifying        : npm-1:6.14.8-1.12.19.0.1.fc31.x86_64                  2/10
  Verifying        : python3-numpy-1:1.18.4-2.fc32.x86_64                  3/10
  Verifying        : python3-numpy-1:1.17.4-2.fc31.x86_64                  4/10
  Verifying        : texlive-texlive.infra-7:20200327-21.fc32.noarch       5/10
  Verifying        : texlive-texlive.infra-7:20190410-8.fc31.noarch        6/10
  Verifying        : filesystem-3.14-2.fc32.x86_64                         7/10
  Verifying        : filesystem-3.12-2.fc31.x86_64                         8/10
  Verifying        : gawk-5.0.1-7.fc32.x86_64                              9/10
  Verifying        : gawk-5.0.1-5.fc31.x86_64                             10/10

Failed:
  filesystem-3.12-2.fc31.x86_64
  filesystem-3.14-2.fc32.x86_64
  gawk-5.0.1-5.fc31.x86_64
  gawk-5.0.1-7.fc32.x86_64
  npm-1:6.14.8-1.12.19.0.1.fc31.x86_64
  npm-1:6.14.12-1.12.22.1.1.fc32.x86_64
  python3-numpy-1:1.17.4-2.fc31.x86_64
  python3-numpy-1:1.18.4-2.fc32.x86_64
  texlive-texlive.infra-7:20190410-8.fc31.noarch
  texlive-texlive.infra-7:20200327-21.fc32.noarch

Error: Transaction failed
$

As it complains that it can not write to the path indicated in the error message, such as, /usr/include/numpy because the directory already exists. To address this, I come out the following solution, i.e., to rename the directory or file that the error message complains about. For instance, we do these,

sudo mv /usr/tmp /usr/tmp.bu
sudo mv /usr/libexec/gawk /usr/libexec/gawk.bu
sudo mv /usr/share/perl5/TeXLive /usr/share/perl5/TeXLive.bu
sudo mv /usr/include/numpy /usr/include/numpy.bu

Then do upgrade again, as in,

sudo dnf upgrade --refresh

After this, don't forget to remove the backup files or directories, such as,

sudo rm -rf /usr/tmp.bu
sudo rm -rf /usr/libexec/gawk.bu
sudo rm -rf /usr/share/perl5/TeXLive.bu
sudo rm -rf /usr/include/numpy.bu

Dealing with Complexities

The above solutions work for some packages. However, there are two complexities. 

1. Some packages has more than one or two files or directories that cause cpio to fail. The sympotom is that we would encounter the similar error. The solution is obvious, just to rename the file or the directory that causes the problem until we can successfully upgrade the package.
2. Sometimes, the error message does not indicate which file or the directory causes the problem, such as, the example below,

$ sudo dnf upgrade --refresh
[sudo] password for hchen:
Docker CE Stable - x86_64                        38 kB/s |  13 kB     00:00
Fedora 33 - x86_64                               11 MB/s |  72 MB     00:06
Fedora 33 openh264 (From Cisco) - x86_64        1.4 kB/s | 2.5 kB     00:01
Fedora Modular 33 - x86_64                      3.1 MB/s | 3.3 MB     00:01
Fedora 33 - x86_64 - Updates                     12 MB/s |  30 MB     00:02
Fedora Modular 33 - x86_64 - Updates            2.8 MB/s | 3.2 MB     00:01
Last metadata expiration check: 0:00:01 ago on Sat 02 Oct 2021 09:14:19 PM EDT.
Dependencies resolved.
================================================================================
 Package   Architecture Version                             Repository     Size
================================================================================
Upgrading:
 npm       x86_64       1:6.14.15-1.14.17.6.1.fc33          updates       3.3 M

Transaction Summary
================================================================================
Upgrade  1 Package

Total download size: 3.3 M
Is this ok [y/N]: y
Downloading Packages:
npm-6.14.15-1.14.17.6.1.fc33.x86_64.rpm         6.2 MB/s | 3.3 MB     00:00
--------------------------------------------------------------------------------
Total                                           4.0 MB/s | 3.3 MB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: npm-1:6.14.15-1.14.17.6.1.fc33.x86_64                  1/1
  Preparing        :                                                        1/1
  Upgrading        : npm-1:6.14.15-1.14.17.6.1.fc33.x86_64                  1/2
Error unpacking rpm package npm-1:6.14.15-1.14.17.6.1.fc33.x86_64
  Verifying        : npm-1:6.14.15-1.14.17.6.1.fc33.x86_64                  1/2
  Verifying        : npm-1:6.14.8-1.12.19.0.1.fc31.x86_64                   2/2

Failed:
  npm-1:6.14.8-1.12.19.0.1.fc31.x86_64   npm-1:6.14.15-1.14.17.6.1.fc33.x86_64

Error: Transaction failed
$

One solution is to download the rpm package, and attempt to upgrade it using rpm. In this way we can observe which file and directory that have caused the problem. For instance, we first download the npmpackage,

sudo dnf download npm

Then we attempt to install it,

$ sudo rpm -U -i -v -h npm-6.14.15-1.14.17.6.1.fc33.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:npm-1:6.14.15-1.14.17.6.1.fc33   ################################# [ 50%]
error: unpacking of archive failed on file /usr/lib/node_modules/npm/docs: cpio: File from package already exists as a directory in system
error: npm-1:6.14.15-1.14.17.6.1.fc33.x86_64: install failed
error: npm-1:6.14.8-1.12.19.0.1.fc31.x86_64: erase skipped
$

which shows that the /usr/lib/node_modules/npm/docs directory is the culprit. We need to rename it following the approach discussed in the above

sudo mv /usr/lib/node_modules/npm/docs /usr/lib/node_modules/npm/docs.bu

After that, we can upgrade the package, for instance,

$ sudo rpm -U -i -v -h npm-6.14.15-1.14.17.6.1.fc33.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:npm-1:6.14.15-1.14.17.6.1.fc33   ################################# [ 50%]
Cleaning up / removing...
   2:npm-1:6.14.8-1.12.19.0.1.fc31    ################################# [100%]
$ sudo dnf upgrade --refresh
Docker CE Stable - x86_64                        22 kB/s | 3.5 kB     00:00
Fedora 33 - x86_64                               47 kB/s |  13 kB     00:00
Fedora 33 openh264 (From Cisco) - x86_64        9.7 kB/s | 990  B     00:00
Fedora Modular 33 - x86_64                      121 kB/s |  12 kB     00:00
Fedora 33 - x86_64 - Updates                     95 kB/s |  11 kB     00:00
Fedora 33 - x86_64 - Updates                    496 kB/s | 430 kB     00:00
Fedora Modular 33 - x86_64 - Updates             49 kB/s |  11 kB     00:00
Dependencies resolved.
Nothing to do.
Complete!
$

Updating Debian Linux Release

I have a Debian Linux system that I haven't kept update-to-date for a while. When I tried to update it, and I got the following error,

$ sudo apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Hit:4 https://download.docker.com/linux/debian buster InRelease
Reading package lists... Done
E: Repository 'http://security.debian.org/debian-security buster/updates InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.6' to '10.10'
E: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
E: Repository 'http://deb.debian.org/debian buster-updates InRelease' changed its 'Suite' value from 'stable-updates' to 'oldstable-updates'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.

This is the result that Debian 10 has updated the release repo from 10.6 to 10.10. To resolve this issue, we can run the following command.

sudo apt-get update --allow-releaseinfo-change

The only interesting part here is that I had to run the command twice to confirm the problem went away. To see this, we can take a look at the transcript showing I ran the command

$ sudo apt-get update --allow-releaseinfo-change
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Hit:4 https://download.docker.com/linux/debian buster InRelease
Get:5 http://deb.debian.org/debian buster/main Sources [7,836 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main Sources [199 kB]
Get:7 http://security.debian.org/debian-security buster/updates/main amd64 Packages [302 kB]
Get:8 http://security.debian.org/debian-security buster/updates/main Translation-en [159 kB]
Get:9 http://deb.debian.org/debian buster-updates/main Sources.diff/Index [7,624 B]
Ign:9 http://deb.debian.org/debian buster-updates/main Sources.diff/Index
Get:10 http://deb.debian.org/debian buster-updates/main amd64 Packages.diff/Index [7,624 B]
Get:11 http://deb.debian.org/debian buster-updates/main Translation-en.diff/Index [5,164 B]
Get:12 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-10-24-2001.35.pdiff [286 B]
Get:13 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-12-24-1401.30.pdiff [286 B]
Get:14 http://deb.debian.org/debian buster-updates/main amd64 Packages 2021-01-29-2000.47.pdiff [408 B]
Get:15 http://deb.debian.org/debian buster-updates/main amd64 Packages 2021-02-07-1359.56.pdiff [2,302 B]
Get:16 http://deb.debian.org/debian buster-updates/main amd64 Packages 2021-04-22-1401.14.pdiff [1,996 B]
Get:17 http://deb.debian.org/debian buster-updates/main amd64 Packages 2021-04-23-1401.02.pdiff [1,127 B]
Get:18 http://deb.debian.org/debian buster-updates/main amd64 Packages 2021-06-21-1401.46.pdiff [656 B]
Get:19 http://deb.debian.org/debian buster-updates/main amd64 Packages 2021-06-23-1401.37.pdiff [4,999 B]
Get:19 http://deb.debian.org/debian buster-updates/main amd64 Packages 2021-06-23-1401.37.pdiff [4,999 B]
Get:20 http://deb.debian.org/debian buster-updates/main Translation-en 2021-02-07-1359.56.pdiff [1,506 B]
Get:21 http://deb.debian.org/debian buster-updates/main Translation-en 2021-04-22-1401.14.pdiff [1,408 B]
Get:22 http://deb.debian.org/debian buster-updates/main Translation-en 2021-06-21-1401.46.pdiff [719 B]
Get:23 http://deb.debian.org/debian buster-updates/main Translation-en 2021-06-23-1401.37.pdiff [6,585 B]
Get:23 http://deb.debian.org/debian buster-updates/main Translation-en 2021-06-23-1401.37.pdiff [6,585 B]
Get:24 http://deb.debian.org/debian buster/main amd64 Packages [7,907 kB]
Get:25 http://deb.debian.org/debian buster/main Translation-en [5,968 kB]
Get:26 http://deb.debian.org/debian buster-updates/main Sources [16.6 kB]
Fetched 22.4 MB in 12s (1,806 kB/s)
Reading package lists... Done
N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.6' to '10.10'
N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: Repository 'http://security.debian.org/debian-security buster/updates InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: Repository 'http://deb.debian.org/debian buster-updates InRelease' changed its 'Suite' value from 'stable-updates' to 'oldstable-updates'
$ sudo apt-get update --allow-releaseinfo-change
Hit:1 http://deb.debian.org/debian buster InRelease
Hit:2 http://security.debian.org/debian-security buster/updates InRelease
Hit:3 http://deb.debian.org/debian buster-updates InRelease
Hit:4 https://download.docker.com/linux/debian buster InRelease
$ sudo apt-get update
Hit:1 https://download.docker.com/linux/debian buster InRelease
Hit:2 http://deb.debian.org/debian buster InRelease
Hit:3 http://deb.debian.org/debian buster-updates InRelease
Hit:4 http://security.debian.org/debian-security buster/updates InRelease
Reading package lists... Done
$

Solving "Permission Denied" Problem When Accessing PostgreSQL Database

I am getting the "permission denied" error when accessing a PostgreSQL database. The PostgreSQL server is version 11.8.

mydb=> select version();
                                                  version

------------------------------------------------------------------------------------------------------------
 PostgreSQL 11.8 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2), 64-bit
(1 row)

mydb=>

I follow the suggestions given in several online discussions to grant necessary privileges to the tables, views, and sequences in the database. These online discussiones include,

• Permission denied in Postgres
• How to fix permission denied for relation some_table_name in PostgreSQL?
• Granted all privileges on my PostGres table, but still am getting a “Permission denied” error

However, the problem remains. The solution that actually works for me is in the discussion below,

• PostgreSQL: Modify OWNER on all tables simultaneously in PostgreSQL

The solution is to alter all tables, views, and sequences' ownership to the database role or user we wish to grant the privileges to and requires a user to enter the password to the database user or role multiple times. To simply this, I rewrite the solution as follows in a shell script.

#!/bin/bash

DB_NAME="mydb"          # replace mydb by the database name
DB_USER="mydbuser"      # repalce mydbuser by the database username
SQL_FILE="fixpermission.sql"

[ -f ${SQL_FILE} ] && rm -f ${SQL_FILE}

for tbl in \
  `psql -qAt -c \
  "select tablename from pg_tables where schemaname = 'public';" $DB_NAME`;
do
  echo "alter table \"$tbl\" owner to $DB_USER;" >> $SQL_FILE
done

for tbl in \
  `psql -qAt -c \
  "select sequence_name from information_schema.sequences where sequence_schema = 'public';" $DB_NAME`;
do
   echo "alter sequence \"$tbl\" owner to $DB_USER;" >> $SQL_FILE
done


for tbl in \
  `psql -qAt -c \
  "select table_name from information_schema.views where table_schema = 'public';" $DB_NAME`;
do
  echo "alter view \"$tbl\" owner to $DB_USER;" >> $SQL_FILE
done

psql -U postgres -h localhost $DB_NAME < $SQL_FILE

rm -f $SQL_FILE

This script only requires a user to enter the data role or user's password for 4 times.

Solving problem updating Windows 10 to version 20H2 or newer

I have been trying to update Windows 10 to Windows 10 version 20H2. The problems that I  encountered are,

1. When using Windows Update, I saw the update process stuck at 61%, eventually failed. I am not the only one, clearly. For instance, there are numerous online threads. Here are several examples, example 1, example 2, and example 3. Want to see more, just Google it. 
2. When using Windows 10 Update Assistant, one of the recommended method, I saw it stuck at 99%, for eons. 

 Finally, I found the right answer, i.e., the method documented in this post,

• Windows 10 Upgrade Assistant is stuck at 99%

This  method worked for me. The caveat is that 1) contrast of using Windows update, I have to wait for two to three hours before it finishes; 2) it updates my Windows to 21H1 directly. Regardless, updating to a new version is what I wanted since Microsoft is going to cease supporting older versions of Windows 10.

The steps as outlined in the article linked in the above are,

1. Open File Explorer, type C:\$GetCurrent, and then press Enter.
2. Copy and paste the Media folder to the desktop. If you don't see the folder, select View and make sure the check box next to Hidden items is selected.
   Restart your PC, open File Explorer, type C:\$GetCurrent in the address bar, and then press Enter.
3. Copy and paste the Media folder from the desktop to C:\$GetCurrent.
4. Open the Media folder and double-click Setup.
5. Follow the instructions to start the upgrade. On the Get important updates screen, select Not right now, and then select Next.
6. Follow the instructions to finish upgrading to Windows 10. After you're done, make sure to install available updates. Select the Start  button, and then select Settings  > Update & Security  > Windows Update > Check for updates.

Installing Python from source on a Fedora Linux system

I have a need to install a new version of Python on a dated Fedora Linux system (Fedora 31). Here is the steps that I followed,

# install neccessary packages to build Python from source
sudo dnf install gcc openssl-devel bzip2-devel libffi-devel zlib-devel

# download Python source package. You may select a different version you wish 
# to download by browsing https://www.python.org/ftp/python/
wget https://www.python.org/ftp/python/3.9.5/Python-3.9.5.tgz

# extract the files from the package
tar xzf Python-3.9.5.tgz

# go to the directory where the extracted files are
cd Python-3.9.5

# enable optimization
sudo ./configure --enable-optimizations

# compile and install to /usr/local
sudo make altinstall

# show the version of the active Python
python --version

# add the newly installed Python to alternatives (I already have a Python2 Python3.7)
sudo alternatives --install /usr/bin/python python /usr/local/bin/python3.9 3

# config alternatives and select 3 as the active Python
sudo alternatives --config python

# check the version of the active Python
python --version

How do we get the "diff" or the patch for the very first commit in Git?

This is a summary of several Stack Overflow discusses about how to get the "diff" or the patch for the very first commit in Git.

1. To get the diff from the very first commit to the empty tree, we can use the following command.

   git diff \
       4b825dc642cb6eb9a060e54bf8d69288fbee4904 \
       $(git log --reverse | head -1 | cut -d' ' -f2)
     

   where 4b825dc642cb6eb9a060e54bf8d69288fbee4904 is the SHA of the empty tree and $(git log --reverse | head -1 | cut -d' ' -f2) retrieves the SHA of the very first commit. Alternatively, we can rewrite the above in a way that perhaps is easier to read.
2. To get the diff from the very first commit to the empty tree, we can use the following command.

   git diff \
       $(printf '' | git hash-object -t tree --stdin) \
       $(git log --reverse | head -1 | cut -d' ' -f2)
     

3. To get the patch from the very first commit to the empty tree, we can use the following command.

   git format-patch \
       --root --stdout \
       $(git log --reverse | head -1 | cut -d' ' -f2)
       

Reference 

1. How to get Git diff of the first commit?  
2. How to use git format-patch on initial commit

Accessing a SSH host via a SSH gateway?

I wrote the notes about how to synchronize two SSH hosts via a middle SSH gateway and how to access a SSH host via a SSH gateway. The assumption there is to use ssh in OpenSSH. But what if we want to do the same with PuTTY? I actually locate the answer on Stack Overflow. This post is just a bookmark to the answer.

• PuTTY configuration equivalent to OpenSSH ProxyCommand

Simply put, it is to use PuTTY's proxy setup on the user interface when we create a session for a host, and the proxy command follows the following format,

plink.exe %user@%proxyhost -P %proxyport -nc %host:%port

or

plink.exe %user@%proxyhost -pw %pass -P %proxyport -nc %host:%port

Note the two comments.

1. plink.exe is part of the PuTTY suite.
2. The second plink.exe example is to use the password authentication. Since the password is part of the command line, someone can reveal the password by showing the process command line arguments, such as, using the methods in the notes written here. So use it with caution.

Synchronizing files on two SSH hosts with one in the middle (accessing a SSH host via a SSH gateway?)

I believe that this is a rather common situation -- I want to copy files from one SSH host to the other; however, the SSH host I want to copy files from or copy files to is behind a SSH gateway. To describe the situation clearly, let's consider three SSH hosts, H, M, and O. I can log on to H via SSH, from H log on M via SSH, but from H not log on to O via SSH. To get to O, I have to log on to M first, and so M is a SSH gateway. In fact, this is a typical setup for many computational cluster, to access the cluster, you SSH to a SSH gateway. How do we copy files between H and O? Of course, we can always copy files from O to M, and M to H, which is somewhat inconvenient. Can we do better? Yes, via SSH's ProxyCommand option. Below we present two two scenarios.

1. SSH to O via M from H

Below is how we get to O "directly" from H assuming the username on M is alice_on_m and that on O alice_on_o.

ssh -o 'ProxyCommand ssh alice_on_m@M -W %h:%p' alice_on_o@O

Note that although we issue this command on H, H does not have to be able to resolve O; however, M must know O.

If you prefer the public key authentication to the password authentication, do the following instead,

ssh -o \
  'ProxyCommand ssh -i ~/.ssh/private_key_for_m alice_on_m@M -W %h:%p' \
  -i ~/.ssh/pri
  vate_key_for_o alice_on_o@O

where we assume that private_key_for_m and private_key_for_o are two private keys on host H, and their respective public keys are in ~alice_on_m/.ssh/authorized_keys at M and ~alice_on_o/.ssh/authorized_keys at O.

To me,  the most convenient method is to set up ~/.ssh/config file on H, with which, we will save some typing on the command line. Below is an example for the ~/.ssh/config file where we add two entries, one for M, and the other for O.

Host M
    HostName M
    ServerAliveInterval 240
    User alice_on_m
    AddKeysToAgent yes
    IdentityFile ~/.ssh/private_key_for_m

Host O
    HostName O
    ServerAliveInterval 240
    User alice_on_o
    AddKeysToAgent yes
    ProxyCommand ssh alice_on_m@M -W %h:%p
    IdentityFile ~/.ssh/private_key_for_o

Once we set up this, we can get on O from H (via M) by simply issing the following command on H,

ssh alice_on_o@O

2. Copying Files from O to H via M using rsync

Assuming we set up ~/.ssh/config correctly, to copy files from O to H via M using rsync, we simply issue a command like the following,

rsync -avzr alice_on_o@O:./foo/ ./foo/

which copies the directory recursively ~alice_on_o/foo at host O to ./foo at host H. Here we also rely on rsync's ability to use SSH as the transport protocol transparently.

Is this neat?

Displaying Command Line Arguments on Windows without Installing Additional Software?

On Windows, can we obtain from the command line a process's command line arguments without installing any additional software like the Process Explorer of the Windows Sysinternal? As documented by a Stack Overflow discussion, the answer is Yes, via WMIC. Here are two examples,

1. List all processes and their command line arguments

   
     WMIC path win32_process get Caption,Processid,Commandline
     

2. List a process, e.g., chrome.exe, and its command line arguments

   
     WMIC path win32_process where "caption='chrome.exe'" get Caption,Processid,Commandline  
     

Inpsecting Python Objects

It is useful to inspect objects when we run Python interactively. Below is a few commands we can use to inspect Python objects.

• type
• dir
• vars
• __dict__

Below are a few examples showing how we may use these:

$ python
>>> # This is to show the version of Pyhon I am using.
>>> import sys
>>> print(sys.version)
3.7.9 (default, Aug 19 2020, 17:05:11)
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)]
>>> 
>>> # create a tempfile.TemporaryDirectory object
>>> import tempfile
>>> d = tempfile.TemporaryDirectory()
>>>
>>> # now inspect the d object
>>> # 1. what is d's data type?
>>> type(d)
<class 'tempfile.TemporaryDirectory'>
>>> # 2. what are d's attributes?
>>> dir(d)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', 
 '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__gt__', 
 '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', 
 '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', 
 '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 
 '__weakref__', '_cleanup', '_finalizer', 'cleanup', 'name']
>>>
>>> # 3. Return the __dict__ attribute for d
>>> vars(d)
{'name': '/tmp/tmpmjddeo49', '_finalizer': }
>>> 
>>> # this is the same
>>> d.__dict__
{'name': '/tmp/tmpmjddeo49', '_finalizer': }
>>> # so d has an "attribute" called 'name', and it is a string!
>>> type(d.name)
<class 'str'>
>>> d.name
'/tmp/tmpmjddeo49'
>>> # d also has an "attribute" called 'cleanup', and it is a method!
>>> type(d.cleanup)
<class 'method'>
>>> d.cleanup()
>>>

Computing File Hash on Windows

Two "build-in" tools we can use to compute file hash values on Windows are Windows PowerShell and CertUtil.

In Windows PowerShell, we can compute a hash value for a file using Get-FileHash command.

C:\> PowerShell
PS C:\> help Get-FileHash

NAME
    Get-FileHash

SYNTAX
    Get-FileHash [-Path] <string[]> [-Algorithm {SHA1 | SHA256 | SHA384 | SHA512 | MACTripleDES | MD5 | RIPEMD160}]
    [<CommonParameters>]

    Get-FileHash -LiteralPath <string[]> [-Algorithm {SHA1 | SHA256 | SHA384 | SHA512 | MACTripleDES | MD5 |
    RIPEMD160}]  [<CommonParameters>]

    Get-FileHash -InputStream <Stream> [-Algorithm {SHA1 | SHA256 | SHA384 | SHA512 | MACTripleDES | MD5 | RIPEMD160}]
     [<CommonParameters>]


ALIASES
    None


REMARKS
    Get-Help cannot find the Help files for this cmdlet on this computer. It is displaying only partial help.
        -- To download and install Help files for the module that includes this cmdlet, use Update-Help.
        -- To view the Help topic for this cmdlet online, type: "Get-Help Get-FileHash -Online" or
           go to https://go.microsoft.com/fwlink/?LinkId=517145.

Windows also comes with a tool called CertUtil that provides an option to compute file hash value as well.

C:\>CertUtil -hashfile -?
Usage:
  CertUtil [Options] -hashfile InFile [HashAlgorithm]
  Generate and display cryptographic hash over a file

Options:
  -Unicode          -- Write redirected output in Unicode
  -gmt              -- Display times as GMT
  -seconds          -- Display times with seconds and milliseconds
  -v                -- Verbose operation
  -privatekey       -- Display password and private key data
  -pin PIN                  -- Smart Card PIN
  -sid WELL_KNOWN_SID_TYPE  -- Numeric SID
            22 -- Local System
            23 -- Local Service
            24 -- Network Service

Hash algorithms: MD2 MD4 MD5 SHA1 SHA256 SHA384 SHA512

CertUtil -?              -- Display a verb list (command list)
CertUtil -hashfile -?    -- Display help text for the "hashfile" verb
CertUtil -v -?           -- Display all help text for all verbs
  

Below are examples.

C:\> CertUtil -hashfile C:\Windows\System32\drivers\etc\hosts sha512
SHA512 hash of C:\Windows\System32\drivers\etc\hosts:
59119e66f5945029f8652c5981589d95cace534adc6780ccea736b7e776615caa0b567c14d161271d6066f57d9bab0d4055850162f5a046c0456264b7b9e7508
CertUtil: -hashfile command completed successfully.

C:\> PowerShell
PS C:\> Get-FileHash -Path C:\Windows\System32\drivers\etc\hosts -Algorithm sha512 | Format-List


Algorithm : SHA512
Hash      : 59119E66F5945029F8652C5981589D95CACE534ADC6780CCEA736B7E776615CAA0B567C14D161271D6066F57D9BAB0D4055850162F5
            A046C0456264B7B9E7508
Path      : C:\Windows\System32\drivers\etc\hosts