Saturday, November 29, 2014

No Space in /boot and Removing Old Kernels

A practice that Linux distributions recommends is to put Linux kernel images in a separate partition that is mounted at /boot. It is a common problem that the /boot partition becomes full, which prevent any new kernel images from being installed to copied to the partition. To resolve this problem, there are commonly two solutions.
  • Solution 1. Increasing the size of the /boot partition.
  • Solution 2. Removing old kernel images from the /boot partition.
This post discusses Solution 2. Previously I took a note how we may remove old kernel images and packages from Ubuntu Linux (or any Debian-based Linux distributions). In this post, I would like to take a note on removing old kernel images and packages from Fedora Linux (or any Redhat Linux-based systems). I have been used two methods that lead to the same results. I would recommend the second method for simplicity. However, if you want to know what packages and images are removed, you may want to use the first method.
  • Using rpm.
    1. First determine what Linux kernel images are present in the /boot partition.
             $ ls /boot/vmlinu*

      The above shows that we have two Linux kernels present in the /boot partition. Now we want to make room in the /boot partition. Once we make sure that we can boot from the /boot/vmlinuz-3.17.3-200.fc20.x86_64, we can remove the old kernel image and associated kernel package.
    2. We now find out what packages that we need to uninstall to remove the old kernel package, in this case, /boot/vmlinuz-3.14.23-100.fc19.x86_64.
             $ rpm -qa | grep 3.14.23
    3. From the above, we can now remove the two packages.
             $ sudo rpm -evh \
                    kernel-devel-3.14.23-100.fc19.x86_64 \

      Alternatively, we can also use yum to remove the two packages.
             sudo yum remove \
                    kernel-devel-3.14.23-100.fc19.x86_64 \
  • Using the yum-utils package.
    1. We first install the yum-utils package.
              sudo yum install yum-utils
    2. The yum-utils package has a tool called package-cleanup. We now use it to remove old kernel images and associated packages, which is done by specifying the number of kernel images that we would like to keep in the system. In this example, we would like to keep only one kernel image. Then, we issue the following command,
              sudo package-cleanup --oldkernels --count=1
    I learned the method of using yum-utils from a web post here, which also discusses how you may configure the system to always keep a given number of kernel images and associated packages.

Upgrading Fedora Linux using Fedup: Key is not Trusted by RPM

When I attempted to upgrade Fedora Linux using FedUp, I encountered an error that basically complains that fedup could not authenticate the downloaded Fedora Linux image because it did not have the public key to verity the image's GPG signature. In the error log, by default in /var/log/fedup.log, the error message resembles the following format,

[   152.927] (II) fedup.yum:check_keyfile() 
             checking keyfile /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-x86_64
[   152.991] (DD) fedup.yum:check_keyfile() 
             keyfile owned by package fedora-release-0:18-6
[   152.992] (DD) fedup.yum:check_keyfile() 
             package was signed with key de7f38bd
[   152.992] (II) fedup.yum:check_keyfile() 
             REJECTED: key de7f38bd is not trusted by rpm
[   152.997] (II) fedup:message() Downloading failed: 
             could not verify GPG signature: No public key

The above example corresponds to the result of upgrading to Fedora Linux 19. In the error message, the key will be different if we upgrade to a different version of Fedora Linux. However, if the system is missing the public key for the GPG signature of the Linux image that we are upgrading to, the same error will occur and the error message will be the same except the value of the public key. To resolve the problem, we need to manually import the public key to the GPG signature of the Linux image. The public keys to GPG signatures of the Linux images of various version of Fedora Linux are available at the following URL


Since it is Fedora Linux 19 whose public key is not present in the system, we shall first locate the public key from the above page as illustrated below,

From the page, we can see that the primary public key is available in two URLs, one at the Fedora Project's website and the other at the website. We can import the key from either URL. The following shows that we import the public key from the Fedora Project's website.

rpm --import

Once we finish importing the public key, we can rerun fedup and the error should go away. The above method should work for other versions of Fedora Linux.

Tuesday, November 25, 2014

Upgrading PostgreSQL Databases

One day after I upgraded my Fedora Linux box, I found that I could not start the PostgreSQL database system service. Examining logs, i.e., /var/log/messages in my case, I found the following log messages,

Oct 23 21:33:56 localhost postgresql-check-db-dir: 
       An old version of the database format was found.
Oct 23 21:33:56 localhost postgresql-check-db-dir: 
       You need to dump and reload before using PostgreSQL 9.3.
Oct 23 21:33:56 localhost postgresql-check-db-dir: 
       See /usr/share/doc/postgresql/README.rpm-dist for more information.
Oct 23 21:33:56 localhost systemd: 
       postgresql.service: control process exited, code=exited status=1
Oct 23 21:33:56 localhost systemd: 
       Failed to start PostgreSQL database server.
Oct 23 21:33:56 localhost systemd: 
       Unit postgresql.service entered failed state.

The above message basically indicates that the database file is in an old format and the file format is no longer supported by the new version of the PostgreSQL database system, in the above, not supported by PostgreSQL 9.3. The solution is to upgrade the database file to the new format supported by the new version of PostgreSQL. According to /usr/share/doc/postgresql/README.rpm-dist, there are two options.

  • Option 1.
    If you are upgrading across more than one major release of PostgreSQL
    (for example, from 8.3.x to 9.0.x), you will need to follow the "traditional"
    dump and reload process to bring your data into the new version.  That is:
    *before* upgrading, run pg_dumpall to extract all your data into a SQL file.
    Shut down the old postmaster, upgrade to the new version RPMs, initdb,
    and run the dump file through psql to restore your data.
  • Options 2.
    In some major releases, the RPMs also support in-place upgrade from the immediately previous major release. Currently, you can upgrade in-place from 9.2.x to 9.3.x.
We now need to determine what version of the PostgreSQL whose format the database files are in, for which we go to the database file directory. In Fedora Linux, by default, it is at /var/lib/pgsql. We can then determine the version of PostgreSQL as follows.

$ sudo cat /var/lib/pgsql/data/PG_VERSION

Since PostgreSQL is version 9.1, we will have to use Option 1 described above. The Option 1 requires us to use pg_dumpall of the old version of PostgreSQL, in this case, version 9.1 to dump the content of the databases to a SQL file. If you observe the eror message mentioned at the beginning of this post, it is very likely that the old version of PostgreSQL has been upgraded to the new version and the old version does not exist on your Linux system.

To upgrade the database file format, I use a workaround in which I download and install the old version of PostgreSQL and use it to complete the database upgrade process as shown below.

  1. Open a terminal window and prepare a directory in which we will install the old version of PostgreSQL.
      mkdir -p $HOME/pgold/source
  2. Download the source code of the olde version of PostgreSQL. In my case, the old version of PostgreSQL is 9.1. Source code of old versions of PostgreSQL are available at the PostgreSQL's web site.
      cd $HOME/pgold/source
      wget -c \
  3. We now extract the source code, compile, and install the old version of PostgreSQL to directory $HOME/pgold/postgresql.
      tar xjvf postgresql-9.1.14.tar.bz2
      cd postgresql-9.1.14
      ./configure --prefix=$HOME/pgold/postgresql
      make install
  4. We now run an instance of the old version of PostgreSQL in foreground. Typically, the PostgreSQL service is running under username postgres. We will first become postgres and then run the old version of PostgreSQL as follows.
      sudo -s       # to become root
      su - postgres # to become postgres
      $HOME/pgold/postgresql/bin/postmaster -D /var/lib/pgsql/data
  5. Open a second terminal window and dump the content of the old databases to a SQL file.
      # in the second terminal window
      sudo -s       # to become root
      su - postgres # to become postgres
      pg_dumpall > /var/lib/pgsql/postgresql_olddb.sql
    If you somehow do not possess the PostgreSQL's password of user postgres or any other database administrator's credential and cannot run pg_dumpall without a successful authentication, you can edit the PostgreSQL Client Authentication Configuration File /var/lib/pgsql/data/pg_hba.conf to trust any Unix domain socket connections to have a line as follows,
      # in /var/lib/pgsql/data/pg_hba.conf
      local   all  all  trust
    You will have to restart the old PostgreSQL program by simply killing processing using CTRL-C and running the postmaster again as described above.
  6. You can now shutdown the old PostgreSQL program. You will go to the terminal window where the postmaster program is running foreground and press CTRL-C to kill the process.
  7. Now backup the old database files. In case of a failure to complete the process, we can restore the database to its original state using the backup. In either the first or the second terminal window, you should remain as user postgres.
      mv /var/lib/pgsql/data /var/lib/pgsql/data_olddb.backup
  8. Now initialize database files using binaries of the new version of PostgreSQL in either terminal window in which you should remain as user postgres.
      /bin/initdb -D /var/lib/pgsql/data
  9. In one of the two terminal window, exit to your shell and then start the new PostgreSQL service.
      exit # exit to your shell
      sudo systemctl start postgresql.service
    Now you should be able to start the new PostgreSQL service.
  10. We now import the content of the old database to the new database. You will need to go to the terminal window in which you remain as user postgres.
      psql < /var/lib/pgsql/postgresql_olddb.sql
We are done and can now celebrate.

Adding Allowed Ports for Firewalld

Fedora Linux, CentOS and Redhat Linux distributions have a firewall called Firewalld.This is a note on adding allowed TCP/UDP ports to the firewall.

  1. Add a port, for instance, add a TCP port 63221 to be allowed by the firewall.
      sudo firewall-cmd --zone=public --add-port=63221/tcp
    Firewalld has a few zones. The public is for accepting incoming connection on a given port from other computers, in the above example, on TCP port 63221, since you do not trust the other computers on networks not to harm your computer.

    Be aware that the addition of the port will be effective upon the completion of the above operation; however, it is not persistent, i.e., it will not survive from reloading firewall or rebooting the system. In next step, we will make the change persistent.
  2. Make the firewall rule change permanent. For instance, to add TCP port 63221 to the allowed list of ports, we do
      sudo firewall-cmd --permanent --zone=public --add-port=63221/tcp
    Be aware that in the above, the rule does not go into effective until Firewalld is reloaded or the system is rebooted. That is to say, without reloading Firewalld or rebooting the system, we will have to run both of the above two steps to make the firewall rule change effective and permanent.
To learn more about firewalld, you may consult the manual page of firewall-cmd and visit the Fedora Project Wiki.

Sunday, November 23, 2014

Configuring SeLinux to Allow Secure Shell Service (SSHD) on Non-Default Port

The Secure Shell Service (SSHD) by default runs on TCP port 22. We sometimes want the Secure Shell Service (SSHD) to listen on a non-default port or to listen on more than one port. When the Security-Ehanced Linux (SeLinux) is turned on, we will have to configure both the Secure Shell Service and SeLinux because SeLinux by default has a policy that allows the Secure Shell Service to listen on TCP port 22 only.

Below are the steps for a recent release of the Fedora Linux distribution in which the Secure Shell Service (SSHD) is provided by OpenSSH and services are managed by systemd to allow the Secure Shell Service to listen on both TCP ports 22 and 64422.

  • Configure OpenSSH to listen on 64422.
    OpenSSH's configuration files in Fedora Linux is at /etc/ssh. Open /etc/ssh/sshd_config and add the following line to the file.
        Port 64422
  • Configure SeLinux to allow the Secure Shell Service to bind to TCP port 64422. We use semanager to configure SeLinux policies. First, let us look up what ports are allowed to bind to the Secure Shell Service.
        $ sudo semanage port -l | grep ssh
        ssh_port_t                     tcp      22
    which shows that the Secure Shell Service is allowed to bind to TCP port 22. We can now add a second port, 64422, to the list of ports that are allowed to bind to the Secure Shell Service.
        $ sudo semanage port -a -t ssh_port_t -p tcp 64422
    The above step usually takes a while to run. Upcon completion, we can now verify that the Secure Shell Service can now bind to both TCP ports 22 and 64422.
        $ sudo semanage port -l | grep ssh
        ssh_port_t                     tcp      64422, 22
  • Restart the Secure Shell Service using systemctl.
        $ sudo systemctl restart sshd.service
    Upon completion, we can verify that the service is now listenting to both TCP port 22 and 64422.
        $ sudo systemctl status sshd.service
        sshd.service - OpenSSH server daemon
              Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled)
              Active: active (running) 
                      since Sun, 23 Nov 2014 00:34:31 -0500; 6s ago
             Process: 2311 ExecStartPre=/usr/sbin/sshd-keygen 
                           (code=exited, status=0/SUCCESS)
            Main PID: 2312 (sshd)
              CGroup: name=systemd:/system/sshd.service
                      └ 2312 /usr/sbin/sshd -D
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on port 64422.
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on :: port 64422.
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on port 22.
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on :: port 22.

Thursday, November 13, 2014

Bibtex Entries for IETF RFCs and Internet-Drafts

Sometimes we want to cite the Internet Engineering Task Force's Request for Comment (RFC) or Internet-Draft documents. The Internet-Draft draft-carpenter-rfc-citation-recs-01 entitled "Recommendations of a committee on RFC citation issues" suggests a format to cite RFCs and Internet-Drafts.  Although the draft recommends that the RFC Editor create and maintain a canonical BibTeX file at a stable public location on the web server "", the BibTex files do not seem to have appeared on the website.

This post provides a means to produce a BibTex entry for a RFC or Internet-Draft document, generally comforming to the recommendation stated in the draft cited above, when the type of documents, RFC or Internet-Draft and the document number are provided. You may then copy and paste the BibTex entry to your .bib file. Note that the Internet-Draft number is not really a number, for instance, for the draft that mentioned above, the number is "draft-carpenter-rfc-citation-recs-01" (without quotation marks).

RFC Document Type and Number

To fill with a bibtex entry.

Wednesday, November 12, 2014

Testing SMTP MSA from Command Line

Sometimes there is a need to test a Simple Mail Transfer Protocol (SMTP) Mail Submission Agent (MSA) from the command line in a terminal window instead of running a mail client. Wikipedia shows an SMTP converstational example. However, you may not run the example as is from the command line in a terminal window on most SMTP servers today since they typically require a secure connection and proper authentication to improve security and reduce SPAMs.

This post demonstrates a procedure to test SMTP transport from a terminal window using Gmail's SMTP server. The idea, of course can be applied to other SMTP servers.

Gmail's SMTP server's setting on the client side at present is as follows,

Outgoing Mail (SMTP) Server - Requires TLS
    Port: 465 or 587
    Requires SSL: Yes
    Requires authentication: Yes

This server is Gmail's SMTP MSA. Based on the setting above,  we shall do the following.
  • First, we need to be able to communicate with the server at using TLS, for which we will use OpenSSL.
  • Second, we will have to be able to authenticate with the SMTP server, for which we will use SASL.

The above are availalbe in most Linux distributions. The following steps are tested on a Ubuntu 14.04 machine.

  1. Open a Linux terminal.
  2. Install OpenSSL and SASL-bin packages
        sudo apt-get install openssl sasl2-bin
  3. Select an authentication method and generate an authentication string. Assume your Gmail e-mail address is with password secretestring and your choose the PLAIN authentication method. We can use the gen-auth tool in the SASL-bin package to generate the authentication string as follows,
       gen-auth PLAIN secretestring
    The output is a hash string as shown below,
       Auth String: AGdyYXkuY2hlbmh1aUBnbWFpbC5jb20Ac3Q5OTg4IUAj
    This hash string will be used in next step.

    By the way, you can actually find out what authentication method a SMTP server supports in the steps follows. You can always run the steps to find it out and then generate the authentication string.
  4. Using openssl to establish a secure connection to Gmail's SMTP server at at port 465
        openssl s_client -crlf -connect
    The last line of the output of the above operation will be something as follows,
        220 ESMTP e23sm3343567vdk.23 - gsmtp
    At this point, we will send a EHLO message to the SMTP server. In the following, we assume that your domain is somesubdomain.somedomain and your IP address is
        EHLO somesubdomain.somedomain
    The output will resemble something below,
    at your service, []
        250-SIZE 35882577
        250 SMTPUTF8
    The message contains the line
    which indicates what authentication method you may use. As discussed above, we use the PLAIN method and have already obtained the authentication string. We can now authenticate with the SMTP server using the authentication string.
  5. We now authenticate with the server by sending an AUTH message as follows,
        AUTH PLAIN AGdyYXkuY2hlbmh1aUBnbWFpbC5jb20Ac3Q5OTg4IUAj
    At this point, you should be successfully authenticated as indicated by the acknowledgement from the SMTP server below,
        235 2.7.0 Accepted
    You can now start a converstation with the SMTP server similar to the Wikipedia example.
  6. Below is a converstation with the SMTP server to send a short e-mail to friend.of.somebody@somesubdomain.somedomain and copy the email to another.friend.of.somebody@somesubdomain.somedomain. Note that the lines starting with "C:" are what you, a client of the SMTP server would enter and the lines starting with "S:" are what the SMTP server would output. Obviously, "C:" should not be part of your input.
       C: mail from:<>
       S: 250 2.1.0 OK ki9sm3203907vdb.16 - gsmtp
       C: rcpt to:<friend.of.somebody@somesubdomain.somedomain>
       S: 250 2.1.5 OK ki9sm3203907vdb.16 - gsmtp
       C: data
       S: 354  Go ahead ki8sm3602907vdb.16 - gsmtp
       C: From: "foo somebody" <>
       C: To: "Friend of Somebody" <friend.of.somebody@somesubdomain.somedomain>
       C: Cc: "Another Friend" <other.friend.of.somebody@somesubdomain.somedomain>
       C: Date: Wed, 12 November 2014 17:29:43 -0500
       C: Subject: Test Message from Command Line
       C: hello friend,
       C: reply me please. I am testing smtp server.
       C: thanks.
       C: your friend
       C: .
       S: 250 2.0.0 OK 1415831485 ki9sm3203907vdb.16 - gsmtp
       C: quit
       S: 221 2.0.0 closing connection ki9sm3203907vdb.16 - gsmtp
       OpenSSL: read:errno=0
    Note in the above,
    • "read:errno=0" is an output from openssl rather than an output from the SMTP server.
    • The blank lines above, i.e, the lines of "C:" indicate that you will enter a new line.
    • The end of conversation is marked with "<CRLF>.<CRLF>". See the last "." in the above converstation. The "-crlf" provided in the "openssl s_client" command line is to convert a line feed to a <CRLF>, a carriage return followed by a line feed.
    • An important item to note is that the "openssl s_client" has the following behavior as described in the manual page of "s_client" that you may view using "man s_client" as follows,
      If a connection is established with an SSL server then any data received from the server is displayed and any key presses will be sent to the server. When used interactively (which means neither -quiet nor -ign_eof have been given), the session will be renegotiated if the line begins with an R, and if the line begins with a Q or if end of file is reached, the connection will be closed down.
      which means that you cannot use letter R and Q in the entire interactive openssl session. If you use R, such as type RCPT TO: ... instead of what is shown in the above, you will encounter an error as follows even though SMTP treats the same.
         C: RCPT TO:<friend.of.somebody@somesubdomain.somedomain>
         OpenSSL: 3073996476:error:1409E0E5:SSL 
           routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:


The following web posts  were excellent references for writing this post,
  • The post here led me to SASL and subsequently the SASL-bin package for the generation of authentication strings.
  • The posts here and here helped me figure out the "RENEGOTIATING" feature of "openssl s_client