Tuesday, December 2, 2014

A Shell Script to Setup Redmine on Fedora Linux

The steps to install and configure Redmine on Fedora Linux 20 are quite lengthy. To simplify the installation and configuration of the Redmine and the Fedora Linux system, I created a shell script to automate the process. The script is available at GitHub.

Below is a typical use case of the script.

# bring Fedora Linux up-to-date
sudo yum -y update


# download instredmine.sh
wget https://raw.githubusercontent.com/graychan/notes/master/redmine/instredmine.sh

# choose database password by editing the script
# vi instredmine.sh

# make the script executable and run the script
chmod +x ./instredmine.sh
sudo ./instredmine.sh

# allow incoming connections to TCP port 80 through firewall
sudo firewall-cmd --zone=public --add-port 80/tcp 
sudo firewall-cmd --permanent --zone=public --add-port 80/tcp
Then, point a browser to the Redmine host. Log in as admin with the initial password admin, and change the admin user's password to a different one.

Monday, December 1, 2014

Installing Redmine (2.6.0) on Fedora Linux (Fedora Linux 20)

I have been running Redmine in Fedora Linux from Fedora Linux 16 to Fedora Linux 20. The instruction I am using to install and upgrade Redmine is from a Redmine's wiki page. In this post, I would like to first briefly document the steps to install Redmine on Feodra Linux, at present Fedora Linux 20, integrate Redmine with the Apache HTTP server, and then discuss a few issues that we would probably need to deal with when upgrading Fedora Linux and  in turn when upgrading Redmine.

Note that the process described below can be automated by a shell script I wrote. See this post for more detail.

Installing Redmine on Fedora Linux 20

The instruction is generally a copy of a Redmine's wiki page with more detailed steps on Fedora Linux and with intention to answer the questions raised in the page here.

Meeting Prerequisites

Before we begin, we will make sure that the Fedora Linux systems has the following software installed,
  • Ruby and Ruby bundler
  • Apache HTTP Server
  • Phusion Passenger.
    Although Redmine can run as a standalone server, it is recommended to integrate it with a HTTP server. One may use Phusion Passenger, FCGI or a Rack server (Unicorn, Thin, Puma, hellip;) to integrate with the Apache HTTP Server. In this post, we use Phusion Passenger.
  • PostgreSQL database server and PostgreSQL development package.
    Among two relational database management system that Redmine supports, MySQL and PostgreSQL, I personally prefer PostgreSQL to MySQL as a result that PostgreSQL's license is less restricted than MySQL's.
  • The GNU C complier. 
  • The image display and manipulation tool ImageMagick and its development package
  • The non-interactive network downloader wget
  • The archiver tool tar
In addition, we assume that a user has an username what is in the "Administrator" group, i.e., the user can switch to user "root" using via <code>sudo</code>.

In this section, we show the steps to meet the prerequisites. If the prerequisites are already met, the following steps are expected not to have any impact on the system.
  1. Installing Ruby and Ruby bundler.
    
      # install the Ruby package
      sudo yum install ruby
    
      # install the Ruby development package
      sudo yum install ruby-devel
    
      # install Ruby's bundler
      sudo yum install rubygem-bundler
      
  2. Installing Apache HTTP Server.
    
      # install the Apache HTTP Server
      sudo yum install httpd
    
      # enable the HTTPD service at system boot time
      sudo systemctl enable httpd.service
    
      # start the HTTPD service
      sudo systemctl start httpd.service
      
  3. Installing Phusion Passenger for Apache HTTP Server.
    
      sudo yum install mod_passenger
      
  4. Installing PostgreSQL database server and PostgreSQL development package.
    
      # install the PostgreSQL database server package
      sudo yum install postgresql-server postgresql-devel
    
      # enable postgresql's start during the boot using following command
      sudo systemctl enable postgresql.service
    
      # initialize the postgresql database
      sudo postgresql-setup initdb
    
      # start the postgresql service. 
      sudo systemctl start postgresql.service
      
  5. Installing the GNU C complier.
    
      sudo yum install gcc
        
  6. Installing the image display and manipulation tool ImageMagick
    
      sudo yum install ImageMagick ImageMagick-devel 
      
  7. Installing the non-interactive network downloader wget.
    
      sudo yum install wget
      
  8. Install the archiver tool tar.
    
      sudo yum install tar
      
  9. Install a few additional packages to work with subversion.
    
      yum install perl-DBI perl-Digest-SHA mod_dav_svn
      

Installing Redmine

  1. Download and extract Redmine source.

    The URL of Redmine sources are available in this page.

    Assume that we are to download Redmine 2.6.0 (released on 2014-10-21) and plan to install it at the directory /var/www/redmine.
    
      cd /var/www/
      sudo wget -q http://www.redmine.org/releases/redmine-2.6.0.tar.gz
      sudo tar -xzf redmine-2.6.0.tar.gz
      
    We can now verify that we have successfully downloaded and extracted the Redmine source.
    
      $ ls -l redmine-2.6.0
      total 76
      drwxrwxr-x. 7 1000 1000 4096 Oct 21 15:13 app
      drwxrwxr-x. 5 1000 1000 4096 Oct 21 15:13 config
      -rw-rw-r--. 1 1000 1000  160 Oct 21 15:13 config.ru
      -rw-rw-r--. 1 1000 1000  240 Oct 21 15:13 CONTRIBUTING.md
      drwxrwxr-x. 3 1000 1000 4096 Oct 21 15:13 db
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 doc
      drwxrwxr-x. 5 1000 1000 4096 Oct 21 15:13 extra
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 files
      -rw-rw-r--. 1 1000 1000 3555 Oct 21 15:13 Gemfile
      drwxrwxr-x. 8 1000 1000 4096 Oct 21 15:13 lib
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 log
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 plugins
      drwxrwxr-x. 7 1000 1000 4096 Oct 21 15:13 public
      -rw-rw-r--. 1 1000 1000  275 Oct 21 15:13 Rakefile
      -rw-rw-r--. 1 1000 1000  205 Oct 21 15:13 README.rdoc
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 script
      drwxrwxr-x. 9 1000 1000 4096 Oct 21 15:13 test
      drwxrwxr-x. 8 1000 1000 4096 Oct 21 15:13 tmp
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 vendor
      
  2. Creating an empty database and accompanying user for Redmine.

    We now create a PostgreSQL database for Redmine. In the commands below, we assume the password is "my_password" for demonstration and should choose a more thoughtful one.

    Also, we also name the database for Redmine as "redmine". It does not have to be and can be any valid PostgreSQL database name.
    
      # switch to user postgres
      sudo su - postgres
    
      # create database role redmine
      echo \
        "CREATE ROLE redmine LOGIN ENCRYPTED
         PASSWORD 'my_password' NOINHERIT VALID UNTIL 'infinity';" | \
        psql
    
      # create database named redmine for Redmine
      echo \
        "CREATE DATABASE redmine WITH ENCODING='UTF8' OWNER=redmine;" | \
        psql
    
      # edit PostgreSQL Client Authentication Configuration File
      sed -e \
        "/host.*all.*all.*::1\/128.*ident/i host redmine redmine ::1/128 md5" \
        --in-place=.bu /var/lib/pgsql/data/pg_hba.conf  
    
      # restart PostgreSQL database server
      sudo systemctl restart postgresql.service
    
      # exit from user postgres
      exit
      
    The change to the PostgreSQL client authentication configuration file, by default at /var/lib/pgsql/data/pg_hba.conf is necessary to Redmine to establish a connection to the database. The working configuration depends on a few factors and can be an issue. More discussion is available. However, the above configuration works on a Fedora Linux distribution with IPv6 enabled, which is the default setting.
  3. Configuring database connection.

    We will follow the instruction of Step 3 in the Redmine wiki page. We will copy ${REDMINE}/config/database.yml.example to ${REDMINE}/config/database.yml and edit the file in order to configure the database settings for "production" environment. In this example, ${REDMINE} is /var/www/redmine-2.6.0.

    With the PostgreSQL database parameters used in the above step, the "production" section of ${REDMINE}/config/database.yml should be replaced as follows,
    
      production:
        adapter: postgresql
        database: redmine
        host: localhost
        username: redmine
        password: my_password
        encoding: utf8
      
    If you do not wish to use the MySQL database management system at all, comment out any sections that refers to the MySQL database management system in the configuration file. In this post, we assume that you do not use the MySQL database management system and your system many not have the MySQL client library and development head files installed. To make the point clear, we list the content of a desired database configuration file as follows,
    
      $ cat ${REDMINE}/config/database.yml
      production:
        adapter: postgresql
        database: redmine
        host: localhost
        username: redmine
        password: my_password
        encoding: utf8  
      
    where ${REDMINE} is /var/www/redmine-2.6.0 in this example. More discussion on potential dependency issue caused by the configuration file is available.
  4. Installing Redmine Ruby dependencies using the Ruby Dependency Management tool bundle.

    Following the generic instruction in the Redmine wiki page, We shall install Redmine's ruby dependencies using bundle.

    There are some subtle issues to install Redmine's Ruby dependencies using the tool. We use a workaround to circumvent the issues, that is, we run bundle under a non-root user with login permission and specify the path in which the dependencies will be installed. In this example, we assume that the non-root username is your username foouser and the directory to install the dependencies is ${REDMINE}/.bundle/ruby where ${REDMINE} is /var/www/redmine-2.6.0 in this example. Replace foouser with your actual username.
    
      # give your user userfoo read and write permission to ${REDMINE}
      sudo chown -R userfoo ${REDMINE}
    
      # make sure that the working directory is ${REDMINE}
      # in this example, ${REDMINE} is /var/www/redmine-2.6.0
      cd ${REDMINE} 
     
      # install Redmine's Ruby dependencies 
      /usr/bin/bundle install \
          --path /var/www/redmine-2.6.0/.bundle \
          --without development test
      
    If success, the output of the above steps will be something similar to the following,
    
      $ /usr/bin/bundle install \
          --path /var/www/redmine-2.6.0/.bundle \
          --without development test
      Fetching gem metadata from https://rubygems.org/.........
      Resolving dependencies...
      Using rake 10.4.0
      Using i18n 0.6.11
      ......
      Your bundle is complete!
      Gems in the groups development and test were not installed.
      It was installed into ./.bundle
      Post-install message from rdoc:
      Depending on your version of ruby, you may need to install ruby rdoc/ri data:
    
      <= 1.8.6 : unsupported
       = 1.8.7 : gem install rdoc-data; rdoc-data --install
       = 1.9.1 : gem install rdoc-data; rdoc-data --install
      >= 1.9.2 : nothing to do! Yay!
      ......
      
    Since on Fedora Linux 20, the version of Ruby is 2.0 or higher, there is nothing else to do for the Ruby dependencies.

    During the above process, a configuration file ${REDMINE}/.bundle/config will be created. The following shows an example of the configuration file created during the process.
    
      $ cat ${REDMINE}/.bundle/config
      ---
      BUNDLE_PATH: /var/www/redmine-2.6.0/.bundle
      BUNDLE_WITHOUT: development:test
      BUNDLE_DISABLE_SHARED_GEMS: '1'
      
    The configuration file informs Redmine where the Ruby dependencies is as indicated by the BUNDLE_PATH variable in the configuraiton file.
  5. Generating session store secret.

    We simply follow the instruction in the Redmine wiki page. Note that in this example, ${REDMINE} is /var/www/redmine-2.6.0.
    
      /usr/bin/bundle exec \
          "${REDMINE}/.bundle/ruby/bin/rake generate_secret_token"
      
  6. Loading Redmine database default data set.

    Again, we simply follow the instruction in the Redmine wiki page.
    
      /usr/bin/bundle exec "RAILS_ENV=production rake db:migrate"
      
  7. Setting up file system permissions.
    Again, we simply follow the instruction in the Redmine wiki page. However, there is a minor deviation since we set up Redmine using your username. We will make necessary changes to integrate it with the Apache HTTP Server in next section. Again ${REDMINE}$ in this example is /var/www/redmine-2.6.0.
    
      mkdir -p ${REDMINE}/tmp ${REDMINE}/tmp/pdf ${REDMINE}/public/plugin_assets
      chmod 755 ${REDMINE}/tmp ${REDMINE}/tmp/pdf ${REDMINE}/public/plugin_assets
      
    Again, in this example, ${REDMINE} is /var/www/redmine-2.6.0.
  8. Testing the Redmine installation.

    We follow the instrucion in the Redmine wiki page.
    
      $ ruby script/rails server webrick -e production
      => Booting WEBrick
      => Rails 3.2.19 application starting in production on http://0.0.0.0:3000
      => Call with -d to detach
      => Ctrl-C to shutdown server
      [2014-11-30 19:05:45] INFO  WEBrick 1.3.1
      [2014-11-30 19:05:45] INFO  ruby 2.0.0 (2013-11-22) [x86_64-linux]
      [2014-11-30 19:05:45] INFO  WEBrick::HTTPServer#start: pid=6454 port=3000
      
    Pointing a browser to http://localhost:3000, you will see Redmine's welcome page, which indicates that Redmine runs OK. However, as indicated in the Redmine wiki page, this is not intended means of running Redmine.

Changing Redmine Admin User's Initial Password

Redmine's has an Admin user "admin" with initial password "admin". Before we make the Redmine available to the outside world, we should change the password of the Admin user "admin". When we test the Redmine in previous step, log in as admin and reset the password.

Integrating Redmine with Apache HTTP Server via Phusion Passenger


    We shall integrate Redmine with the Apache HTTP Server via Phusion Passenger. Fedora Linux by default has SeLinux enabled. SeLinux can be a source of problems. We will defer configuration for SeLinux in next section. Then, we will temporarily disable SeLinux.

  1. Temporarily disabling SeLinux
    
      sudo setenforce false
      
  2. Change Redmine installaiton's ownership
    To create a workaround to install Redmine's Ruby dependencies using bundle, we changed the ownership of the Redmine installation to your username. We now change the ownership to the user as whom the Apache HTTP Server, i.e., user apache is running and remove access to the installation from any users but user apache and group apache.
    
      sudo chown -R apache:apache ${REDMINE}
      sudo chmod o-rwx ${REDMINE}
      
    Note that ${REDMINE} is /var/www/redmine-2.6.0 in this example.
  3. Configuring the Phusion Passenger module to integrate Redmine with the Apache Web server.

    When the Phusion Passenger module package is installed, an Apache HTTP Server configuration file will be installed at /etc/httpd/conf.d/. The configuration file is /etc/httpd/conf.d/passenger.conf. We will append the following lines to the configuration file.
          
          <VirtualHost *:80>
            SetEnv GEM_HOME /var/www/redmine-2.6.0/.bundle/ruby
            ServerName localhost.localdomain
            DocumentRoot /var/www/redmine-2.6.0/public
            <Directory /var/www/redmine-2.6.0/public>
              AllowOverride all
              Options -MultiViews
            </Directory>
          </VirtualHost>
          
    In the above, the environment variable GEM_HOME is important. Without its pointing to the location of the Gem files installed by bundle, i.e., /var/www/redmine-2.6.0/.bundle/ruby, Redmine will complains that it cannot find the required Gem files. More discussion on this issue is available.

Configuring SeLinux for Redmine

Redmine has a wiki page Redmine, Phusion Passenger, Ruby Enterprise Edition, Apache and ... SELinux that is our beginning point. Fedora Linux 20 has a SeLinux policy module that properly set up SeLinux permissions for Phusion Passenger, which makes the steps simpler than the Redmine's wiki page. The following steps is a minor revision of the RedmineAndSELinux script in the Redmine wiki page.


  APACHE_USER=apache
  REDMINE=/var/www/redmine-2.6.0
  GEM_HOME=${REDMINE}/.bundle

  # Base permissions.
  sudo chown -R ${APACHE_USER}:${APACHE_USER} ${REDMINE}
  sudo chmod -R u=rw,g=r,o-rwx ${REDMINE}
  sudo chmod -R ug+X ${REDMINE}
  sudo chcon -R -u system_u -t httpd_sys_content_t ${REDMINE}

  # Writable directories.
  sudo chown -R ${APACHE_USER}:${APACHE_USER} ${REDMINE}/log
  sudo chcon -R -t httpd_log_t ${REDMINE}/log

  sudo chown -R ${APACHE_USER}:${APACHE_USER} ${REDMINE}/tmp
  sudo chcon -R -t httpd_tmpfs_t ${REDMINE}/tmp

  sudo chown -R ${APACHE_USER}:${APACHE_USER} ${REDMINE}/files
  sudo chcon -R -t httpd_sys_script_rw_t ${REDMINE}/files

  sudo chown -R ${APACHE_USER}:${APACHE_USER} ${REDMINE}/public/plugin_assets
  sudo chcon -R -t httpd_sys_script_rw_t ${REDMINE}/public/plugin_assets

  # Set permission to the shared objects used by Ruby GEM files
  sudo find -P ${GEM_HOME} -type f -name "*.so*" -exec chcon -t lib_t {} \;

  # enable SeLinux
  sudo setenforce 1

  # allow Apache to connect to database
  sudo setsebool -P httpd_can_network_connect_db 1

  # restart the Apache HTTP Server
  sudo systemctl restart httpd.service
  
Alternatively, you may download and execute the shell script selinuxforredmine.sh at Github, for instance,

  sudo ./selinuxforredmine.sh /var/www/redmine-2.6.0
  sudo setenforce 1
  sudo systemctl restart httpd.service
  

Configuring Firewall

To allow the outside world to connect to the Redmine instance on the Fedora Linux system, we will have to configure the firewall to allow incoming connection at TCP port 80. One of my previous post discusses how we can add an allowed port in Fedora's firewalld.

This is final step, we are now done.



Common Issues During Redmine Installation on Fedora Linux

Authentication with PostgreSQL Database

In the above, the PostgreSQL client authentication configuration file works for a Fedora Linux system with IPv6 enabled since ::1/128 is an IPv6 address. IPv6 is enabled on Fedora Linux 20 by default.

If the IPv6 is disabled, then the line to be added to the configuration file, by default at /var/lib/pgsql/data/hg_hba.conf should be,

  host    redmine    redmine    127.0.0.1/32  md5
  
Be aware that the order of such configuration lines in the configuration file matters. The line must appear before a more generic configuration line, as such, the segment of the configuration upon addition of the line should resemble the following,

  host    redmine    redmine    127.0.0.1/32  md5
  host    all        all        127.0.0.1/32  ident
  
If you place the line after the more generic configuration line, the more restrictive line wil have no effect. For instance, the segment of configuration is as follows, a database client program will attempt to connect to the datbase using the "ident" method and fail to establish the connection.

  host    all        all        127.0.0.1/32  ident
  host    redmine    redmine    127.0.0.1/32  md5
  

Potential Dependency Issues Caused by database.yml

The configuration file ${REDMINE}/config/database.yml specifies what database management system and databases that Redmine relies on. For instance, the configuration file has a section as follows,

  development:
    adapter: mysql2
    database: redmine_development
    host: localhost
    username: root
    password: ""
    encoding: utf8
  
The configuration file creates a dependency on the MySQL database management system and the Ruby Dependency Management Tool bundle will attempt to install the MySQL Gem file. If the system does not have required the MySQL client library and the MySQL development header files, the bundle installation process will fail as shown in the example below,

$ /usr/bin/bundle install \
  --path /var/www/redmine-2.6.0/.bundle \
  --without development test
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Using rake 10.4.0
Using i18n 0.6.11
Using multi_json 1.10.1
......
Using jquery-rails 3.1.2

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

    /usr/bin/ruby extconf.rb
checking for ruby/thread.h... yes
checking for rb_thread_call_without_gvl() in ruby/thread.h... yes
......
checking for mysql_query() in -lmysqlclient... no
......
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
          .....
        --without-mysqlclientlib
        --with-mygcclib
        --without-mygcclib
        --with-mysqlclientlib
        --without-mysqlclientlib


Gem files will remain installed in /var/www/redmine-2.6.0/.bundle/ruby/gems/mysql2-0.3.17 for inspection.
Results logged to /var/www/redmine-2.6.0/.bundle/ruby/gems/mysql2-0.3.17/ext/mysql2/gem_make.out

An error occurred while installing mysql2 (0.3.17), and Bundler cannot continue.
Make sure that `gem install mysql2 -v '0.3.17'` succeeds before bundling.
  
The existence of such a dependency is a fact. Therefore, the error remains even if you instruct bundle explicitly not to install MySQL Gem files, e.g.,

  $ /usr/bin/bundle install \
    --path /var/www/redmine-2.6.0/.bundle \
    --without development test mysql mysql2
  

Using Ruby Dependency Management Tool

We install Redmine's ruby dependencies using bundle. The instruction in the Redmine wiki page is a generic description that some appear to be confused with what to do.

In the Redmine's wiki page, The Ruby bundle is installed via gem as in gem install bundler. However, it will install it in your local directory unless you run gem as root. When you install it as root, i.e., sudo gem install bundler. The bundler will be located in /usr/local. You will to make sure that /usr/local/bin in your search path. In the step above, I simplify it by installing Fedora Linux's bundler package, and then bundle will be in /usr/bin.

We need to first identify a user under whom the Redmine's ruby dependencies should be installed. In this post, we assume that we will integrate Redmine with the Apache web server. Then the user under whom Redmine's ruby dependencies should be installed is the user apache since in Fedora Linux, the Apache web server is run under user apache.

The challenge is that in a typical Fedora Linux installation, you cannot switch to user apache since the user is created without the login permission. Furthermore, if you attempt to install the Redmine's ruby dependencies under user root, you will observe an error as shown below,

  $ sudo bundle install --without development test
  Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
  installing your bundle as root will break this application for all 
  non-root users on this machine.
  Could not locate Gemfile
  
In the above, we use a workaround. Assume that your username is userfoo. We install Redmine's ruby dependencies under the username userfoo at a specific directory, in this example, ${REDMINE}/.bundle/ruby. Again, ${REDMINE} is /var/www/redmine-2.6.0.

Locating Gem Files

You may install Gem files to any directory using bundle. However, you will set the value of environmental variable GEM_HOME to the directory. In the above, we set the environment variable in the Apache HTTP Server's configuration file /etc/httpd/conf.d/passenger.conf. For instance, if we comment out the SetEnv line from the configuration file, we will observe an error with the error messages as follows,

  You have already activated rake 10.0.4, but your Gemfile requires rake 10.4.0. 
  Prepending `bundle exec` to your command may solve this. (Gem::LoadError)
  /usr/share/gems/gems/bundler-1.7.6/lib/bundler/runtime.rb:34:in `block in setup'
  /usr/share/gems/gems/bundler-1.7.6/lib/bundler/runtime.rb:19:in `setup'
  /usr/share/gems/gems/bundler-1.7.6/lib/bundler.rb:121:in `setup'
  ......
  
The effect of the environment variable can be demonstrated using the two examples below, one without the environment variable's value being set, one with the value being set.

  $ gem env
  RubyGems Environment:
  - RUBYGEMS VERSION: 2.1.11
  - RUBY VERSION: 2.0.0 (2013-11-22 patchlevel 353) [i386-linux]
  ......
  - GEM PATHS:
     - /home/foouser/.gem/ruby
     - /usr/share/gems
     - /usr/local/share/gems
  ......
  $ gem list
  
Note the above, GEM PATHS do not include the directory /var/www/redmine-2.6.0/.bundle/ruby. Then, Ruby will not be able to locate the Gem files installed using bundle.

   $ gem list

   *** LOCAL GEMS ***

   bigdecimal (1.2.0)
   bundler (1.7.6)
   daemon_controller (1.1.8)
   io-console (0.4.2)
   json (1.7.7)
   net-http-persistent (2.9.4)
   passenger (4.0.18)
   psych (2.0.0)
   rack (1.5.2)
   rake (10.0.4)
   rdoc (4.0.1)
   thor (0.18.1)
   
We see from the above that the activated Rake is of version 10.0.4 that is indicated in the error message shown above. However, we will observe different results if we set the GEM_HOME directory to the directory where we install Gem files using bundle.

   $ GEM_HOME=/var/www/redmine-2.6.0/.bundle/ruby gem env
   RubyGems Environment:
   ......
   - GEM PATHS:
     - /var/www/redmine-2.6.0/.bundle/ruby
     - /home/foouser/.gem/ruby
     - /usr/share/gems
     - /usr/local/share/gems
   .....
   
Now /var/www/redmine-2.6.0/.bundle/ruby is in GEM PATHS.

   $ GEM_HOME=/var/www/redmine-2.6.0/.bundle/ruby gem list

   *** LOCAL GEMS ***

   actionmailer (3.2.19)
   ......
   rake (10.4.0, 10.0.4)
   ......
   
We now see that the Gem files installed using bundle are listed. It also shows that it finds two version of Rake.

Saturday, November 29, 2014

No Space in /boot and Removing Old Kernels

A practice that Linux distributions recommends is to put Linux kernel images in a separate partition that is mounted at /boot. It is a common problem that the /boot partition becomes full, which prevent any new kernel images from being installed to copied to the partition. To resolve this problem, there are commonly two solutions.
  • Solution 1. Increasing the size of the /boot partition.
  • Solution 2. Removing old kernel images from the /boot partition.
This post discusses Solution 2. Previously I took a note how we may remove old kernel images and packages from Ubuntu Linux (or any Debian-based Linux distributions). In this post, I would like to take a note on removing old kernel images and packages from Fedora Linux (or any Redhat Linux-based systems). I have been used two methods that lead to the same results. I would recommend the second method for simplicity. However, if you want to know what packages and images are removed, you may want to use the first method.
  • Using rpm.
    1. First determine what Linux kernel images are present in the /boot partition.
      
             $ ls /boot/vmlinu*
             /boot/vmlinuz-3.14.23-100.fc19.x86_64
             /boot/vmlinuz-3.17.3-200.fc20.x86_64
             

      The above shows that we have two Linux kernels present in the /boot partition. Now we want to make room in the /boot partition. Once we make sure that we can boot from the /boot/vmlinuz-3.17.3-200.fc20.x86_64, we can remove the old kernel image and associated kernel package.
    2. We now find out what packages that we need to uninstall to remove the old kernel package, in this case, /boot/vmlinuz-3.14.23-100.fc19.x86_64.
      
             $ rpm -qa | grep 3.14.23
             kernel-devel-3.14.23-100.fc19.x86_64
             kernel-3.14.23-100.fc19.x86_64
             
    3. From the above, we can now remove the two packages.
      
             $ sudo rpm -evh \
                    kernel-devel-3.14.23-100.fc19.x86_64 \
                    kernel-3.14.23-100.fc19.x86_64
             

      Alternatively, we can also use yum to remove the two packages.
      
             sudo yum remove \
                    kernel-devel-3.14.23-100.fc19.x86_64 \
                    kernel-3.14.23-100.fc19.x86_64
             
  • Using the yum-utils package.
    1. We first install the yum-utils package.
      
              sudo yum install yum-utils
              
    2. The yum-utils package has a tool called package-cleanup. We now use it to remove old kernel images and associated packages, which is done by specifying the number of kernel images that we would like to keep in the system. In this example, we would like to keep only one kernel image. Then, we issue the following command,
      
              sudo package-cleanup --oldkernels --count=1
              
    I learned the method of using yum-utils from a web post here, which also discusses how you may configure the system to always keep a given number of kernel images and associated packages.

Upgrading Fedora Linux using Fedup: Key is not Trusted by RPM

When I attempted to upgrade Fedora Linux using FedUp, I encountered an error that basically complains that fedup could not authenticate the downloaded Fedora Linux image because it did not have the public key to verity the image's GPG signature. In the error log, by default in /var/log/fedup.log, the error message resembles the following format,

             ......
[   152.927] (II) fedup.yum:check_keyfile() 
             checking keyfile /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-x86_64
[   152.991] (DD) fedup.yum:check_keyfile() 
             keyfile owned by package fedora-release-0:18-6
[   152.992] (DD) fedup.yum:check_keyfile() 
             package was signed with key de7f38bd
[   152.992] (II) fedup.yum:check_keyfile() 
             REJECTED: key de7f38bd is not trusted by rpm
[   152.997] (II) fedup:message() Downloading failed: 
             could not verify GPG signature: No public key
             ......

The above example corresponds to the result of upgrading to Fedora Linux 19. In the error message, the key will be different if we upgrade to a different version of Fedora Linux. However, if the system is missing the public key for the GPG signature of the Linux image that we are upgrading to, the same error will occur and the error message will be the same except the value of the public key. To resolve the problem, we need to manually import the public key to the GPG signature of the Linux image. The public keys to GPG signatures of the Linux images of various version of Fedora Linux are available at the following URL

           https://fedoraproject.org/keys

Since it is Fedora Linux 19 whose public key is not present in the system, we shall first locate the public key from the above page as illustrated below,


From the page, we can see that the primary public key is available in two URLs, one at the Fedora Project's website and the other at the gngpg.net website. We can import the key from either URL. The following shows that we import the public key from the Fedora Project's website.

rpm --import https://fedoraproject.org/static/BA094068.txt

Once we finish importing the public key, we can rerun fedup and the error should go away. The above method should work for other versions of Fedora Linux.

Tuesday, November 25, 2014

Upgrading PostgreSQL Databases

One day after I upgraded my Fedora Linux box, I found that I could not start the PostgreSQL database system service. Examining logs, i.e., /var/log/messages in my case, I found the following log messages,

Oct 23 21:33:56 localhost postgresql-check-db-dir: 
       An old version of the database format was found.
Oct 23 21:33:56 localhost postgresql-check-db-dir: 
       You need to dump and reload before using PostgreSQL 9.3.
Oct 23 21:33:56 localhost postgresql-check-db-dir: 
       See /usr/share/doc/postgresql/README.rpm-dist for more information.
Oct 23 21:33:56 localhost systemd: 
       postgresql.service: control process exited, code=exited status=1
Oct 23 21:33:56 localhost systemd: 
       Failed to start PostgreSQL database server.
Oct 23 21:33:56 localhost systemd: 
       Unit postgresql.service entered failed state.

The above message basically indicates that the database file is in an old format and the file format is no longer supported by the new version of the PostgreSQL database system, in the above, not supported by PostgreSQL 9.3. The solution is to upgrade the database file to the new format supported by the new version of PostgreSQL. According to /usr/share/doc/postgresql/README.rpm-dist, there are two options.

  • Option 1.
    If you are upgrading across more than one major release of PostgreSQL
    (for example, from 8.3.x to 9.0.x), you will need to follow the "traditional"
    dump and reload process to bring your data into the new version.  That is:
    *before* upgrading, run pg_dumpall to extract all your data into a SQL file.
    Shut down the old postmaster, upgrade to the new version RPMs, initdb,
    and run the dump file through psql to restore your data.
  • Options 2.
    In some major releases, the RPMs also support in-place upgrade from the immediately previous major release. Currently, you can upgrade in-place from 9.2.x to 9.3.x.
We now need to determine what version of the PostgreSQL whose format the database files are in, for which we go to the database file directory. In Fedora Linux, by default, it is at /var/lib/pgsql. We can then determine the version of PostgreSQL as follows.

$ sudo cat /var/lib/pgsql/data/PG_VERSION
9.1

Since PostgreSQL is version 9.1, we will have to use Option 1 described above. The Option 1 requires us to use pg_dumpall of the old version of PostgreSQL, in this case, version 9.1 to dump the content of the databases to a SQL file. If you observe the eror message mentioned at the beginning of this post, it is very likely that the old version of PostgreSQL has been upgraded to the new version and the old version does not exist on your Linux system.

To upgrade the database file format, I use a workaround in which I download and install the old version of PostgreSQL and use it to complete the database upgrade process as shown below.

  1. Open a terminal window and prepare a directory in which we will install the old version of PostgreSQL.
    
      mkdir -p $HOME/pgold/source
      
  2. Download the source code of the olde version of PostgreSQL. In my case, the old version of PostgreSQL is 9.1. Source code of old versions of PostgreSQL are available at the PostgreSQL's web site.
    
      cd $HOME/pgold/source
      wget -c \
        https://ftp.postgresql.org/pub/source/v9.1.14/postgresql-9.1.14.tar.bz2
      
  3. We now extract the source code, compile, and install the old version of PostgreSQL to directory $HOME/pgold/postgresql.
    
      tar xjvf postgresql-9.1.14.tar.bz2
      cd postgresql-9.1.14
      ./configure --prefix=$HOME/pgold/postgresql
      make
      make install
      
  4. We now run an instance of the old version of PostgreSQL in foreground. Typically, the PostgreSQL service is running under username postgres. We will first become postgres and then run the old version of PostgreSQL as follows.
    
      sudo -s       # to become root
      su - postgres # to become postgres
      $HOME/pgold/postgresql/bin/postmaster -D /var/lib/pgsql/data
      
  5. Open a second terminal window and dump the content of the old databases to a SQL file.
    
      # in the second terminal window
      sudo -s       # to become root
      su - postgres # to become postgres
      pg_dumpall > /var/lib/pgsql/postgresql_olddb.sql
      
    If you somehow do not possess the PostgreSQL's password of user postgres or any other database administrator's credential and cannot run pg_dumpall without a successful authentication, you can edit the PostgreSQL Client Authentication Configuration File /var/lib/pgsql/data/pg_hba.conf to trust any Unix domain socket connections to have a line as follows,
    
      # in /var/lib/pgsql/data/pg_hba.conf
      local   all  all  trust
      
    You will have to restart the old PostgreSQL program by simply killing processing using CTRL-C and running the postmaster again as described above.
  6. You can now shutdown the old PostgreSQL program. You will go to the terminal window where the postmaster program is running foreground and press CTRL-C to kill the process.
  7. Now backup the old database files. In case of a failure to complete the process, we can restore the database to its original state using the backup. In either the first or the second terminal window, you should remain as user postgres.
    
      mv /var/lib/pgsql/data /var/lib/pgsql/data_olddb.backup
      
  8. Now initialize database files using binaries of the new version of PostgreSQL in either terminal window in which you should remain as user postgres.
    
      /bin/initdb -D /var/lib/pgsql/data
      
  9. In one of the two terminal window, exit to your shell and then start the new PostgreSQL service.
    
      exit # exit to your shell
      sudo systemctl start postgresql.service
      
    Now you should be able to start the new PostgreSQL service.
  10. We now import the content of the old database to the new database. You will need to go to the terminal window in which you remain as user postgres.
    
      psql < /var/lib/pgsql/postgresql_olddb.sql
      
We are done and can now celebrate.

Adding Allowed Ports for Firewalld

Fedora Linux, CentOS and Redhat Linux distributions have a firewall called Firewalld.This is a note on adding allowed TCP/UDP ports to the firewall.

  1. Add a port, for instance, add a TCP port 63221 to be allowed by the firewall.
    
      sudo firewall-cmd --zone=public --add-port=63221/tcp
      
    Firewalld has a few zones. The public is for accepting incoming connection on a given port from other computers, in the above example, on TCP port 63221, since you do not trust the other computers on networks not to harm your computer.

    Be aware that the addition of the port will be effective upon the completion of the above operation; however, it is not persistent, i.e., it will not survive from reloading firewall or rebooting the system. In next step, we will make the change persistent.
  2. Make the firewall rule change permanent. For instance, to add TCP port 63221 to the allowed list of ports, we do
    
      sudo firewall-cmd --permanent --zone=public --add-port=63221/tcp
      
    Be aware that in the above, the rule does not go into effective until Firewalld is reloaded or the system is rebooted. That is to say, without reloading Firewalld or rebooting the system, we will have to run both of the above two steps to make the firewall rule change effective and permanent.
To learn more about firewalld, you may consult the manual page of firewall-cmd and visit the Fedora Project Wiki.



Sunday, November 23, 2014

Configuring SeLinux to Allow Secure Shell Service (SSHD) on Non-Default Port

The Secure Shell Service (SSHD) by default runs on TCP port 22. We sometimes want the Secure Shell Service (SSHD) to listen on a non-default port or to listen on more than one port. When the Security-Ehanced Linux (SeLinux) is turned on, we will have to configure both the Secure Shell Service and SeLinux because SeLinux by default has a policy that allows the Secure Shell Service to listen on TCP port 22 only.

Below are the steps for a recent release of the Fedora Linux distribution in which the Secure Shell Service (SSHD) is provided by OpenSSH and services are managed by systemd to allow the Secure Shell Service to listen on both TCP ports 22 and 64422.

  • Configure OpenSSH to listen on 64422.
    OpenSSH's configuration files in Fedora Linux is at /etc/ssh. Open /etc/ssh/sshd_config and add the following line to the file.
    
        Port 64422
        
  • Configure SeLinux to allow the Secure Shell Service to bind to TCP port 64422. We use semanager to configure SeLinux policies. First, let us look up what ports are allowed to bind to the Secure Shell Service.
    
        $ sudo semanage port -l | grep ssh
        ssh_port_t                     tcp      22
        
    which shows that the Secure Shell Service is allowed to bind to TCP port 22. We can now add a second port, 64422, to the list of ports that are allowed to bind to the Secure Shell Service.
    
        $ sudo semanage port -a -t ssh_port_t -p tcp 64422
        
    The above step usually takes a while to run. Upcon completion, we can now verify that the Secure Shell Service can now bind to both TCP ports 22 and 64422.
    
        $ sudo semanage port -l | grep ssh
        ssh_port_t                     tcp      64422, 22
        
  • Restart the Secure Shell Service using systemctl.
    
        $ sudo systemctl restart sshd.service
        
    Upon completion, we can verify that the service is now listenting to both TCP port 22 and 64422.
    
        $ sudo systemctl status sshd.service
        sshd.service - OpenSSH server daemon
              Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled)
              Active: active (running) 
                      since Sun, 23 Nov 2014 00:34:31 -0500; 6s ago
             Process: 2311 ExecStartPre=/usr/sbin/sshd-keygen 
                           (code=exited, status=0/SUCCESS)
            Main PID: 2312 (sshd)
              CGroup: name=systemd:/system/sshd.service
                      └ 2312 /usr/sbin/sshd -D
    
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on 0.0.0.0 port 64422.
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on :: port 64422.
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on 0.0.0.0 port 22.
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on :: port 22.