Tuesday, December 2, 2014

A Shell Script to Setup Redmine on Fedora Linux

The steps to install and configure Redmine on Fedora Linux 20 are quite lengthy. To simplify the installation and configuration of the Redmine and the Fedora Linux system, I created a shell script to automate the process. The script is available at GitHub.

Below is a typical use case of the script.

# bring Fedora Linux up-to-date
sudo yum -y update


# download instredmine.sh
wget https://raw.githubusercontent.com/graychan/notes/master/redmine/instredmine.sh

# choose database password by editing the script
# vi instredmine.sh

# make the script executable and run the script
chmod +x ./instredmine.sh
sudo ./instredmine.sh

# allow incoming connections to TCP port 80 through firewall
sudo firewall-cmd --zone=public --add-port 80/tcp 
sudo firewall-cmd --permanent --zone=public --add-port 80/tcp
Then, point a browser to the Redmine host. Log in as admin with the initial password admin, and change the admin user's password to a different one.

Monday, December 1, 2014

Installing Redmine (2.6.0) on Fedora Linux (Fedora Linux 20)

I have been running Redmine in Fedora Linux from Fedora Linux 16 to Fedora Linux 20. The instruction I am using to install and upgrade Redmine is from a Redmine's wiki page. In this post, I would like to first briefly document the steps to install Redmine on Feodra Linux, at present Fedora Linux 20, integrate Redmine with the Apache HTTP server, and then discuss a few issues that we would probably need to deal with when upgrading Fedora Linux and  in turn when upgrading Redmine.

Note that the process described below can be automated by a shell script I wrote. See this post for more detail.

Installing Redmine on Fedora Linux 20

The instruction is generally a copy of a Redmine's wiki page with more detailed steps on Fedora Linux and with intention to answer the questions raised in the page here.

Meeting Prerequisites

Before we begin, we will make sure that the Fedora Linux systems has the following software installed,
  • Ruby and Ruby bundler
  • Apache HTTP Server
  • Phusion Passenger.
    Although Redmine can run as a standalone server, it is recommended to integrate it with a HTTP server. One may use Phusion Passenger, FCGI or a Rack server (Unicorn, Thin, Puma, hellip;) to integrate with the Apache HTTP Server. In this post, we use Phusion Passenger.
  • PostgreSQL database server and PostgreSQL development package.
    Among two relational database management system that Redmine supports, MySQL and PostgreSQL, I personally prefer PostgreSQL to MySQL as a result that PostgreSQL's license is less restricted than MySQL's.
  • The GNU C complier. 
  • The image display and manipulation tool ImageMagick and its development package
  • The non-interactive network downloader wget
  • The archiver tool tar
In addition, we assume that a user has an username what is in the "Administrator" group, i.e., the user can switch to user "root" using via <code>sudo</code>.

In this section, we show the steps to meet the prerequisites. If the prerequisites are already met, the following steps are expected not to have any impact on the system.
  1. Installing Ruby and Ruby bundler.
    
      # install the Ruby package
      sudo yum install ruby
    
      # install the Ruby development package
      sudo yum install ruby-devel
    
      # install Ruby's bundler
      sudo yum install rubygem-bundler
      
  2. Installing Apache HTTP Server.
    
      # install the Apache HTTP Server
      sudo yum install httpd
    
      # enable the HTTPD service at system boot time
      sudo systemctl enable httpd.service
    
      # start the HTTPD service
      sudo systemctl start httpd.service
      
  3. Installing Phusion Passenger for Apache HTTP Server.
    
      sudo yum install mod_passenger
      
  4. Installing PostgreSQL database server and PostgreSQL development package.
    
      # install the PostgreSQL database server package
      sudo yum install postgresql-server postgresql-devel
    
      # enable postgresql's start during the boot using following command
      sudo systemctl enable postgresql.service
    
      # initialize the postgresql database
      sudo postgresql-setup initdb
    
      # start the postgresql service. 
      sudo systemctl start postgresql.service
      
  5. Installing the GNU C complier.
    
      sudo yum install gcc
        
  6. Installing the image display and manipulation tool ImageMagick
    
      sudo yum install ImageMagick ImageMagick-devel 
      
  7. Installing the non-interactive network downloader wget.
    
      sudo yum install wget
      
  8. Install the archiver tool tar.
    
      sudo yum install tar
      
  9. Install a few additional packages to work with subversion.
    
      yum install perl-DBI perl-Digest-SHA mod_dav_svn
      

Installing Redmine

  1. Download and extract Redmine source.

    The URL of Redmine sources are available in this page.

    Assume that we are to download Redmine 2.6.0 (released on 2014-10-21) and plan to install it at the directory /var/www/redmine.
    
      cd /var/www/
      sudo wget -q http://www.redmine.org/releases/redmine-2.6.0.tar.gz
      sudo tar -xzf redmine-2.6.0.tar.gz
      
    We can now verify that we have successfully downloaded and extracted the Redmine source.
    
      $ ls -l redmine-2.6.0
      total 76
      drwxrwxr-x. 7 1000 1000 4096 Oct 21 15:13 app
      drwxrwxr-x. 5 1000 1000 4096 Oct 21 15:13 config
      -rw-rw-r--. 1 1000 1000  160 Oct 21 15:13 config.ru
      -rw-rw-r--. 1 1000 1000  240 Oct 21 15:13 CONTRIBUTING.md
      drwxrwxr-x. 3 1000 1000 4096 Oct 21 15:13 db
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 doc
      drwxrwxr-x. 5 1000 1000 4096 Oct 21 15:13 extra
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 files
      -rw-rw-r--. 1 1000 1000 3555 Oct 21 15:13 Gemfile
      drwxrwxr-x. 8 1000 1000 4096 Oct 21 15:13 lib
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 log
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 plugins
      drwxrwxr-x. 7 1000 1000 4096 Oct 21 15:13 public
      -rw-rw-r--. 1 1000 1000  275 Oct 21 15:13 Rakefile
      -rw-rw-r--. 1 1000 1000  205 Oct 21 15:13 README.rdoc
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 script
      drwxrwxr-x. 9 1000 1000 4096 Oct 21 15:13 test
      drwxrwxr-x. 8 1000 1000 4096 Oct 21 15:13 tmp
      drwxrwxr-x. 2 1000 1000 4096 Oct 21 15:13 vendor
      
  2. Creating an empty database and accompanying user for Redmine.

    We now create a PostgreSQL database for Redmine. In the commands below, we assume the password is "my_password" for demonstration and should choose a more thoughtful one.

    Also, we also name the database for Redmine as "redmine". It does not have to be and can be any valid PostgreSQL database name.
    
      # switch to user postgres
      sudo su - postgres
    
      # create database role redmine
      echo \
        "CREATE ROLE redmine LOGIN ENCRYPTED
         PASSWORD 'my_password' NOINHERIT VALID UNTIL 'infinity';" | \
        psql
    
      # create database named redmine for Redmine
      echo \
        "CREATE DATABASE redmine WITH ENCODING='UTF8' OWNER=redmine;" | \
        psql
    
      # edit PostgreSQL Client Authentication Configuration File
      sed -e \
        "/host.*all.*all.*::1\/128.*ident/i host redmine redmine ::1/128 md5" \
        --in-place=.bu /var/lib/pgsql/data/pg_hba.conf  
    
      # restart PostgreSQL database server
      sudo systemctl restart postgresql.service
    
      # exit from user postgres
      exit
      
    The change to the PostgreSQL client authentication configuration file, by default at /var/lib/pgsql/data/pg_hba.conf is necessary to Redmine to establish a connection to the database. The working configuration depends on a few factors and can be an issue. More discussion is available. However, the above configuration works on a Fedora Linux distribution with IPv6 enabled, which is the default setting.
  3. Configuring database connection.

    We will follow the instruction of Step 3 in the Redmine wiki page. We will copy ${REDMINE}/config/database.yml.example to ${REDMINE}/config/database.yml and edit the file in order to configure the database settings for "production" environment. In this example, ${REDMINE} is /var/www/redmine-2.6.0.

    With the PostgreSQL database parameters used in the above step, the "production" section of ${REDMINE}/config/database.yml should be replaced as follows,
    
      production:
        adapter: postgresql
        database: redmine
        host: localhost
        username: redmine
        password: my_password
        encoding: utf8
      
    If you do not wish to use the MySQL database management system at all, comment out any sections that refers to the MySQL database management system in the configuration file. In this post, we assume that you do not use the MySQL database management system and your system many not have the MySQL client library and development head files installed. To make the point clear, we list the content of a desired database configuration file as follows,
    
      $ cat ${REDMINE}/config/database.yml
      production:
        adapter: postgresql
        database: redmine
        host: localhost
        username: redmine
        password: my_password
        encoding: utf8  
      
    where ${REDMINE} is /var/www/redmine-2.6.0 in this example. More discussion on potential dependency issue caused by the configuration file is available.
  4. Installing Redmine Ruby dependencies using the Ruby Dependency Management tool bundle.

    Following the generic instruction in the Redmine wiki page, We shall install Redmine's ruby dependencies using bundle.

    There are some subtle issues to install Redmine's Ruby dependencies using the tool. We use a workaround to circumvent the issues, that is, we run bundle under a non-root user with login permission and specify the path in which the dependencies will be installed. In this example, we assume that the non-root username is your username foouser and the directory to install the dependencies is ${REDMINE}/.bundle/ruby where ${REDMINE} is /var/www/redmine-2.6.0 in this example. Replace foouser with your actual username.
    
      # give your user userfoo read and write permission to ${REDMINE}
      sudo chown -R userfoo ${REDMINE}
    
      # make sure that the working directory is ${REDMINE}
      # in this example, ${REDMINE} is /var/www/redmine-2.6.0
      cd ${REDMINE} 
     
      # install Redmine's Ruby dependencies 
      /usr/bin/bundle install \
          --path /var/www/redmine-2.6.0/.bundle \
          --without development test
      
    If success, the output of the above steps will be something similar to the following,
    
      $ /usr/bin/bundle install \
          --path /var/www/redmine-2.6.0/.bundle \
          --without development test
      Fetching gem metadata from https://rubygems.org/.........
      Resolving dependencies...
      Using rake 10.4.0
      Using i18n 0.6.11
      ......
      Your bundle is complete!
      Gems in the groups development and test were not installed.
      It was installed into ./.bundle
      Post-install message from rdoc:
      Depending on your version of ruby, you may need to install ruby rdoc/ri data:
    
      <= 1.8.6 : unsupported
       = 1.8.7 : gem install rdoc-data; rdoc-data --install
       = 1.9.1 : gem install rdoc-data; rdoc-data --install
      >= 1.9.2 : nothing to do! Yay!
      ......
      
    Since on Fedora Linux 20, the version of Ruby is 2.0 or higher, there is nothing else to do for the Ruby dependencies.

    During the above process, a configuration file ${REDMINE}/.bundle/config will be created. The following shows an example of the configuration file created during the process.
    
      $ cat ${REDMINE}/.bundle/config
      ---
      BUNDLE_PATH: /var/www/redmine-2.6.0/.bundle
      BUNDLE_WITHOUT: development:test
      BUNDLE_DISABLE_SHARED_GEMS: '1'
      
    The configuration file informs Redmine where the Ruby dependencies is as indicated by the BUNDLE_PATH variable in the configuraiton file.
  5. Generating session store secret.

    We simply follow the instruction in the Redmine wiki page. Note that in this example, ${REDMINE} is /var/www/redmine-2.6.0.
    
      /usr/bin/bundle exec \
          "${REDMINE}/.bundle/ruby/bin/rake generate_secret_token"
      
  6. Loading Redmine database default data set.

    Again, we simply follow the instruction in the Redmine wiki page.
    
      /usr/bin/bundle exec "RAILS_ENV=production rake db:migrate"
      
  7. Setting up file system permissions.
    Again, we simply follow the instruction in the Redmine wiki page. However, there is a minor deviation since we set up Redmine using your username. We will make necessary changes to integrate it with the Apache HTTP Server in next section. Again ${REDMINE}$ in this example is /var/www/redmine-2.6.0.
    
      mkdir -p ${REDMINE}/tmp ${REDMINE}/tmp/pdf ${REDMINE}/public/plugin_assets
      chmod 755 ${REDMINE}/tmp ${REDMINE}/tmp/pdf ${REDMINE}/public/plugin_assets
      
    Again, in this example, ${REDMINE} is /var/www/redmine-2.6.0.
  8. Testing the Redmine installation.

    We follow the instrucion in the Redmine wiki page.
    
      $ ruby script/rails server webrick -e production
      => Booting WEBrick
      => Rails 3.2.19 application starting in production on http://0.0.0.0:3000
      => Call with -d to detach
      => Ctrl-C to shutdown server
      [2014-11-30 19:05:45] INFO  WEBrick 1.3.1
      [2014-11-30 19:05:45] INFO  ruby 2.0.0 (2013-11-22) [x86_64-linux]
      [2014-11-30 19:05:45] INFO  WEBrick::HTTPServer#start: pid=6454 port=3000
      
    Pointing a browser to http://localhost:3000, you will see Redmine's welcome page, which indicates that Redmine runs OK. However, as indicated in the Redmine wiki page, this is not intended means of running Redmine.

Changing Redmine Admin User's Initial Password

Redmine's has an Admin user "admin" with initial password "admin". Before we make the Redmine available to the outside world, we should change the password of the Admin user "admin". When we test the Redmine in previous step, log in as admin and reset the password.

Integrating Redmine with Apache HTTP Server via Phusion Passenger


    We shall integrate Redmine with the Apache HTTP Server via Phusion Passenger. Fedora Linux by default has SeLinux enabled. SeLinux can be a source of problems. We will defer configuration for SeLinux in next section. Then, we will temporarily disable SeLinux.

  1. Temporarily disabling SeLinux
    
      sudo setenforce false
      
  2. Change Redmine installaiton's ownership
    To create a workaround to install Redmine's Ruby dependencies using bundle, we changed the ownership of the Redmine installation to your username. We now change the ownership to the user as whom the Apache HTTP Server, i.e., user apache is running and remove access to the installation from any users but user apache and group apache.
    
      sudo chown -R apache:apache ${REDMINE}
      sudo chmod o-rwx ${REDMINE}
      
    Note that ${REDMINE} is /var/www/redmine-2.6.0 in this example.
  3. Configuring the Phusion Passenger module to integrate Redmine with the Apache Web server.

    When the Phusion Passenger module package is installed, an Apache HTTP Server configuration file will be installed at /etc/httpd/conf.d/. The configuration file is /etc/httpd/conf.d/passenger.conf. We will append the following lines to the configuration file.
          
          <VirtualHost *:80>
            SetEnv GEM_HOME /var/www/redmine-2.6.0/.bundle/ruby
            ServerName localhost.localdomain
            DocumentRoot /var/www/redmine-2.6.0/public
            <Directory /var/www/redmine-2.6.0/public>
              AllowOverride all
              Options -MultiViews
            </Directory>
          </VirtualHost>
          
    In the above, the environment variable GEM_HOME is important. Without its pointing to the location of the Gem files installed by bundle, i.e., /var/www/redmine-2.6.0/.bundle/ruby, Redmine will complains that it cannot find the required Gem files. More discussion on this issue is available.

Configuring SeLinux for Redmine

Redmine has a wiki page Redmine, Phusion Passenger, Ruby Enterprise Edition, Apache and ... SELinux that is our beginning point. Fedora Linux 20 has a SeLinux policy module that properly set up SeLinux permissions for Phusion Passenger, which makes the steps simpler than the Redmine's wiki page. The following steps is a minor revision of the RedmineAndSELinux script in the Redmine wiki page.


  APACHE_USER=apache
  REDMINE=/var/www/redmine-2.6.0
  GEM_HOME=${REDMINE}/.bundle

  # Base permissions.
  sudo chown -R ${APACHE_USER}:${APACHE_USER} ${REDMINE}
  sudo chmod -R u=rw,g=r,o-rwx ${REDMINE}
  sudo chmod -R ug+X ${REDMINE}
  sudo chcon -R -u system_u -t httpd_sys_content_t ${REDMINE}

  # Writable directories.
  sudo chown -R ${APACHE_USER}:${APACHE_USER} ${REDMINE}/log
  sudo chcon -R -t httpd_log_t ${REDMINE}/log

  sudo chown -R ${APACHE_USER}:${APACHE_USER} ${REDMINE}/tmp
  sudo chcon -R -t httpd_tmpfs_t ${REDMINE}/tmp

  sudo chown -R ${APACHE_USER}:${APACHE_USER} ${REDMINE}/files
  sudo chcon -R -t httpd_sys_script_rw_t ${REDMINE}/files

  sudo chown -R ${APACHE_USER}:${APACHE_USER} ${REDMINE}/public/plugin_assets
  sudo chcon -R -t httpd_sys_script_rw_t ${REDMINE}/public/plugin_assets

  # Set permission to the shared objects used by Ruby GEM files
  sudo find -P ${GEM_HOME} -type f -name "*.so*" -exec chcon -t lib_t {} \;

  # enable SeLinux
  sudo setenforce 1

  # allow Apache to connect to database
  sudo setsebool -P httpd_can_network_connect_db 1

  # restart the Apache HTTP Server
  sudo systemctl restart httpd.service
  
Alternatively, you may download and execute the shell script selinuxforredmine.sh at Github, for instance,

  sudo ./selinuxforredmine.sh /var/www/redmine-2.6.0
  sudo setenforce 1
  sudo systemctl restart httpd.service
  

Configuring Firewall

To allow the outside world to connect to the Redmine instance on the Fedora Linux system, we will have to configure the firewall to allow incoming connection at TCP port 80. One of my previous post discusses how we can add an allowed port in Fedora's firewalld.

This is final step, we are now done.



Common Issues During Redmine Installation on Fedora Linux

Authentication with PostgreSQL Database

In the above, the PostgreSQL client authentication configuration file works for a Fedora Linux system with IPv6 enabled since ::1/128 is an IPv6 address. IPv6 is enabled on Fedora Linux 20 by default.

If the IPv6 is disabled, then the line to be added to the configuration file, by default at /var/lib/pgsql/data/hg_hba.conf should be,

  host    redmine    redmine    127.0.0.1/32  md5
  
Be aware that the order of such configuration lines in the configuration file matters. The line must appear before a more generic configuration line, as such, the segment of the configuration upon addition of the line should resemble the following,

  host    redmine    redmine    127.0.0.1/32  md5
  host    all        all        127.0.0.1/32  ident
  
If you place the line after the more generic configuration line, the more restrictive line wil have no effect. For instance, the segment of configuration is as follows, a database client program will attempt to connect to the datbase using the "ident" method and fail to establish the connection.

  host    all        all        127.0.0.1/32  ident
  host    redmine    redmine    127.0.0.1/32  md5
  

Potential Dependency Issues Caused by database.yml

The configuration file ${REDMINE}/config/database.yml specifies what database management system and databases that Redmine relies on. For instance, the configuration file has a section as follows,

  development:
    adapter: mysql2
    database: redmine_development
    host: localhost
    username: root
    password: ""
    encoding: utf8
  
The configuration file creates a dependency on the MySQL database management system and the Ruby Dependency Management Tool bundle will attempt to install the MySQL Gem file. If the system does not have required the MySQL client library and the MySQL development header files, the bundle installation process will fail as shown in the example below,

$ /usr/bin/bundle install \
  --path /var/www/redmine-2.6.0/.bundle \
  --without development test
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Using rake 10.4.0
Using i18n 0.6.11
Using multi_json 1.10.1
......
Using jquery-rails 3.1.2

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

    /usr/bin/ruby extconf.rb
checking for ruby/thread.h... yes
checking for rb_thread_call_without_gvl() in ruby/thread.h... yes
......
checking for mysql_query() in -lmysqlclient... no
......
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
          .....
        --without-mysqlclientlib
        --with-mygcclib
        --without-mygcclib
        --with-mysqlclientlib
        --without-mysqlclientlib


Gem files will remain installed in /var/www/redmine-2.6.0/.bundle/ruby/gems/mysql2-0.3.17 for inspection.
Results logged to /var/www/redmine-2.6.0/.bundle/ruby/gems/mysql2-0.3.17/ext/mysql2/gem_make.out

An error occurred while installing mysql2 (0.3.17), and Bundler cannot continue.
Make sure that `gem install mysql2 -v '0.3.17'` succeeds before bundling.
  
The existence of such a dependency is a fact. Therefore, the error remains even if you instruct bundle explicitly not to install MySQL Gem files, e.g.,

  $ /usr/bin/bundle install \
    --path /var/www/redmine-2.6.0/.bundle \
    --without development test mysql mysql2
  

Using Ruby Dependency Management Tool

We install Redmine's ruby dependencies using bundle. The instruction in the Redmine wiki page is a generic description that some appear to be confused with what to do.

In the Redmine's wiki page, The Ruby bundle is installed via gem as in gem install bundler. However, it will install it in your local directory unless you run gem as root. When you install it as root, i.e., sudo gem install bundler. The bundler will be located in /usr/local. You will to make sure that /usr/local/bin in your search path. In the step above, I simplify it by installing Fedora Linux's bundler package, and then bundle will be in /usr/bin.

We need to first identify a user under whom the Redmine's ruby dependencies should be installed. In this post, we assume that we will integrate Redmine with the Apache web server. Then the user under whom Redmine's ruby dependencies should be installed is the user apache since in Fedora Linux, the Apache web server is run under user apache.

The challenge is that in a typical Fedora Linux installation, you cannot switch to user apache since the user is created without the login permission. Furthermore, if you attempt to install the Redmine's ruby dependencies under user root, you will observe an error as shown below,

  $ sudo bundle install --without development test
  Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
  installing your bundle as root will break this application for all 
  non-root users on this machine.
  Could not locate Gemfile
  
In the above, we use a workaround. Assume that your username is userfoo. We install Redmine's ruby dependencies under the username userfoo at a specific directory, in this example, ${REDMINE}/.bundle/ruby. Again, ${REDMINE} is /var/www/redmine-2.6.0.

Locating Gem Files

You may install Gem files to any directory using bundle. However, you will set the value of environmental variable GEM_HOME to the directory. In the above, we set the environment variable in the Apache HTTP Server's configuration file /etc/httpd/conf.d/passenger.conf. For instance, if we comment out the SetEnv line from the configuration file, we will observe an error with the error messages as follows,

  You have already activated rake 10.0.4, but your Gemfile requires rake 10.4.0. 
  Prepending `bundle exec` to your command may solve this. (Gem::LoadError)
  /usr/share/gems/gems/bundler-1.7.6/lib/bundler/runtime.rb:34:in `block in setup'
  /usr/share/gems/gems/bundler-1.7.6/lib/bundler/runtime.rb:19:in `setup'
  /usr/share/gems/gems/bundler-1.7.6/lib/bundler.rb:121:in `setup'
  ......
  
The effect of the environment variable can be demonstrated using the two examples below, one without the environment variable's value being set, one with the value being set.

  $ gem env
  RubyGems Environment:
  - RUBYGEMS VERSION: 2.1.11
  - RUBY VERSION: 2.0.0 (2013-11-22 patchlevel 353) [i386-linux]
  ......
  - GEM PATHS:
     - /home/foouser/.gem/ruby
     - /usr/share/gems
     - /usr/local/share/gems
  ......
  $ gem list
  
Note the above, GEM PATHS do not include the directory /var/www/redmine-2.6.0/.bundle/ruby. Then, Ruby will not be able to locate the Gem files installed using bundle.

   $ gem list

   *** LOCAL GEMS ***

   bigdecimal (1.2.0)
   bundler (1.7.6)
   daemon_controller (1.1.8)
   io-console (0.4.2)
   json (1.7.7)
   net-http-persistent (2.9.4)
   passenger (4.0.18)
   psych (2.0.0)
   rack (1.5.2)
   rake (10.0.4)
   rdoc (4.0.1)
   thor (0.18.1)
   
We see from the above that the activated Rake is of version 10.0.4 that is indicated in the error message shown above. However, we will observe different results if we set the GEM_HOME directory to the directory where we install Gem files using bundle.

   $ GEM_HOME=/var/www/redmine-2.6.0/.bundle/ruby gem env
   RubyGems Environment:
   ......
   - GEM PATHS:
     - /var/www/redmine-2.6.0/.bundle/ruby
     - /home/foouser/.gem/ruby
     - /usr/share/gems
     - /usr/local/share/gems
   .....
   
Now /var/www/redmine-2.6.0/.bundle/ruby is in GEM PATHS.

   $ GEM_HOME=/var/www/redmine-2.6.0/.bundle/ruby gem list

   *** LOCAL GEMS ***

   actionmailer (3.2.19)
   ......
   rake (10.4.0, 10.0.4)
   ......
   
We now see that the Gem files installed using bundle are listed. It also shows that it finds two version of Rake.

Saturday, November 29, 2014

No Space in /boot and Removing Old Kernels

A practice that Linux distributions recommends is to put Linux kernel images in a separate partition that is mounted at /boot. It is a common problem that the /boot partition becomes full, which prevent any new kernel images from being installed to copied to the partition. To resolve this problem, there are commonly two solutions.
  • Solution 1. Increasing the size of the /boot partition.
  • Solution 2. Removing old kernel images from the /boot partition.
This post discusses Solution 2. Previously I took a note how we may remove old kernel images and packages from Ubuntu Linux (or any Debian-based Linux distributions). In this post, I would like to take a note on removing old kernel images and packages from Fedora Linux (or any Redhat Linux-based systems). I have been used two methods that lead to the same results. I would recommend the second method for simplicity. However, if you want to know what packages and images are removed, you may want to use the first method.
  • Using rpm.
    1. First determine what Linux kernel images are present in the /boot partition.
      
             $ ls /boot/vmlinu*
             /boot/vmlinuz-3.14.23-100.fc19.x86_64
             /boot/vmlinuz-3.17.3-200.fc20.x86_64
             

      The above shows that we have two Linux kernels present in the /boot partition. Now we want to make room in the /boot partition. Once we make sure that we can boot from the /boot/vmlinuz-3.17.3-200.fc20.x86_64, we can remove the old kernel image and associated kernel package.
    2. We now find out what packages that we need to uninstall to remove the old kernel package, in this case, /boot/vmlinuz-3.14.23-100.fc19.x86_64.
      
             $ rpm -qa | grep 3.14.23
             kernel-devel-3.14.23-100.fc19.x86_64
             kernel-3.14.23-100.fc19.x86_64
             
    3. From the above, we can now remove the two packages.
      
             $ sudo rpm -evh \
                    kernel-devel-3.14.23-100.fc19.x86_64 \
                    kernel-3.14.23-100.fc19.x86_64
             

      Alternatively, we can also use yum to remove the two packages.
      
             sudo yum remove \
                    kernel-devel-3.14.23-100.fc19.x86_64 \
                    kernel-3.14.23-100.fc19.x86_64
             
  • Using the yum-utils package.
    1. We first install the yum-utils package.
      
              sudo yum install yum-utils
              
    2. The yum-utils package has a tool called package-cleanup. We now use it to remove old kernel images and associated packages, which is done by specifying the number of kernel images that we would like to keep in the system. In this example, we would like to keep only one kernel image. Then, we issue the following command,
      
              sudo package-cleanup --oldkernels --count=1
              
    I learned the method of using yum-utils from a web post here, which also discusses how you may configure the system to always keep a given number of kernel images and associated packages.

Upgrading Fedora Linux using Fedup: Key is not Trusted by RPM

When I attempted to upgrade Fedora Linux using FedUp, I encountered an error that basically complains that fedup could not authenticate the downloaded Fedora Linux image because it did not have the public key to verity the image's GPG signature. In the error log, by default in /var/log/fedup.log, the error message resembles the following format,

             ......
[   152.927] (II) fedup.yum:check_keyfile() 
             checking keyfile /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-x86_64
[   152.991] (DD) fedup.yum:check_keyfile() 
             keyfile owned by package fedora-release-0:18-6
[   152.992] (DD) fedup.yum:check_keyfile() 
             package was signed with key de7f38bd
[   152.992] (II) fedup.yum:check_keyfile() 
             REJECTED: key de7f38bd is not trusted by rpm
[   152.997] (II) fedup:message() Downloading failed: 
             could not verify GPG signature: No public key
             ......

The above example corresponds to the result of upgrading to Fedora Linux 19. In the error message, the key will be different if we upgrade to a different version of Fedora Linux. However, if the system is missing the public key for the GPG signature of the Linux image that we are upgrading to, the same error will occur and the error message will be the same except the value of the public key. To resolve the problem, we need to manually import the public key to the GPG signature of the Linux image. The public keys to GPG signatures of the Linux images of various version of Fedora Linux are available at the following URL

           https://fedoraproject.org/keys

Since it is Fedora Linux 19 whose public key is not present in the system, we shall first locate the public key from the above page as illustrated below,


From the page, we can see that the primary public key is available in two URLs, one at the Fedora Project's website and the other at the gngpg.net website. We can import the key from either URL. The following shows that we import the public key from the Fedora Project's website.

rpm --import https://fedoraproject.org/static/BA094068.txt

Once we finish importing the public key, we can rerun fedup and the error should go away. The above method should work for other versions of Fedora Linux.

Tuesday, November 25, 2014

Upgrading PostgreSQL Databases

One day after I upgraded my Fedora Linux box, I found that I could not start the PostgreSQL database system service. Examining logs, i.e., /var/log/messages in my case, I found the following log messages,

Oct 23 21:33:56 localhost postgresql-check-db-dir: 
       An old version of the database format was found.
Oct 23 21:33:56 localhost postgresql-check-db-dir: 
       You need to dump and reload before using PostgreSQL 9.3.
Oct 23 21:33:56 localhost postgresql-check-db-dir: 
       See /usr/share/doc/postgresql/README.rpm-dist for more information.
Oct 23 21:33:56 localhost systemd: 
       postgresql.service: control process exited, code=exited status=1
Oct 23 21:33:56 localhost systemd: 
       Failed to start PostgreSQL database server.
Oct 23 21:33:56 localhost systemd: 
       Unit postgresql.service entered failed state.

The above message basically indicates that the database file is in an old format and the file format is no longer supported by the new version of the PostgreSQL database system, in the above, not supported by PostgreSQL 9.3. The solution is to upgrade the database file to the new format supported by the new version of PostgreSQL. According to /usr/share/doc/postgresql/README.rpm-dist, there are two options.

  • Option 1.
    If you are upgrading across more than one major release of PostgreSQL
    (for example, from 8.3.x to 9.0.x), you will need to follow the "traditional"
    dump and reload process to bring your data into the new version.  That is:
    *before* upgrading, run pg_dumpall to extract all your data into a SQL file.
    Shut down the old postmaster, upgrade to the new version RPMs, initdb,
    and run the dump file through psql to restore your data.
  • Options 2.
    In some major releases, the RPMs also support in-place upgrade from the immediately previous major release. Currently, you can upgrade in-place from 9.2.x to 9.3.x.
We now need to determine what version of the PostgreSQL whose format the database files are in, for which we go to the database file directory. In Fedora Linux, by default, it is at /var/lib/pgsql. We can then determine the version of PostgreSQL as follows.

$ sudo cat /var/lib/pgsql/data/PG_VERSION
9.1

Since PostgreSQL is version 9.1, we will have to use Option 1 described above. The Option 1 requires us to use pg_dumpall of the old version of PostgreSQL, in this case, version 9.1 to dump the content of the databases to a SQL file. If you observe the eror message mentioned at the beginning of this post, it is very likely that the old version of PostgreSQL has been upgraded to the new version and the old version does not exist on your Linux system.

To upgrade the database file format, I use a workaround in which I download and install the old version of PostgreSQL and use it to complete the database upgrade process as shown below.

  1. Open a terminal window and prepare a directory in which we will install the old version of PostgreSQL.
    
      mkdir -p $HOME/pgold/source
      
  2. Download the source code of the olde version of PostgreSQL. In my case, the old version of PostgreSQL is 9.1. Source code of old versions of PostgreSQL are available at the PostgreSQL's web site.
    
      cd $HOME/pgold/source
      wget -c \
        https://ftp.postgresql.org/pub/source/v9.1.14/postgresql-9.1.14.tar.bz2
      
  3. We now extract the source code, compile, and install the old version of PostgreSQL to directory $HOME/pgold/postgresql.
    
      tar xjvf postgresql-9.1.14.tar.bz2
      cd postgresql-9.1.14
      ./configure --prefix=$HOME/pgold/postgresql
      make
      make install
      
  4. We now run an instance of the old version of PostgreSQL in foreground. Typically, the PostgreSQL service is running under username postgres. We will first become postgres and then run the old version of PostgreSQL as follows.
    
      sudo -s       # to become root
      su - postgres # to become postgres
      $HOME/pgold/postgresql/bin/postmaster -D /var/lib/pgsql/data
      
  5. Open a second terminal window and dump the content of the old databases to a SQL file.
    
      # in the second terminal window
      sudo -s       # to become root
      su - postgres # to become postgres
      pg_dumpall > /var/lib/pgsql/postgresql_olddb.sql
      
    If you somehow do not possess the PostgreSQL's password of user postgres or any other database administrator's credential and cannot run pg_dumpall without a successful authentication, you can edit the PostgreSQL Client Authentication Configuration File /var/lib/pgsql/data/pg_hba.conf to trust any Unix domain socket connections to have a line as follows,
    
      # in /var/lib/pgsql/data/pg_hba.conf
      local   all  all  trust
      
    You will have to restart the old PostgreSQL program by simply killing processing using CTRL-C and running the postmaster again as described above.
  6. You can now shutdown the old PostgreSQL program. You will go to the terminal window where the postmaster program is running foreground and press CTRL-C to kill the process.
  7. Now backup the old database files. In case of a failure to complete the process, we can restore the database to its original state using the backup. In either the first or the second terminal window, you should remain as user postgres.
    
      mv /var/lib/pgsql/data /var/lib/pgsql/data_olddb.backup
      
  8. Now initialize database files using binaries of the new version of PostgreSQL in either terminal window in which you should remain as user postgres.
    
      /bin/initdb -D /var/lib/pgsql/data
      
  9. In one of the two terminal window, exit to your shell and then start the new PostgreSQL service.
    
      exit # exit to your shell
      sudo systemctl start postgresql.service
      
    Now you should be able to start the new PostgreSQL service.
  10. We now import the content of the old database to the new database. You will need to go to the terminal window in which you remain as user postgres.
    
      psql < /var/lib/pgsql/postgresql_olddb.sql
      
We are done and can now celebrate.

Adding Allowed Ports for Firewalld

Fedora Linux, CentOS and Redhat Linux distributions have a firewall called Firewalld.This is a note on adding allowed TCP/UDP ports to the firewall.

  1. Add a port, for instance, add a TCP port 63221 to be allowed by the firewall.
    
      sudo firewall-cmd --zone=public --add-port=63221/tcp
      
    Firewalld has a few zones. The public is for accepting incoming connection on a given port from other computers, in the above example, on TCP port 63221, since you do not trust the other computers on networks not to harm your computer.

    Be aware that the addition of the port will be effective upon the completion of the above operation; however, it is not persistent, i.e., it will not survive from reloading firewall or rebooting the system. In next step, we will make the change persistent.
  2. Make the firewall rule change permanent. For instance, to add TCP port 63221 to the allowed list of ports, we do
    
      sudo firewall-cmd --permanent --zone=public --add-port=63221/tcp
      
    Be aware that in the above, the rule does not go into effective until Firewalld is reloaded or the system is rebooted. That is to say, without reloading Firewalld or rebooting the system, we will have to run both of the above two steps to make the firewall rule change effective and permanent.
To learn more about firewalld, you may consult the manual page of firewall-cmd and visit the Fedora Project Wiki.



Sunday, November 23, 2014

Configuring SeLinux to Allow Secure Shell Service (SSHD) on Non-Default Port

The Secure Shell Service (SSHD) by default runs on TCP port 22. We sometimes want the Secure Shell Service (SSHD) to listen on a non-default port or to listen on more than one port. When the Security-Ehanced Linux (SeLinux) is turned on, we will have to configure both the Secure Shell Service and SeLinux because SeLinux by default has a policy that allows the Secure Shell Service to listen on TCP port 22 only.

Below are the steps for a recent release of the Fedora Linux distribution in which the Secure Shell Service (SSHD) is provided by OpenSSH and services are managed by systemd to allow the Secure Shell Service to listen on both TCP ports 22 and 64422.

  • Configure OpenSSH to listen on 64422.
    OpenSSH's configuration files in Fedora Linux is at /etc/ssh. Open /etc/ssh/sshd_config and add the following line to the file.
    
        Port 64422
        
  • Configure SeLinux to allow the Secure Shell Service to bind to TCP port 64422. We use semanager to configure SeLinux policies. First, let us look up what ports are allowed to bind to the Secure Shell Service.
    
        $ sudo semanage port -l | grep ssh
        ssh_port_t                     tcp      22
        
    which shows that the Secure Shell Service is allowed to bind to TCP port 22. We can now add a second port, 64422, to the list of ports that are allowed to bind to the Secure Shell Service.
    
        $ sudo semanage port -a -t ssh_port_t -p tcp 64422
        
    The above step usually takes a while to run. Upcon completion, we can now verify that the Secure Shell Service can now bind to both TCP ports 22 and 64422.
    
        $ sudo semanage port -l | grep ssh
        ssh_port_t                     tcp      64422, 22
        
  • Restart the Secure Shell Service using systemctl.
    
        $ sudo systemctl restart sshd.service
        
    Upon completion, we can verify that the service is now listenting to both TCP port 22 and 64422.
    
        $ sudo systemctl status sshd.service
        sshd.service - OpenSSH server daemon
              Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled)
              Active: active (running) 
                      since Sun, 23 Nov 2014 00:34:31 -0500; 6s ago
             Process: 2311 ExecStartPre=/usr/sbin/sshd-keygen 
                           (code=exited, status=0/SUCCESS)
            Main PID: 2312 (sshd)
              CGroup: name=systemd:/system/sshd.service
                      └ 2312 /usr/sbin/sshd -D
    
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on 0.0.0.0 port 64422.
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on :: port 64422.
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on 0.0.0.0 port 22.
        Nov 23 00:34:31 localhost.localdomain sshd[2312]: 
                        Server listening on :: port 22.
       
        

Thursday, November 13, 2014

Bibtex Entries for IETF RFCs and Internet-Drafts

Sometimes we want to cite the Internet Engineering Task Force's Request for Comment (RFC) or Internet-Draft documents. The Internet-Draft draft-carpenter-rfc-citation-recs-01 entitled "Recommendations of a committee on RFC citation issues" suggests a format to cite RFCs and Internet-Drafts.  Although the draft recommends that the RFC Editor create and maintain a canonical BibTeX file at a stable public location on the web server "www.rfc-editor.org", the BibTex files do not seem to have appeared on the website.

This post provides a means to produce a BibTex entry for a RFC or Internet-Draft document, generally comforming to the recommendation stated in the draft cited above, when the type of documents, RFC or Internet-Draft and the document number are provided. You may then copy and paste the BibTex entry to your .bib file. Note that the Internet-Draft number is not really a number, for instance, for the draft that mentioned above, the number is "draft-carpenter-rfc-citation-recs-01" (without quotation marks).


RFC Document Type and Number



To fill with a bibtex entry.

Wednesday, November 12, 2014

Testing SMTP MSA from Command Line

Sometimes there is a need to test a Simple Mail Transfer Protocol (SMTP) Mail Submission Agent (MSA) from the command line in a terminal window instead of running a mail client. Wikipedia shows an SMTP converstational example. However, you may not run the example as is from the command line in a terminal window on most SMTP servers today since they typically require a secure connection and proper authentication to improve security and reduce SPAMs.

This post demonstrates a procedure to test SMTP transport from a terminal window using Gmail's SMTP server. The idea, of course can be applied to other SMTP servers.

Gmail's SMTP server's setting on the client side at present is as follows,

Outgoing Mail (SMTP) Server - Requires TLS
    smtp.gmail.com
    Port: 465 or 587
    Requires SSL: Yes
    Requires authentication: Yes

This server is Gmail's SMTP MSA. Based on the setting above,  we shall do the following.
  • First, we need to be able to communicate with the server at smtp.gmail.com using TLS, for which we will use OpenSSL.
  • Second, we will have to be able to authenticate with the SMTP server, for which we will use SASL.

The above are availalbe in most Linux distributions. The following steps are tested on a Ubuntu 14.04 machine.

  1. Open a Linux terminal.
  2. Install OpenSSL and SASL-bin packages
    
        sudo apt-get install openssl sasl2-bin
        
  3. Select an authentication method and generate an authentication string. Assume your Gmail e-mail address is foo.somebody@gmail.com with password secretestring and your choose the PLAIN authentication method. We can use the gen-auth tool in the SASL-bin package to generate the authentication string as follows,
    
       gen-auth PLAIN foo.somebody@gmail.com secretestring
       
    The output is a hash string as shown below,
    
       Auth String: AGdyYXkuY2hlbmh1aUBnbWFpbC5jb20Ac3Q5OTg4IUAj
       
    This hash string will be used in next step.

    By the way, you can actually find out what authentication method a SMTP server supports in the steps follows. You can always run the steps to find it out and then generate the authentication string.
  4. Using openssl to establish a secure connection to Gmail's SMTP server at smtp.gmail.com at port 465
    
        openssl s_client -crlf -connect smtp.gmail.com:465
        
    The last line of the output of the above operation will be something as follows,
    
        220 mx.google.com ESMTP e23sm3343567vdk.23 - gsmtp
        
    At this point, we will send a EHLO message to the SMTP server. In the following, we assume that your domain is somesubdomain.somedomain and your IP address is 10.0.2.15.
    
        EHLO somesubdomain.somedomain
        
    The output will resemble something below,
    
        250-mx.google.com at your service, [10.0.2.15]
        250-SIZE 35882577
        250-8BITMIME
        250-AUTH LOGIN PLAIN XOAUTH XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER
        250-ENHANCEDSTATUSCODES
        250-PIPELINING
        250-CHUNKING
        250 SMTPUTF8
        
    The message contains the line
    
        250-AUTH LOGIN PLAIN XOAUTH XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER
        
    which indicates what authentication method you may use. As discussed above, we use the PLAIN method and have already obtained the authentication string. We can now authenticate with the SMTP server using the authentication string.
  5. We now authenticate with the server by sending an AUTH message as follows,
    
        AUTH PLAIN AGdyYXkuY2hlbmh1aUBnbWFpbC5jb20Ac3Q5OTg4IUAj
        
    At this point, you should be successfully authenticated as indicated by the acknowledgement from the SMTP server below,
    
        235 2.7.0 Accepted
        
    You can now start a converstation with the SMTP server similar to the Wikipedia example.
  6. Below is a converstation with the SMTP server to send a short e-mail to friend.of.somebody@somesubdomain.somedomain and copy the email to another.friend.of.somebody@somesubdomain.somedomain. Note that the lines starting with "C:" are what you, a client of the SMTP server would enter and the lines starting with "S:" are what the SMTP server would output. Obviously, "C:" should not be part of your input.
    
       C: mail from:<foo.somebody@gmail.com>
       S: 250 2.1.0 OK ki9sm3203907vdb.16 - gsmtp
       C: rcpt to:<friend.of.somebody@somesubdomain.somedomain>
       S: 250 2.1.5 OK ki9sm3203907vdb.16 - gsmtp
       C: data
       S: 354  Go ahead ki8sm3602907vdb.16 - gsmtp
       C: From: "foo somebody" <foo.somebody@gmail.com>
       C: To: "Friend of Somebody" <friend.of.somebody@somesubdomain.somedomain>
       C: Cc: "Another Friend" <other.friend.of.somebody@somesubdomain.somedomain>
       C: Date: Wed, 12 November 2014 17:29:43 -0500
       C: Subject: Test Message from Command Line
       C: hello friend,
       C: 
       C: reply me please. I am testing smtp server.
       C: 
       C: thanks.
       C:
       C: your friend
       C:
       C: .
       C:
       S: 250 2.0.0 OK 1415831485 ki9sm3203907vdb.16 - gsmtp
       C: quit
       S: 221 2.0.0 closing connection ki9sm3203907vdb.16 - gsmtp
       OpenSSL: read:errno=0
       
    Note in the above,
    • "read:errno=0" is an output from openssl rather than an output from the SMTP server.
    • The blank lines above, i.e, the lines of "C:" indicate that you will enter a new line.
    • The end of conversation is marked with "<CRLF>.<CRLF>". See the last "." in the above converstation. The "-crlf" provided in the "openssl s_client" command line is to convert a line feed to a <CRLF>, a carriage return followed by a line feed.
    • An important item to note is that the "openssl s_client" has the following behavior as described in the manual page of "s_client" that you may view using "man s_client" as follows,
      If a connection is established with an SSL server then any data received from the server is displayed and any key presses will be sent to the server. When used interactively (which means neither -quiet nor -ign_eof have been given), the session will be renegotiated if the line begins with an R, and if the line begins with a Q or if end of file is reached, the connection will be closed down.
      which means that you cannot use letter R and Q in the entire interactive openssl session. If you use R, such as type RCPT TO: ... instead of what is shown in the above, you will encounter an error as follows even though SMTP treats the same.
      
         C: RCPT TO:<friend.of.somebody@somesubdomain.somedomain>
         OpenSSL: RENEGOTIATING
         OpenSSL: 3073996476:error:1409E0E5:SSL 
           routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:
         

Acknowledgement 

The following web posts  were excellent references for writing this post,
  • The post here led me to SASL and subsequently the SASL-bin package for the generation of authentication strings.
  • The posts here and here helped me figure out the "RENEGOTIATING" feature of "openssl s_client


Thursday, October 16, 2014

Firefox iMacro Add-on Error (0x8000ffff NS_ERROR_UNEXPECTED)

I was running iMacro Add-on for Firefox on a Linux machine. iMacro reported the following error,

Error: NS_ERROR_UNEXPECTED: 
  Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) 
  [nsIPrefBranch.getIntPref], line 6 (Error code: -1001)

I tried the same script on two Linux machines, the same error occurred. The versions of the two cases are as follows,

  • iMacros for Firefox 8.8.2
  • Firefox 33.0
  • Ubuntu 14.04
A few others saw the similar problem, as reported here in Feburary 2014.

To verify the problem exists, you may run the demo script coming from iMaco add-on, such as, Loop-Csv-2-Web.iim by default at $HOME/iMacros/Macros/Demo-Firefox. In my case, running the demo program would yield the same error.

Apparently, no fixes exist. One suggested workaround is to use an old version of Firefix, as indicated in the discussion mentioned above.

We can download old versions of Firefox from here. I tried Firefox 27.0b9. It works fine with the iMacro demo scripts and my scripts.

Since you would probably have another copy of Firefox installed, it is important that you close all existing Firefox window and run the older version of Firefox using its path. In addition, it is also important to check "Never check for updates (not reocmmended:security risk)" from the Preference under the "Advanced" tab; otherwise, the old Firefox would automatically download updates and the updates would be applied when the old Firefox restarts. Below is the list of steps that I performed,

wget https://ftp.mozilla.org/pub/mozilla.org/\
firefox/releases/27.0b9/linux-i686/en-US/firefox-27.0b9.tar.bz2

tar -xjvf firefox-27.0b9.tar.bz2

firefox/firefox -preferences # disable updates

firefox/firefox



Friday, October 3, 2014

Conditionally Processing LaTeX Documents

Sometimes one may wish to process a LaTeX document based on a condition. For instance, one may wish to conditional process a resume written in LaTeX depending where she is sending the document, i.e., she may wish to expand certain sections, shrink certain sections, or exclude certain sections, in which way, she does not need to maintain multiple different documents that share significant amount of content.

We can use LaTex ifthen package to achieve the purpose. An example is given in this post.

However, some (this and this) suggests that the package is obsolete and we should use the etoolbox package.

I have two examples that use the ifthen package and the etoolbox package, respectively. The complete examples are in Github. I tested the examples on a Linux machine.

In the first example, we use  the etoolbox package and create a "toggle" file (toggle.tex) in the Makefile

ex_cond_1_long.pdf: ex_cond_1.tex
 [ ! -f toggle.tex ] || rm -f toggle.tex
 echo "\\\\toggletrue{long}" > toggle.tex
 pdflatex -jobname ex_cond_1_long ex_cond_1.tex
 
ex_cond_1_short.pdf: ex_cond_1.tex
 [ ! -f toggle.tex ] || rm -f toggle.tex
 echo "\\\\togglefalse{long}" > toggle.tex
 pdflatex -jobname ex_cond_1_short ex_cond_1.tex

The content of toggle.tex is now either,

\toggletrue{long}
or

\togglefalse{long}
Then, in the LaTex file (ex_cond_1.tex), one of two files (long.tex or short.tex) will be conditionally included,

\documentclass[11pt, letterpaper]{article}
\usepackage{etoolbox}
\newtoggle{long}

\input{toggle.tex}

\iftoggle{long}{
    \input{long.tex}
}{
    \input{short.tex}
}

The second example uses the ifthen packag. We demonstrate that we can pass a value from the command line using jobname as shown in the Makefile,

ex_cond_2_long.pdf: ex_cond_2.tex
 pdflatex -jobname ex_cond_2_long ex_cond_2.tex

ex_cond_2_short.pdf: ex_cond_2.tex
 pdflatex -jobname ex_cond_2_short ex_cond_2.tex

Based on whether the jobname is ex_cond_2_long or ex_cond_2_short, one of two files (long.tex or short.tex) will be conditionally included,

\ifthenelse{\equal{\detokenize{ex_cond_2_long}}{\jobname}}{
    \input{long.tex}
}{
    \input{short.tex}
}




Querying Package Status in Linux Systems

I am interested in querying the status of packages on Linux systems. Of course, you can query the status using a number of GUI applications, such as, the "Synaptic Package Manager" on Ubuntu Linux and "GNOME Software" on Federa Linux. However, I am more interested a command line approach.

For Linux systems using APT, such as, Ubuntu Linux, we can use the utilities that comes with APT, for instance, use dpkg-query, as pointed out in this post. In addition, this post gives a few more elaborated queries.

Here are a few examples.
  • To query if a package is installed, i.e, to report status of specified package,

    
    $ dpkg-query -s gnome-shell
    dpkg-query: package 'gnome-shell' is not installed and no information is available
    ......
    

    
    $ dpkg -s gnome-session-flashback
    Package: gnome-session-flashback
    Status: install ok installed
    ......
     
  • To List packages matching given pattern,
    
    $ dpkg-query -l gnome-shell
    Desired=Unknown/Install/Remove/Purge/Hold
    | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
    |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    ||/ Name           Version      Architecture Description
    +++-==============-============-============-=================================
    un  gnome-shell    <none>       <none>       (no description available)
    
    
    $ dpkg-query -l gnome-session-flashback
    Desired=Unknown/Install/Remove/Purge/Hold
    | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
    |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    ||/ Name           Version      Architecture Description
    +++-==============-============-============-=================================
    ii  gnome-session- 1:3.8.0-1ubu all          GNOME Session Manager - GNOME Fla
    
    
    $ dpkg-query -l gnome*
    Desired=Unknown/Install/Remove/Purge/Hold
    | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
    |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    ||/ Name           Version      Architecture Description
    +++-==============-============-============-=================================
    un  gnome          <none>       <none>       (no description available)
    ii  gnome-accessib 3.10.0-1ubun all          Accessibility themes for the GNOM
    ii  gnome-applets  3.5.92-0ubun i386         Various applets for the GNOME pan
    ......
    
For Linux systems using RPM, such as, Fedora Linux, we can use yum and rpm. These two pages, on yum and on rpm respectively provide a few good examples of the two.

Thursday, October 2, 2014

Wildcard Targets in a Makefile (Pattern Rules)

You may use "Wildcard" targets in GNU makefile. In GNU make, this is called Pattern Rules. I find this post is very helpful.  For those who do not  have the patience to read the documentation of GNU Make or to jump to the post, here is a few examples that serves as a summary.

To apply a rule to compile any C++ files, we can write a rule as follows,

CPP=g++
CLFAGS=-Wall

%.o:%.cpp
       $(CPP) $(CFLAGS) $< -o $@

To apply to rule to convert any LaTeX files to dvi files, any dvi files to PostScript files, and then any PostScript files to PDF files, we may write a set of rules as follows,

%.pdf: %.ps
 ps2pdf \
              -dPDFSETTINGS=/prepress \
              -dSubsetFonts=true \
              -dEmbedAllFonts=true \
              -dMaxSubsetPct=100 \
              -dCompatibilityLevel=1.4 $<

%.ps: %.dvi
       dvips -t letter $<

 
%.dvi: %.tex
       latex $<



Monday, September 22, 2014

SQL and NoSQL

This is just a note on NoSQL.

The company I worked with many years ago had a discussion among the developers and managers on how they would port their code base that managed and queried much data in the company's core business from using a B-tree library to a relational database. It was compelling that a relational database would offer much advantage of the B-tree library -- SQL is widely supported in many languages and platform, the database engine would actually use a B+ tree for efficient query of the data, and the relational databases can also manages transactions and ensures integrity constriants.

Well, the tide is now turned. Data management based on key-value stores, mostly using hash table or tree structures to ensure efficient query becomes popular again. The development of network applications appears to pose a serious challenge to relational database business and NoSQL is gaining momentum. NoSQL aappears to threat some traditional database company's profit margin. There have been many discussions about NoSQL and relatinoal database management systems (RDBMS) powered by SQL, such as,


You can find more by do a search in Google Scholar or just a Google search. It appears that it is not that NoSQL can do something RDMBS cannot, rather it is that NoSQL has successfully created some an excellent APIs and platforms that work well on a few special and popular set of on-line applications.

Many NoSQL engines are available. The news article cited above mentionds MongoDB that has a good and interactive tutorial on their website; by the way, the language used in the tutorial is JavaScript whose popularity, in my opinion, an exellent footnote on the increasing popularity of NoSQL -- well, is it the same set of online applications that also helped the increased popularity of JavaScript?

Sunday, September 21, 2014

Problem with Mercurial (hg) Subrepo: "about: HTTP Error 404: Not Found"

Mercurial Subrepository allows us to clone multiple repositories as a group. Following the steps in the documentation, I created a main repository and a few sub-repositories. I did not encounter any problem when I clone the main repository or any of the sub-repositories. However,I observed an error when I attempted to push a local subrepository to the server. The error message reads,

abort: HTTP Error 404: Not Found
What gives me the hint what might be wrong is this post. In the repository's hgweb file, I had,

[paths]
/ = /home/hg/repo-private/*

Since we have subrepostories in the repository, the above configuration is incorrect -- it should have been two *'s instead of one *, i.e.,

[paths]
/ = /home/hg/repo-private/**

Indeed, as pointed out by the post cited above, <code>hg help hgweb</code> gives a good explanation and example how we may configure the <code>hgweb</code> file.

Friday, September 19, 2014

Audio Playing and Recording with Octave

Octave Forge has an audio package. At present, the package is in the "unmaintained" status. However, once installed successfully, it has all the functionality that I need:
  • Record audio data
  • Play audio data
I primarily use it on Linux virtual machines on Windows hosts. The following procedure was tested with Octave 3.6 on a Ubuntu 13.10 (Saucy) guest operating system on a Windows 8.1 host.

It is quite straight forward to install the audio package,

  1. Install Octave and Octave development package.
     
    sudo apt-get install octave liboctave-dev

    The latter is required to install Octave packages from Octave Forge.
  2. Install the audio package from Octave Forge.
    
         sudo octave
         
    In Octave, run
    
         pkg install -forge audio
         
    We may list installed packages in Octave to verify that it is successfully installed by
    
         pkg list
         
    The output should look like the following,
    
         Package Name  | Version | Installation directory
         --------------+---------+-----------------------
                audio *|   1.1.4 | /usr/share/octave/packages/audio-1.1.4
         
    which shows that the audio package is version 1.1.4 and installed at /usr/share/octave/packages/audio-1.1.4. The "*" next to the package name indicates that the package is loaded and ready to use.

    If you run Octave as a non-root user, the Octave package will be installed under the user's home directory and can only be used by the user. By running Octave as root (i.e., using sudo octave), we install the Octave package for everyone.

  3. The audio package uses some external programs to play and record audio data. Sox is sufficient for playing sound. We now install Sox in Linux.
    
         apt-get install sox
         
  4. The audio package uses OSS to record audio data. Ubuntu 13.10 does not have an OSS package. Instead, it provides an ALSA wrapper for OSS. We now install the wrapper.
    
        apt-get install alsa-oss
        
  5. To use the ALSA wrapper for OSS, we must invoke the aoss command. Then to run Octave, we run,
    
        aoss octave
        
    Without using aoss to invoke octave, you would encounter an error indicating that it failed to open /dev/dsp that is supposedly created by OSS. Since we are emulating OSS using ALSA, the system does not have the device file.

  6. Now, let us play a tone of 2 seconds at 400 Hz in Octave.
    
      y = cos(2 * pi * 400 * (0:1/44100:2));
      sound(6, 44100);
      

    Note that the sampling rate is 44100Hz in the above.
  7. To record and then play the recorded audio data, we may do
    
      [y, fs, ch] = aurecord(5, 44100, 2);
      sound(y, fs, ch);
      
  8. Finally, if you do not hear anything when you play the sound, check your virtual machine setting. It must have an appropriate audio card. The following shows the audio card setting for the Ubuntu virtual machine running in VMware Player.


    At the same time, the mixer of the Windows host should show that the virtual machine is a source as shown below. You can open the mixer by right-cliking on the "speaker" icon in the system tray, then choose "Open Volume Mixer".



    If you are running VirtualBox, you will have more than one audio driver to choose from when configuring the virtual machine. My experience is that not all driver works on a particular host. You can test the driver one by one until you find one that works. When the driver works and the virtual machine is running, you should observe that Windows Mixer shows the virtual machine as a sound source similar to the figure above.

Thursday, September 18, 2014

Creating Small Footprint Linux Virtual Machine

Although it may be easier to download a Linux virtual appliance, it can be fun to create linux virtual machines yourself. In particular, we may be interested in creating small footprint Linux virtual machines -- it would be easy to put it on a USB drive or let other people to download it. This post documents the process that I use to create Ubuntu 14.04 (Trusty) Linux virtual machines and reduce the image size of the virtual machines. The key idea is (1) to remove non-essential packages as much as possible; (2) to increase the compression of virtual machine images by zeroing virtual disks; and (3) to compact and compress the virtual machine image.
  1. Download 32-bit Ubuntu 14.04 (Trusty) ISO file and install it as a guest operating system under VMware Player or VirtualBox.  
  2. To make copy & paste instructions easier, install VMware Tools.  
    1. Install Linux headers and the build essential packages

      sudo apt-get install linux-headers-generic build-essential

    2. Then install VMware Tools
    3. Restart the virtual machine.
  3. By default, Ubuntu 14.04 runs the Ubuntu Unity desktop environment. It prevents us from entering the VMware Unity mode. I now get rid of the Ubuntu Unity desktop environment. I learned the command from this post.
    
    sudo apt-get purge \
         unity unity-common unity-services \
         unity-lens-\* unity-scope-\* unity-webapps-\* \
         gnome-control-center-unity \
         hud libunity-core-6\* libunity-misc4 \
         libunity-webapps\* appmenu-gtk \
         appmenu-gtk3 appmenu-qt\* overlay-scrollbar\* \
         activity-log-manager-control-center \
         firefox-globalmenu thunderbird-globalmenu \
         libufe-xidgetter0 xul-ext-unity \
         xul-ext-webaccounts webaccounts-extension-common \
         xul-ext-websites-integration \
         gnome-control-center gnome-session

  4. Then, install GNOME 3.
    
    sudo apt-get install gnome-session-fallback
    

  5. Now, remove the default games.
    
    sudo apt-get game\* purge ace-of-penguins gnome-mahjongg gnome-mines \
                 berusky biniax2 gnurobbo groundhog gtkballs hitori \
                 four-in-a-row gnect gbrainy gnome-sudoku chess\* \
                 nibbles\* solitaire\* iagno\* klotski\* tetravex\* \
                 drascula gtkboard oneisenough openyahtzee ri-li \
                 scribble swell-foop quadrapassel\* lightsoff \
                 five-or-more gnome-robots gtali
    

  6. Remove a few network applications.
    
    sudo apt-get purge webbrowser-app thunderbird\*
    

  7. Remove Libre Office. This can save a few hundred megabytes.
    
    sudo apt-get purge libreoffice\*
    

  8. Now get rid of the "Ubuntu Software Center".
    
    sudo apt-get purge software-center\*
    

  9. The /var/cache directory can easily have a few hundred megabytes data. We can trim it down.
    
    sudo apt-get clean
    

  10. Do an automove,
    
    sudo apt-get autoremove
    

  11. Now remove GNOME and Ubuntu documentation.
    
      sudo apt-get purge yelp\*
      

  12. Now, you can install whatever important software you want to include in the virtual machine, e.g., I am installing Octave,

    
    sudo apt-get install octave
    

  13. We would probably want to compress the virtual machine image before we pass it to someone. We need to replace every byte of free space on the disk by 0's.
    
    dd if=/dev/zero of=zerofile; \
       sleep 5; sync; \
       sleep 5; sync; \
       sleep 5; sync; \
       rm -f zerofile;
    

  14. I am using VMware Player. Using it, I do a "defragment" and then "compact" the virtual disk. The guest system must be off before you can do it. Of course, you can do the same using VirtualBox.
  15. Finally, I compress the folder of the virtual machine. The zip archieve file is about 1 GB. 
In addition, you may save a few megabytes, by removing all log and temporary files. The tool bleachbit can help. Before you zero the hard disk, install and run the tool.


sudo apt-get install bleachbit; sudo bleachbit

Tuesday, September 16, 2014

Querying Wi-Fi Link Speed on Windows

Many have expressed the interest to query Wi-Fi link speeed on Windows and provided answers, such as, this, this, and this, which are the main references for this post.

On Windows 7,  we can view network link speed easily and conveniently using the Windows Task Manager, as shown in the figure below,



On Windows 8, the Task Manager has different interface and does not show the link speed any more. If we want to know the link speed, we can view it using the Windows Control Panel, which works on both Windows 7 and Windows 8. A shortcut is to right-click on the  "network" icon in the system tray, then open the "Network and Sharing Center", finally click on the Wi-Fi connection on the right. The result will be something similar to below,



which clealy shows the link speed.

The following methods may be more interesting,

netsh

Open a Windows Command Prompt window, you can issue the following command,

netsh wlan show interface 


which list all Wi-Fi interfaces and their properties.

Querying WMI with WQL

We can use Windows PowerShell to issue WMI queries. Below is an example,


C:\>powershell
Windows PowerShell
Copyright (C) 2013 Microsoft Corporation. All rights reserved.

PS C:\> $query = "SELECT Description, DeviceID, Speed FROM CIM_NetworkAdapter
WHERE Description like '%Wi-Fi%' OR Description like '%802.11%'";
PS C:\>  Get-WmiObject -Query $query


These two posts are particularly helpful for me to write WQL queries to query the WMI.

GetAdaptersAddresses API

To me, this is the most interesting one. This stackoverflow thread provides a sample program in C# that uses P/Invoke to call the GetAdaptersAddresses API.  I made some simple revisions and put all the code together,
You can create a project in Visual Studio to compile the program. However, for this simple example, I would prefer a command line approach,


C:\notes\wifistatus\windows>"c:\Program Files (x86)\Microsoft Visual 
Studio 12.0\Common7\Tools\vsvars32.bat"
C:\notes\wifistatus\windows>csc /warn:4 WifiStatus.cs IPIntertop.cs
Microsoft (R) Visual C# Compiler version 12.0.30501.0
for C# 5
Copyright (C) Microsoft Corporation. All rights reserved.

C:\notes\wifistatus\windows>WifiStatus.exe
GetAdaptersAddresses yields:
        Length of the IP_ADAPTER_ADDRESS struct: 448
        IfIndex (IPv4 interface): 4
        Adapter name: {......}
        DNS Suffix: foo.com
        Description: Broadcom 802.11n Network Adapter
        Friendly name: Wi-Fi
        Physical address: ......
        Flags: 453
        Mtu: 1500
        IfType: IF_TYPE_IEEE80211
        OperStatus: IfOperStatusUp
        Ipv6IfIndex (IPv6 interface): 4
        ZoneIndices (hex): 4 4 4 4 1 1 1 1 1 1 1 1 1 1 0 1
        Transmit link speed: 65000000
        Receive link speed: 144000000
        Number of IP Adapter Prefix entries: 5


GetAllNetworkInterfaces yields:
        Current Wi-Fi Speed on Broadcom 802.11n Network Adapter: 144000000

C:\notes\wifistatus\windows>

Alternatively, perhaps, more naturally, you can do the same in C/C++,
To compile and run it on the Windows Command Prompt command line, do,

C:\notes\wifistatus\windows>"c:\Program Files (x86)\Microsoft Visual 
Studio 12.0\Common7\Tools\vsvars32.bat"

C:\notes\wifistatus\windows>cl /W4 getwifiadapteraddress.cpp
Microsoft (R) C/C++ Optimizing Compiler Version 18.00.30501 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.

getwifiadapteraddress.cpp
Microsoft (R) Incremental Linker Version 12.00.30501.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:getwifiadapteraddress.exe
getwifiadapteraddress.obj

C:\notes\wifistatus\windows>getwifiadapteraddress.exe
       Length of the IP_ADAPTER_ADDRESS struct: 376
       IfIndex (IPv4 interface): 4
       Adapter name: {......}
       DNS Suffix: foo.com
       Description: Broadcom 802.11n Network Adapter
       Friendly name: Wi-Fi
       Physical address: ......
       Flags: 453
       Mtu: 1500
       IfType: 71
       OperStatus: 1
       Ipv6IfIndex (IPv6 interface): 4
       ZoneIndices (hex): 4 4 4 4 1 1 1 1 1 1 1 1 1 1 0 1
       Transmit link speed: 65000000
       Receive link speed: 144000000
       Number of IP Adapter Prefix entries: 8
C:\notes\wifistatus\windows>

Discussion

Be aware of the difference between the speed (or bandwidth) of the interface card, the negotiated link speed (or bandwidth), and the throughput. The speed of the interface card is the specified speed that the interface card could achieve, i.e., a part of the hardware specification. The negotiated link speed is a reflection of the signal strength. The throughput is the one that you actually achieve. NetworkInterface gives you the speed of the interface card.GetAdaptersAddresses can give you the negotiated link speed. Finally, the throughput can only be determined by a measurement based approach. IPInterfaceStatistics and IPv4InterfaceStatistics can help you estimate it.