Friday, October 3, 2014

Querying Package Status in Linux Systems

I am interested in querying the status of packages on Linux systems. Of course, you can query the status using a number of GUI applications, such as, the "Synaptic Package Manager" on Ubuntu Linux and "GNOME Software" on Federa Linux. However, I am more interested a command line approach.

For Linux systems using APT, such as, Ubuntu Linux, we can use the utilities that comes with APT, for instance, use dpkg-query, as pointed out in this post. In addition, this post gives a few more elaborated queries.

Here are a few examples.
  • To query if a package is installed, i.e, to report status of specified package,

    
    $ dpkg-query -s gnome-shell
    dpkg-query: package 'gnome-shell' is not installed and no information is available
    ......
    

    
    $ dpkg -s gnome-session-flashback
    Package: gnome-session-flashback
    Status: install ok installed
    ......
     
  • To List packages matching given pattern,
    
    $ dpkg-query -l gnome-shell
    Desired=Unknown/Install/Remove/Purge/Hold
    | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
    |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    ||/ Name           Version      Architecture Description
    +++-==============-============-============-=================================
    un  gnome-shell    <none>       <none>       (no description available)
    
    
    $ dpkg-query -l gnome-session-flashback
    Desired=Unknown/Install/Remove/Purge/Hold
    | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
    |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    ||/ Name           Version      Architecture Description
    +++-==============-============-============-=================================
    ii  gnome-session- 1:3.8.0-1ubu all          GNOME Session Manager - GNOME Fla
    
    
    $ dpkg-query -l gnome*
    Desired=Unknown/Install/Remove/Purge/Hold
    | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
    |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    ||/ Name           Version      Architecture Description
    +++-==============-============-============-=================================
    un  gnome          <none>       <none>       (no description available)
    ii  gnome-accessib 3.10.0-1ubun all          Accessibility themes for the GNOM
    ii  gnome-applets  3.5.92-0ubun i386         Various applets for the GNOME pan
    ......
    
For Linux systems using RPM, such as, Fedora Linux, we can use yum and rpm. These two pages, on yum and on rpm respectively provide a few good examples of the two.

Thursday, October 2, 2014

Wildcard Targets in a Makefile (Pattern Rules)

You may use "Wildcard" targets in GNU makefile. In GNU make, this is called Pattern Rules. I find this post is very helpful.  For those who do not  have the patience to read the documentation of GNU Make or to jump to the post, here is a few examples that serves as a summary.

To apply a rule to compile any C++ files, we can write a rule as follows,

CPP=g++
CLFAGS=-Wall

%.o:%.cpp
       $(CPP) $(CFLAGS) $< -o $@

To apply to rule to convert any LaTeX files to dvi files, any dvi files to PostScript files, and then any PostScript files to PDF files, we may write a set of rules as follows,

%.pdf: %.ps
 ps2pdf \
              -dPDFSETTINGS=/prepress \
              -dSubsetFonts=true \
              -dEmbedAllFonts=true \
              -dMaxSubsetPct=100 \
              -dCompatibilityLevel=1.4 $<

%.ps: %.dvi
       dvips -t letter $<

 
%.dvi: %.tex
       latex $<



Monday, September 22, 2014

SQL and NoSQL

This is just a note on NoSQL.

The company I worked with many years ago had a discussion among the developers and managers on how they would port their code base that managed and queried much data in the company's core business from using a B-tree library to a relational database. It was compelling that a relational database would offer much advantage of the B-tree library -- SQL is widely supported in many languages and platform, the database engine would actually use a B+ tree for efficient query of the data, and the relational databases can also manages transactions and ensures integrity constriants.

Well, the tide is now turned. Data management based on key-value stores, mostly using hash table or tree structures to ensure efficient query becomes popular again. The development of network applications appears to pose a serious challenge to relational database business and NoSQL is gaining momentum. NoSQL aappears to threat some traditional database company's profit margin. There have been many discussions about NoSQL and relatinoal database management systems (RDBMS) powered by SQL, such as,


You can find more by do a search in Google Scholar or just a Google search. It appears that it is not that NoSQL can do something RDMBS cannot, rather it is that NoSQL has successfully created some an excellent APIs and platforms that work well on a few special and popular set of on-line applications.

Many NoSQL engines are available. The news article cited above mentionds MongoDB that has a good and interactive tutorial on their website; by the way, the language used in the tutorial is JavaScript whose popularity, in my opinion, an exellent footnote on the increasing popularity of NoSQL -- well, is it the same set of online applications that also helped the increased popularity of JavaScript?

Sunday, September 21, 2014

Problem with Mercurial (hg) Subrepo: "about: HTTP Error 404: Not Found"

Mercurial Subrepository allows us to clone multiple repositories as a group. Following the steps in the documentation, I created a main repository and a few sub-repositories. I did not encounter any problem when I clone the main repository or any of the sub-repositories. However,I observed an error when I attempted to push a local subrepository to the server. The error message reads,

abort: HTTP Error 404: Not Found
What gives me the hint what might be wrong is this post. In the repository's hgweb file, I had,

[paths]
/ = /home/hg/repo-private/*

Since we have subrepostories in the repository, the above configuration is incorrect -- it should have been two *'s instead of one *, i.e.,

[paths]
/ = /home/hg/repo-private/**

Indeed, as pointed out by the post cited above, <code>hg help hgweb</code> gives a good explanation and example how we may configure the <code>hgweb</code> file.

Friday, September 19, 2014

Audio Playing and Recording with Octave

Octave Forge has an audio package. At present, the package is in the "unmaintained" status. However, once installed successfully, it has all the functionality that I need:
  • Record audio data
  • Play audio data
I primarily use it on Linux virtual machines on Windows hosts. The following procedure was tested with Octave 3.6 on a Ubuntu 13.10 (Saucy) guest operating system on a Windows 8.1 host.

It is quite straight forward to install the audio package,

  1. Install Octave and Octave development package.
     
    sudo apt-get install octave liboctave-dev

    The latter is required to install Octave packages from Octave Forge.
  2. Install the audio package from Octave Forge.
    
         sudo octave
         
    In Octave, run
    
         pkg install -forge audio
         
    We may list installed packages in Octave to verify that it is successfully installed by
    
         pkg list
         
    The output should look like the following,
    
         Package Name  | Version | Installation directory
         --------------+---------+-----------------------
                audio *|   1.1.4 | /usr/share/octave/packages/audio-1.1.4
         
    which shows that the audio package is version 1.1.4 and installed at /usr/share/octave/packages/audio-1.1.4. The "*" next to the package name indicates that the package is loaded and ready to use.

    If you run Octave as a non-root user, the Octave package will be installed under the user's home directory and can only be used by the user. By running Octave as root (i.e., using sudo octave), we install the Octave package for everyone.

  3. The audio package uses some external programs to play and record audio data. Sox is sufficient for playing sound. We now install Sox in Linux.
    
         apt-get install sox
         
  4. The audio package uses OSS to record audio data. Ubuntu 13.10 does not have an OSS package. Instead, it provides an ALSA wrapper for OSS. We now install the wrapper.
    
        apt-get install alsa-oss
        
  5. To use the ALSA wrapper for OSS, we must invoke the aoss command. Then to run Octave, we run,
    
        aoss octave
        
    Without using aoss to invoke octave, you would encounter an error indicating that it failed to open /dev/dsp that is supposedly created by OSS. Since we are emulating OSS using ALSA, the system does not have the device file.

  6. Now, let us play a tone of 2 seconds at 400 Hz in Octave.
    
      y = cos(2 * pi * 400 * (0:1/44100:2));
      sound(6, 44100);
      

    Note that the sampling rate is 44100Hz in the above.
  7. To record and then play the recorded audio data, we may do
    
      [y, fs, ch] = aurecord(5, 44100, 2);
      sound(y, fs, ch);
      
  8. Finally, if you do not hear anything when you play the sound, check your virtual machine setting. It must have an appropriate audio card. The following shows the audio card setting for the Ubuntu virtual machine running in VMware Player.


    At the same time, the mixer of the Windows host should show that the virtual machine is a source as shown below. You can open the mixer by right-cliking on the "speaker" icon in the system tray, then choose "Open Volume Mixer".



    If you are running VirtualBox, you will have more than one audio driver to choose from when configuring the virtual machine. My experience is that not all driver works on a particular host. You can test the driver one by one until you find one that works. When the driver works and the virtual machine is running, you should observe that Windows Mixer shows the virtual machine as a sound source similar to the figure above.

Thursday, September 18, 2014

Creating Small Footprint Linux Virtual Machine

Although it may be easier to download a Linux virtual appliance, it can be fun to create linux virtual machines yourself. In particular, we may be interested in creating small footprint Linux virtual machines -- it would be easy to put it on a USB drive or let other people to download it. This post documents the process that I use to create Ubuntu 14.04 (Trusty) Linux virtual machines and reduce the image size of the virtual machines. The key idea is (1) to remove non-essential packages as much as possible; (2) to increase the compression of virtual machine images by zeroing virtual disks; and (3) to compact and compress the virtual machine image.
  1. Download 32-bit Ubuntu 14.04 (Trusty) ISO file and install it as a guest operating system under VMware Player or VirtualBox.  
  2. To make copy & paste instructions easier, install VMware Tools.  
    1. Install Linux headers and the build essential packages

      sudo apt-get install linux-headers-generic build-essential

    2. Then install VMware Tools
    3. Restart the virtual machine.
  3. By default, Ubuntu 14.04 runs the Ubuntu Unity desktop environment. It prevents us from entering the VMware Unity mode. I now get rid of the Ubuntu Unity desktop environment. I learned the command from this post.
    
    sudo apt-get purge \
         unity unity-common unity-services \
         unity-lens-\* unity-scope-\* unity-webapps-\* \
         gnome-control-center-unity \
         hud libunity-core-6\* libunity-misc4 \
         libunity-webapps\* appmenu-gtk \
         appmenu-gtk3 appmenu-qt\* overlay-scrollbar\* \
         activity-log-manager-control-center \
         firefox-globalmenu thunderbird-globalmenu \
         libufe-xidgetter0 xul-ext-unity \
         xul-ext-webaccounts webaccounts-extension-common \
         xul-ext-websites-integration \
         gnome-control-center gnome-session

  4. Then, install GNOME 3.
    
    sudo apt-get install gnome-session-fallback
    

  5. Now, remove the default games.
    
    sudo apt-get game\* purge ace-of-penguins gnome-mahjongg gnome-mines \
                 berusky biniax2 gnurobbo groundhog gtkballs hitori \
                 four-in-a-row gnect gbrainy gnome-sudoku chess\* \
                 nibbles\* solitaire\* iagno\* klotski\* tetravex\* \
                 drascula gtkboard oneisenough openyahtzee ri-li \
                 scribble swell-foop quadrapassel\* lightsoff \
                 five-or-more gnome-robots gtali
    

  6. Remove a few network applications.
    
    sudo apt-get purge webbrowser-app thunderbird\*
    

  7. Remove Libre Office. This can save a few hundred megabytes.
    
    sudo apt-get purge libreoffice\*
    

  8. Now get rid of the "Ubuntu Software Center".
    
    sudo apt-get purge software-center\*
    

  9. The /var/cache directory can easily have a few hundred megabytes data. We can trim it down.
    
    sudo apt-get clean
    

  10. Do an automove,
    
    sudo apt-get autoremove
    

  11. Now remove GNOME and Ubuntu documentation.
    
      sudo apt-get purge yelp\*
      

  12. Now, you can install whatever important software you want to include in the virtual machine, e.g., I am installing Octave,

    
    sudo apt-get install octave
    

  13. We would probably want to compress the virtual machine image before we pass it to someone. We need to replace every byte of free space on the disk by 0's.
    
    dd if=/dev/zero of=zerofile; \
       sleep 5; sync; \
       sleep 5; sync; \
       sleep 5; sync; \
       rm -f zerofile;
    

  14. I am using VMware Player. Using it, I do a "defragment" and then "compact" the virtual disk. The guest system must be off before you can do it. Of course, you can do the same using VirtualBox.
  15. Finally, I compress the folder of the virtual machine. The zip archieve file is about 1 GB. 
In addition, you may save a few megabytes, by removing all log and temporary files. The tool bleachbit can help. Before you zero the hard disk, install and run the tool.


sudo apt-get install bleachbit; sudo bleachbit

Tuesday, September 16, 2014

Querying Wi-Fi Link Speed on Windows

Many have expressed the interest to query Wi-Fi link speeed on Windows and provided answers, such as, this, this, and this, which are the main references for this post.

On Windows 7,  we can view network link speed easily and conveniently using the Windows Task Manager, as shown in the figure below,



On Windows 8, the Task Manager has different interface and does not show the link speed any more. If we want to know the link speed, we can view it using the Windows Control Panel, which works on both Windows 7 and Windows 8. A shortcut is to right-click on the  "network" icon in the system tray, then open the "Network and Sharing Center", finally click on the Wi-Fi connection on the right. The result will be something similar to below,



which clealy shows the link speed.

The following methods may be more interesting,

netsh

Open a Windows Command Prompt window, you can issue the following command,

netsh wlan show interface 


which list all Wi-Fi interfaces and their properties.

Querying WMI with WQL

We can use Windows PowerShell to issue WMI queries. Below is an example,


C:\>powershell
Windows PowerShell
Copyright (C) 2013 Microsoft Corporation. All rights reserved.

PS C:\> $query = "SELECT Description, DeviceID, Speed FROM CIM_NetworkAdapter
WHERE Description like '%Wi-Fi%' OR Description like '%802.11%'";
PS C:\>  Get-WmiObject -Query $query


These two posts are particularly helpful for me to write WQL queries to query the WMI.

GetAdaptersAddresses API

To me, this is the most interesting one. This stackoverflow thread provides a sample program in C# that uses P/Invoke to call the GetAdaptersAddresses API.  I made some simple revisions and put all the code together,
You can create a project in Visual Studio to compile the program. However, for this simple example, I would prefer a command line approach,


C:\notes\wifistatus\windows>"c:\Program Files (x86)\Microsoft Visual 
Studio 12.0\Common7\Tools\vsvars32.bat"
C:\notes\wifistatus\windows>csc /warn:4 WifiStatus.cs IPIntertop.cs
Microsoft (R) Visual C# Compiler version 12.0.30501.0
for C# 5
Copyright (C) Microsoft Corporation. All rights reserved.

C:\notes\wifistatus\windows>WifiStatus.exe
GetAdaptersAddresses yields:
        Length of the IP_ADAPTER_ADDRESS struct: 448
        IfIndex (IPv4 interface): 4
        Adapter name: {......}
        DNS Suffix: foo.com
        Description: Broadcom 802.11n Network Adapter
        Friendly name: Wi-Fi
        Physical address: ......
        Flags: 453
        Mtu: 1500
        IfType: IF_TYPE_IEEE80211
        OperStatus: IfOperStatusUp
        Ipv6IfIndex (IPv6 interface): 4
        ZoneIndices (hex): 4 4 4 4 1 1 1 1 1 1 1 1 1 1 0 1
        Transmit link speed: 65000000
        Receive link speed: 144000000
        Number of IP Adapter Prefix entries: 5


GetAllNetworkInterfaces yields:
        Current Wi-Fi Speed on Broadcom 802.11n Network Adapter: 144000000

C:\notes\wifistatus\windows>

Alternatively, perhaps, more naturally, you can do the same in C/C++,
To compile and run it on the Windows Command Prompt command line, do,

C:\notes\wifistatus\windows>"c:\Program Files (x86)\Microsoft Visual 
Studio 12.0\Common7\Tools\vsvars32.bat"

C:\notes\wifistatus\windows>cl /W4 getwifiadapteraddress.cpp
Microsoft (R) C/C++ Optimizing Compiler Version 18.00.30501 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.

getwifiadapteraddress.cpp
Microsoft (R) Incremental Linker Version 12.00.30501.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:getwifiadapteraddress.exe
getwifiadapteraddress.obj

C:\notes\wifistatus\windows>getwifiadapteraddress.exe
       Length of the IP_ADAPTER_ADDRESS struct: 376
       IfIndex (IPv4 interface): 4
       Adapter name: {......}
       DNS Suffix: foo.com
       Description: Broadcom 802.11n Network Adapter
       Friendly name: Wi-Fi
       Physical address: ......
       Flags: 453
       Mtu: 1500
       IfType: 71
       OperStatus: 1
       Ipv6IfIndex (IPv6 interface): 4
       ZoneIndices (hex): 4 4 4 4 1 1 1 1 1 1 1 1 1 1 0 1
       Transmit link speed: 65000000
       Receive link speed: 144000000
       Number of IP Adapter Prefix entries: 8
C:\notes\wifistatus\windows>

Discussion

Be aware of the difference between the speed (or bandwidth) of the interface card, the negotiated link speed (or bandwidth), and the throughput. The speed of the interface card is the specified speed that the interface card could achieve, i.e., a part of the hardware specification. The negotiated link speed is a reflection of the signal strength. The throughput is the one that you actually achieve. NetworkInterface gives you the speed of the interface card.GetAdaptersAddresses can give you the negotiated link speed. Finally, the throughput can only be determined by a measurement based approach. IPInterfaceStatistics and IPv4InterfaceStatistics can help you estimate it.