Monday, September 22, 2014


This is just a note on NoSQL.

The company I worked with many years ago had a discussion among the developers and managers on how they would port their code base that managed and queried much data in the company's core business from using a B-tree library to a relational database. It was compelling that a relational database would offer much advantage of the B-tree library -- SQL is widely supported in many languages and platform, the database engine would actually use a B+ tree for efficient query of the data, and the relational databases can also manages transactions and ensures integrity constriants.

Well, the tide is now turned. Data management based on key-value stores, mostly using hash table or tree structures to ensure efficient query becomes popular again. The development of network applications appears to pose a serious challenge to relational database business and NoSQL is gaining momentum. NoSQL aappears to threat some traditional database company's profit margin. There have been many discussions about NoSQL and relatinoal database management systems (RDBMS) powered by SQL, such as,

You can find more by do a search in Google Scholar or just a Google search. It appears that it is not that NoSQL can do something RDMBS cannot, rather it is that NoSQL has successfully created some an excellent APIs and platforms that work well on a few special and popular set of on-line applications.

Many NoSQL engines are available. The news article cited above mentionds MongoDB that has a good and interactive tutorial on their website; by the way, the language used in the tutorial is JavaScript whose popularity, in my opinion, an exellent footnote on the increasing popularity of NoSQL -- well, is it the same set of online applications that also helped the increased popularity of JavaScript?

Sunday, September 21, 2014

Problem with Mercurial (hg) Subrepo: "about: HTTP Error 404: Not Found"

Mercurial Subrepository allows us to clone multiple repositories as a group. Following the steps in the documentation, I created a main repository and a few sub-repositories. I did not encounter any problem when I clone the main repository or any of the sub-repositories. However,I observed an error when I attempted to push a local subrepository to the server. The error message reads,

abort: HTTP Error 404: Not Found
What gives me the hint what might be wrong is this post. In the repository's hgweb file, I had,

/ = /home/hg/repo-private/*

Since we have subrepostories in the repository, the above configuration is incorrect -- it should have been two *'s instead of one *, i.e.,

/ = /home/hg/repo-private/**

Indeed, as pointed out by the post cited above, <code>hg help hgweb</code> gives a good explanation and example how we may configure the <code>hgweb</code> file.

Friday, September 19, 2014

Audio Playing and Recording with Octave

Octave Forge has an audio package. At present, the package is in the "unmaintained" status. However, once installed successfully, it has all the functionality that I need:
  • Record audio data
  • Play audio data
I primarily use it on Linux virtual machines on Windows hosts. The following procedure was tested with Octave 3.6 on a Ubuntu 13.10 (Saucy) guest operating system on a Windows 8.1 host.

It is quite straight forward to install the audio package,

  1. Install Octave and Octave development package.
    sudo apt-get install octave liboctave-dev

    The latter is required to install Octave packages from Octave Forge.
  2. Install the audio package from Octave Forge.
         sudo octave
    In Octave, run
         pkg install -forge audio
    We may list installed packages in Octave to verify that it is successfully installed by
         pkg list
    The output should look like the following,
         Package Name  | Version | Installation directory
                audio *|   1.1.4 | /usr/share/octave/packages/audio-1.1.4
    which shows that the audio package is version 1.1.4 and installed at /usr/share/octave/packages/audio-1.1.4. The "*" next to the package name indicates that the package is loaded and ready to use.

    If you run Octave as a non-root user, the Octave package will be installed under the user's home directory and can only be used by the user. By running Octave as root (i.e., using sudo octave), we install the Octave package for everyone.

  3. The audio package uses some external programs to play and record audio data. Sox is sufficient for playing sound. We now install Sox in Linux.
         apt-get install sox
  4. The audio package uses OSS to record audio data. Ubuntu 13.10 does not have an OSS package. Instead, it provides an ALSA wrapper for OSS. We now install the wrapper.
        apt-get install alsa-oss
  5. To use the ALSA wrapper for OSS, we must invoke the aoss command. Then to run Octave, we run,
        aoss octave
    Without using aoss to invoke octave, you would encounter an error indicating that it failed to open /dev/dsp that is supposedly created by OSS. Since we are emulating OSS using ALSA, the system does not have the device file.

  6. Now, let us play a tone of 2 seconds at 400 Hz in Octave.
      y = cos(2 * pi * 400 * (0:1/44100:2));
      sound(6, 44100);

    Note that the sampling rate is 44100Hz in the above.
  7. To record and then play the recorded audio data, we may do
      [y, fs, ch] = aurecord(5, 44100, 2);
      sound(y, fs, ch);
  8. Finally, if you do not hear anything when you play the sound, check your virtual machine setting. It must have an appropriate audio card. The following shows the audio card setting for the Ubuntu virtual machine running in VMware Player.

    At the same time, the mixer of the Windows host should show that the virtual machine is a source as shown below. You can open the mixer by right-cliking on the "speaker" icon in the system tray, then choose "Open Volume Mixer".

    If you are running VirtualBox, you will have more than one audio driver to choose from when configuring the virtual machine. My experience is that not all driver works on a particular host. You can test the driver one by one until you find one that works. When the driver works and the virtual machine is running, you should observe that Windows Mixer shows the virtual machine as a sound source similar to the figure above.

Thursday, September 18, 2014

Creating Small Footprint Linux Virtual Machine

Although it may be easier to download a Linux virtual appliance, it can be fun to create linux virtual machines yourself. In particular, we may be interested in creating small footprint Linux virtual machines -- it would be easy to put it on a USB drive or let other people to download it. This post documents the process that I use to create Ubuntu 14.04 (Trusty) Linux virtual machines and reduce the image size of the virtual machines. The key idea is (1) to remove non-essential packages as much as possible; (2) to increase the compression of virtual machine images by zeroing virtual disks; and (3) to compact and compress the virtual machine image.
  1. Download 32-bit Ubuntu 14.04 (Trusty) ISO file and install it as a guest operating system under VMware Player or VirtualBox.  
  2. To make copy & paste instructions easier, install VMware Tools.  
    1. Install Linux headers and the build essential packages

      sudo apt-get install linux-headers-generic build-essential

    2. Then install VMware Tools
    3. Restart the virtual machine.
  3. By default, Ubuntu 14.04 runs the Ubuntu Unity desktop environment. It prevents us from entering the VMware Unity mode. I now get rid of the Ubuntu Unity desktop environment. I learned the command from this post.
    sudo apt-get purge \
         unity unity-common unity-services \
         unity-lens-\* unity-scope-\* unity-webapps-\* \
         gnome-control-center-unity \
         hud libunity-core-6\* libunity-misc4 \
         libunity-webapps\* appmenu-gtk \
         appmenu-gtk3 appmenu-qt\* overlay-scrollbar\* \
         activity-log-manager-control-center \
         firefox-globalmenu thunderbird-globalmenu \
         libufe-xidgetter0 xul-ext-unity \
         xul-ext-webaccounts webaccounts-extension-common \
         xul-ext-websites-integration \
         gnome-control-center gnome-session

  4. Then, install GNOME 3.
    sudo apt-get install gnome-session-fallback

  5. Now, remove the default games.
    sudo apt-get game\* purge ace-of-penguins gnome-mahjongg gnome-mines \
                 berusky biniax2 gnurobbo groundhog gtkballs hitori \
                 four-in-a-row gnect gbrainy gnome-sudoku chess\* \
                 nibbles\* solitaire\* iagno\* klotski\* tetravex\* \
                 drascula gtkboard oneisenough openyahtzee ri-li \
                 scribble swell-foop quadrapassel\* lightsoff \
                 five-or-more gnome-robots gtali

  6. Remove a few network applications.
    sudo apt-get purge webbrowser-app thunderbird\*

  7. Remove Libre Office. This can save a few hundred megabytes.
    sudo apt-get purge libreoffice\*

  8. Now get rid of the "Ubuntu Software Center".
    sudo apt-get purge software-center\*

  9. The /var/cache directory can easily have a few hundred megabytes data. We can trim it down.
    sudo apt-get clean

  10. Do an automove,
    sudo apt-get autoremove

  11. Now remove GNOME and Ubuntu documentation.
      sudo apt-get purge yelp\*

  12. Now, you can install whatever important software you want to include in the virtual machine, e.g., I am installing Octave,

    sudo apt-get install octave

  13. We would probably want to compress the virtual machine image before we pass it to someone. We need to replace every byte of free space on the disk by 0's.
    dd if=/dev/zero of=zerofile; \
       sleep 5; sync; \
       sleep 5; sync; \
       sleep 5; sync; \
       rm -f zerofile;

  14. I am using VMware Player. Using it, I do a "defragment" and then "compact" the virtual disk. The guest system must be off before you can do it. Of course, you can do the same using VirtualBox.
  15. Finally, I compress the folder of the virtual machine. The zip archieve file is about 1 GB. 
In addition, you may save a few megabytes, by removing all log and temporary files. The tool bleachbit can help. Before you zero the hard disk, install and run the tool.

sudo apt-get install bleachbit; sudo bleachbit

Tuesday, September 16, 2014

Querying Wi-Fi Link Speed on Windows

Many have expressed the interest to query Wi-Fi link speeed on Windows and provided answers, such as, this, this, and this, which are the main references for this post.

On Windows 7,  we can view network link speed easily and conveniently using the Windows Task Manager, as shown in the figure below,

On Windows 8, the Task Manager has different interface and does not show the link speed any more. If we want to know the link speed, we can view it using the Windows Control Panel, which works on both Windows 7 and Windows 8. A shortcut is to right-click on the  "network" icon in the system tray, then open the "Network and Sharing Center", finally click on the Wi-Fi connection on the right. The result will be something similar to below,

which clealy shows the link speed.

The following methods may be more interesting,


Open a Windows Command Prompt window, you can issue the following command,

netsh wlan show interface 

which list all Wi-Fi interfaces and their properties.

Querying WMI with WQL

We can use Windows PowerShell to issue WMI queries. Below is an example,

Windows PowerShell
Copyright (C) 2013 Microsoft Corporation. All rights reserved.

PS C:\> $query = "SELECT Description, DeviceID, Speed FROM CIM_NetworkAdapter
WHERE Description like '%Wi-Fi%' OR Description like '%802.11%'";
PS C:\>  Get-WmiObject -Query $query

These two posts are particularly helpful for me to write WQL queries to query the WMI.

GetAdaptersAddresses API

To me, this is the most interesting one. This stackoverflow thread provides a sample program in C# that uses P/Invoke to call the GetAdaptersAddresses API.  I made some simple revisions and put all the code together,
You can create a project in Visual Studio to compile the program. However, for this simple example, I would prefer a command line approach,

C:\notes\wifistatus\windows>"c:\Program Files (x86)\Microsoft Visual 
Studio 12.0\Common7\Tools\vsvars32.bat"
C:\notes\wifistatus\windows>csc /warn:4 WifiStatus.cs IPIntertop.cs
Microsoft (R) Visual C# Compiler version 12.0.30501.0
for C# 5
Copyright (C) Microsoft Corporation. All rights reserved.

GetAdaptersAddresses yields:
        Length of the IP_ADAPTER_ADDRESS struct: 448
        IfIndex (IPv4 interface): 4
        Adapter name: {......}
        DNS Suffix:
        Description: Broadcom 802.11n Network Adapter
        Friendly name: Wi-Fi
        Physical address: ......
        Flags: 453
        Mtu: 1500
        IfType: IF_TYPE_IEEE80211
        OperStatus: IfOperStatusUp
        Ipv6IfIndex (IPv6 interface): 4
        ZoneIndices (hex): 4 4 4 4 1 1 1 1 1 1 1 1 1 1 0 1
        Transmit link speed: 65000000
        Receive link speed: 144000000
        Number of IP Adapter Prefix entries: 5

GetAllNetworkInterfaces yields:
        Current Wi-Fi Speed on Broadcom 802.11n Network Adapter: 144000000


Alternatively, perhaps, more naturally, you can do the same in C/C++,
To compile and run it on the Windows Command Prompt command line, do,

C:\notes\wifistatus\windows>"c:\Program Files (x86)\Microsoft Visual 
Studio 12.0\Common7\Tools\vsvars32.bat"

C:\notes\wifistatus\windows>cl /W4 getwifiadapteraddress.cpp
Microsoft (R) C/C++ Optimizing Compiler Version 18.00.30501 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.

Microsoft (R) Incremental Linker Version 12.00.30501.0
Copyright (C) Microsoft Corporation.  All rights reserved.


       Length of the IP_ADAPTER_ADDRESS struct: 376
       IfIndex (IPv4 interface): 4
       Adapter name: {......}
       DNS Suffix:
       Description: Broadcom 802.11n Network Adapter
       Friendly name: Wi-Fi
       Physical address: ......
       Flags: 453
       Mtu: 1500
       IfType: 71
       OperStatus: 1
       Ipv6IfIndex (IPv6 interface): 4
       ZoneIndices (hex): 4 4 4 4 1 1 1 1 1 1 1 1 1 1 0 1
       Transmit link speed: 65000000
       Receive link speed: 144000000
       Number of IP Adapter Prefix entries: 8


Be aware of the difference between the speed (or bandwidth) of the interface card, the negotiated link speed (or bandwidth), and the throughput. The speed of the interface card is the specified speed that the interface card could achieve, i.e., a part of the hardware specification. The negotiated link speed is a reflection of the signal strength. The throughput is the one that you actually achieve. NetworkInterface gives you the speed of the interface card.GetAdaptersAddresses can give you the negotiated link speed. Finally, the throughput can only be determined by a measurement based approach. IPInterfaceStatistics and IPv4InterfaceStatistics can help you estimate it.

Thursday, September 11, 2014

A Note on JavaScript

JavaScript appears to be a growingly popular choice of programming languages for many things, perhaps, as a result of HTML5 development, a result of being able to run in a sandbox, a result of being able to create closure, and others ... ... I am particularly impressed by BablylonJS as I came across it today.

It is the time for me to create, well, a bookmark for a few JavaScript libraries, frameworks, IDEs, and every other thing about JavaScript that I have come across. I'll keep it updated when I come across something interesting.


Server-Side JavaScript

Client-Side Javascript

Tutorials and Examples

Sunday, September 7, 2014

All About Food

This servers as a bookmark for various cooking shows, blogs, books and many others that I like. I will keep updating this post. 

Recipe Blogs & Sites

Personal Website

Ubuntu 14.04 (Trusty) and VMWare Unity Mode

Unity mode is a nice feature of VMWare's virtualization software. However, you will find that you cannot switch to VMWare Unity mode once you install the most recent release of Ubuntu including Ubuntu 14.04 (Trusty) and its predecessor (Ubuntu 13.10 or Saucy).

It turns out that VMWare does not like Ubuntu's default desktop environment called Ubuntu Unity. It could be confusing that the Ubuntu Unity desktop environment and the VMWare Unity mode share the same word "Unity", and it has created a difficulty  for us who try to find an answer  on the web since depending on what search terms you enter, web searches can give you answers related to both and it becomes a diffcult task to find a solution that may help you.

Since VMWare cannot switch to Unity mode is a result that VMWare does not like Ubuntu's default desktop environment called Ubuntu Unity, the solution becomes to find a Desktop Environment that VMWare can work with.

On Ubuntu 13.10 (Saucy), I found that Gnome 3 works with the VMWare Unity mode. Therefore, the solution is to install Gnome 3 and to use Gnome 3. To install Gnome 3, you may issue command below,
sudo apt-get update; sudo apt-get install gnome-session-fallback

I have been happily running Ubuntu 13.10 (Sauncy) as a guest operating system and using it in VMWare Unity mode for a quite while, well, until I upgrade it to Ubuntu 14.04 (Trusty). What is the problem? I checked the versions of Gnome 3 before and after the upgrade from Ubuntu 13.10 (Sauncy) to Ubuntu 14.04 (Trusty), they are the same,

$ gnome-session --version
gnome-session 3.9.90

I believe that it is still the issue with the Desktop environment that Ubuntu is running -- not VMWare tries not to support such a feature. Then, the task is to find out which desktop environment works with the VMWare Unity mode on Ubuntu 14.04 (Trusty). I found the list of Desktop environments that I may try from this post, 
I did not try all of them. However, I am happily to report that the following Desktop environments that work with VMWare Unity mode on Ubuntu 14.04 (Trusty),

  • Xfce
    To install it. Run
          sudo apt-get install xfce4
    or if you want more bells & whitsles, 
          apt-get install xubuntu-desktop
  • LXDE
    To install it. Run
          sudo apt-get install lxde
    or if you want the full desktop environment,
          apt-get install lubuntu-desktop
  • Mate
    To install it. Run
    sudo apt-add-repository ppa:ubuntu-mate-dev/ppa
    sudo apt-add-repository ppa:ubuntu-mate-dev/trusty-mate
    sudo apt-get update
    sudo apt-get upgrade
    sudo apt-get install --no-install-recommends \
                        ubuntu-mate-core ubuntu-mate-desktop
    At this point, you can switch to VMware Unity mode. However, the application menu of the VMWare Unity mode is empty. This is the result that the VMware tools at present only recognoizes three desktop environments, Gnome, KDE, and Xfce -- if you run vmware-xdg-detect-de in your Ubuntu guest, you will see that the output of the script is empty. I made one minor change to the script,  and the VMWare Unity mode menu appeared. Below is the diff result on the script before and after the revision.
    diff \
      /usr/bin/vmware-xdg-detect-de \
    <   elif `dbus-send --print-reply --dest=org.freedesktop.DBus \
    <       /org/freedesktop/DBus \
    <       org.freedesktop.DBus.GetNameOwner \
    <       string:org.mate.SessionManager > /dev/null 2>&1` ; \
    <       then DE=gnome; # pretent to be gnome

    The complete script is at GitHub.
  • GNOME Flashback/Fallback

    As described before, to install it, run,
    sudo apt-get update; sudo apt-get install gnome-session-fallback
    Initially, I thought GNOME Flashback/Fallback stopped working with VMware Unity mode on Ubuntu 14.04. However, once I investigated further, I discovered that Ubuntu 14.04 reports the value of environmental variable XDG_CURRENT_DESKTOP as Unity. The solution is still to let vmware-xdg-detect-de return gnome. What I did is shown in this diff result,
    diff /usr/bin/vmware-xdg-detect-de \
    < DE=gnome
    < xdgDE=gnome

    The complete script using the approach described above (although not identical) is at GitHub. It is worth noting that when I manually set the environmental variable to GNOME in the user profile, e.g., adding the following line to $HOME/.profile, VMware Unity mode remains NOT working. 
    It begins to work only after I made the revision to vmware-xdg-detect-de as described above.

    A new problem I have encountered is that once VMware enters the Unity mode, in some of my Ubuntu virtual machines, the application menu is empty -- if you always launch programs from command line, it may not be too much of a problem. A further investigation indicates that this problem only occurs on the Ubuntu virtual machines that are upgraded from previous revision. I reinstalled the gnome-session-flashback and the problem went away. To reinstall the package, do,
    sudo apt-get install --reinstall gnome-session-flashback

    Just be aware that when you have GNOME Flashback/Fallback installed, you will have two options on the login screen, "GNOME Flashback (compwiz)" and "GNOME Flashback (metacity)". VMware Unity mode only works with the latter. In fact, VMware lists "a modern version of Metacity or KDE" as a prerequist to use the Unity mode on its website.

To conclude, below is what I have tried.

Testing Result of Desktop Environment for VMWare Unity Mode on Ubuntu 14.04 (Trusty)
Desktop EnvironmentDisplay Name in Login ScreenVMWare Unity Mode Works?
Ubuntu UnityUbuntu DefaultNo
GNOME 3GNOME Flashback (Compwiz)No
GNOME 3GNOME Flashback (Metacity)Yes (See the note on Gnome above and
download vmware-xdg-detect-de)
MateMateYes (See the note on Mate above and
download vmware-xdg-detect-de)

However, if you try a different desktop envionment on Ubuntu 14.04 (Trusty) and it works or does not work with the VMWare Unity mode, please share it with me. I will update this post to include your result.

Saturday, September 6, 2014

LaTeX: Embedding All Fonts in PDF

Many publishers are cautions about readers' experience and want to ensure all fonts are embedded in PDF files when you send your final camera-ready PDF files in.

We commonly use LaTeX to produce publication grade PDF files. However, some publishers warn about the use of pdflatex to compile LaTeX files to produce PDF files, suggesting pdflatex is the culprit. Perhaps, the problem is rather not with pdflatex, but with the configuration of pdflatex. My experience indicates that default configuration of most LaTeX distributions has already got the configuration right. However, my experience indicates that regardless I use pdflatex or the combination of latex, dvips, and ps2pdf, I have many occasions that not all fonts are embedded.

The issue is actually not with latex, but with figures generated by some third party tools. For instance, if you use Matlab to save figure as an eps file, for instance, the following Matlab script produces an eps file example_signal.eps showing a cosine signal.

num_periods = 10;

freq = 5;
sampling_interval = 1/100;
signal_length = num_periods * 1 / freq;
t = 1:sampling_interval:signal_length;
y = 10 * cos(2 * pi * freq * t + pi / 4);

plot(t, y);

set(gcf, 'PaperSize', [5 5*0.618]);
set(gcf, 'PaperPosition', [0 0 5 5*0.618]);
print('-depsc', 'example_signal');

Let us convert the eps file to a PDF file.

ps2pdf example_signal.eps
pdfcrop example_signal.pdf

Now let us verify if the resulting PDF file has had all fonts embedded,

pdffonts ./example_signal.pdf
name       type    encoding  emb sub uni object ID
---------- ------- --------- --- --- --- ---------
Helvetica  Type 1  Custom    no  no  no       8  0

The above shows that font Helvetica is not embedded. How do we make sure all fonts are embedded. You must include as the example shows, 5 switches in the ps2pdf command,

ps2pdf \
       -dPDFSETTINGS=/prepress \
       -dSubsetFonts=true \
       -dEmbedAllFonts=true \
       -dMaxSubsetPct=100 \
       -dCompatibilityLevel=1.4 ./example_signal.eps

pdffonts ./example_signal.pdf
name              type    encoding emb sub uni object ID
----------------- ------- -------- --- --- --- ---------
FHQQIQ+Helvetica  Type 1C Custom   yes yes no       8  0

We now see the font is embedded.

Note the following,
  • You must include the 5 switches in the ps2pdf command. If you miss one, fonts may not be all embedded. 
  • The command has no effect on PDF files, even though you can run ps2pdf against PDF files and no error or warning will be reported.

If you have a PDF file, but not its ps or eps file, you can use the steps below to convert the PDF file to a ps file and then convert the ps file back to a PDF file. The new PDF file will have all the fonts embedded, for instance,

pdf2ps example.pdf
ps2pdf \
       -dPDFSETTINGS=/prepress \
       -dSubsetFonts=true \
       -dEmbedAllFonts=true \
       -dMaxSubsetPct=100 \
       -dCompatibilityLevel=1.4 ./
That is all.

Friday, September 5, 2014

The Stem Crisis: Shortage or Abundance of STEM Graduates

From time to time, some minor voice on the shortage of  STEM graduates starts attract attention. Contrary to more dominant view that not only does STEM shortage exist, it may also be a crisis in the making, a recent IEEE Spectrum article questions whether such a crisis is real, which got my attention. A debate ensures and the discussion continues.

I would like to document the opinions of both sides that  I come across.  I'll come back to update this post when I come across new articles.

Motion: There is a STEM Crisis

For the Motion

Against the Motion


    Monday, September 1, 2014

    Bypassing SIM Card and Motoblur Account Setup on Atrix 4G

    From time to time, I get the job to deal with a Motorola Atrix 4G phone. Yes, they are an old model, but we do have a couple of them. Fresh out-of-box, the phone always asks the user to insert a SIM card and set up a Motoblur account. To bypassing SIM card and Motoblur account setup, I follow the instruction in this post, which is pretty handy. Just a convenient jump for myselt, click the instruction below for the steps,

    And to quote, the steps are,
    1. Power off the phone
    2. While holding the "Volumn Down" key, power up the phone
    3. As it is powering up, the phone pauses the boot process and displays boot option. Press the "Volumn Down" button a few times, until "Boot Android (No BP)" appears
    4. Now press the "Volumn Up" key to choose the boot option. The phone will resume the boot process.
    5. When it finishes the boot process, the phone displays a language drop-down menu. Thelanguage selection screen is for Motoblur account setup -- the phone has bypassed the "No SIM card/Emergency calls only" screen.
    6. Hold the "Menu" key at the language selection screen of the Motorblur account setup until the soft keyboard appears
    7. Then press the "?123, ALT, ABC, e" key sequence twice in a row, then type the key sequence "b l u r o f f". Note that quotation marks are not part of the key sequences.