Sunday, September 29, 2013

Convert VirtualBox VDI to VMWare VMDK

You can run virtual machine instances created in one virtualization software in another. QEMU is a good tool to do this. The following example converts a VirtualBox VDI file to a VMDK file that can be loaded in VMWare.

I did the conversion on a Windows host. You can download the precompiled Windows binaries of QEMU, provided by Prashant Satish,  or those provided by Eric Lassauge.

The procedure is the same in Linux hosts.

  1. To convert VirtualBox virtual appliance to a VMDK file.
    qemu-img convert  virtualbox.vdi  -O vmdk vmware.vmdk
  2. If you have VMWare Workstation or more luxury version, you can start with creating a new virtual machine. During the process, you have the option of selecting an "existing disk". In this case, you simply select the .vmdk file.

    However, I would like to stick to the free VMWare Player that does not provide the option of selecting an "existing disk". I have to use a workaround.

    Start VMWare Player, then create a new virtual machine, for instance, a virtual machine called "vboxvmware".  You must select "I will install the operating system later." option as the two images show. 



    Since you created or downloaded the VirtualBox virtual machine, you should know what guest operating system it is and what memory capacity is allocated. Choose them accordingly.



    Next step, choose the option of "Store Virtual Disk as a Single File".


    Upon confirming your choice, you will see that the virtual machine is created.



    The last step is to edit the VMWare virtual machine configuration file (.vmx file). Go to the directory where the virtual machine is located, locate the .vmx file, and edit the file using your favorite text editor. In the editor, locate the line of "scsi0:0.fileName = ..." and replace the virtual disk file by the .vmdk file previously converted using QEMU. Note that when you edit the file, make sure the path of the .vmdk file is correct if it is not copied to the directory. Then, you can safely remove the vboxvwmare.vmdk file that your virtual machine does not really use. You can then safely launch the virtual machine from the summary screen.



Thursday, September 26, 2013

Tracing System Calls and System Events

I found strace and ltrace were good tools on Linux to diagnose many problems.

On Windows, similar tools actually exist. The equivalent to strace, perhaps is, Process Monitor and that to ltrace, Logger and LogViewer.

 In addition, Windows' event instrumentation is useful.

A good article is http://jbremer.org/intercepting-system-calls-on-x86_64-windows/ . The post is based on the discussion at http://stackoverflow.com/questions/3847745/systrace-for-windows.




Tuesday, September 24, 2013

Web Pages Never Expire

Heard this website http://perma.cc from Marketplace Tech. "Some of the most prestigious institutions in the country are taking efforts to create a new, permanent archive of scholarly work online. new, permanent archive of scholarly work online" as it states. Sounds quite interesting.The site is still in beta -- I hope that it will go alive soon.

Sunday, September 15, 2013

Physical Address Extension (PAE) on 32-bit Windows

One friend complains to me that his company issued computer has more than 4GB physical memory but the system property shows that he can only access 3GB. This information can be revealed from Windows Control Panel (Control Panel\System and Security\System on Windows Vista and above).  It shows you how much physical memory installed and how much the operating system can get access to. If you have a 32-bit Windows installed, like my friend, your system can address up to 4GB physical memory. Microsoft provides memory limits for various Windows releases. See the page. What makes the situation worse is that your 32-bit Windows typically can access even less than 4GB memory. It happens when some of I/O devices are operated using memory-mapped I/O. You may refer to Microsoft Knowledge Base KB929605 for more detailed discussion.  However, I simply cite the "cause" section as follows,

Various devices in a typical computer require memory-mapped access. This is known as memory-mapped I/O (MMIO). For the MMIO space to be available to 32-bit operating systems, the MMIO space must reside within the first 4 GB of address space.

For example, if you have a video card that has 256 MB of onboard memory, that memory must be mapped within the first 4 GB of address space. If 4 GB of system memory is already installed, part of that address space must be reserved by the graphics memory mapping. Graphics memory mapping overwrites a part of the system memory. These conditions reduce the total amount of system memory that is available to the operating system.

The reduction in available system memory depends on the devices that are installed in the computer. However, to avoid potential driver compatibility issues, the 32-bit versions of Windows Vista limit the total available memory to 3.12 GB. See the "More information" section for information about potential driver compatibility issues.

If a computer has many installed devices, the available memory may be reduced to 3 GB or less. However, the maximum memory available in 32-bit versions of Windows Vista is typically 3.12 GB.

Oops! How would I access more the rest of physical memory then? The memory is important to system performance. Of course, you may upgrade your Windows to a 64-bit version. However, you may avoid the hassle by enabling Physical Address Extension (PAE) on your 32-bit Windows. Microsoft has an entry (aa366796) on PAE in MSDN library. To make it simple, you may following the following steps on your 32-bit Windows,

  1. Run a Windows command window as  administrator. (Search cmd, then right click on the short cut, choose Run as Administrator)
  2. In the command window, run

    bcdedit /set pae ForceEnable

    which allows Windows on x86-based systems to access up to 64 GB or 128 GB of physical memory, depending on the physical address size of the processor.Wikipedia has a table that lists the physical address bus size.


Friday, September 13, 2013

Vim as Hex Editor


Vi Improved (vim) is a great tool. It can serve as a good Hex editor if the file you are viewing is not too big.

Once you open a file using Vim, use command %!xxd to enter hex mode. To exit hex mode, use %!xxd -r.

Thursday, September 12, 2013

Emulating LaTeX Look & Feel in Microsoft World

Following the instruction , you can find a Word template and the URL to Computer Modern fonts that should be used along with the template.

Using the template and the fonts, you can generate Word documents that remarkable resemble those generated using LaTeX.

I found downloading the fonts manually a pain since there are many. Perhaps, a simpler way is to use a shellscript, which I ran on an Ubuntu Linux guest OS,

wget http://www.ctan.org/tex-archive/fonts/cm/ps-type1/bakoma/ttf/ -O - -q | \
    grep "mirror.*ttf" | \
    sed -e 's/^.*href=\"//g' | \
    sed -e 's/\" title=.*$//g' | \
    xargs wget -c -nv

Well, it may not be a script since it is just a one-liner. If you are unfamiliar with Unix/Linux shells, note that there is no space after "\" that simply indicates a line continuation







Thursday, September 5, 2013

Testing Exchange Server Connectivity


To test Microsft Exchange servers, Microsoft has a service that can help. The service is at 
which allows you to test the connectivity between your host and a Microsoft Exchange server.

Even if you are not setting up an Exchange server, the service can help you determine what configuration that your Exchange e-mail provider may be, e.g., does it work with Windows Mail? does it provide the Outlook Anyware connectivity?



Monday, September 2, 2013

Linux Capabilities

It had been for a long time that processes' permission on UNIX/Linux systems are differentiated into two categories, privileged or non-privileged processes. The effective user ID of privileged processes is 0 while that of non-privileged processes is nonzero. User ID 0 belongs to the superuser or root. Such a granularity were viewed as too coarse by many. Starting from Kernel 2.2,  Linux introduces the concept of capabilities that divides the privileges that traditionally associated with superuser into many categories. Linux manual page Capabilities has a good discussion on this topic.

Apparently,  many Linux programmers do  not seem to have a good understanding on this new development. Michael Kerrisk has statistics on the usage of different Linux capabilities. Perhaps, it is easier to get what you need by just assuming the privilege of the superuser than figuring out what you do not really need, which requires perhaps higher cognitive load and activity.

This post demonstrates a few usage of capabilities from an application programmer point of view. 

Packet Socket

Packet socket requires that the opening process has effective UID 0 or the CAP_NET_RAW capability. The following example program sends a message over an Ethernet.

#include <arpa/inet.h>
#include <net/ethernet.h>
#include <netinet/ether.h>
#include <netpacket/packet.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
    int sockfd;
    struct sockaddr_ll dest_addr;

    if (argc < 3) {
        printf("Usage: %s destination_host message\n", argv[0]);
        exit(0);
    }

    sockfd = socket(AF_PACKET, SOCK_DGRAM, htons(ETH_P_ALL));

    if (sockfd == -1) {
        perror("Error calling socket(AF_PACKET, SOCK_DGRAM ...): ");
        exit(1);
    }


    /* When you send packets it is enough to specify sll_family, sll_addr,
     * sll_halen, sll_ifindex. The other fields should be 0. */
    memset(&dest_addr, '\0', sizeof(dest_addr));

    dest_addr.sll_family = AF_PACKET;
    dest_addr.sll_ifindex = 1;
    dest_addr.sll_halen = ETH_ALEN;

    if (ether_aton_r(argv[1],
            (struct ether_addr*)&(dest_addr.sll_addr)) == NULL) {
        fprintf(stderr,
            "Error: %s is not in the hex-digits-and-colons format.\n",
            argv[1]);
    }

   if (sendto(sockfd, argv[2], strlen(argv[2]), 0,
            (struct sockaddr*)&dest_addr, sizeof(dest_addr)) == -1) {
        perror("Error calling sendto(...): ");
        exit(1);
    }

    printf("Info: packet sent successful\n");

    close(sockfd);
    return 0;
}

The program takes two command line arguments. The first argument is the Ethernet address of destination host and the second argument is the message to send.

When you run it as a non-privileged user, for instance, as follows,

        $ ./sendpacket 00:0c:29:89:7a:4d "Hello, World"

you would receive an "Operation not permitted" error,

       Error calling socket(AF_PACKET, SOCK_DGRAM ...): : Operation not permitted

Two methods that we can use to make it work. First, run it under root, the traditional method,

        $ sudo ./sendpacket 00:0c:29:89:7a:4d "Hello, World"
        Info: packet sent successful

A new method, which is a better and preferred method, is to give the program minimal but necessary privilege -- since the packet socket requires the program with CAP_NET_RAW privilege, we ought to give the program the privilege, but only the privilege.

However, before that, let us check what privilege the program has,


        $ /sbin/getcap ./sendpacket

It outputs nothing, which means the program does not any privilege. Now we can give the program the privilege by

        $ sudo /sbin/setcap cap_net_raw=ep ./sendpacket

Now check the program's privilege again,

        $ /sbin/getcap ./sendpacket
        ./sendpacket = cap_net_raw+ep

Now the output indicates that the program has its effective privilege set as  CAP_NET_RAW. Run the program again as a non-privileged user,

        $ ./sendpacket 00:0c:29:89:7a:4d "Hello, World"
        Info: packet sent successful


Notable Issues

When I tried to set capability for the program on a Virtual Machine, I received an error:

        Failed to set capabilities on file `./sendpacket' (Operation not supported)

This is because that the file system that the file was on is actually a VMWare HGFS that moutns a Windows NTFS. The Windows NTFS does not support the security capability. When I copied the file to an ext4 file system, the problem went away.

libcap Library

For programming Linux capabilities, you need the libcap library.

On Ubuntu,

          sudo apt-get install libcap-dev

On CentOS/Fedora Linux,

          sudo yum install libcap-devel


Reference and Further Reading

  1. http://www.cis.syr.edu/~wedu/seed/Labs/Documentation/Linux/How_Linux_Capability_Works.pdf
  2. http://man7.org/linux/man-pages/man7/packet.7.html
  3. http://www.linuxjournal.com/article/5737
  4. http://ols.fedoraproject.org/OLS/Reprints-2008/hallyn-reprint.pdf
  5. http://www.cis.gvsu.edu/~kalafuta/cis458/f12/labs/lab3.html

Sunday, September 1, 2013

Search Windows Drivers for Unknown/Other Devices Online

It is often that we encounter the problem of unidentified devices in Microsoft Windows. A web search, perhaps, is the most common method to locate the drivers for the devices.

Having located Windows drivers for a few "other device" or "unknown devices", I learned that the correct method to do the web searches is to use the devices "Hardware Ids" listed under the "Details" tab of the selected device in Windows Device Manager. See the figures below.



Basically, you ought to open Windows Device Manager from Control Panel, right click on the unknown device or other device, then select Properties.  Click the Details tab, choose Hardware Ids from the dropdown box, and press CTRL-C to copy the details shown under  the Value box. Then, press CTRL-V in a web search box along with model and maker of your computer. In this particular, example, I entered,

USB\VID_138A&PID_003C HP EliteBook 2760p driver

The first two returned results would give you sufficient information to locate the driver for the "uknown device", which is, in fact, a Validity Fingerprint sensor.



Other Devices/Unknown Devices When Installing Windows 8 on HP Elitebook 2760p

Having installed Windows 8 on an HP Elitebook 2760p, found the following unrecognized devices in Windows Device Manager,
  • Other devices
    • Base System Devices
    • Unknown devices
The "Base System Devices" has the following hardware ids,
   Hardware Ids
     PCI\VEN_197B&DEV_2392&SUBSYS_162A103C&REV_30
     PCI\VEN_197B&DEV_2392&SUBSYS_162A103C
     PCI\VEN_197B&DEV_2392&CC_088000
     PCI\VEN_197B&DEV_2392&CC_0880


The "Unknown Device" has the following hardware ids,
   Hardware Ids
     USB\VID_138A&PID_003C&REV_0086
     USB\VID_138A&PID_003C


These two devices actually are

For convenience you may download the drivers for different versions of Windows below:
    HP Elitebook 2760p Drivers for Windows.