Saturday, May 30, 2020

Problem: Keyboard Input Echoed Back to Serial Terminal using PuTTY to Linux System

I enabled the serial console on a Debian Linux system. I used PuTTY to connect to the serial console. However, I encountered a problem, i.e., any keyboard inputs were echoed back to the serial console. The following screen shot perhaps explains the problem, e.g., when I enter user name, the user name is echoed back again, and when I enter the password, by default, the terminal shouldn't show anything, but it echoes it back.


A Stack Overflow thread discusses a similar problem, and I linked it below,
The thread led me to experimenting the terminal setting in PuTTY. The default setting of the "Line discipline options" is "Auto" for "Local echo", and "Auto" for "Local line editing" shown in the screen capture below. When using these settings, I encountered the problem I described in the above.


My fiddling with the settings tells me that the solution is to check "Force off"  for "Local echo", and "Force off" for "Local line editing" shown in the screen shot below,


Having made the setting changes, I saw that the problem went away. Well, we can see it from the screen shot below,



Thursday, May 28, 2020

Adding new disk to a system? MBR Partition Table or GUID Partition Table?

I wanted to add a new virtual harddisk drive to a Debian Linux system on an Oracle VirtualBox virtual machine. A little bit planning goes a long way. One question we ought to ask is, what type of partition table should we choose?

For PCs, there are two types of partitions one can choose from,
  1. Master Boot Record Partition Table (MBR Partition Table, or MBR), and
  2. GUID Partition Table (GPT).
The former was developed in 1980's with DOS and enjoys a wide support, but has a major limitation.  The major limitation is rooted from the fact that the logical address to a sector on a MBR partition is 32-bit wide. Given sector size is 512 bytes, the largest partition is limited to

232*512=241=2*240=2 TB

To overcome this limitation, GPT uses 64-bit logical addresses. This arguably allows us to address the number of bytes of an astronomical scale.

Master Boot Record Partition Table

The complete process of adding a harddisk drive to a Debian Linux sytem using the MBR partition is as follows,
  1. Look up the name the attached harddisk drive using lsblk
  2. Create an extended partition using the fdisk tool
  3. Create a logical partition on the extended partition using the fdisk tool
  4. Create a file system on the logical partition
  5. Mount the file system and use it
This method is in fact similar to the method to create GUID partition table. The details are thus skipped here.

GUID Partition Table

There are a couple of tools we can use to create GUID partition tables, e.g.,
  • fdisk. The new version of fdisk has GPT support. We can know whether the version installed in the system has the GPT support by viewing its manual page, e.g., the manual page of the version of fdisk on the Linux system I am using states,

    "
    fdisk is a dialog-driven program for creation and manipulation of partition tables. It understands GPT, MBR, Sun, SGI and BSD partition tables.
    "
  • gdisk. It is a tool much like fdisk. The manual page states that it is a

    "
    GPT fdisk (aka gdisk) is a text-mode menu-driven program for creation and manipulation of partition tables.
    "
  • parted. This is a more powerful tool than fdisk and gdisk.

Below we show 3 examples using these 3 tools to create a GPT.

Using fdisk

  1. Look up the name the attached harddisk drive using lsblk
  2. Create a GPT.
    
    $ sudo fdisk /dev/sdb
    
    Welcome to fdisk (util-linux 2.33.1).
    Changes will remain in memory only, until you decide to write them.
    Be careful before using the write command.
    
    Command (m for help): m
    
    Help:
    
      GPT
       M   enter protective/hybrid MBR
    
      Generic
       d   delete a partition
       F   list free unpartitioned space
       l   list known partition types
       n   add a new partition
       p   print the partition table
       t   change a partition type
       v   verify the partition table
       i   print information about a partition
    
      Misc
       m   print this menu
       x   extra functionality (experts only)
    
      Script
       I   load disk layout from sfdisk script file
       O   dump disk layout to sfdisk script file
    
      Save & Exit
       w   write table to disk and exit
       q   quit without saving changes
    
      Create a new label
       g   create a new empty GPT partition table
       G   create a new empty SGI (IRIX) partition table
       o   create a new empty DOS partition table
       s   create a new empty Sun partition table
    
    
    Command (m for help): g
    
    Created a new GPT disklabel (GUID: 08486B58-59CB-E849-B814-38DD8434D43C).
    
    Command (m for help): n
    Partition number (1-128, default 1):
    First sector (2048-32767966, default 2048):
    Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-32767966, default 32767966):
    
    Created a new partition 1 of type 'Linux filesystem' and of size 15.6 GiB.
    Partition #1 contains a ext4 signature.
    $
    
  3. Display partition information
    
    $ sudo fdisk -l /dev/sdb
    Disk /dev/sdb: 15.6 GiB, 16777216000 bytes, 32768000 sectors
    Disk model: VBOX HARDDISK
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: gpt
    Disk identifier: 85CF1437-0605-4C91-BE0C-F3B416BEA1D8
    
    Device     Start      End  Sectors  Size Type
    /dev/sdb1   2048 32767966 32765919 15.6G Linux filesystem
    $
    
  4. Make the system to re-load the partition tables
    
    $ sudo partprobe
    $
    
  5. Mount the file system on the partition
    
    $ sudo mount -t ext4 /dev/sdb1 /mnt
    $
    

Using gdisk

  1. Look up the name the attached harddisk drive using lsblk
  2. Create a GPT.
    
    $ sudo gdisk /dev/sdb
    GPT fdisk (gdisk) version 1.0.3
    
    Partition table scan:
      MBR: MBR only
      BSD: not present
      APM: not present
      GPT: not present
    
    
    ***************************************************************
    Found invalid GPT and valid MBR; converting MBR to GPT format
    in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
    typing 'q' if you don't want to convert your MBR partitions
    to GPT format!
    ***************************************************************
    
    
    Command (? for help): ?
    b       back up GPT data to a file
    c       change a partition's name
    d       delete a partition
    i       show detailed information on a partition
    l       list known partition types
    n       add a new partition
    o       create a new empty GUID partition table (GPT)
    p       print the partition table
    q       quit without saving changes
    r       recovery and transformation options (experts only)
    s       sort partitions
    t       change a partition's type code
    v       verify disk
    w       write table to disk and exit
    x       extra functionality (experts only)
    ?       print this menu
    
    Command (? for help): n
    Partition number (1-128, default 1):
    First sector (34-32767966, default = 2048) or {+-}size{KMGTP}:
    Last sector (2048-32767966, default = 32767966) or {+-}size{KMGTP}:
    Current type is 'Linux filesystem'
    Hex code or GUID (L to show codes, Enter = 8300):
    Changed type of partition to 'Linux filesystem'
    
    Command (? for help): w
    
    Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
    PARTITIONS!!
    
    Do you want to proceed? (Y/N): y
    OK; writing new GUID partition table (GPT) to /dev/sdb.
    Warning: The kernel is still using the old partition table.
    The new table will be used at the next reboot or after you
    run partprobe(8) or kpartx(8)
    The operation has completed successfully.
    $
    
  3. Display partition information
    
    $ sudo gdisk -l /dev/sdb
    GPT fdisk (gdisk) version 1.0.3
    
    Partition table scan:
      MBR: protective
      BSD: not present
      APM: not present
      GPT: present
    
    Found valid GPT with protective MBR; using GPT.
    Disk /dev/sdb: 32768000 sectors, 15.6 GiB
    Model: VBOX HARDDISK
    Sector size (logical/physical): 512/512 bytes
    Disk identifier (GUID): 08486B58-59CB-E849-B814-38DD8434D43C
    Partition table holds up to 128 entries
    Main partition table begins at sector 2 and ends at sector 33
    First usable sector is 2048, last usable sector is 32767966
    Partitions will be aligned on 2048-sector boundaries
    Total free space is 0 sectors (0 bytes)
    
    Number  Start (sector)    End (sector)  Size       Code  Name
       1            2048        32767966   15.6 GiB    8300
    $
    
  4. Make the system to re-load the partition tables
    
    $ sudo partprobe
    $
    
  5. Mount the file system on the partition
    
    $ sudo mount -t ext4 /dev/sdb1 /mnt
    $
    

Using parted

  1. Look up the name the attached harddisk drive using lsblk
  2. Create a GPT.
    
    $ sudo parted /dev/sdb
    GNU Parted 3.2
    Using /dev/sdb
    Welcome to GNU Parted! Type 'help' to view a list of commands.
    (parted) mklabel gpt
    Warning: The existing disk label on /dev/sdb will be destroyed and all data on
    this disk will be lost. Do you want to continue?
    Yes/No? Yes
    (parted) mkpart primary 1 100%
    (parted) align-check optimal 1
    1 aligned
    (parted) print
    Model: ATA VBOX HARDDISK (scsi)
    Disk /dev/sdb: 16.8GB
    Sector size (logical/physical): 512B/512B
    Partition Table: gpt
    Disk Flags:
    
    Number  Start   End     Size    File system  Name     Flags
     1      1049kB  16.8GB  16.8GB               primary
    
    (parted) quit
    Information: You may need to update /etc/fstab.
    .
    $
    
  3. Display partition information
    
    $ sudo parted -l /dev/sdb
    ...
    
    
    Model: ATA VBOX HARDDISK (scsi)
    Disk /dev/sdb: 16.8GB
    Sector size (logical/physical): 512B/512B
    Partition Table: gpt
    Disk Flags:
    
    Number  Start   End     Size    File system  Name     Flags
     1      1049kB  16.8GB  16.8GB  ext4         primary
    
    
    $
    
  4. Make the system to re-load the partition tables
    
    $ sudo partprobe
    $
    
  5. Create an ext4 file system on the partition.
    
    $ sudo mkfs -t ext4 /dev/sdb1
    mke2fs 1.44.5 (15-Dec-2018)
    /dev/sdb1 contains a ext4 file system
            last mounted on Thu May 28 21:45:35 2020
    Proceed anyway? (y,N) y
    Creating filesystem with 4095488 4k blocks and 1024000 inodes
    Filesystem UUID: 6b43a1a4-ab37-4788-acef-df13149f7bc6
    Superblock backups stored on blocks:
            32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208
    
    Allocating group tables: done
    Writing inode tables: done
    Creating journal (16384 blocks): done
    Writing superblocks and filesystem accounting information: done
    $
    
  6. Mount the file system on the partition
    
    $ sudo mount -t ext4 /dev/sdb1 /mnt
    $
    
From the above example, we see that parted is a more powerful tool. For instance, it has a
align-check command. This is to check if partition satisfies some alignment constraint.

Mouting File System using UUID

We can mount the file system using its UUID. Below is an example,

  1. Display UUID

    
    $ sudo blkid /dev/sdb1
    /dev/sdb1: UUID="6b43a1a4-ab37-4788-acef-df13149f7bc6" TYPE="ext4" PARTLABEL="primary" PARTUUID="d9b95367-7e63-4789-9b6d-65ef8e7c61c0"
    $
    
  2. Mount the file system using its UUID

    
    $ sudo mount --uuid=6b43a1a4-ab37-4788-acef-df13149f7bc6 /mnt
    $
    

Enabling Serial Console on Debian Linux

I wanted to enable the serial console on a Linux system. The system is a Debian Linux release 10 (a.k.a., "buster"). The official documentation has the following description about the serial console

"
If you are booting with a serial console, generally the kernel will autodetect this. If you have a videocard (framebuffer) and a keyboard also attached to the computer which you wish to boot via serial console, you may have to pass the console=device argument to the kernel, where device is your serial device, which is usually something like ttyS0

You may need to specify parameters for the serial port, such as speed and parity, for instance console=ttyS0,9600n8; other typical speeds may be 57600 or 115200. Be sure to specify this option after ---, so that it is copied into the bootloader configuration for the installed system (if supported by the installer for the bootloader). 

In order to ensure the terminal type used by the installer matches your terminal emulator, the parameter TERM=type can be added. Note that the installer only supports the following terminal types: linux, bterm, ansi, vt102 and dumb. The default for serial console in debian-installer is vt102. If you are using an IPMI console, or a virtualization tool which does not provide conversion into such terminals types itself, e.g. QEMU/KVM, you can start it inside a screen session. That will indeed perform translation into the screen terminal type, which is very close to vt102.

"

It also states,

"
Those used to change inittab to enable/disable virtual or serial consoles will notice that that file is gone from clean installs. This is all managed through systemd directly now. For example, you can enable a serial console on COM1 with: 

systemctl enable serial-getty@ttyS0.service
systemctl start serial-getty@ttyS0.service
 

However, it is generally preferable to add console=ttyS0 on the kernel commandline, since this also enables kernel output on reboots. This is done by adding the following to /etc/default/grub
 
GRUB_CMDLINE_LINUX="console=ttyS0"
 
... and running update-grub. This will take effect only on the next reboot, however.

"

It is clearly that the documentation offers two solutions, using the systemd service or using the boot parameter. Perhaps, the former has the shortcoming that the serial console may not start if the systemd does not get a chance to run because we happen to have a boot problem. The preferred method, as suggested is to pass a boot parameter to the kernel. Since I have the grub boot loader installed, this becomes a simple two step approach,

  1. Edit /etc/default/grub file. In my case, replace
    
    GRUB_CMDLINE_LINUX_DEFAULT="quiet"
    

    by
    
    GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0 console=tty0"
    

  2. Update the grub.cfg file
    
    update-grub -o /boot/grub/grub.cfg
    

I happened to back up the old grub.cfg file. By comparing the old and new grub.cfg file, we can easily see the difference between the two files,

$ diff /boot/grub/grub.cfg /boot/grub/grub.cfg.bu01
diff /boot/grub/grub.cfg /boot/grub/grub.cfg.bu01
119c119
<       linux   /boot/vmlinuz-4.19.0-8-686 root=UUID=123456789 ro  console=ttyS0 console=tty0
---
>       linux   /boot/vmlinuz-4.19.0-8-686 root=UUID=123456789 ro  quiet
137c137
<               linux   /boot/vmlinuz-4.19.0-8-686 root=UUID=123456789 ro  console=ttyS0 console=tty0
---
>               linux   /boot/vmlinuz-4.19.0-8-686 root=UUID=123456789 ro  quiet
$

Sunday, May 24, 2020

"grep" does not work on WSL?

Does "grep" not always work on the Windows Subsystem for Linux (WSL) on Windows 10? I has been investigating this problem that bothered me a great deal. Here is what I saw,

$ vi josh.txt

What I saw in vi is,

  Josh
  Anonymous
~       
~                                                                        ~  

OK, let's grep something ...

$ grep "Josh" josh.txt
$ echo $?
1

Should I have seen a match and exit-code 0 instead? I haven't gotten a clue until I ran strace,

$ strace grep "Josh" josh.txt
...
openat(AT_FDCWD, "josh.txt", O_RDONLY|O_NOCTTY) = 3
fstat(3, {st_mode=S_IFREG|0777, st_size=42, ...}) = 0
read(3, "\377\376 \0 \0J\0o\0s\0h\0 \0\r\0\n\0 \0 \0A\0n\0o\0n\0"..., 98304) = 42
read(3, "", 98304)                      = 0
close(3)                                = 0
...
$

Good, I saw 'J', 'o, ..., but what are these '\377', '\376', ... Instead of doing octal numbers to hexadecimal number conversion, I let strace do this for me, and

$ strace grep "Josh" josh.txt
...
openat(AT_FDCWD, "josh.txt", O_RDONLY|O_NOCTTY) = 3
fstat(3, {st_mode=S_IFREG|0777, st_size=42, ...}) = 0
read(3, "\xff\xfe\x20\x00\x20\x00\x4a\x00\x6f\x00\x73\x00\x68\x00\x20\x00\x0d\x00\x0a\x00\x20\x00\x20\x00\x41\x00\x6e\x00\x6f\x00\x6e\x00"..., 98304) = 42
read(3, "", 98304)                      = 0
close(3)                                = 0                            = 0
...
$

Huh? No characters? What are these "\xff\xfe\x20\x00..."? How about

$ cat josh.txt
    J o s h
     A n o n y m o u s
$

At this moment, I realized that the character encoding is neither ASCII nor UTF-8, and it must be something else, and the leading bytes are the "Byte Order Marks (BOM)". Windows API documentation has a page that has the following,

Byte order markDescription
EF BB BF UTF-8
FF FE UTF-16, little endian
FE FF UTF-16, big endian
FF FE 00 00 UTF-32, little endian
00 00 FE FF UTF-32, big-endian
Note
A byte order mark is not a control character that selects the byte order of the text.

It turns out the text file is encoded in "UTF-16, little endian". Just for fun, I ran file,

$ file josh.txt
josh.txt: Little-endian UTF-16 Unicode text, with CRLF line terminators
$

That's it! I got this file from downloading it in Webex on the Windows host, and Webex must have encoded it in the Windows default encoding scheme, "UTF-16, little endian".

How do I grep this file? There might be many other methods. But I just use the iconv command  to convert the encoding from utf-16 to utf-8, and then redirect the output to grep, like,

$ iconv -f utf-16le -t utf-8 josh.txt | grep "Josh"
  Josh
$ echo $?
0
$

Problem solved!

Saturday, May 23, 2020

How to redirect the output of a command that reads a here document to another command in a shell script?

I wrote a long title here, "how to redirect the output of a command that reads a here document to another command in a shell script?"

I found this Stack Overflow post answers this question very well.
Following the answer over there, Below is an example,
{
 cut -d':' -f2 << END
A1: 3
A2: 4
A3: 1
A4: 3 
END
} | awk 'BEGIN {s=0;} {s+=$1;} END {print "sum =", s;}'

Thursday, May 21, 2020

Linux System and Network Diagnosing Tools

This is a short note about the growing list of Linux system and network diagnosing tools,
  • ngrep
  • perf
  • wireshark/tcpdump
  • strace & ltrace
  • stap
  • eBPF
  • /proc
  • netstat
  • ps
  • dstat
  • iostat

Sunday, May 17, 2020

Failed to Start Virtual Machine in VMWare Workstation Player

I upgraded VMWare Workstation Player to 15.5.2 on a Windows 10 host. It failed to start any virtual machines after the upgrade. The error message is

VMware Player and Device/Credential Guard are not compatible. VMware
Player can be run after disabling Device/Credential Guard. Please visit
http://www.vmware.com/go/turnoff_CG_Dg for more details. 
 

I followed the instruction to make sure that the Device Guide was disabled. Since I have a Windows Pro, to disable Device Guard, I run the Local Group Policy Editor

mmc gpedit.msc


However, the problem remains. It turned out an additional step was need, that was to run the following command as the system adiministrator and reboot the Windows host,

bcdedit /set hypervisorlaunchtype off 

Wednesday, May 13, 2020

Configuring Windows LAN Manager Authentication Level

Windows LAN Manager authentication level can cause interoperability issues between Windows servers and Samba clients, between Windows clients and Samba servers, and sometimes between Samba servers and clients, and Windows servers and clients.

On Windows, the authentication level is in the Windows Registry at

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\LmCompatibilityLevel




There are two methods to change the authentication level. Regardless on which edition of Windows, one can always directly edit the Windows Registry. The value of the LmCompatibilityLevel can be 0, 1, 2, 3, 4, and 5 on Windows 10. According to the official Windows 10 documentation, the meaning of the 6 levels are as follows,



SettingDescription Registry security level
Send LM & NTLM responses Client devices use LM and NTLM authentication, and they never use NTLMv2 session security. Domain controllers accept LM, NTLM, and NTLMv2 authentication. 0
Send LM & NTLM – use NTLMv2 session security if negotiated Client devices use LM and NTLM authentication, and they use NTLMv2 session security if the server supports it. Domain controllers accept LM, NTLM, and NTLMv2 authentication. 1
Send NTLM response only Client devices use NTLMv1 authentication, and they use NTLMv2 session security if the server supports it. Domain controllers accept LM, NTLM, and NTLMv2 authentication. 2
Send NTLMv2 response only Client devices use NTLMv2 authentication, and they use NTLMv2 session security if the server supports it. Domain controllers accept LM, NTLM, and NTLMv2 authentication. 3
Send NTLMv2 response only. Refuse LM Client devices use NTLMv2 authentication, and they use NTLMv2 session security if the server supports it. Domain controllers refuse to accept LM authentication, and they will accept only NTLM and NTLMv2 authentication. 4
Send NTLMv2 response only. Refuse LM & NTLM Client devices use NTLMv2 authentication, and they use NTLMv2 session security if the server supports it. Domain controllers refuse to accept LM and NTLM authentication, and they will accept only NTLMv2 authentication. 5


LmCompatibilityLevel 1 provides the highest level of compatibility, but isn't recommended is it permits dated and less secure LM and NTMLv1.

On a Windows Professional edition system, we can also use the Local Group Policy Editor. In an elevated Windows Command Prompt window, i.e., a Windows Command Prompt window ran as the system administrator, open the Local Group Policy Editor by the following command,

mmc gpedit.msc

Then, locate the option via the following navigation sequence,
  1. (In the Left Navigation Pan) Computer Configuration
  2. (In the Left Navigation Pan)Windows Settings
  3. (In the Left Navigation Pan)Security Settings
  4. (In the Left Navigation Pan)Local Policies
  5. (In the Left Navigation Pan)Security Options
  6. (In the Right Policy Option Pan) Network security: LAN Manager authentication level
Double-click on the option, and select one of the 6 options that corresponds to the 6 LmCompatibilityLevel values.



Alternatively, one may directly open the Local Security policies by the command,

mmc secpol.msc

With this, there would be a shorter navigation sequence,
  1. (In the Left Navigation Pan)Local Policies
  2. (In the Left Navigation Pan)Security Options
  3. (In the Right Policy Option Pan) Network security: LAN Manager authentication level

Friday, May 8, 2020

Running a Windows Command Line Job in Background

If we want to run a Windows command line task in background, like UNIX's "&" do, we can do the following from the Windows command line,

start /B cmd /C call program [arg1 [...]]

We explain each part as follows,
  • /B is the command line option to the start command. The documentation of the command start states the following,
    • /B. Start application without creating a new window. The application has ^C handling ignored. Unless the application enables ^C processing, ^Break is the only way to interrupt the application.
  • cmd is the program to run from the command line, i.e., C:\Windows\System32\cmd.exe
  • /C is the command line option given to cmd. The manual states,
    • /C. Carries out the command specified by string and then terminates
  • call is a command to cmd, it is to "call one batch program from another." The program does not have to be batch program, and can be any executable file.
  • program is the program to be called by the call command.
  • arg1 [...] are optional one or more command line arguments given to program.
For example, if we'd like to to run a Java program HelloWorld from the command line in background, we would do,

start /B cmd /C call java HelloWord

A Unix equivalent of this command would be java HelloWorld &.

Clearing IPMI Log Entries

I saw this error message on my FreeNAS system,

IPMI SEL low space left: 0 bytes (100% used).
This can happen to any system. In my case, it is that motherboard battery somehow has low voltage and needs replacement. However, one issue is to clear the IPMI SEL log entries to make some space. The following command allows me to clear the log entries,

sudo ipmitool sel clear

Thursday, May 7, 2020

Removing Files Whose Names Start with "-"

I wanted to remove files whose names begin with "-" on a Linux system. After some searches, I found out that the following method worked.

rm -- -foo
where the file name is -foo.