"grep" does not work on WSL?

Does "grep" not always work on the Windows Subsystem for Linux (WSL) on Windows 10? I has been investigating this problem that bothered me a great deal. Here is what I saw,

$ vi josh.txt

What I saw in vi is,

  Josh
  Anonymous
~       
~                                                                        ~  

OK, let's grep something ...

$ grep "Josh" josh.txt
$ echo $?
1

Should I have seen a match and exit-code 0 instead? I haven't gotten a clue until I ran strace,

$ strace grep "Josh" josh.txt
...
openat(AT_FDCWD, "josh.txt", O_RDONLY|O_NOCTTY) = 3
fstat(3, {st_mode=S_IFREG|0777, st_size=42, ...}) = 0
read(3, "\377\376 \0 \0J\0o\0s\0h\0 \0\r\0\n\0 \0 \0A\0n\0o\0n\0"..., 98304) = 42
read(3, "", 98304)                      = 0
close(3)                                = 0
...
$

Good, I saw 'J', 'o, ..., but what are these '\377', '\376', ... Instead of doing octal numbers to hexadecimal number conversion, I let strace do this for me, and

$ strace grep "Josh" josh.txt
...
openat(AT_FDCWD, "josh.txt", O_RDONLY|O_NOCTTY) = 3
fstat(3, {st_mode=S_IFREG|0777, st_size=42, ...}) = 0
read(3, "\xff\xfe\x20\x00\x20\x00\x4a\x00\x6f\x00\x73\x00\x68\x00\x20\x00\x0d\x00\x0a\x00\x20\x00\x20\x00\x41\x00\x6e\x00\x6f\x00\x6e\x00"..., 98304) = 42
read(3, "", 98304)                      = 0
close(3)                                = 0                            = 0
...
$

Huh? No characters? What are these "\xff\xfe\x20\x00..."? How about

$ cat josh.txt
    J o s h
     A n o n y m o u s
$

At this moment, I realized that the character encoding is neither ASCII nor UTF-8, and it must be something else, and the leading bytes are the "Byte Order Marks (BOM)". Windows API documentation has a page that has the following,

Byte order mark	Description
EF BB BF	UTF-8
FF FE	UTF-16, little endian
FE FF	UTF-16, big endian
FF FE 00 00	UTF-32, little endian
00 00 FE FF	UTF-32, big-endian

Note

A byte order mark is not a control character that selects the byte order of the text.

It turns out the text file is encoded in "UTF-16, little endian". Just for fun, I ran file,

$ file josh.txt
josh.txt: Little-endian UTF-16 Unicode text, with CRLF line terminators
$

That's it! I got this file from downloading it in Webex on the Windows host, and Webex must have encoded it in the Windows default encoding scheme, "UTF-16, little endian".

How do I grep this file? There might be many other methods. But I just use the iconv command  to convert the encoding from utf-16 to utf-8, and then redirect the output to grep, like,

$ iconv -f utf-16le -t utf-8 josh.txt | grep "Josh"
  Josh
$ echo $?
0
$

Problem solved!

How to redirect the output of a command that reads a here document to another command in a shell script?

I wrote a long title here, "how to redirect the output of a command that reads a here document to another command in a shell script?"

I found this Stack Overflow post answers this question very well.

• How do you use output redirection in combination with here-documents and cat?

Following the answer over there, Below is an example,

{
 cut -d':' -f2 << END
A1: 3
A2: 4
A3: 1
A4: 3 
END
} | awk 'BEGIN {s=0;} {s+=$1;} END {print "sum =", s;}'

Linux System and Network Diagnosing Tools

This is a short note about the growing list of Linux system and network diagnosing tools,

• ngrep
• perf
• wireshark/tcpdump
• strace & ltrace
• stap
• eBPF
• /proc
• netstat
• ps
• dstat
• iostat

Failed to Start Virtual Machine in VMWare Workstation Player

I upgraded VMWare Workstation Player to 15.5.2 on a Windows 10 host. It failed to start any virtual machines after the upgrade. The error message is

VMware Player and Device/Credential Guard are not compatible. VMware
Player can be run after disabling Device/Credential Guard. Please visit
http://www.vmware.com/go/turnoff_CG_Dg for more details. 

I followed the instruction to make sure that the Device Guide was disabled. Since I have a Windows Pro, to disable Device Guard, I run the Local Group Policy Editor

mmc gpedit.msc

However, the problem remains. It turned out an additional step was need, that was to run the following command as the system adiministrator and reboot the Windows host,

bcdedit /set hypervisorlaunchtype off 

Configuring Windows LAN Manager Authentication Level

Windows LAN Manager authentication level can cause interoperability issues between Windows servers and Samba clients, between Windows clients and Samba servers, and sometimes between Samba servers and clients, and Windows servers and clients.

On Windows, the authentication level is in the Windows Registry at

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\LmCompatibilityLevel

There are two methods to change the authentication level. Regardless on which edition of Windows, one can always directly edit the Windows Registry. The value of the LmCompatibilityLevel can be 0, 1, 2, 3, 4, and 5 on Windows 10. According to the official Windows 10 documentation, the meaning of the 6 levels are as follows,

Setting	Description	Registry security level
Send LM & NTLM responses	Client devices use LM and NTLM authentication, and they never use NTLMv2 session security. Domain controllers accept LM, NTLM, and NTLMv2 authentication.	0
Send LM & NTLM – use NTLMv2 session security if negotiated	Client devices use LM and NTLM authentication, and they use NTLMv2 session security if the server supports it. Domain controllers accept LM, NTLM, and NTLMv2 authentication.	1
Send NTLM response only	Client devices use NTLMv1 authentication, and they use NTLMv2 session security if the server supports it. Domain controllers accept LM, NTLM, and NTLMv2 authentication.	2
Send NTLMv2 response only	Client devices use NTLMv2 authentication, and they use NTLMv2 session security if the server supports it. Domain controllers accept LM, NTLM, and NTLMv2 authentication.	3
Send NTLMv2 response only. Refuse LM	Client devices use NTLMv2 authentication, and they use NTLMv2 session security if the server supports it. Domain controllers refuse to accept LM authentication, and they will accept only NTLM and NTLMv2 authentication.	4
Send NTLMv2 response only. Refuse LM & NTLM	Client devices use NTLMv2 authentication, and they use NTLMv2 session security if the server supports it. Domain controllers refuse to accept LM and NTLM authentication, and they will accept only NTLMv2 authentication.	5

LmCompatibilityLevel 1 provides the highest level of compatibility, but isn't recommended is it permits dated and less secure LM and NTMLv1.

On a Windows Professional edition system, we can also use the Local Group Policy Editor. In an elevated Windows Command Prompt window, i.e., a Windows Command Prompt window ran as the system administrator, open the Local Group Policy Editor by the following command,

mmc gpedit.msc

Then, locate the option via the following navigation sequence,

1. (In the Left Navigation Pan) Computer Configuration
2. (In the Left Navigation Pan)Windows Settings
3. (In the Left Navigation Pan)Security Settings
4. (In the Left Navigation Pan)Local Policies
5. (In the Left Navigation Pan)Security Options
6. (In the Right Policy Option Pan) Network security: LAN Manager authentication level

Double-click on the option, and select one of the 6 options that corresponds to the 6 LmCompatibilityLevel values.

Alternatively, one may directly open the Local Security policies by the command,

mmc secpol.msc

With this, there would be a shorter navigation sequence,

1. (In the Left Navigation Pan)Local Policies
2. (In the Left Navigation Pan)Security Options
3. (In the Right Policy Option Pan) Network security: LAN Manager authentication level

Running a Windows Command Line Job in Background

If we want to run a Windows command line task in background, like UNIX's "&" do, we can do the following from the Windows command line,

start /B cmd /C call program [arg1 [...]]

We explain each part as follows,

• /B is the command line option to the start command. The documentation of the command start states the following,
  • /B. Start application without creating a new window. The application has ^C handling ignored. Unless the application enables ^C processing, ^Break is the only way to interrupt the application.
• cmd is the program to run from the command line, i.e., C:\Windows\System32\cmd.exe
• /C is the command line option given to cmd. The manual states,
  • /C. Carries out the command specified by string and then terminates
• call is a command to cmd, it is to "call one batch program from another." The program does not have to be batch program, and can be any executable file.
• program is the program to be called by the call command.
• arg1 [...] are optional one or more command line arguments given to program.

For example, if we'd like to to run a Java program HelloWorld from the command line in background, we would do,

start /B cmd /C call java HelloWord

A Unix equivalent of this command would be java HelloWorld &.

Clearing IPMI Log Entries

I saw this error message on my FreeNAS system,

IPMI SEL low space left: 0 bytes (100% used).

This can happen to any system. In my case, it is that motherboard battery somehow has low voltage and needs replacement. However, one issue is to clear the IPMI SEL log entries to make some space. The following command allows me to clear the log entries,

sudo ipmitool sel clear