Monitoring transient network traffic session

 Sometimes there is a need to investigate network traffic that is transient. To make the problem clearer, let's examine this example. The firewall indicates some network traffic was blocked:

Block IPv4 link-local (1000000102) 192.168.99.99:35018 169.254.169.254:80 TCP:S 

We  want to figure out which process that sent out the packets. So, we would do something like

sudo netstat -anp | grep 35018

Unfortunately, this yields nothing because at the time we issue the netstat command port 35018 is not open. It turns out the network traffic is short-lived. How do we figure out which process sends out the packets? Of course, we could try to capture the packets:

tcpdump -XX -i any host 169.254.169.254 and port 80

which indeed captures the packets, and also shows the header and content of the packets captured. Sometimes, the packet header and the content are sufficiently for us to figure out what progress sent out the packets. However, what if the packet header and the content do not offer a clue?

It turns out, we can use sysdig, for instance, we can use it in this way:

sysdig -p '*%evt.num  %evt.time   %evt.cpu   %proc.name   (%thread.tid %proc.ppid)   %evt.dir %evt.type %evt.info' fd.rip=169.254.169.254 and fd.rport=80

which tells us the process that sent out the packets and the parent process PID. The process that sent out the packets may have gone, but it is offen that the parent process is still around. This solves us the problem because it offers a way to investigate further.

Binding Process to TCP/UDP Port Failure on Windows

Windows has the concept of reserved TCP/UDP ports. These ports can nonetheless be used by any other application. These can be annoying because the reserved ports would not shown be used when we query used ports using netstat. For instance, if we want to bind TCP port 23806 to an application, we determine the availability using the netstat command, such as

C:> netstat -anp tcp | find ":23806"

C:>

The output is blank, which means that the port is unused. However, when we attempt to bind the port to a process of our choice, we encounter an error, such as

bind [127.0.0.1]:23806: Permission denied

This is annoying. The reason is that the port somehow becomes a reserved port. To see this, we can query reserved ports, e.g.,

C:> netsh int ipv4 show excludedportrange protocol=tcp

Protocol tcp Port Exclusion Ranges

Start Port    End Port
----------    --------
      1155        1254
      ...          ...
     23733       23832
     23833       23932
     50000       50059     *

* - Administered port exclusions.


C:>
  

which shows that 23806 is now a served port. What is really annoying is that the range can be updated by Windows dynamically. There are several methods to deal with this.

1. Method 1. Stop and start the Windows NAT Driver service.

   
     net stop winnat
     net start winnat
     

   After this, query the reserved the ports again. It is often the reserved ports are much limited when compared to before, e.g.,

   
   C:>netsh int ipv4 show excludedportrange protocol=tcp
   
   Protocol tcp Port Exclusion Ranges
   
   Start Port    End Port
   ----------    --------
         2869        2869
         5357        5357
        50000       50059     *
   
   * - Administered port exclusions.
   
   C:>
     

2. Method 2. If you don't wish to use this feature of Windows, we can disable reserved ports.

   
   reg add HKLM\SYSTEM\CurrentControlSet\Services\hns\State /v EnableExcludedPortRange /d 0 /f
   

Ressetting Network Stack on Windows

Sometimes, I want to reset the network stack on Windows. I found that Intel has a good documentation for it. I copy the steps below:

Resetting the network stack

ipconfig /release
ipconfig /flushdns
ipconfig /renew
netsh int ip reset
netsh winsock reset

Reinstalling disappeared ddclient

I upgraded a CentOS 8 host to CentOS Stream 8. Later I discovered the ddclient package disappeared. I attempted to install it, and this error occurred. 

$ sudo dnf install ddclient
 Problem: conflicting requests
  - nothing provides perl(Digest::SHA1) needed by ddclient-3.9.1-1.el8.noarch
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
$

A research on the web tells me that I need the PowerTools package. To find out the exact name of the package, 

$ dnf repolist all | grep -i tool
powertools                               CentOS Stream 8 - PowerTools   enabled
powertools-source                        CentOS Stream 8 - PowerTools - disabled
$

Following this, I enable the repo

$ sudo dnf install ddclient
CentOS Stream 8 - PowerTools                    2.5 MB/s | 4.2 MB     00:01
......
......
Complete!
$

Which ISP's routers support hairpinning (NAT loopback)?

NAT loopback, also known as hairpinning is that two hosts on the LAN behind the NAT gateway can communicate with each other via the NAT gateway's external address. Which ISP's router out of the box supports NAT loopback? Based on experience thus far in October, 2020, my experience is as follows,

• Optimum. Yes
• Verizon Fios. Yes
• Spectrum. No
  

Configuring Windows Dynamic Port Allocation?

Some Windows applications can reserve TCP or UDP ports which may cause some other applications fail to run. These threads provide solutions to this problem,

• Reserving ports in the Windows 10 Dynamic Port Range  on Superuser.com
• Solution for common Docker not starting problem after latest update
• Cannot bind to some ports due to permission denied on StackOverflow.com 

The solution are generally three.

1. Removing/disabling the offending application or Windows feature. For instance, the threads point out Hyper-V is often the culprit because it reserves ports dynamically and unpredictably. We can disable it if we aren't using it,

   
   REM disable Hyper-V
   dism.exe /Online /Disable-Feature:Microsoft-Hyper-V
     

   In PowerShell, do this to remove all Hyper-V adapters,

   
   Get-HNSNetwork | Remove-HNSNetwork
     

2. Reset dynamic port range from which the offending application reserves ports. For instance, for Hyper-V, for instance, we can do this,

   
   dism.exe /Online /Disable-Feature:Microsoft-Hyper-V
   
   netsh int ipv4 add excludedportrange protocol=tcp startport=50051 numberofports=1
   
   dism.exe /Online /Enable-Feature:Microsoft-Hyper-V /All
       

3. Disable dynamic port allocation, i.e.,

   
   reg add HKLM\SYSTEM\CurrentControlSet\Services\hns\State /v EnableExcludedPortRange /d 0 /f
       

Verizon Fios G3100 Router Fort Forwarding

Verizon Fios G3100 Router's manual doesn't explain clearly how to set up port forwarding. This is just a note to clarify a few concepts related to it, in particular, I saw there were a few discussions about port forwarding. 

To set up port forwarding, we go to https://192.168.1.1/. Note that the certificate is self-signed, and we have to accept it when the browser complains about it. 

Port Forwarding Rules

From the UI, follow the navigation path of "Network | Advanced | Network Settings | Port Forwarding Rules", we arrive at the UI to set up "Port Forwarding Rules". This is actually for you to define a network service you may wish to expose from you home network, i.e., to define an incoming port to the router from the outside the home network. 

With this in mind, the following example is to most is a mistake because we specify the source port as 80.

 

The source port is actually the port of the application that attempts to connect to port 80 at your home web server. It is very unlikely a web client is connecting from port 80. So the correct setup is likely to be the following for the Web service.

 

What is the service used for? This service is used to create port forwarding firewall rules. To set up port forwarding for a network service hosted at your home, you can select one of these rules to populate some fields for setting up port forwarding for a network service, so these rules only provide some convenience, but isn't necessary. I would say, forget about these.

Setting up Port Forwarding

To set up port forwarding, from the UI, follow the navigation path of "Network | Firewall  | Port Forwarding". The router's manual does not say much, in particular, advanced port forwarding. So, I believe that there is a confusion among users. 

The UI looks like this,

I marked "Add" and "Advanced". These are actually "buttons" you can click. To create advanced port forwarding rules, i.e., forwarding the traffic coming in to one port on the router to a different port on a host at your home, click "Advanced. Once, you are done, don't forget to click "Add".

The actually confusion comes from the UI of the "Advanced Port Forwarding Setup" interface, which is different from many other routers where you only specify two ports, like "external" and "incoming". Here you are given 3 fields to fill up 3 ports as shown below,

 

Source Ports

These aren't the ports open on your router. These are the ports belong to the network application running on a host outside of your home in the world. Let's say, you want to connect to the Plex server at your home from your office, these are the ports of your web browser on your office computer. For most applications, we don't know these ports since they are assigned by the operating systems dynamically. In generally, they should be "Any"

Destination Ports

These are the ports on the Fios router. We often call them the "service" ports. At your office, you open your home plex server at URL like http://your_home.example.com:51400/web/index.html, then you fill it up with 51400. Don't confuse this with the port open on a home server behind the Fios router. 

Forward to Port

This is actually a port open at your home server, in this figure, at 192.168.1.55. Using the example in the above, at your office, if you open your home plex server at URL like http://your_home.example.com:51400/web/index.html, the browser at your office computer connects to port 51400 at the Fios router at your home. The router then forwards the incoming network traffic to the home server at the "Forward to Port" you specify.

Testing Your Ports

You should test your ports. But you are at home. So we have to rely on some services. I saw a few recommended services like https://www.yougetsignal.com/tools/open-ports/. Be ware that lots of these tools only check TCP ports, e.g., if your game runs on UDP ports, find a better tool to test your ports.