Monitoring transient network traffic session

 Sometimes there is a need to investigate network traffic that is transient. To make the problem clearer, let's examine this example. The firewall indicates some network traffic was blocked:

Block IPv4 link-local (1000000102) 192.168.99.99:35018 169.254.169.254:80 TCP:S 

We  want to figure out which process that sent out the packets. So, we would do something like

sudo netstat -anp | grep 35018

Unfortunately, this yields nothing because at the time we issue the netstat command port 35018 is not open. It turns out the network traffic is short-lived. How do we figure out which process sends out the packets? Of course, we could try to capture the packets:

tcpdump -XX -i any host 169.254.169.254 and port 80

which indeed captures the packets, and also shows the header and content of the packets captured. Sometimes, the packet header and the content are sufficiently for us to figure out what progress sent out the packets. However, what if the packet header and the content do not offer a clue?

It turns out, we can use sysdig, for instance, we can use it in this way:

sysdig -p '*%evt.num  %evt.time   %evt.cpu   %proc.name   (%thread.tid %proc.ppid)   %evt.dir %evt.type %evt.info' fd.rip=169.254.169.254 and fd.rport=80

which tells us the process that sent out the packets and the parent process PID. The process that sent out the packets may have gone, but it is offen that the parent process is still around. This solves us the problem because it offers a way to investigate further.