Destination unreachable, Port unreachable. Which firewall rule is blocking me?

I was trying to connect an application on z/OS through a server to my laptop – so three systems involved.

On the connection from the server to my laptop, using Wireshark I could see no traffic from the application.

When I used Wireshark on the z/OS to server connection I got

   Source   Destination port Protocol info 
>1 10.1.1.2 10.1.0.2    2175 TCP ..
<2 10.1.1.1 10.1.1.2    2175 ICMP     Destination unreachable (Port unreachable)

This means

1. There was a TCP/IP Packet from 10.1.1.2 (z/OS) to 10.1.0.2 (mylaptop) port 2175
2. Response:Destination unreachable (Port unreachable)

This was a surprise because I could ping from z/OS through the server to the laptop.

Looking in the firewall log ( /var/log/ufw) I found

[UFW BLOCK] IN=tap0 OUT=eno1 MAC=... SRC=10.1.1.2 DST=10.1.0.2 ... PROTO=TCP SPT=1050 DPT=2175 ...

This says

• Packet was blocked. When using the ufw firewall – all of its messages and definitions contain ufw.
• From 10.1.1.2
• To 10.1.0.2
• Source port 1050
• Destination port 2175

With the command

sudo ufw route allow in on tap0 out on eno1

This allows traffic to be routed through this node from interface tap0 to interface eno1, and solved my problem.

What caused the problem?

iptables allows the systems administrator to define rules (or chains of rules – think subroutines) to control the flow of packets through the Linux kernel. For example

• control input input packets destined for this system
• control output packets from this system
• control forwarded packets flowing through this system.

ufw is an interface to iptables which makes it easier to define rules.

You can use

sudo ufw status

to display the ufw definitions, for example

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere                  
Anywhere on eno1           ALLOW       Anywhere                  
Anywhere on tap0           ALLOW       Anywhere  (log) # ‘colin-ethernet’

You can use

sudo iptables -L -v

to display the iptables. The -v options show you how many times the rules have been used.

sudo iptables-save reports on all of the rules. For example (a very small subset of my rules)

-A FORWARD -j ufw-before-forward
-A ufw-before-forward -j ufw-user-forward
-A ufw-user-forward -i tap0 -o eno1 -j ACCEPT
-A ufw-user-forward -i eno1 -o tap0 -j ACCEPT

-A ufw-skip-to-policy-forward -j REJECT --reject-with icmp-port-unreachable

Where

• -A FORWARD.… says when doing forwarding use the rule (subroutine) called ufw-before-forward. You can have many of these statements
• -A ufw-before-forward -j ufw-user-forward add to the end of subroutine ufw-before-forward, call (-jump to) subroutine ufw-user-forward
• -A ufw-user-forward -i tap0 -o eno1 -j ACCEPT in subroutine ufw-user-forward, if the input interface is tap0, and the output interface is eno1, then ACCEPT the traffic, and pass it on to interface eno1.
• -A ufw-user-forward -i eno1 -o tap0 -j ACCEPT in subroutine ufw-user-forward, if the input interface is eno1, and the output interface is tap0, then ACCEPT the traffic, and pass it on to interface eno1.
• -A ufw-skip-to-policy-forward -j REJECT –reject-with icmp-port-unreachable. In this subroutine do not allow the packet to pass through, but send back a response icmp-port-unreachable. This is the response I saw in Wireshark.

With -j REJECT you can specify

icmp-net-unreachable
icmp-host-unreachable
icmp-port-unreachable
icmp-proto-unreachable
icmp-net-prohibited
icmp-host-prohibited
icmp-admin-prohibiteda

The processing starts at the top of the tree and goes into each relevant “subroutine” in sequence till it finds and ACCEPT or REJECT.

If you use sudo iptables -L -v it lists all the rules and the use count. For example

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target              prot opt in     out     source               destination         
 ...            
  259 16364 ufw-before-forward  all  --  any    any     anywhere             anywhere            
 
Chain ufw-before-forward (1 references)
 pkts bytes target            prot opt in     out     source               destination         
 ...
   77  4620 ufw-user-forward  all  --  any    any     anywhere             anywhere  

Chain ufw-user-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  eno1   tap2    anywhere             anywhere            
    0     0 ACCEPT     all  --  tap2   eno1    anywhere             anywhere            
    9   540 ACCEPT     all  --  tap0   eno1    anywhere             anywhere            
    0     0 ACCEPT     all  --  eno1   tap0    anywhere             anywhere 

Chain ufw-reject-forward (1 references)
 pkts bytes target  ...         
   45  2700 REJECT  ... reject-with icmp-port-unreachable

• For the packet forwarding it processed a number of “rules”
  • 259 packets were processed by subroutine ufw-before-forward
• Within ufw-before-forward, there were several calls to subroutines
  • 77 packets were processed by subroutine ufw-user-forward
• Within ufw-user-forward the line (in bold) said there were 9 packets, which were forwarded when the input interface was tap0 and the output was eno1.
• Within the subroutine ufw-reject-forward, 45 packets were rejected with icmp-port-unreachable.

The ufw-reject-forward was the only instance of icmp-port-unreachable with packet count > 0. This was the rule which blocked me.

Log file

In the /var/log/ufw.log was an entry for [UFW BLOCK] for the address and port,