I could not ping my Linux Server, and could not issue a traceroute command. It turns out the firewall was blocking the traceroute flow.
This blog posts describes how I checked this, and fixed the firewall problem.
Traceroute sends (by default) a UDP packet to a port address in the range 33434-33523. It usually responds with a “timed out” type response. If there is no response then there is a good chance that the packet is being dropped by a firewall.
See Understanding traceroute (or tracerte).
Using wireshark I could see UDP packets going in to my Linux, but there was no corresponding reply being returned.
When traceroute worked I got the out inbound UDP packet, and the outbound response with “destination unreachable” (which looks like a problem but actually shows normal behaviour) as shown in the data below. Wireshark highlights it with a black background, because it thinks it is a problem.
Source | Destination | Dst Port | port | Protocol | Info |
2001:db8::2 | 2001:db8::7 | 33434 | 52119 | UDP | 52119 â 33434 Len=24 |
2001:db8::7 | 2001:db8::7 | 33434 | 52119 | ICMPV6 | Destination Unreachable (Port unreachable) |
When traceroute failed I only got the inbound UDP packet
Source | Destination | Dst Port | port | Protocol | Info |
2001:db8::2 | 2001:db8::7 | 33434 | 52119 | UDP | 52119 â 33434 Len=24 |
If the packets is blocked by a firewall, then the traceroute output will have “*” as the node name.
Useful Fire Wall (ufw) is documented here.
I was on Ubuntu Linux 20.04.
Display the status of the firewall
sudo ufw status verbose
This gave me
Status: active Logging: off (low) Default: deny (incoming), allow (outgoing), deny (routed) New profiles: skip To Action From -- ------ ---- Anywhere ALLOW IN 10.1.1.2 22/tcp ALLOW IN Anywhere 20,21,10000:10100/tcp ALLOW IN Anywhere 21/tcp ALLOW IN Anywhere 20/tcp ALLOW IN Anywhere 22/tcp (v6) ALLOW IN Anywhere (v6) 20,21,10000:10100/tcp (v6) ALLOW IN Anywhere (v6) 21/tcp (v6) ALLOW IN Anywhere (v6) 20/tcp (v6) ALLOW IN Anywhere (v6)
By default,
- incoming data is blocked
- outbound data is allowed
- routed data is blocked.
Logging is off, and problems are not reported.
The displays shows there are no rules for UDP – so any incoming UDP request is blocked (quietly dropped = dropped without telling anyone).
You may want to issue the command and pipe the output to a file, ufw.txt, to keep a record of the status before you make any changes. If you make any changes, they persist – even across reboot.
Enable logging to see what is being blocked
sudo ufw logging on
Rerun your traceroute or command.
At the bottom of /var/log/ufw I had (this has been reformatted to make it display better)
Nov 28 12:27:43 colinpaice kernel: [ 3317.641508] [UFW BLOCK] IN=enp0s31f6 OUT= MAC=8c:16:45:36:f4:8a:00:d8:61:e9:31:2a:86:dd SRC=2001:0db8:0000:0000:0000:0000:0000:0002 DST=2001:0db8:0000:0000:0000:0000:0000:0007
LEN=80
TC=0
HOPLIMIT=1
FLOWLBL=924186
PROTO=UDP
SPT=48582
DPT=33434
LEN=40
Wireshark gave me
Frame 4: 94 bytes on wire (752 bits), 94 bytes captured (752 bits) on interface enp0s31f6, id 0 Ethernet II, Src: Micro-St_e9:31:2a (00:d8:61:e9:31:2a), Dst: LCFCHeFe_36:f4:8a (8c:16:45:36:f4:8a) Internet Protocol Version 6, Src: 2001:db8::2, Dst: 2001:db8::7 0110 .... = Version: 6 .... 0000 0000 .... .... .... .... .... = Traffic Class: 0x00 .... .... .... 1110 0001 1010 0001 1010 = Flow Label: 0xe1a1a Payload Length: 40 Next Header: UDP (17) Hop Limit: 1 Source: 2001:db8::2 Destination: 2001:db8::7 User Datagram Protocol, Src Port: 48582, Dst Port: 33434 Source Port: 48582 Destination Port: 33434 Length: 40 Checksum: 0x6ebd [unverified] [Checksum Status: Unverified] [Stream index: 0] [Timestamps] Data (32 bytes)
From this, we can see the fields match up
- flow label (0xe1a1a = 924186)
- source 2001:db8::2 = 2001:0db8:0000:0000:0000:0000:0000:0002
- destination 2001:db8::7 = 2001:0db8:0000:0000:0000:0000:0000:0007
- source port 48582
- destination port 33434.
Port 33434 is used by traceroute, so this is a good clue this is a traceroute packet.
The reason the record was written to the log is [UFW BLOCK]. The firewall blocked it.
The request came in over interface enp0s31f6.
How to enable it.
You can specify different filters, and granularity of parameters.
For example
- sudo ufw rule allow log proto udp from 2001:db8::2
- sudo ufw rule allow in on enp0s31f6 log comment ‘colin-ethernet’
- sudo ufw rule allow proto udp to 2001:db8::7 port 33434:33523 from 2001:db8::2
Where enp0s31f6 is the name of the ethernet link where the traffic comes from.
When running with either of these, I had in the log file
Nov 28 17:03:12 colinpaice kernel: [19847.112045] [UFW ALLOW] IN=enp0s31f6 OUT= MAC=8c:16:45:36:f4:8a:00:d8:61:e9:31:2a:86:dd SRC=2001:0db8:0001:0000:0000:0000:0000:0009 DST=2001:0db8:0000:0000:0000:0000:0000:0007 LEN=60 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=UDP SPT=33434 DPT=33440 LEN=20
and the traceroute worked.
Note: The comment ‘…’ is an administration aid to give a description. It does not come out in the logs.
Display the rules
sudo ufw status numbered
gave
Status: active To Action From -- ------ ---- [ 1] Anywhere ALLOW IN 10.1.1.2 [ 2] 22/tcp ALLOW IN Anywhere [ 3] 20,21,10000:10100/tcp ALLOW IN Anywhere [ 4] 21/tcp ALLOW IN Anywhere [ 5] 20/tcp ALLOW IN Anywhere [ 6] Anywhere on enp0s31f6 ALLOW IN Anywhere (log) [ 7] 22/tcp (v6) ALLOW IN Anywhere (v6) [ 8] 20,21,10000:10100/tcp (v6) ALLOW IN Anywhere (v6) [ 9] 21/tcp (v6) ALLOW IN Anywhere (v6) [10] 20/tcp (v6) ALLOW IN Anywhere (v6) [11] Anywhere (v6) on enp0s31f6 ALLOW IN Anywhere (v6) (log) [12] 2001:db8::7 33434/udp ALLOW IN 2001:db8::2 (log)
There is now a rule [12] for udp to 2001:db8::7 port 33434
You can use commands like
sudo ufw delete 6
to delete a row.
Note: Always display before delete. Having deleted the rule 6 – rule 7 now becomes rule 6, etc.
Now that it works…
Any changes to ufw are remembered across reboots.
You may want to turn off the logging, until the next problem
sudo ufw logging off
and remove the log from the fire wall rules, by deleting and re-adding the rule.
sudo ufw rule delete allow log proto udp from 2001:db8::2
sudo ufw rule allow proto udp from 2001:db8::2