Understanding radvd with IPV6 on Linux.

My two day project to deploy IP V6 dynamic routing, turned into an eight week project before I got it to work.

I am documenting a lot of what I learned – today’s experience is understanding what radvd is and what the configuration options mean. I found a lot of documentation – most of which either assumed you know a lot, or only provided an incomplete description.

High level view of what radvd provides

There are several ways of providing configuration to TCPIP. One way is using radvd (on Linux).

In a configuration file you give

  • An interface name
    • Which IP addresses (and address ranges) are available at the remote end of the connection.
    • Which IP addresses (and address ranges) are available at the local end of the connection

From the information about the local end, the remote end can build its routing tables.

Background IP routing

With IP you can have

  • static routing, where you explicitly give the routes to a destination – such as to get to a.b.c.d – go via xyz interface. You have to do a lot of administration defining the addresses of each end of an interface (think Ethernet cable)
  • dynamic routing, and neighbourhood discovery, where the system automatically finds the neighbours and there is less work for an administrator to do.

With IPv6 you have

  • global addresses with IP addresses like 2001:db8:1:16…..
  • link-local addresses. These are specific to an interface. An Ethernet connection can have many terminals hanging off ‘the bus’. The link-local address is only used within the connection. A different Ethernet cable can use the same address. There is no problem as the addresses are only used with the cable.
  • Neighbour Discovery. Rather than specify every thing as you do with static routing, IP V6 supports Neighbour Discovery, where each node can tell connected nodes, what routes and IP addresses it knows about. This is documented in the specification. This supports
    • Router Advertisement (RA), (“Hello, I’m a router, I know about the following addresses and routes”),
    • Router Solicitation (RS), (“Hello, I’ve just started up, are there any routers out there?”),
    • Neighbour Solicitation(NS) (“Does anyone have this IP address?”), and
    • Neighbour Advertisement (Usually in response to a Neighbour Solicitation, “I have this address”).

What is radvd?

The radvd program is a Router Advertisment Daemon (RADvd) which provides fakes router information – but is not a router.

You specify a configuration file. The syntax is defined here.

You can specify that this interface provides a “default” route. See here.

Example definition

I have a Linux Server, and a laptop running Linux connected by an Ethernet cable.

For the server, the radvd.conf file has

# define the ethernet connection
interface eno1
{
AdvSendAdvert on;
MaxRtrAdvInterval 60;
MinDelayBetweenRAs 60;

prefix 2001:ccc::/64
{
# AdvOnLink on;
# AdvAutonomous on;
AdvRouterAddr on;
};

# route 2001:ff99::/64
# {
# # AdvRoutePreference medium;
# # 3600 = 1 hour
# AdvRouteLifetime 3600;
#
# };
};

The key information is

Comments

Data following #

The name of the interface

interface eno1{….};

prefix statement

prefix 2001:ccc::/64{…}

2001:ccc::/64 is the ipv6 address range or, to say it another way, ipv6 addresses with the left 64 bits beginning with 2001:0ccc:0000:0000. In IP V6 this is known as the prefix.

Basically this prefix statement means “this interface is a route to the specified prefix”.

This creates some addresses on the server for the connection.

eno1    inet6 2001:ccc::e02a:943b:3642:1d73/64 scope global temporary dynamic...       
eno1    inet6 2001:ccc::dbf:5c90:61a6:20ae/64 scope global dynamic mngtmpaddr...
eno1    inet6 2001:ccc::c48b:e8f1:495c:5b52/64 scope global temporary dynamic ...       
eno1    inet6 2001:ccc::2d8:61ff:fee9:312a/64 scope global dynamic mngtmpaddr ...

create routes

The prefix statement creates a route on the server to get to the laptop

2001:ccc::/64 dev eno1 proto ra metric 100 pref medium
2001:ccc::/64 dev eno1 proto kernel metric 256 ...

This says that on the server, if there is a request for an address in the range 2001:ccc::/64 send it via device eno1.

route 2001:ff99::/64 statement.

If present this passes information to the remote end of the connection, see below. It says “I (the server) know how to route to 2001:ff99::/64”.

It the routing address does not show up in any ip -6 commands on the server.

I could see the Route Information data flowing across the network within a Router Advertisment packet. But I could not see it being used at the remote end.

I think the best thing to do is ignore the route statement.

Polling

The radvd code periodically sends information along the connection to the other end, at an interval you specify. See radvdump below on how to display it.

At the other end of the connection…

At the remote (laptop) end of the connection, using WiresShark to display the data received, shows a Router Advertisement with

Internet Protocol Version 6, Src: fe80::a2f0:9936:ddfd:95fa, Dst: ff02::1
Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    ...
    ICMPv6 Option (Prefix information : 2001:ccc::/64)
    ICMPv6 Option (Route Information : Medium 2001:ff99::/64)
    ICMPv6 Option (Source link-layer address : 00:d8:61:e9:31:2a)

Where

  • fe80::a2f0:9936:ddfd:95fa is the link-local address on the server machine
  • ff02::1 the multicast address “All nodes” on a link (link-local scope)”
  • Prefix information : 2001:ccc::7/64 the prefix of the IP address 2001:ccc:0:0…
  • Route Information : Medium 2001:ff99::/64 This is from the “route” in the radvd configuration file on the Linux Server. It tells the laptop that this connection knows about routing to 2001:ccc::/64 .

From the route information it dynamically creates a route on the laptop to the server.

2001:ff99::/64 via fe80::a2f0:9936:ddfd:95fa dev enp0s31f6 proto ra metric 100 ...

This creates a route to 2001:ff99::/64 via the IP address fe80::….95fa, from the laptop end of the Ethernet connection with name enp0s31f6 to where-ever the connection goes to (in this example it goes to my server).

If I ping 2001:ffcc::9 from the laptop, it will use this route on its way to the z/OS server.

Connection to z/OS

Within the radvd.conf file is the definition to get to z/OS. This interface looks like an Ethernet (the ip -6 link command gives link/ether). This is for a different radvd configuration file to the earlier example.

interface  tap1
{
   AdvSendAdvert on; 
   MaxRtrAdvInterval 60;
   MinDelayBetweenRAs  60; 
   AdvManagedFlag  on;
   AdvOtherConfigFlag on;
   
   prefix 2001:db8:1::/64
   {
     AdvOnLink on;
     AdvAutonomous on;
     AdvRouterAddr on;
    
   };
   prefix 2001:db8:1::99/128
   {
     AdvOnLink on;
     AdvAutonomous on;
     AdvRouterAddr on;
    
   };

   route 2001:db8::/64
   {
     AdvRoutePreference medium;
     AdvRouteLifetime 3100;
   
   };
};

This says there are IP addresses in the range 2001:db8:1::/64 via this connection. z/OS reads the Router Advertisement and creates a route for them. TSO Netstat route gives

DestIP:   2001:db8:1::/64 
  Gw:     :: 
  Intf:   IFPORTCP6         Refcnt:  0000000000 
  Flgs:   UD                MTU:     9000 

The explicit IP address, with the 128 to specify use the whole value, rather than just the prefix,

 prefix 2001:db8:1::99/128
   {
     AdvOnLink on;
     AdvAutonomous on;
     AdvRouterAddr on;    
   };

creates a route in z/OS; from TSO NETSTAT ROUTE

DestIP:   2001:db8:1::99/128 
  Gw:     :: 
  Intf:   IFPORTCP6         Refcnt:  0000000000 
  Flgs:   UHD               MTU:     9000 

In UHD, the U says the interface is Up, the H says this is for a host (a specific end point), and the D says this is dynamically created.

If you try to ping 2001:db8:1::99 from the server, a Neighbour Solicitation request is sent from the server to z/OS, asking “do you have 2001:db8:1:99?”. My z/OS did not have that defined and so did not respond.

When I defined this address on z/OS TCPIP using the obeyfile

INTERFACE IFPORTCP6  DELETE 
INTERFACE IFPORTCP6 
    DEFINE IPAQENET6 
    CHPIDTYPE OSD 
    PORTNAME PORTCP 
    INTFID 7:7:7:7 
                                             
INTERFACE IFPORTCP6 
    ADDADDR 2001:DB8:1::9 
INTERFACE IFPORTCP6 
    ADDADDR 2001:DB8:9::9 
INTERFACE IFPORTCP6 
    ADDADDR 2001:DB8:1::99 

(And used

  • v tcpip,,sto,ifportcp6
  • v tcpip,,obeyfile,USER.Z24C.TCPPARMS(IFACE61)
  • v tcpip,,sta,ifportcp6

to activate it)

After this, the ping was successful because there was a neighbour solicitation for 2001:db8:1::99, and z/OS replied with Neighbour Advertisement of 2001:db8:1::9 saying I have it.

Create a default route

If you specify AdvDefaultLifetime 0 on the interface, this indicates that the router is not a default router and should not appear on the default router list in the Router Advertiser broadcasts. If the value is non zero, the recipient, can use this connection as a default route, for example there is no existing default route. A statically defined default will be used in preference to a dynamically defined one.

Use radvdump to display what it sent in the RA

You can use sudo radvdump to display what is being sent in the Router Advertiser message broadcast over multicast. It looks just like the radvd.conf file, but with all of the values filled in, so you can see the defaults.

How do I see what’s changing in my Linux network configuration?

I was trying to find out what changes were being made to my IP V6 network.

The short answer is the command

sudo nmcli general logging level DEBUG domains ALL

and reset it to the default with

sudo nmcli general logging level INFO domains ALL

You see the last 5 minutes worth of trace using

journalctl -u NetworkManager -S -5m >nw.txt

See man NetworkManager.conf.

Debian Man gives slightly different information including

ALL : all log domains
DEFAULT : default log domains
DHCP : shortcut for “DHCP4,DHCP6”
IP : shortcut for “IP4,IP6”

ADSL : ADSL device operations
AGENTS : Secret agents operations and communication
AUDIT : Audit records
AUTOIP4 : AutoIP operations
BOND : Bonding operations
BRIDGE : Bridging operations
BT : Bluetooth operations
CONCHECK : Connectivity check
CORE : Core daemon and policy operations
DBUS_PROPS : D-Bus property changes
DCB : Data Center Bridging (DCB) operations
DEVICE : Activation and general interface operations
DHCP4 : DHCP for IPv4
DHCP6 : DHCP for IPv6
DISPATCH : Dispatcher scripts
DNS : Domain Name System related operations
ETHER : Ethernet device operations
FIREWALL : FirewallD related operations
INFINIBAND : InfiniBand device operations
IP4 : IPv4-related operations
IP6 : IPv6-related operations
MB : Mobile broadband operations
NONE : when given by itself logging is disabled
OLPC : OLPC Mesh device operations
PLATFORM : OS (platform) operations
PPP : Point-to-point protocol operations
PROXY : logging messages for proxy handling
RFKILL : RFKill subsystem operations
SETTINGS : Settings/config service operations
SHARING : Connection sharing. With TRACE level log queries for dnsmasq instance
SUPPLICANT : WPA supplicant related operations
SUSPEND : Suspend/resume
SYSTEMD : Messages from internal libsystemd
TEAM : Teaming operations
VLAN : VLAN operations
VPN : Virtual Private Network connections and operations
VPN_PLUGIN : logging messages from VPN plugins
WIFI : Wi-Fi device operations
WIFI_SCAN : Wi-Fi scanning operations
WIMAX : WiMAX device operations

it produces data like

<debug> [747.1872] ndisc-lndp[...,"eno1"]: received router advertisement at 17121
<debug> ndisc[...,"eno1"]: scheduling next now/lifetime check: 163 seconds
<debug> ndisc[...,"eno1"]: neighbor discovery configuration changed [GAR]:
<debug> ndisc[...,"eno1"]:   dhcp-level none
<debug> ndisc[...,"eno1"]:   hop limit      : 64
<debug> ndisc[...,"eno1"]:   gateway fe80::a2f0:9936:ddfd:95fa pref medium exp 179.8153
<debug> ndisc[...,"eno1"]:   gateway fe80::9b07:33a1:aa30:e272 pref medium exp 162.8153
<debug> ndisc[...,"eno1"]:   address 2001:bbb::573e:5c69:2ab3:4ae6 exp 81751.8153
<debug> ndisc[...,"eno1"]:   address 2001:ccc::dbf:5c90:61a6:20ae exp 86399.8153
<debug> ndisc[...,"eno1"]:   route 2001:db8::99/128 via :: pref medium exp 86399.8153
<debug> ndisc[...,"eno1"]:   route 2001:ff99::/64 via fe80::a2f0:9936:ddfd:95fa pref medium exp 3099.8153
<debug> ndisc[...,"eno1"]:   route 2001:ccc::/64 via :: pref medium exp 86399.8153
<debug> ndisc[...,"eno1"]:   route 2001:bbb::/64 via :: pref medium exp 81751.8153
<debug>  platform: (eno1) route: append     IPv6 route: 2001:db8::/80 via :: dev 2 metric 100 mss 0 rt-src ndisc
<debug>  platform: (eno1) signal: route   6   added: 2001:db8::/80 via :: dev 2 metric 100 mss 0 rt-src rt-ra

Is “via” needed when creating a Linux IP route?

To get static routing working I needed a route like one of

# specific destination
sudo ip -6 route add fc:1::9/128 via fc::2 dev enp0s31f6r
sudo ip -6 route add fc:1::9/128  via fc::2 
#range of addresses
sudo ip -6 route add fc:1::/64 via fc::2  dev enp0s31f6
sudo ip -6 route add fc:1::/64 via fc::2 

If I a route without the via

sudo ip -6 route add fc:1::9/128 dev enp0s31f6

then it ignored my static routing and did Neighbor Solicitation; it asked adjacent systems if they had knew about the IP address fc:1::9. This is an IP V6 Neighbour Discovery facility.

There were hints around the internet that if the next hop address is not specified, then the “next hop router” will try to locate the passed address.

So the short answer to the question is: “yes. You should specify it when using static routing”.

Why Linux is not responding – it’s the flaming file-wall!

I could not ping my Linux Server, and could not issue a traceroute command. It turns out the firewall was blocking the traceroute flow.

This blog posts describes how I checked this, and fixed the firewall problem.

Traceroute sends (by default) a UDP packet to a port address in the range 33434-33523. It usually responds with a “timed out” type response. If there is no response then there is a good chance that the packet is being dropped by a firewall.

See Understanding traceroute (or tracerte).

Using wireshark I could see UDP packets going in to my Linux, but there was no corresponding reply being returned.

When traceroute worked I got the out inbound UDP packet, and the outbound response with “destination unreachable” (which looks like a problem but actually shows normal behaviour) as shown in the data below. Wireshark highlights it with a black background, because it thinks it is a problem.

SourceDestinationDst PortportProtocolInfo
2001:db8::22001:db8::73343452119UDP52119 → 33434 Len=24
2001:db8::7 2001:db8::7 33434 52119 ICMPV6 Destination Unreachable (Port unreachable)

When traceroute failed I only got the inbound UDP packet

SourceDestinationDst PortportProtocolInfo
2001:db8::22001:db8::73343452119UDP52119 → 33434 Len=24

If the packets is blocked by a firewall, then the traceroute output will have “*” as the node name.

Useful Fire Wall (ufw) is documented here.

I was on Ubuntu Linux 20.04.

Display the status of the firewall

sudo ufw status verbose

This gave me

Status: active
Logging: off (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
Anywhere                   ALLOW IN    10.1.1.2                  
22/tcp                     ALLOW IN    Anywhere                  
20,21,10000:10100/tcp      ALLOW IN    Anywhere                  
21/tcp                     ALLOW IN    Anywhere                  
20/tcp                     ALLOW IN    Anywhere                  
22/tcp (v6)                ALLOW IN    Anywhere (v6)             
20,21,10000:10100/tcp (v6) ALLOW IN    Anywhere (v6)             
21/tcp (v6)                ALLOW IN    Anywhere (v6)             
20/tcp (v6)                ALLOW IN    Anywhere (v6)         

By default,

  • incoming data is blocked
  • outbound data is allowed
  • routed data is blocked.

Logging is off, and problems are not reported.

The displays shows there are no rules for UDP – so any incoming UDP request is blocked (quietly dropped = dropped without telling anyone).

You may want to issue the command and pipe the output to a file, ufw.txt, to keep a record of the status before you make any changes. If you make any changes, they persist – even across reboot.

Enable logging to see what is being blocked

sudo ufw logging on

Rerun your traceroute or command.

At the bottom of /var/log/ufw I had (this has been reformatted to make it display better)

Nov 28 12:27:43 colinpaice kernel: [ 3317.641508] [UFW BLOCK] IN=enp0s31f6 OUT= MAC=8c:16:45:36:f4:8a:00:d8:61:e9:31:2a:86:dd SRC=2001:0db8:0000:0000:0000:0000:0000:0002 DST=2001:0db8:0000:0000:0000:0000:0000:0007
LEN=80
TC=0
HOPLIMIT=1
FLOWLBL=924186
PROTO=UDP
SPT=48582
DPT=33434
LEN=40

Wireshark gave me

Frame 4: 94 bytes on wire (752 bits), 94 bytes captured (752 bits) on interface enp0s31f6, id 0   
Ethernet II, Src: Micro-St_e9:31:2a (00:d8:61:e9:31:2a), Dst: LCFCHeFe_36:f4:8a (8c:16:45:36:f4:8a)
Internet Protocol Version 6, Src: 2001:db8::2, Dst: 2001:db8::7
    0110 .... = Version: 6
    .... 0000 0000 .... .... .... .... .... = Traffic Class: 0x00 
    .... .... .... 1110 0001 1010 0001 1010 = Flow Label: 0xe1a1a
    Payload Length: 40
    Next Header: UDP (17)
    Hop Limit: 1
    Source: 2001:db8::2
    Destination: 2001:db8::7
User Datagram Protocol, Src Port: 48582, Dst Port: 33434
    Source Port: 48582
    Destination Port: 33434
    Length: 40
    Checksum: 0x6ebd [unverified]
    [Checksum Status: Unverified]
    [Stream index: 0]
    [Timestamps]
Data (32 bytes)

From this, we can see the fields match up

  • flow label (0xe1a1a = 924186)
  • source 2001:db8::2 = 2001:0db8:0000:0000:0000:0000:0000:0002
  • destination 2001:db8::7 = 2001:0db8:0000:0000:0000:0000:0000:0007
  • source port 48582
  • destination port 33434.

Port 33434 is used by traceroute, so this is a good clue this is a traceroute packet.

The reason the record was written to the log is [UFW BLOCK]. The firewall blocked it.

The request came in over interface enp0s31f6.

How to enable it.

You can specify different filters, and granularity of parameters.

For example

  • sudo ufw rule allow log proto udp from 2001:db8::2
  • sudo ufw rule allow in on enp0s31f6 log comment ‘colin-ethernet’
  • sudo ufw rule allow proto udp to 2001:db8::7 port 33434:33523 from 2001:db8::2

Where enp0s31f6 is the name of the ethernet link where the traffic comes from.

When running with either of these, I had in the log file

Nov 28 17:03:12 colinpaice kernel: [19847.112045] 
[UFW ALLOW] 
IN=enp0s31f6 OUT= MAC=8c:16:45:36:f4:8a:00:d8:61:e9:31:2a:86:dd SRC=2001:0db8:0001:0000:0000:0000:0000:0009 
DST=2001:0db8:0000:0000:0000:0000:0000:0007 
LEN=60 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=UDP SPT=33434 DPT=33440 LEN=20

and the traceroute worked.

Note: The comment ‘…’ is an administration aid to give a description. It does not come out in the logs.

Display the rules

sudo ufw status numbered

gave

Status: active

     To                         Action      From
     --                         ------      ----
[ 1] Anywhere                   ALLOW IN    10.1.1.2                  
[ 2] 22/tcp                     ALLOW IN    Anywhere                  
[ 3] 20,21,10000:10100/tcp      ALLOW IN    Anywhere                  
[ 4] 21/tcp                     ALLOW IN    Anywhere                  
[ 5] 20/tcp                     ALLOW IN    Anywhere                  
[ 6] Anywhere on enp0s31f6      ALLOW IN    Anywhere                   (log)
[ 7] 22/tcp (v6)                ALLOW IN    Anywhere (v6)             
[ 8] 20,21,10000:10100/tcp (v6) ALLOW IN    Anywhere (v6)             
[ 9] 21/tcp (v6)                ALLOW IN    Anywhere (v6)             
[10] 20/tcp (v6)                ALLOW IN    Anywhere (v6)             
[11] Anywhere (v6) on enp0s31f6 ALLOW IN    Anywhere (v6)              (log)
[12] 2001:db8::7 33434/udp      ALLOW IN    2001:db8::2                (log)

There is now a rule [12] for udp to 2001:db8::7 port 33434

You can use commands like

sudo ufw delete 6

to delete a row.

Note: Always display before delete. Having deleted the rule 6 – rule 7 now becomes rule 6, etc.

Now that it works…

Any changes to ufw are remembered across reboots.

You may want to turn off the logging, until the next problem

sudo ufw logging off

and remove the log from the fire wall rules, by deleting and re-adding the rule.

sudo ufw rule delete allow log proto udp from 2001:db8::2

sudo ufw rule allow proto udp from 2001:db8::2

Why is Ubuntu is running out of space? It is /var/log/journal/…

Low Disk space on “Filesystem root”

I’ve been getting this message more frequently – and I’ve found out why.

It could be

  • the system journal file
  • snap files in the cache
  • stuff in /tmp

You may get this message during installation of a large set of packages. Packages get unpacked into a temporary file – which is deleted afterwards, so you get a temporary hump in usage.

System journal file

There is a “systemd journal file” with content like

Jul 12 15:50:09 colinpaice rtkit-daemon[1385]: Successfully made thread 2682 of process 2540 owned by ‘1000’ RT at priority 10.
Jul 12 16:44:41 colinpaice rtkit-daemon[1385]: Supervising 5 threads of 3 processes of 1 users.
Jul 12 16:45:01 colinpaice CRON[7075]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 12 16:45:01 colinpaice CRON[7076]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 12 16:45:01 colinpaice CRON[7075]: pam_unix(cron:session): session closed for user root
Jul 12 15:58:32 colinpaice kernel: irq_thread+0xda/0x170

This goes back to when I first installed Ubuntu about 4 years ago, but I think a month’s worth of data would be enough.

You can display the disk space used by using

sudo journalctl –disk-usage

and display the contents of the file using

sudo journalctl -n 50 |less

Note: Without sudo you get the userid’s log size… with sudo you get total log size.

The log file is in /var/log/journal/ and was 1.4 GB in size. The size of this file is controlled by the /etc/systemd/journald.conf configuration file. I edited this file (using sudo gedit /etc/systemd.journald.conf).

  • I uncommented SystemMaxFileSize and gave it a value of 500M.
  • I uncommented SystemMaxFiles and gave it a value of 10

You can either reboot, or use

service systemd-journald restart

to restart the systemd journal.

Although I set the value to 500M, after the journal was restarted – it had size 100MB!

I think 100MB is plenty big enough, and I get a log of disk space back.

Snap files in the cache

sudo du -hs /var/lib/snapd/cache/

gives you the space used.

I then used

sudo rm -r /var/lib/snapd/cache/

Other rubbish

The disk usage analyser gives you a picture of all the space on a file system. Click on “Show Applications” and select Disk Usage Analyser

Why is this Linux slower to download than that one

I have a laptop which is my primary work station, and an under desk server for running my z/OS system on top of the same Linux.

Running “apt update” on the laptop was always faster on the laptop compared to the server. Was this because all traffic for the server was going through my laptop? How do I tell?

The boxes are connected with an Ethernet cable, I had to purchase a wireless dongle for my server, my laptop has a built in wireless adapter.

The linux ifconfig or the ip command gives information about the configuration. For example ip a

eno1: flags=4163 mtu 1500
    inet 10.1.0.3 netmask 255.255.255.0 broadcast 10.1.0.255
    inet6 fe80::.... prefixlen 64 scopeid 0x20
    ether 00:... txqueuelen 1000 (Ethernet)
    RX packets 5136 bytes 1445665 (1.4 MB)
    RX errors 0 dropped 0 overruns 0 frame 0
    TX packets 4933 bytes 1692274 (1.6 MB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
    device interrupt 17 memory 0xb1200000-b1220000  
...
wlxd037450ab7ac: flags=4163 mtu 1500
    inet 192.... netmask 255.255.255.0 broadcast 192....
    inet6 2a00:... prefixlen 64 scopeid 0x0
    inet6 fe80::... prefixlen 64 scopeid 0x20
    inet6 2a00:... prefixlen 64 scopeid 0x0
    ether d0:... txqueuelen 1000 (Ethernet)
    RX packets 42427 bytes 60919847 (60.9 MB)
    RX errors 0 dropped 1 overruns 0 frame 0
    TX packets 25996 bytes 2397812 (2.3 MB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
  • The EtherNet connection eno1 has received 5136 packets and 1.4 MB of data
  • The WireLess connection wlx037450ab7ac has received 42427 packets and 60.9 MB of data.

As I had just done an apt upgrade, the wireless had all of the traffic to download the files, so the traffic was not coming through my laptop.

Once the system was updated, the only traffic flowing was down the Ethernet cable as I used the server from my laptop.

Ping

A ping from each system gave a similar response time.

traceroute

traceroute shows you the hops to the destination.

For example

traceroute abc.xyz.com

To specify the interface you need to run as a superuser.

sudo traceroute abc.xyz.com -i wlxd037450ab7ac

gave

1 bthub.home (192....) 4.654 ms 38.438 ms 38.425 ms
2 * * *
3 * * *
4 31.55.185.184 (31.55.185.184) 75.897 ms 75.890 ms 75.861 ms

If you are not running as a superuser you will get:

setsockopt SO_BINDTODEVICE: Operation not permitted

What else is there to help me?

On z/OS the netstat command gives a lot of information about the session, for example the send window size, the receive windows size etc. This information tends not to be available on other platforms.

On linux there is the ss (Socket Statistics) command.

Example output from ss -t -i included

  • ESTAB 0 0 192.168.1.223:58212 192.0.78.12:
  • https
  • cubic the congestion algorithm name, the default congestion algorithm is “cubic”
  • wscale:9,7 if window scale option is used, this field shows the send scale factor and receive scale factor
  • rto:232
  • rtt:29.236/5.693
  • ato:40 mss:1452
  • pmtu:1500
  • rcvmss:880
  • advmss:1460
  • cwnd:10 congestion window size
  • bytes_sent:68335
  • bytes_acked:68336
  • bytes_received:16334
  • segs_out:276
  • segs_in:202
  • data_segs_out:151
  • data_segs_in:123
  • send 4.0Mbps number of bits sent/time of send.
  • lastsnd:376 how long time since the last packet sent, the unit is millisecond
  • lastrcv:348 how long time since the last packet received, the unit is millisecond
  • lastack:348 how long time since the last packet acknowledged, the unit is millisecond
  • pacing_rate 7.9Mbps
  • delivery_rate 2.6Mbps
  • delivered:152
  • app_limited busy:3700ms
  • rcv_space:14600
  • rcv_ssthresh:64076
  • minrtt:23.773

I did not find most of this information very useful. It is all to easy for a developer(I have done it myself) to provide statistics from information which is readily available, rather than ask what information would be useful to debug problems – then collect and publish that information.

Using CISCO openconnect to tunnel to another system from Linux

I needed to use openconnect from CISCO to be able to logon from my Ubuntu system to someone else’s z/OS system.

This was pretty easy, but understanding some of the under the cover’s bits took a bit of time.

Basic install

  • Use sudo apt install openconnect
  • Download the VPNC script from http://www.infradead.org/ 
  • Create the configuration script
    • I saved the script as vpnc-script.sh
    • Using ls /etc/vpnc showed the directory did not exist. Create it and move the file
    • sudo mkdir /etc/vpnc/
    • sudo mv vpnc-script.sh /etc/vpnc/
    • sudo chmod +x /etc/vpnc/vpnc-script.sh
  • You need information from the owners of the vpn server.
    • vpn userid
    • vpn password
    • name of their system
    • IP address of their internal system
    • tso userid
    • tso password
  • I created a script (openc.sh), where XXXXX is the short userid, and password is your long userid:
    • printf ‘%s’ “password” | sudo openconnect –user=XXXXXX –script=/etc/vpnc/vpnc-script.sh vpn.customer.com
  • When you run openc.sh it prompts for your su password on the machine. The print… means you can store the password in the shell script. If you do not specify it, openconnect will prompt you for it.
  • Once the connection is made you can use ping 10.66.77.88, or x3270 -model 5 10.66.77.88 to access the system, where 10.66.77.88 is the IP address the owner of the vpn server gave you.

x3270

The owner of the vpn server gave me the address of the z/OS machine, my userid and password.

I then used

x3270 -model 5 10.66.77.88 to logon to the system.

Hot key

I like to hot key to my z/OS sessions. I used Ubuntu “Settings”-> Keyboard shortcuts, and added a shortcut

  • name: mvsCust
  • Command: wmctrl -a 10.66.77.88
  • Hot key: Ctrl + H

The wmctl -a says make the window active which has 10.66.77.88 in the window page title.

When I press Ctrl +H it makes the customers x3270 session the active window.

Change the x3270 colours

I wanted to change the screen colours, to distinguish it from other 3270 sessions. See Making x3270 green screens blue or red, or yellow with green bits.

FTP

I had to use SFTP colin@10.66.77.88 to ftp to the remote z/OS system (where colin is my TSO userid).

What happens with openconnect, under the covers.

The handshake has several stages

  • Establish a TLS session using the certificate from the server. Once this has completed, any traffic is encrypted. In my case I used the vpn userid and password. The vpn server can be configured to accept certificates instead of userid and password.
  • The server sends down configuration information from the vpn server’s configuration. For example
    • The IP addresses it supports , such as 10.66.0.0 and netmask 255.255.0.0
    • Any changes to the DNS configuration, so it knows to route 10.66.77.78 via the VPN session.
    • The “banner” such as “Welcome to mycom.com. Users of this system do so at their own risk”.
    • A default domain.
    • Which tunnelling device to use – such as tun0.
    • How many configuration statements.
    • Each set of configuration statements.
    • You can see this information by using the -v option on the openconnect command.
  • Using the information sent from the the vpn server, the openconnect client creates environment variables.
  • The script defined (or defaulted, for example /etc/vpnc/vpnc-script.sh) on the openconnect command is invoked, and it uses these environment variables to manage the ip and dns configuration, changing files like /etc/resolv.conf (the local DNS file).

Oh p*x, I’ve lost my changes

I have been using pax to backup the files in my Unix Services directory and needed to restore a file so I could compare it with the last version (and work out why my updates didn’t work). Unfortunately I managed to overwrite my latest version instead of creating a copy.
I backed up my directory using

pax -W “seqparms=’space=(cyl,(10,10))'” -wzvf “//’COLIN.PAX.PYMQI2′” -x os390 /u/tmp/pymqi2/

This created a data set COLIN.PAX.PYMQI2 with the give space parameters, and os390 format.
(If you do not want all the files listed, use –wzf)

To list the contents of this file use

pax -f “//’COLIN.PAX.PYMQI2′”

To display a subset of the files use

pax -f “//’COLIN.PAX.PYMQI2′” /u/tmp/pymqi2/code

which gave

/u/tmp/pymqi2/code/
/u/tmp/pymqi2/code/pymqi/
/u/tmp/pymqi2/code/pymqi/__init__.py
/u/tmp/pymqi2/code/pymqi/old__init__.old
/u/tmp/pymqi2/code/pymqi/aa

And provide more information using the -v option

drwxrwxrwx 1 COLIN    1000      0 Jan 22 17:04 /u/tmp/pymqi2/code/
drwxr-xr-x 1 COLIN    1000      0 Feb 11 13:10 /u/tmp/pymqi2/code/pymqi/
-rw-r--r-- 1 OMVSKERN 1000 133011 Feb 22 13:15 /u/tmp/pymqi2/code/pymqi/init.py
-rw-r----- 1 COLIN    1000 119592 Feb  3 12:59 /u/tmp/pymqi2/code/pymqi/old__init__.old
-rwx------ 1 OMVSKERN 1000 119565 Jan 22 16:43 /u/tmp/pymqi2/code/pymqi/aa

The whoops

To restore an individual file and overwrite the original I used the -r option.

pax -rf “//’COLIN.PAX.PYMQI2′” /u/tmp/pymqi2/pymqi/__init__.py

I was expecting the file to be restored relative to the directory I was in; No – because I had backed up the files using an absolute path it restored the file to the same place, and so it overwrote my changes to the file. I had changed to a temporary directory, but I had not realised how the command worked.

There are several ways of doing it properly.

Restore with rename

pax -rf “//’COLIN.PAX.PYMQI2′” -i /u/tmp/pymqi2/pymqi/__init__.py

The -i option means rename.

I ran the command and it prompted me to rename it

Rename “/u/tmp/pymqi2/pymqi/__init__.py” as…

/tmp/oldinit.py

Set “do not overwrite”

I could also have used the -k option which prevents the overwriting of existing files.

Rename on restore

I could also have used the rename

pax -rf “//’COLIN.PAX.PYMQI2′” -s#/u/tmp/pymqi2/pymqi#/tmp/# /u/tmp/pymqi2/pymqi/__init__.py

Where the -s#/u/tmp/pymqi2/pymqi#/tmp/# / says use the regular expression to change /u/tmp/pymqi2/pymqi to /tmp and so restore it to a different place. Note: The more obvious -s/abc/xyz/, where / is used as the delimiter, would not work, as there is a ‘/’ in the file path.

All of the above

I could have use all of the options -i -k -s…. .

A better way to backup.

I had specified an absolute directory /u/tmp/pymi2/. If I was in this directory when I did the backup I could have used

pax … -x os390 .

Where the . at the end means from this directory, and so backup a relative directory.

If I list the files I get

pax -f “//’COLIN.PAX.PYMQI2A'” ./aa
./aa

And now if I restore the file…

pax -rf “//’COLIN.PAX.PYMQI2A'” ./aa

It restored the file into my working directory /tmp/aa .

So out of all the good ways of backing up and restoring – I chose the worst one. It only took me about 2 hours to remake all the changes I had lost.

How do I look at TLS 1.3 handshakes?

With TLS before 1.3, encryption of the data on the session occurred after the handshake had completed, so the handshake was visible in Wireshark. With TLS 1.3 the traffic is encrypted after the “Client Hello”, so you cannot immediately see the remainder of the handshake.

Tools like OpenSSL, can write out the magic data needed for decryption. For example

openssl s_client -keylogfile /tmp/kl -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile /home/colinpaice/ssl/ssl2/colinpaice.pem

This writes information to the specified file, in my case /tmp/kl.

The file has data like

SSL/TLS secrets log file, generated by OpenSSL
SERVER_HANDSHAKE_TRAFFIC_SECRET 05a42762…
EXPORTER_SECRET 05a42762dc…
SERVER_TRAFFIC_SECRET_0 05a4276…
CLIENT_HANDSHAKE_TRAFFIC_SECRET 05a42762dc…
CLIENT_TRAFFIC_SECRET_0 05a42762d…

On Linux (Ubuntu) you can tell Wireshark to use this through

edit -> preferences -> protocols -> SSL -> (pre)-master-secret log filename

Different versions of Wireshark have TLS or SSL, use whichever one is available to you.

Specify the name of your file (/tmp/kl in my case) and Wireshark will be able to decrypt the data.

Can I define a disk Read Only to z/OS?

As part of migrating z/OS to a new service level, I wanted to mount old volumes Read-Only, so they were not updated when the new level was used. (For example z/OS updates the dataset last access time in the VTOC). I was running on zPDT, or z/OS on top of Linux, so all of the hardware is emulated. On a real machine you may be able to configure the storage subsystem.

I had four options

  • Make the disk on Linux read only – this worked, and was easy.
  • Copy the disks of interest so I had write access to a copy. This worked, and was easy.
  • Use the zPDT command awsmount 0ac5 -m /mnt/zimages/zOS/A4USR1 –readonly . This worked and was easy.
  • Update the Hardware Configuration Definition (HCD) to make a disk read only. I could define it, but not activate it because this read-only support is for PPRC mirrored disks. I could not vary the address online.

This blog post describes how I changed the HCD to add a read only disk.

This was a journey going into areas I had not been in before (creating IODFs).

The Hardware Configuration Definition(HCD) defines the configuration of the hardware. In day’s gone by the systems programmer would have to do a “sysgen” and used macros to define devices, then assemble it and use it. Nowadays you can maintain the configuration using ISPF panels.

What does the HCD do, and what is an OSCONFIG?

The documentation is not very clear about HCD. There are tiny clues, where it mentions making disks read-only, in OSCONFIG, but does not explain how to display and use the OSCONFIG. Now I know, it is easy.

  • You define each device, or group of similar devices in the HCD.
  • For each OS Configuration (OSCONFIG) you define each operating system image, and which devices belong in which OSCONFIG. See, … simple!

For example you define your configuration, including production and test devices, in the HCD. You then configure

  • A test system with only the test volumes
  • A production system with only the production volumes
  • The sysprog’s system with both test and production devices. From this machine, the systems programmer can create production or test configurations.

Getting started with HCD

The HCD is panel driven from ISPF.

You have to work with a copy of the IODF, and the system will generate a copy for you (suffixed with .WORK). I created a copy, made changes, then created a new IODF.

What is currently being used?

From the main HCD panel

  • 2. Activate or process configuration data
    • 5. View active configuration

Create a copy

From main menu use

  • 6. Maintain I/O definition files
    • 2. Copy I/O definition file

and follow the prompts.

On the home page it has the name of the current IODF being worked on, update it if necessary.

Display the OSCONFIG

Use the ISPF configuration panels for HCD:

  • 1. Define, modify, or view configuration data
    • 1. Operating system configurations

It then lists the available OSCONFIGs. Use / to select one, then select

  • 7. Work with attached devices

This lists the devices. You can scroll or use “L AF0” to locate the devices.

Put / in front to display the options. At the right it gives the command, so

  • 8. Delete . . . . . . . . . . . . . . (d)

I can either use /, and 8, or use ‘d’ (instead of the /) to delete an entry.

PF3 to return to “Define, Modify, or View Configuration Data”.

Add new devices

Use

  • 5. I/O devices

This lists the devices. Use F11 to add

  • Device number 0af0
  • Number of devices 16
  • Device type 3390

Press enter.

It displays a list of OS Configs, select one.

  • option 1 select

You are prompted to configure the devices

  • OFFLINE No Device considered online or offline at IPL
  • DYNAMIC Yes Device supports dynamic configuration
  • LOCANY No UCB can reside in 31 bit storage
  • WLMPAV Yes Device supports work load manager
  • READ-ONLY Sec Restrict access to read requests (SEC or NO)
  • SHARED No Device shared with other systems
  • SHAREDUP No Shared when system physically partitioned

Press enter. To make this read-only I specified Shared=no and read-only=sec. (Sec is for secondary device. The read write copy of the mirrored is is the primary device).

Use PF3 to return.

Activate the configuration

From the HCD home page,

  • 2. Activate or process configuration data
    • 1. Build production I/O definition file

Create production eg “‘SYS1.IODF88”

then

  • 6. Activate or verify configuration dynamically

This displays

  • Currently active IODF . : SYS1.IODF99
  • IODF to be activated . : SYS1.IODF88
  • Test only . . . . . . . . Yes (Yes or No)

Use Test only = YES to validate it, then repeat with Test only = NO. This will make it live.

For me, the SYS1.IODFxx dataset, was created on the wrong volume. It has to be on the same volume as the SYS1.IPLPARM and other IPL information for a successful IPL.

Move the SYS1.IODF to the IPL parm volume.

Change your IPL loadxx member in SYS1.IPLPARM to point to the new IODF.

Although I had specified A4SYS1 as the volume for the SYS1.IODF88, SMS allocation routines allocated it on a different volume. I had to move it to the correct volume. See here.

Once I had IPLed with the new IODF

The command

D U,,,,0AF0,1 gave

UNIT TYPE STATUS   VOLSER     VOLSTATE      SS   
0AF0 3390 F-NRD-RO                /RSDNT     0   

Which says there is no device mounted, but it has been defined as RO.

I varied it online and I got

V 0AF0,ONLINE
IEE103I UNIT 0AF0 NOT BROUGHT ONLINE
IEE763I NAME= IECDINIT CODE= 000000000110088F
IEA434I DEVICE ONLINE IS NOT ALLOWED, R/O SEC PPRC STATE NOT VALID
IEE764I END OF IEE103I RELATED MESSAGES

Which means it was unable to mount my disk as it was not part of a PPRC mirrored DASD environment. I had defined a disk as Read Only, but was not able to use it.