Category: linux

Restore files from Linux using Duplicity

Duplicity is a program which manages backup and restore of files on your Linux machine.

What is duplicity?

Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.

Backing up files

You run the backup application, and can specify the directories to backup. You can specify which directories to ignore.

The backups can be stored

  • Google drive
  • Networks server
  • Locally attached drive
  • Local Folder

That’s the easy bit.

What files are backed up?

I have my backups going to an external USB drive /media/colinpaice/UbuntuBackup1/home/Backup2024 on my Linux machine. The files have names like

duplicity-full.20240308T084825Z.vol295.difftar.gpg

The command

duplicity list-current-files file:///media/colinpaice/UbuntuBackup1/home/Backup2024 >files2024

Restore a file

duplicity restore -t 3D –file-to-restore ~/ssl/ssl2/rsaca256.csr file:///media/colinpaice/UbuntuBackup/home/Backup2021 ~/ssl/ssl2/rsa256.csr

This restores a file

  • -t 3D from 3 days ago
  • –file-to-restore ~/ssl/ssl2/rsaca256.csr
  • file:///media/colinpaice/UbuntuBackup/home/Backup2021 from this device
  • ~/ssl/ssl2/rsa256.csr to this file

What next

Duplicity can do much more than this. Ive just provided information on the most basic stuff that I have used. See the products web page or man duplicity

Setting up for PING can be difficult!

  • Setting up ping within one machine is trivial
  • Setting up ping between two machines is relatively easy
  • Setting up ping with 3 machines can be hard to get right and to get working.

Most documentation describes how to set up PING between two machines, and does not mention anything more complex.

Ancient history (well 1980’s)

To understand modern TCPIP networks, it helps to know about the origins of Ethernet.

Originally with Ethernet there was a bus; a very long cable. You plugged your computer into this bus. Each computer on the bus had a unique 48 bit MAC address. You could send a request to another computer on this bus using their Ethernet address. You could send a request to ALL computers on this bus. These days this might be, send to all computers:”does anyone have this IP address…?”

An Ethernet bus is connected to an Ethernet router which can route packets to other routers.

Ethernet has evolved and instead of a long bus shared by all users, you have a switch device, and you plug the Ethernet cable from your computer to the switch. Conceptually it is the same, but with the switch you get much better performance and usability. You still have routers to get between different Ethernet environments. The request send to all computers:”does anyone have this IP address…?” goes to the switch, and the switch sends it to all plugged in computers.

When you start TCPIP, it configures the Ethernet hardware adapter with the addresses is should listen for.

Other terminology and concepts

  • A router is a networking device which forwards packets between different networks. Packets of data get sent from the originator through zero or more routers to the final destination.
  • A gateway has several meanings
    • It can connect different network types, for example act as a protocol translator, it can have a built-in firewall
    • It can route IP traffic between different IP subnets
    • It can loosely mean a router

Usually a router or gateway is a dedicated device or hardware.

You can have a computer act as a router or gateway; it takes data from one interface and routes it to a different interface. The computer can pass the data through a fire wall, or transform it, for example converting internal IP addresses to external IP addresses (NAT translation).

At a concept level, a computer’s network adapter is configured only to listen for packets which match the IP addresses of the interface, and ignore the rest.

If you want a computer to act as a router or gateway, you need to configure the network adapter to listen to all traffic, and to process the traffic. This is important when routing traffic from computer A through computer B to computer C.

A simple one hop ping, first time

I have set up my laptop to talk to a server over Ethernet.

I have configured my laptop using

sudo ip -6 addr add 2001:7::/64 dev enp0s31f6

This says for any IP V6 address starting with 2001:0007:0000:0000 then try sending it down the Ethernet cable with interface name enp0s31f6.

Ping a non existing address

If I try to ping an address say 2001:7::99 a packet is sent to all computers on the Ethernet bus

from myipAddress to everyone on the Ethernet bus, does anyone have IP address 2001:7::99?

There is no reply because no computer on the bus, has the address.

Ping the adjacent box

On the server, the IP address of the Ethernet cable is 2001:7::2.

If I ping this address, there are the following flows

From myipAddress (and myMAC) to everyone on the Ethernet bus, does anyone have IP address 2001:7::2?
From 2001:7::2 to myipAddress (MAC), yes I have that IP address. My Ethernet MAC address is …

My laptop puts the remote IP address and its MAC address into its neighbour cache, then issues the ping request.

The ping request looks in the neighbour cache find the IP address and MAC address and issues.

From myipAddress to 2001:7::2 at macAddress .. ping request.

The second and later times I issue a ping

The ping request looks in the neighbour cache find the IP address and MAC address and issues the ping

From myipAddress to 2001:7::2 at macAddress .. ping request.

So while the neighbour cache still has the IP of the target, and it’s MAC address, the ping can be routed to the next hop.

Setting up a multi hop ping

I have my laptop connected via a physical Ethernet cable to my server. The server is connected to z/OS via a virtual Ethernet connection. The IP address of the z/OS end of the virtual Ethernet cable is 2001:7::4.

The ping 2001:7::4 request on my laptop, does not work (as we saw above), because TCP asked everyone on the Ethernet bus if it has address 2001:7::4 – and no machine replies.

You need to define the link to the server machine as a gateway or router-like device which can handle IP addresses which are not on the Ethernet bus. You define it like

sudo ip -6 route add 2001:7::/64 via 2001:7::2

This says for any traffic with IP address 2001:0007:0000:0000:* send it via address 2001:7::2.

This requires 2001:7::2 to be known about, and so needs the following to be configured first

sudo ip -6 route add 2001:7::2 dev enp0s31f6

for TCPIP to know where to send the traffic to.

The route add 2001:7::2 dev enp0s31f6 command sends a broadcast on the Ethernet bus asking – does anyone have 2001:7::2. My server replies saying yes I have it. This is the same as for the ping above.

In summary

To send traffic on the same Ethernet bus you use

sudo ip -6 route add 2001:7::2 dev enp0s31f6

To route it via a router, switch, or computer acting as a router or switch you need.

sudo ip -6 route add 2001:7::/64 via 2001:7::2 or
sudo ip -6 route add 2001:7::/64 via 2001:7::2 <dev enp0s31f6>

The computer acting as a router or switch may need additional configuration. For example to allow it to route traffic from one Ethernet bus to another Ethernet bus you need to enable packet forwarding. On Linux to enable forwarding for all interfaces use

sysctl -w net.ipv6.conf.all.forwarding=1

On z/OS you use a static route such as

ROUTE 2001:7::/64 2001:7::3 JFPORTCP6

Where 2001:7::/64 is the range of addresses 2001:7::3 is (one of) the addresses of the interface at the remote end of the cabl, and JFPORTCP6 is the interface name. This is similar to the Linux route statement above.

You might need to set up the firewall. On my server I needed

sudo ufw route allow in on eno1 out on tap2

What IP Address does the sender have?

This is where is starts to get more complex.

Every network connection has at least one IP address.

With IP V6

  • Each interface gets an “internal” IP address such as fe80::9b07:33a1:aa30:e272
  • You can allocate an external address using the Linux command sudo ip -6 addr add 2001:7::1 dev enp0s31f6
  • If the interface has one external IP address configured, then this will be used.
  • If the interface has more than one external address configured then the first in the list may be used (not always).
  • If the interface does not have an external IP address, then TCPIP will find one from another interface, or allocate one for the duration of the request, such as 2a00:23c5:978f:9999:210a:1e9b:94a4:c8e. This address comes from my wireless connection

When my laptop is started, only the wireless connection has an IPV6 address. A ping request to 2001:7::4 had the origin IP address of 2a00:23c5:978f:9999:210a:1e9b:94a4:c8e which is the first address in the list for the wireless connection.

You can tell ping which address to use, for example

ping -I 2a00:23c5:978f:9999:cff1:dc13:4fc6:f21b 2007:1::4 (this fails because the server does now know to send the response to the requester)

I defined a new address for the interface using

sudo ip -6 addr rep 2002:7::1 dev enp0s31f6

I could issue ping -I 2002:7::1 2001:7::4, but this failed to get a response, because the back-end and intermediate nodes, did not know how to get the response back to 2002:7::1

Does this sender address matter?

Yes, because the remote end needs to have a definition to be able to send the response back.

I had my laptop connected to a Linux server over Ethernet, which in turn was connected to z/OS over a virtual Ethernet.

On z/OS I could see the ping request arrive, but it could not send the response back because it did not know how to get to 2a00:23c5:978f:9999….

I configured the laptop end of the Ethernet to give it an IP address 2001:7::1

sudo ip -6 addr add 2001:7::1/64 dev enp0s31f6

I configured the server to laptop to have an IP address of 2001:7::2

sudo ip -6 route add 2001:7::1/128 dev eno1

I configured the server to z/OS with an IP address and a route

sudo ip -6 addr add 2001:7::3/128 dev tap2
sudo ip -6 route add 2001:7::4/128 dev tap2

Now when I did the ping, the originator was 2001:7::1.

I configured the z/OS interface to send stuff back

INTERFACE JFPORTCP6 
    DEFINE IPAQENET6 
    CHPIDTYPE OSD 
    PORTNAME PORTC 
                                   
INTERFACE JFPORTCP6 
    ADDADDR 2001:DB8:8::9 
INTERFACE JFPORTCP6 
    ADDADDR 2001:DB8::9 
INTERFACE JFPORTCP6 
    ADDADDR 2001:7::4 
START JFPORTCP6

and the routing

BEGINRoutes 
;     Destination      Gateway                 LinkName  Size 
ROUTE 2001:7::/64      2001:7::3              JFPORTCP6   MTU 5000 
...
ENDRoutes

This says that all traffic with destination address 2001:0007:0000:000…. send to interface JFPORTCP6. This interface is connected to a gateway with the address of the remote end of the Ethernet (so on the server) of 2001:7::3.

The server machine needs to act as a router between the different Ethernet buses. You can display and configure this using

sysctl -a |grep forwarding
sudo sysctl -w net.ipv6.conf.all.forwarding=1

Packet forwarding on z/OS

By default the z/OS interface only listens for packets with one of the IP addresses of the interface. For z/OS to be able to be a router; accept all packets, and route them to other interfaces you need:

  • IPCONFIG DATAGRAMFWD in the TCP/IP Profile
  • PRIROUTER on the Interface definition . This configures the Ethernet adapter (hardware) so If a datagram is received at this device for an unknown IP address, the datagram is routed to this TCP/IP instance.

But normally if you are running z/OS you use a cheaper, physical router, rather than use the z/OS to do your routing. It might only be people like me who run z/OS on their laptop who want to try routing through z/OS.

Why doesn’t ctrl-s work in ISPF edit? – ah it does now.

I had been editing a file, saving it, and finding the changes were not being picked up. Looking back, it was obvious; I was using CTRL-S the familiar Linux command, instead of F3 on ISPF.

I fixed this by configuring X3270 (on Linux).

My file /home/colin/.x3270pro now has

...
x3270.keymap: mine
! Definition of the 'mine' keymap
x3270.keymap.mine: #override \
    Alt<Key>4:          String("\\x00a2")\n\
    Ctrl<Key>backslash: String("\\x00a2")\n\
    <Key>Escape:    Clear()\n\
    <Key>End:        FieldEnd()\n\
    Ctrl<Key>Delete:   EraseEOF()\n\
    Ctrl<Key>Right:    NextWord()\n\
    Ctrl<Key>Left:    PreviousWord()\n\
    Ctrl<Key>Up:    Home()\n\
    <Key>Control_L: Reset()\n\
    <Key>Control_R: Reset()\n\
    <Key>Prior: PF(7)\n\
    <Key>Next: PF(8)\n\
    <Btn3Down>:   PA(1)\n\
    Ctrl<Key>1:   PA(1)\n\
    Ctrl<Key>s:   MoveCursor(3,15) String("save") Enter()\n\

When I started a new X3270 session, Ctrl -S went to the command line, typed save and pressed enter. Job Done ! The numbers are 0 based, so 3 means line 4 on the screen.

This makes life so much easier!

Setting up z/OS for TLS clients

There is a lot of configuration needed when setting up TLS(SSL) between a server and a client. There are many options and it is easy to misconfigure. The diagnostic information you get when the TLS handshake fails is usually insufficient to identify any problems.

You need the following on z/OS:

  • One or more Certificate Authority certificates. You can create and use your own for testing. If you want to work with external sites you’ll need a proper (external) CA, but for validation and proof of concept you can create your own CA. You could set up a top level CA CN=CA,O=MYORG, and another one (signed by CA=CA,O=MYORG), called CN=CA,OU=TEST,O=MYORG. Either or both of the public CA certificates will need to be sent to the clients in imported into their keystore.
  • A private/public key, signed by a CA, (such as signed by CA=CA,OU=TEST,O=MYORG).
  • The private key is associated with a userid.
    • The signing operation takes the data (the public key), does a hash sum calculation on the data, encrypts this hash sum, and stores the encrypted hash value, and CA public certificate with the original data. To check the signature, the receiving application compares the CA with its local copy, if that matches, does the same checksum calculation, decodes the encrypted hash sum – and checks the decrypted and locally calculated values match.
    • A certificate is created using one from a list of algorithms. (For example, Elliptic Curve, RSA). When the certificate is sent to the client, the client needs to support the algorithm. Either end can be configured, for example, to support Elliptic Curve, but not RSA.
  • A keyring to contain your private key(s) – this can also contain CA public certificates of the partners (clients or servers).
  • A “site” keyring (public keystore, or trust ring) which holds the public CA certificates of all the other sites you work with. If you have only one keyring per user or application, you need to update each of them if you need to an a new CA to your environment. Many applications are only designed to work with one keyring. Java applications tend to have a key store(for the private key) and a trust store for the CAs.
  • Some applications can support more than one private certificate on a keyring. The certificate needs to match what the client can support.
  • For certificates which are sent to your server, you need a copy of the CA(s) used to sign the incoming certificate. If you have a copy of the CA, then you can validate any certificate that the CA signed. This means you do not have to have a copy of the public certificate of every client. You just need the CA.
    • Some application need access to just one CA in the chain, other applications need access to all certificates in the CA chain.
  • As part of the TLS handshake
    • the client sends up a list of the valid cipher specs it supports (which algorithms, and size of key)
    • the server sends down a subset of the list of cipher spec to use (from the client’s list)
    • the server can also send down its certificate, which contains information such as the distinguished name CN=zSERVER, OU=TEST, O= MYORG, and host name.
    • the client can validate these names – to make sure the host name in the certificate matches the host, and what it was expecting.
    • if requested, the client can send up its certificate for identification. The server can validate the certificate, and can optionally map it to a userid on the server.
  • A userid can be given permission to read certificate in another user’s keyring. A userid needs a higher level of authority to be able to access the private key in another id’s keyring.

Create the Certificate Authority

//IBMRACF  JOB 1,MSGCLASS=H                               
//S1  EXEC PGM=IKJEFT01,REGION=0M                         
//SYSPRINT DD SYSOUT=*                                    
//SYSTSPRT DD SYSOUT=*                                    
//SYSTSIN DD * 
RACDCERT certauth LIST(label('DOCZOSCA')) 
RACDCERT CERTAUTH DELETE(LABEL('DOCZOSCA'))               
RACDCERT GENCERT  -                                         
  CERTAUTH -                                                
  SUBJECTSDN(CN('DocZosCA')- 
             O('COLIN') -                                   
             OU('CA')) - 
  NOTAFTER(   DATE(2027-07-02  ))-                          
  KEYUSAGE(   CERTSIGN )  -      
  SIZE(2048) -                                              
  WITHLABEL('DOCZOSCA') 
/*
//

This certificate is created against “user” CERTAUTH. Keyusage CERTSIGN means it can be used to sign certificates. “user” CERTAUTH is often displayed internally as “irrcerta”.

Once it has been created the certificate should be connected to every ring that may use it, see below.

Export the CA certificate to a file so, clients can access it 

RACDCERT CERTAUTH EXPORT(LABEL('DOCZOSCA')) -
  DSN('IBMUSER.CERT.DOC.CA.PEM') -
  FORMAT(CERTB64) -
  PASSWORD('password')

The file looks like

-----BEGIN CERTIFICATE-----                                        
MIIDYDCCAkigAwIBAgIBADANBgkqhkiG9w0BAQsFADAwMQ4wDAYDVQQKEwVDT0xJ   
TjELMAkGA1UECxMCQ0ExETAPBgNVBAMTCERvY1pvc0NBMB4XDTIyMTAwOTAwMDAw 
...

This can be sent to the clients, so they can validate certificates sent from the server. This file could be sent using cut and paste, or FTP.

Create the keyring for user START1.

The instructions below lists the ring first – in case you need to know what it was before you deleted it”

RACDCERT LISTRING(TN3270)  ID(START1) 

RACDCERT DELRING(TN3270) ID(START1) 

RACDCERT ADDRING(TN3270) ID(START1)                                                          

RACDCERT LISTRING(TN3270)  ID(START1) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Connect the CA to every keyring that needs to use it

RACDCERT ID(START1) CONNECT(RING(TN3270) - 
                            CERTAUTH LABEL('DOCZOSCA'))

Create a user certificate and sign it on z/OS

This creates a certificate and gets is signed – as one operation. You can create a certificate, export it, sent it off to a remote CA, import it, and add it to a userid.

RACDCERT ID(START1) DELETE(LABEL('NISTECC521')) 
                                                                
RACDCERT ID(START1) GENCERT -                                   
  SUBJECTSDN(CN('10.1.1.2') - 
             O('NISTECC521') -                                  
             OU('SSS')) -                                       
   ALTNAME(IP(10.1.1.2))-                                       
   NISTECC - 
   KEYUSAGE(HANDSHAKE) - 
   SIZE(521) - 
   SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -                      
   WITHLABEL('NISTECC521')     -                                
                                                                
RACDCERT id(START1) ALTER(LABEL('NISTECC521'))TRUST             

RACDCERT ID(START1) CONNECT(RING(TN3270) -                      
                            ID(START1)  -                       
                            DEFAULT  - 
                            LABEL('NISTECC521') )               
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh                    
RACDCERT LIST(LABEL('NISTECC521' )) ID(START1)                  
RACDCERT LISTRING(TN3270)  ID(START1)

This creates a certificate with type Elliptic Curve (NISTECC) with a key size of 521. It is signed with the CA certificate created above.

The ALTNAME, is a field the client can verify that the Source Name in the certificate matches the IP address of the connection.

It is connected to the user’s keyring as the DEFAULT. The default certificate is used if the label of a certificate is not specified when using the keyring.

Give a user access to the keyring

PERMIT START1.TN3270.LST CLASS(RDATALIB)  -    
    ID(COLIN )  ACCESS(UPDATE )                          
SETROPTS RACLIST(RDATALIB) refresh
  • Update access give userid COLIN access to the private key.
  • Read access only gives access to the public keys in the ring.

You would typically give a group of userids access, not just to individual userids.

Import the client’s CA’s used to sign the client certificates

This is the opposite to Export the CA certificate to a file so clients can access it above.

Copy the certificate to z/OS. This can be done using FTP or cut and paste.

Use it!

I used it in AT-TLS

TTLSConnectionAdvancedParms       TNCOonAdvParms 
{ 
 ServerCertificateLabel  NISTECC521
 ...
} 
TTLSSignatureParms                TNESigParms 
{ 
   CLientECurves Any 
} 
TTLSEnvironmentAction                 TNEA 
{ 
  HandshakeRole                       ServerWithClientAuth 
  TTLSKeyringParms 
  { 
    Keyring                   start1/TN3270 
  } 
...
}

Linux ls command timestamps in microseconds is easy-ish

Any of the following work

ls -la --time-style=full-iso ...
ls --full-time ...

Which gave me

-rw-r--r-- 1 root root 1534 2023-01-01 16:46:58.394054373 +0000 group

Where the format is

The TIME_STYLE argument can be full-iso, long-iso, iso, locale, or +FORMAT. FORMAT is interpreted like in date(1).

But during installing/removing a package it touched the file, and I have

-rw-r--r-- 1 root root 3784 2022-12-30 11:14:15.436236905 +0000 passwd
-rw-r--r-- 1 root root 3764 2022-12-30 11:14:15.000000000 +0000 passwd-
and 
-rw-r--r-- 1 root root 1534 2023-01-01 16:46:58.394054373 +0000 group
-rw-r--r-- 1 root root 1523 2022-12-30 11:14:15.000000000 +0000 group-

so the temporary files have .000000 microseconds – so there is something else going on!

You can use

alias lt=’ls -ltr –full-time –color=auto’

to make a command “lt” which is the ls command, plus options.

Why has my ethernet connection stopped connecting?

This morning my Ethernet connection between my two Linux systems stopped working. I could see IPV6 stuff flowing over the network, but Linux did not say connected. Also there was no IPV4 address. It took me almost a day to work out what the problem was. Googling and following the advice may have made it worse!

I also include some useful commands for next time it happens.

The high level problem

It looks like the Network Manager has changed.

A week ago, I had files like

/etc/NetworkManager/system-connections/enp0s31f6

containing the definitions for my Ethernet.

Now Network Manager uses

/etc/NetworkManager/system-connections/BTHub6-9999.nmconnection

and these configuration files were missing configuration data. I have a .nmconnection file going back to November, so something has changed.

Further study shows that the

nmcli connection migrate

converts from old format to .nmconnection files, so perhaps this was done under the covers.

Network manager files

Files in /etc/NetworkManager/system-connections/ must be owned by root and be readably only by root – otherwise NetworkManager will ignore it.

Some of my files had the wrong permissions, and so were ignored.

I used

sudo chmod -R 600 /etc/NetworkManager/system-connections/

and restarted NetworkManager

sudo systemctl restart NetworkManager

and missing files reappeared in Network Manager.

For more information about the files see man nm-settings-keyfile.

The detailed problem

Using Wireshark I could see IPV6 traffic flowing over the connection, so the cable was OK, and some of the definitions were OK.

The ip addr command showed there was an IPV6 address for the connection, but no IPV4 address.

I could not find a log for Network Manager with its error messages, see log below for the messages on syslog.

Looking online, there were suggestions that you delete your existing definition and recreate it, also use nm-connection-editor. This may have been a bad move; it is always better to rename than to delete.

Comparing the definitions currently in use /etc/NetworkManager/system-connections/ with a backup version, I could see that the .nmconnection files were in use.

I used Network Manager to change my Ethernet definitions. Under the IPv4 tab

  • IPv4 method: change from Automatic(DHCP) to Manual
  • Address: Added 10.1.0.2 Netmask 255.255.255.0
  • Route: Added 10.1.0.3 Netmask 255.255.255.255.0 Gateway 10.1.0.2

The route statement says to get to 10.1.0.3 go via 10.1.0.2 .

Once I restarted the connection it became active, and the ip -4 addr command showed it had an IPv4 address.

For the other end of the connection I did the matching changes and the end to end connection burst into life!

For my Ethernet connection my file was 

[connection]
id=Wired connection 1
uuid=ecc4df76-4733-45f5-9b67-9fba9ef2d3bf
type=ethernet
interface-name=enp0s31f6
permissions=
timestamp=1673353909

[ethernet]
mac-address-blacklist=

[ipv4]
address1=10.1.0.2/24

dns-priority=100
dns-search=
method=manual
route1=10.1.0.3/24,10.1.0.2

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

[proxy]

This defines the IP address 10.1.0.2, and a route to 10.1.0.3 via 10.1.0.2 .

Useful commands

Display the devices

nmcli d

nmcli device

gives

DEVICE             TYPE      STATE         CONNECTION         
wlp4s0             wifi      connected     BTHub6-78RQ        
enp0s31f6          ethernet  connected     Wired connection 1 
virbr0             bridge    connected     virbr0  d

so for my en0s31f6 device, the connection file is ‘Wired connection 1’

Display the connection

nmcli c

nmcli connection

NAME                UUID                                  TYPE       DEVICE    
BTHub6-78RQ         fc74c8e0-6f96-4e8b-a8ba-6389abbe3396  wifi       wlp4s0    
Wired connection 1  ecc4df76-4733-45f5-9b67-9fba9ef2d3bf  ethernet   enp0s31f6 
virbr0              386a5a3a-023b-41d9-9138-04202d8dfda6  bridge     virbr0

Display more information

nmcli -f all c |less

Display only some fields

nmcli -f name,device,FILENAME c |less

gives

NAME                DEVICE     FILENAME                                                                    
BTHub6-78RQ         wlp4s0     /etc/NetworkManager/system-connections/BTHub6-78RQ.nmconnection.old2        
Wired connection 1  enp0s31f6  /etc/NetworkManager/system-connections/Wired connection 1.nmconnection.old2 
virbr0              virbr0     /run/NetworkManager/system-connections/virbr0.nmconnection

Displaying trace

When the Ethernet connection work, /var/log/syslog had entries

 NetworkManager[11240]: <info>  [....0612] device (enp0s31f6): Activation: starting connection 'enp0s31f6' (c066ca29-2253-41ef-8e69-2251fb15f7b8)
 NetworkManager[11240]: <info>  [....0617] audit: op="connection-activate" uuid="c066ca29-2253-41ef-8e69-2251fb15f7b8" name="enp0s31f6" pid=2585 uid=1000 result="success"
 NetworkManager[11240]: <info>  [....0636] device (enp0s31f6): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
 NetworkManager[11240]: <info>  [....0678] device (enp0s31f6): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
 NetworkManager[11240]: <info>  [....0718] device (enp0s31f6): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
 avahi-daemon[1123]: Joining mDNS multicast group on interface enp0s31f6.IPv6 with address fe80::9b07:33a1:aa30:e272.
 avahi-daemon[1123]: New relevant interface enp0s31f6.IPv6 for mDNS.
 avahi-daemon[1123]: Registering new address record for fe80::9b07:33a1:aa30:e272 on enp0s31f6.*.
 avahi-daemon[1123]: Joining mDNS multicast group on interface enp0s31f6.IPv4 with address 10.1.0.2.
 avahi-daemon[1123]: New relevant interface enp0s31f6.IPv4 for mDNS.
 avahi-daemon[1123]: Registering new address record for 10.1.0.2 on enp0s31f6.IPv4.
 NetworkManager[11240]: <info>  [....0830] device (enp0s31f6): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
 NetworkManager[11240]: <info>  [....1005] device (enp0s31f6): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
 NetworkManager[11240]: <info>  [....1008] device (enp0s31f6): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
 NetworkManager[11240]: <info>  [....1021] device (enp0s31f6): Activation: successful, device activated.

When the connection was defined as DHCP the trace was 

NetworkManager: <info>  [...] device (enp0s31f6): state change: ip-config -> deactivating (reason 'user-requested', sys-iface-state: 'managed')
NetworkManager: <info>  [...] audit: op="device-disconnect" interface="enp0s31f6" ifindex=2 pid=2585 uid=1000 result="success"
NetworkManager: <info>  [...] device (enp0s31f6): state change: deactivating -> disconnected (reason 'user-requested', sys-iface-state: 'managed')
avahi-daemon: Withdrawing address record for fe80::78e8:9e55:9f3f:768 on enp0s31f6.
avahi-daemon: Leaving mDNS multicast group on interface enp0s31f6.IPv6 with address fe80::78e8:9e55:9f3f:768.
avahi-daemon: Interface enp0s31f6.IPv6 no longer relevant for mDNS.
NetworkManager: <info>  [...] dhcp4 (enp0s31f6): canceled DHCP transaction
NetworkManager: <info>  [...] dhcp4 (enp0s31f6): state changed unknown -> done
NetworkManager: <info>  [...] device (enp0s31f6): Activation: starting connection 'Wired connection 1' (ecc4df76-4733-45f5-9b67-9fba9ef2d3bf)
NetworkManager: <info>  [...] device (enp0s31f6): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
NetworkManager: <info>  [...] device (enp0s31f6): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
NetworkManager: <info>  [...] device (enp0s31f6): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
NetworkManager: <info>  [...] dhcp4 (enp0s31f6): activation: beginning transaction (timeout in 45 seconds)
avahi-daemon: Joining mDNS multicast group on interface enp0s31f6.IPv6 with address fe80::78e8:9e55:9f3f:768.
avahi-daemon: New relevant interface enp0s31f6.IPv6 for mDNS.
avahi-daemon: Registering new address record for fe80::78e8:9e55:9f3f:768 on enp0s31f6.*.

This has entries about DHCP.

My machine did not have a DHCP server installed – so any request for DHCP will fail to get an address.

Backing up on Linux is easy – how do I restore?

I’ve been using deja-dup front end to duplicity to backup my Ubuntu Linux. This has worked fine. The only problems I had were trying to restore files I had not backed up (/etc….) – a Defective End User problem.

Restoring one of these files was a little challenge, with bumps in the road… here’s how I did it. I’ve listed the errors I received while using duplicity.

A high level thought – should you change the repository name every year – such as including the year?

Where am I backing up to (where do I restore from)?

Use the deja-dup application (use the Linux show applications and find backup).

Under Storage location will be location and folder.

If this is an external drive it may be mounted under /mnt/… or /media/.

High level view

duplicity collection-status file:///media/colinpaice/UbuntuBackup/home/Backup2023

gave

Last full backup date: Sun Jan  1 17:02:48 2023
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive directory: /home/colinpaice/.cache/duplicity/52b5c01605d52d0976f1434440a7c11a

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Sun Jan  1 17:02:48 2023
Chain end time: Tue Jan 10 08:19:05 2023
Number of contained backup sets: 10
Total number of contained volumes: 250
 Type of backup set:                            Time:   Number of volumes:
                Full         Sun Jan  1 17:02:48 2023               238
         Incremental         Mon Jan  2 08:25:38 2023                 1
         Incremental         Tue Jan  3 08:50:23 2023                 1
         Incremental         Wed Jan  4 08:53:01 2023                 1
         Incremental         Thu Jan  5 08:22:41 2023                 1
         Incremental         Fri Jan  6 08:36:31 2023                 4
         Incremental         Sat Jan  7 10:23:00 2023                 1
         Incremental         Sun Jan  8 12:16:58 2023                 1
         Incremental         Mon Jan  9 09:02:31 2023                 1
         Incremental         Tue Jan 10 08:19:05 2023                 1
-------------------------
No orphaned or incomplete backup sets found.

List files in the backup

duplicity list-current-files file:///media/colinpaice/UbuntuBackup/home/Backup2023/

I saved the list of files using

duplicity list-current-files file:///media/colinpaice/UbuntuBackup/home/Backup2023 >dup.lst
Synchronising remote metadata to local cache…
GnuPG passphrase for decryption:

This took over a minute (it has to check thousands of files names). I think it scans every “signature” file to build a list. The more signature files – the longer it takes.

Inside dup.lst it had entries like

Tue Aug 14 11:41:33 2018 etc/NetworkManager/system-connections/LG-D855_2292

This says the backed up file was from the given date in 2018.

List files available on a given date

I used

duplicity list-current-files –time 5D file:///media/colinpaice/UbuntuBackup/home/Backup2023 > aa.lst

to get a list of files available 5 days ago.

Restore the file

duplicity restore -t 3D –file-to-restore etc/NetworkManager/system-connections/LG-D855_2292 file:///media/colinpaice/UbuntuBackup/home/Backup2023 /home/colinpaice/secret/abc

where

  • -t ( or –time) specify a time period
  • 3D for three days ago. Other options s, m, h, D, W, M, or Y. 1M3d is 33 days ago.
  • etc/Network…. this is the file I wanted to restore. Note no leading / .
  • file:///media…/home/Backup2023 from this backup location.
  • /home/colinpaice/secret/abc and put the file here. The file/directory must not exist.

Restore a directory

Specify the folder (ending in /) and specify a target directory (/home/colinpaice/secret/abcd). It will create the directory if needed.

If you specify a name like etc/…/* it will report it was not found in archive.

duplicity restore -t 3D –file-to-restore etc/NetworkManager/system-connections/ file:///media/colinpaice/UbuntuBackup/home/Backup2023 /home/colinpaice/secret/abcd

Error messages

gpg: WARNING: unsafe permissions on homedir

This says the contents of your secret directory are publicly available. Change the permissions.

chmod 600 /home/colinpaice/.gnupg/*
chmod 700 /home/colinpaice/.gnupg/

Local and Remote metadata are synchronized, no sync needed.

This is OK. It means the metadata is the same in the local cache as in the remote store.

duplicity.dup_collections.CollectionsError: No signature chains found

You have the wrong backup repository name,

  • it needs ///
  • ls should give a filename like /media/colinpaice/UbuntuBackup/…./duplicity-new-signatures.20230101T170248Z.to.20230102T082538Z.sigtar.gpg

Error ‘[Errno 1] Operation not permitted: b’/home/colinpaice/secret/abc” processing .

After the restore, duplicity was not able to use chown on the file to restore it to the original owner.

GPGError: GPG Failed, see log below:

===== Begin GnuPG log =====
gpg: AES256 encrypted data
gpg: gcry_kdf_derive failed: Invalid data
gpg: encrypted with 1 passphrase
gpg: decryption failed: No secret key
===== End GnuPG log =====

I got this asking for a date before backups were taken. See

duplicity collection-status file:///media/colinpaice/UbuntuBackup/home/Backup2023

I was looking for a backup in 2002 – which was not in the 2003 backup.

What does tso netstat neighbour give you?

The command TSO NETSTAT ND gave me 

Query Neighbor cache for 2001:db8:1:0:8024:bff:fe45:840c 
  IntfName: IFPORTCP6          IntfType: IPAQENET6 
  LinkLayerAddr: 82240B45840C  State: Reachable 
  Type: Router                 AdvDfltRtr: No 

Query Neighbor cache for fe80::8024:bff:fe45:840c 
  IntfName: IFPORTCP6          IntfType: IPAQENET6 
  LinkLayerAddr: 82240B45840C  State: Reachable 
  Type: Router                 AdvDfltRtr: No 

Query Neighbor cache for fe80::9863:1eff:fe13:1408 
  IntfName: JFPORTCP6          IntfType: IPAQENET6 
  LinkLayerAddr: 9A631E131408  State: Reachable 
  Type: Router                 AdvDfltRtr: No

On Linux the

ip -6 addr

command gave me

tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 1000
    inet6 2001:db8:1:0:b0fd:f92b:8362:577b/64 ...
    inet6 2001:db8:1:0:8024:bff:fe45:840c/64 ...
    inet6 fe80::8024:bff:fe45:840c/64 ...

tap2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 1000
    inet6 fe80::9863:1eff:fe13:1408/64 ...

The TSO output means

  • Query Neighbor cache for 2001:db8:1:0:8024:bff:fe45:840c. The address is one of the addresses on the remote end of the connection. There is an entry because some traffic came via the address.
  • IntfName: IFPORTCP6 The z/OS Interface name used to create the defintion
  • IntfType: IPAQENET6 the OSA-Express QDIO interfaces statement
  • LinkLayerAddr: 82240B45840C
  • State: Reachable Other options can include stale, which means z/OS has not heard anything from this address for a while
  • Type: Router
  • AdvDfltRtr: No. The information passed in the Router Advertisement, said this was connection does not Advertise a Default Router(AdvDfltRtr).

From the NETSTAT ND output we can see data has been received from

  • IFPORTCP6:2001:db8:1:0:8024:bff:fe45:840c
  • IFPORTCP6:fe80::8024:bff:fe45:840c
  • JFPORTCP6:fe80::9863:1eff:fe13:1408

To get data to flow down the 2001…. address I had to use

ping -I 2001:db8:1:0:8024:bff:fe45:840c 2001:db8:1::9

Where the -I says use the interface address.

You can get information about bytes processed by interface (not by address) using the TSO NETSTAT DEVLINKS command.

Understanding radvd with IPV6 on Linux.

My two day project to deploy IP V6 dynamic routing, turned into an eight week project before I got it to work.

I am documenting a lot of what I learned – today’s experience is understanding what radvd is and what the configuration options mean. I found a lot of documentation – most of which either assumed you know a lot, or only provided an incomplete description.

High level view of what radvd provides

There are several ways of providing configuration to TCPIP. One way is using radvd (on Linux).

In a configuration file you give

  • An interface name
    • Which IP addresses (and address ranges) are available at the remote end of the connection.
    • Which IP addresses (and address ranges) are available at the local end of the connection

From the information about the local end, the remote end can build its routing tables.

Background IP routing

With IP you can have

  • static routing, where you explicitly give the routes to a destination – such as to get to a.b.c.d – go via xyz interface. You have to do a lot of administration defining the addresses of each end of an interface (think Ethernet cable)
  • dynamic routing, and neighbourhood discovery, where the system automatically finds the neighbours and there is less work for an administrator to do.

With IPv6 you have

  • global addresses with IP addresses like 2001:db8:1:16…..
  • link-local addresses. These are specific to an interface. An Ethernet connection can have many terminals hanging off ‘the bus’. The link-local address is only used within the connection. A different Ethernet cable can use the same address. There is no problem as the addresses are only used with the cable.
  • Neighbour Discovery. Rather than specify every thing as you do with static routing, IP V6 supports Neighbour Discovery, where each node can tell connected nodes, what routes and IP addresses it knows about. This is documented in the specification. This supports
    • Router Advertisement (RA), (“Hello, I’m a router, I know about the following addresses and routes”),
    • Router Solicitation (RS), (“Hello, I’ve just started up, are there any routers out there?”),
    • Neighbour Solicitation(NS) (“Does anyone have this IP address?”), and
    • Neighbour Advertisement (Usually in response to a Neighbour Solicitation, “I have this address”).

What is radvd?

The radvd program is a Router Advertisment Daemon (RADvd) which provides fakes router information – but is not a router.

You specify a configuration file. The syntax is defined here.

You can specify that this interface provides a “default” route. See here.

Example definition

I have a Linux Server, and a laptop running Linux connected by an Ethernet cable.

For the server, the radvd.conf file has 

# define the ethernet connection
interface  eno1
{
   AdvSendAdvert on; 
   MaxRtrAdvInterval 60;
   MinDelayBetweenRAs  60; 
   
   prefix 2001:ccc::/64 
   {
   #     AdvOnLink on;
   #     AdvAutonomous on;
     AdvRouterAddr on; 
   };
  
#   route 2001:ff99::/64
#   {
#   #   AdvRoutePreference medium;
#   # 3600 = 1 hour
#      AdvRouteLifetime 3600;
#   
#   };
};

The key information is

Comments

Data following #

The name of the interface

interface eno1{….};

prefix statement

prefix 2001:ccc::/64{…}

2001:ccc::/64 is the ipv6 address range or, to say it another way, ipv6 addresses with the left 64 bits beginning with 2001:0ccc:0000:0000. In IP V6 this is known as the prefix.

Basically this prefix statement means “this interface is a route to the specified prefix”.

This creates some addresses on the server for the connection.

eno1    inet6 2001:ccc::e02a:943b:3642:1d73/64 scope global temporary dynamic...       
eno1    inet6 2001:ccc::dbf:5c90:61a6:20ae/64 scope global dynamic mngtmpaddr...
eno1    inet6 2001:ccc::c48b:e8f1:495c:5b52/64 scope global temporary dynamic ...       
eno1    inet6 2001:ccc::2d8:61ff:fee9:312a/64 scope global dynamic mngtmpaddr ...

create routes

The prefix statement creates a route on the server to get to the laptop

2001:ccc::/64 dev eno1 proto ra metric 100 pref medium
2001:ccc::/64 dev eno1 proto kernel metric 256 ...

This says that on the server, if there is a request for an address in the range 2001:ccc::/64 send it via device eno1.

route 2001:ff99::/64 statement.

If present this passes information to the remote end of the connection, see below. It says “I (the server) know how to route to 2001:ff99::/64”.

It the routing address does not show up in any ip -6 commands on the server.

I could see the Route Information data flowing across the network within a Router Advertisment packet. But I could not see it being used at the remote end.

I think the best thing to do is ignore the route statement.

Polling

The radvd code periodically sends information along the connection to the other end, at an interval you specify. See radvdump below on how to display it.

At the other end of the connection…

At the remote (laptop) end of the connection, using WiresShark to display the data received, shows a Router Advertisement with

Internet Protocol Version 6, Src: fe80::a2f0:9936:ddfd:95fa, Dst: ff02::1
Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    ...
    ICMPv6 Option (Prefix information : 2001:ccc::/64)
    ICMPv6 Option (Route Information : Medium 2001:ff99::/64)
    ICMPv6 Option (Source link-layer address : 00:d8:61:e9:31:2a)

Where

  • fe80::a2f0:9936:ddfd:95fa is the link-local address on the server machine
  • ff02::1 the multicast address “All nodes” on a link (link-local scope)”
  • Prefix information : 2001:ccc::7/64 the prefix of the IP address 2001:ccc:0:0…
  • Route Information : Medium 2001:ff99::/64 This is from the “route” in the radvd configuration file on the Linux Server. It tells the laptop that this connection knows about routing to 2001:ccc::/64 .

From the route information it dynamically creates a route on the laptop to the server.

2001:ff99::/64 via fe80::a2f0:9936:ddfd:95fa dev enp0s31f6 proto ra metric 100 ...

This creates a route to 2001:ff99::/64 via the IP address fe80::….95fa, from the laptop end of the Ethernet connection with name enp0s31f6 to where-ever the connection goes to (in this example it goes to my server).

If I ping 2001:ffcc::9 from the laptop, it will use this route on its way to the z/OS server.

Connection to z/OS

Within the radvd.conf file is the definition to get to z/OS. This interface looks like an Ethernet (the ip -6 link command gives link/ether). This is for a different radvd configuration file to the earlier example.

interface  tap1
{
   AdvSendAdvert on; 
   MaxRtrAdvInterval 60;
   MinDelayBetweenRAs  60; 
   AdvManagedFlag  on;
   AdvOtherConfigFlag on;
   
   prefix 2001:db8:1::/64
   {
     AdvOnLink on;
     AdvAutonomous on;
     AdvRouterAddr on;
    
   };
   prefix 2001:db8:1::99/128
   {
     AdvOnLink on;
     AdvAutonomous on;
     AdvRouterAddr on;
    
   };

   route 2001:db8::/64
   {
     AdvRoutePreference medium;
     AdvRouteLifetime 3100;
   
   };
};

This says there are IP addresses in the range 2001:db8:1::/64 via this connection. z/OS reads the Router Advertisement and creates a route for them. TSO Netstat route gives

DestIP:   2001:db8:1::/64 
  Gw:     :: 
  Intf:   IFPORTCP6         Refcnt:  0000000000 
  Flgs:   UD                MTU:     9000

The explicit IP address, with the 128 to specify use the whole value, rather than just the prefix,

 prefix 2001:db8:1::99/128
   {
     AdvOnLink on;
     AdvAutonomous on;
     AdvRouterAddr on;    
   };

creates a route in z/OS; from TSO NETSTAT ROUTE

DestIP:   2001:db8:1::99/128 
  Gw:     :: 
  Intf:   IFPORTCP6         Refcnt:  0000000000 
  Flgs:   UHD               MTU:     9000

In UHD, the U says the interface is Up, the H says this is for a host (a specific end point), and the D says this is dynamically created.

If you try to ping 2001:db8:1::99 from the server, a Neighbour Solicitation request is sent from the server to z/OS, asking “do you have 2001:db8:1:99?”. My z/OS did not have that defined and so did not respond.

When I defined this address on z/OS TCPIP using the obeyfile

INTERFACE IFPORTCP6  DELETE 
INTERFACE IFPORTCP6 
    DEFINE IPAQENET6 
    CHPIDTYPE OSD 
    PORTNAME PORTCP 
    INTFID 7:7:7:7 
                                             
INTERFACE IFPORTCP6 
    ADDADDR 2001:DB8:1::9 
INTERFACE IFPORTCP6 
    ADDADDR 2001:DB8:9::9 
INTERFACE IFPORTCP6 
    ADDADDR 2001:DB8:1::99

(And used

  • v tcpip,,sto,ifportcp6
  • v tcpip,,obeyfile,USER.Z24C.TCPPARMS(IFACE61)
  • v tcpip,,sta,ifportcp6

to activate it)

After this, the ping was successful because there was a neighbour solicitation for 2001:db8:1::99, and z/OS replied with Neighbour Advertisement of 2001:db8:1::9 saying I have it.

Create a default route

If you specify AdvDefaultLifetime 0 on the interface, this indicates that the router is not a default router and should not appear on the default router list in the Router Advertiser broadcasts. If the value is non zero, the recipient, can use this connection as a default route, for example there is no existing default route. A statically defined default will be used in preference to a dynamically defined one.

Use radvdump to display what it sent in the RA

You can use sudo radvdump to display what is being sent in the Router Advertiser message broadcast over multicast. It looks just like the radvd.conf file, but with all of the values filled in, so you can see the defaults.

How do I see what’s changing in my Linux network configuration?

I was trying to find out what changes were being made to my IP V6 network.

The short answer is the command

sudo nmcli general logging level DEBUG domains ALL

and reset it to the default with

sudo nmcli general logging level INFO domains ALL

You see the last 5 minutes worth of trace using

journalctl -u NetworkManager -S -5m >nw.txt

See man NetworkManager.conf.

Debian Man gives slightly different information including

ALL : all log domains
DEFAULT : default log domains
DHCP : shortcut for “DHCP4,DHCP6”
IP : shortcut for “IP4,IP6”

ADSL : ADSL device operations
AGENTS : Secret agents operations and communication
AUDIT : Audit records
AUTOIP4 : AutoIP operations
BOND : Bonding operations
BRIDGE : Bridging operations
BT : Bluetooth operations
CONCHECK : Connectivity check
CORE : Core daemon and policy operations
DBUS_PROPS : D-Bus property changes
DCB : Data Center Bridging (DCB) operations
DEVICE : Activation and general interface operations
DHCP4 : DHCP for IPv4
DHCP6 : DHCP for IPv6
DISPATCH : Dispatcher scripts
DNS : Domain Name System related operations
ETHER : Ethernet device operations
FIREWALL : FirewallD related operations
INFINIBAND : InfiniBand device operations
IP4 : IPv4-related operations
IP6 : IPv6-related operations
MB : Mobile broadband operations
NONE : when given by itself logging is disabled
OLPC : OLPC Mesh device operations
PLATFORM : OS (platform) operations
PPP : Point-to-point protocol operations
PROXY : logging messages for proxy handling
RFKILL : RFKill subsystem operations
SETTINGS : Settings/config service operations
SHARING : Connection sharing. With TRACE level log queries for dnsmasq instance
SUPPLICANT : WPA supplicant related operations
SUSPEND : Suspend/resume
SYSTEMD : Messages from internal libsystemd
TEAM : Teaming operations
VLAN : VLAN operations
VPN : Virtual Private Network connections and operations
VPN_PLUGIN : logging messages from VPN plugins
WIFI : Wi-Fi device operations
WIFI_SCAN : Wi-Fi scanning operations
WIMAX : WiMAX device operations

it produces data like

<debug> [747.1872] ndisc-lndp[...,"eno1"]: received router advertisement at 17121
<debug> ndisc[...,"eno1"]: scheduling next now/lifetime check: 163 seconds
<debug> ndisc[...,"eno1"]: neighbor discovery configuration changed [GAR]:
<debug> ndisc[...,"eno1"]:   dhcp-level none
<debug> ndisc[...,"eno1"]:   hop limit      : 64
<debug> ndisc[...,"eno1"]:   gateway fe80::a2f0:9936:ddfd:95fa pref medium exp 179.8153
<debug> ndisc[...,"eno1"]:   gateway fe80::9b07:33a1:aa30:e272 pref medium exp 162.8153
<debug> ndisc[...,"eno1"]:   address 2001:bbb::573e:5c69:2ab3:4ae6 exp 81751.8153
<debug> ndisc[...,"eno1"]:   address 2001:ccc::dbf:5c90:61a6:20ae exp 86399.8153
<debug> ndisc[...,"eno1"]:   route 2001:db8::99/128 via :: pref medium exp 86399.8153
<debug> ndisc[...,"eno1"]:   route 2001:ff99::/64 via fe80::a2f0:9936:ddfd:95fa pref medium exp 3099.8153
<debug> ndisc[...,"eno1"]:   route 2001:ccc::/64 via :: pref medium exp 86399.8153
<debug> ndisc[...,"eno1"]:   route 2001:bbb::/64 via :: pref medium exp 81751.8153
<debug>  platform: (eno1) route: append     IPv6 route: 2001:db8::/80 via :: dev 2 metric 100 mss 0 rt-src ndisc
<debug>  platform: (eno1) signal: route   6   added: 2001:db8::/80 via :: dev 2 metric 100 mss 0 rt-src rt-ra