Backing up on Linux is easy – how do I restore?

I’ve been using deja-dup front end to duplicity to backup my Ubuntu Linux. This has worked fine. The only problems I had were trying to restore files I had not backed up (/etc….) – a Defective End User problem.

Restoring one of these files was a little challenge, with bumps in the road… here’s how I did it. I’ve listed the errors I received while using duplicity.

A high level thought – should you change the repository name every year – such as including the year?

Where am I backing up to (where do I restore from)?

Use the deja-dup application (use the Linux show applications and find backup).

Under Storage location will be location and folder.

If this is an external drive it may be mounted under /mnt/… or /media/.

High level view

duplicity collection-status file:///media/colinpaice/UbuntuBackup/home/Backup2023

gave

Last full backup date: Sun Jan  1 17:02:48 2023
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive directory: /home/colinpaice/.cache/duplicity/52b5c01605d52d0976f1434440a7c11a

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Sun Jan  1 17:02:48 2023
Chain end time: Tue Jan 10 08:19:05 2023
Number of contained backup sets: 10
Total number of contained volumes: 250
 Type of backup set:                            Time:   Number of volumes:
                Full         Sun Jan  1 17:02:48 2023               238
         Incremental         Mon Jan  2 08:25:38 2023                 1
         Incremental         Tue Jan  3 08:50:23 2023                 1
         Incremental         Wed Jan  4 08:53:01 2023                 1
         Incremental         Thu Jan  5 08:22:41 2023                 1
         Incremental         Fri Jan  6 08:36:31 2023                 4
         Incremental         Sat Jan  7 10:23:00 2023                 1
         Incremental         Sun Jan  8 12:16:58 2023                 1
         Incremental         Mon Jan  9 09:02:31 2023                 1
         Incremental         Tue Jan 10 08:19:05 2023                 1
-------------------------
No orphaned or incomplete backup sets found.

List files in the backup

duplicity list-current-files file:///media/colinpaice/UbuntuBackup/home/Backup2023/

I saved the list of files using

duplicity list-current-files file:///media/colinpaice/UbuntuBackup/home/Backup2023 >dup.lst
Synchronising remote metadata to local cache…
GnuPG passphrase for decryption:

This took over a minute (it has to check thousands of files names). I think it scans every “signature” file to build a list. The more signature files – the longer it takes.

Inside dup.lst it had entries like

Tue Aug 14 11:41:33 2018 etc/NetworkManager/system-connections/LG-D855_2292

This says the backed up file was from the given date in 2018.

List files available on a given date

I used

duplicity list-current-files –time 5D file:///media/colinpaice/UbuntuBackup/home/Backup2023 > aa.lst

to get a list of files available 5 days ago.

Restore the file

duplicity restore -t 3D –file-to-restore etc/NetworkManager/system-connections/LG-D855_2292 file:///media/colinpaice/UbuntuBackup/home/Backup2023 /home/colinpaice/secret/abc

where

  • -t ( or –time) specify a time period
  • 3D for three days ago. Other options s, m, h, D, W, M, or Y. 1M3d is 33 days ago.
  • etc/Network…. this is the file I wanted to restore. Note no leading / .
  • file:///media…/home/Backup2023 from this backup location.
  • /home/colinpaice/secret/abc and put the file here. The file/directory must not exist.

Restore a directory

Specify the folder (ending in /) and specify a target directory (/home/colinpaice/secret/abcd). It will create the directory if needed.

If you specify a name like etc/…/* it will report it was not found in archive.

duplicity restore -t 3D –file-to-restore etc/NetworkManager/system-connections/ file:///media/colinpaice/UbuntuBackup/home/Backup2023 /home/colinpaice/secret/abcd

Error messages

gpg: WARNING: unsafe permissions on homedir

This says the contents of your secret directory are publicly available. Change the permissions.

chmod 600 /home/colinpaice/.gnupg/*
chmod 700 /home/colinpaice/.gnupg/

Local and Remote metadata are synchronized, no sync needed.

This is OK. It means the metadata is the same in the local cache as in the remote store.

duplicity.dup_collections.CollectionsError: No signature chains found

You have the wrong backup repository name,

  • it needs ///
  • ls should give a filename like /media/colinpaice/UbuntuBackup/…./duplicity-new-signatures.20230101T170248Z.to.20230102T082538Z.sigtar.gpg

Error ‘[Errno 1] Operation not permitted: b’/home/colinpaice/secret/abc” processing .

After the restore, duplicity was not able to use chown on the file to restore it to the original owner.

GPGError: GPG Failed, see log below:

===== Begin GnuPG log =====
gpg: AES256 encrypted data
gpg: gcry_kdf_derive failed: Invalid data
gpg: encrypted with 1 passphrase
gpg: decryption failed: No secret key
===== End GnuPG log =====

I got this asking for a date before backups were taken. See

duplicity collection-status file:///media/colinpaice/UbuntuBackup/home/Backup2023

I was looking for a backup in 2002 – which was not in the 2003 backup.

What does tso netstat neighbour give you?

The command TSO NETSTAT ND gave me

Query Neighbor cache for 2001:db8:1:0:8024:bff:fe45:840c 
  IntfName: IFPORTCP6          IntfType: IPAQENET6 
  LinkLayerAddr: 82240B45840C  State: Reachable 
  Type: Router                 AdvDfltRtr: No 

Query Neighbor cache for fe80::8024:bff:fe45:840c 
  IntfName: IFPORTCP6          IntfType: IPAQENET6 
  LinkLayerAddr: 82240B45840C  State: Reachable 
  Type: Router                 AdvDfltRtr: No 

Query Neighbor cache for fe80::9863:1eff:fe13:1408 
  IntfName: JFPORTCP6          IntfType: IPAQENET6 
  LinkLayerAddr: 9A631E131408  State: Reachable 
  Type: Router                 AdvDfltRtr: No 

On Linux the

ip -6 addr

command gave me

tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 1000
    inet6 2001:db8:1:0:b0fd:f92b:8362:577b/64 ...
    inet6 2001:db8:1:0:8024:bff:fe45:840c/64 ...
    inet6 fe80::8024:bff:fe45:840c/64 ...

tap2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 1000
    inet6 fe80::9863:1eff:fe13:1408/64 ...

The TSO output means

  • Query Neighbor cache for 2001:db8:1:0:8024:bff:fe45:840c. The address is one of the addresses on the remote end of the connection. There is an entry because some traffic came via the address.
  • IntfName: IFPORTCP6 The z/OS Interface name used to create the defintion
  • IntfType: IPAQENET6 the OSA-Express QDIO interfaces statement
  • LinkLayerAddr: 82240B45840C
  • State: Reachable Other options can include stale, which means z/OS has not heard anything from this address for a while
  • Type: Router
  • AdvDfltRtr: No. The information passed in the Router Advertisement, said this was connection does not Advertise a Default Router(AdvDfltRtr).

From the NETSTAT ND output we can see data has been received from

  • IFPORTCP6:2001:db8:1:0:8024:bff:fe45:840c
  • IFPORTCP6:fe80::8024:bff:fe45:840c
  • JFPORTCP6:fe80::9863:1eff:fe13:1408

To get data to flow down the 2001…. address I had to use

ping -I 2001:db8:1:0:8024:bff:fe45:840c 2001:db8:1::9

Where the -I says use the interface address.

You can get information about bytes processed by interface (not by address) using the TSO NETSTAT DEVLINKS command.

Understanding radvd with IPV6 on Linux.

My two day project to deploy IP V6 dynamic routing, turned into an eight week project before I got it to work.

I am documenting a lot of what I learned – today’s experience is understanding what radvd is and what the configuration options mean. I found a lot of documentation – most of which either assumed you know a lot, or only provided an incomplete description.

High level view of what radvd provides

There are several ways of providing configuration to TCPIP. One way is using radvd (on Linux).

In a configuration file you give

  • An interface name
    • Which IP addresses (and address ranges) are available at the remote end of the connection.
    • Which IP addresses (and address ranges) are available at the local end of the connection

From the information about the local end, the remote end can build its routing tables.

Background IP routing

With IP you can have

  • static routing, where you explicitly give the routes to a destination – such as to get to a.b.c.d – go via xyz interface. You have to do a lot of administration defining the addresses of each end of an interface (think Ethernet cable)
  • dynamic routing, and neighbourhood discovery, where the system automatically finds the neighbours and there is less work for an administrator to do.

With IPv6 you have

  • global addresses with IP addresses like 2001:db8:1:16…..
  • link-local addresses. These are specific to an interface. An Ethernet connection can have many terminals hanging off ‘the bus’. The link-local address is only used within the connection. A different Ethernet cable can use the same address. There is no problem as the addresses are only used with the cable.
  • Neighbour Discovery. Rather than specify every thing as you do with static routing, IP V6 supports Neighbour Discovery, where each node can tell connected nodes, what routes and IP addresses it knows about. This is documented in the specification. This supports
    • Router Advertisement (RA), (“Hello, I’m a router, I know about the following addresses and routes”),
    • Router Solicitation (RS), (“Hello, I’ve just started up, are there any routers out there?”),
    • Neighbour Solicitation(NS) (“Does anyone have this IP address?”), and
    • Neighbour Advertisement (Usually in response to a Neighbour Solicitation, “I have this address”).

What is radvd?

The radvd program is a Router Advertisment Daemon (RADvd) which provides fakes router information – but is not a router.

You specify a configuration file. The syntax is defined here.

You can specify that this interface provides a “default” route. See here.

Example definition

I have a Linux Server, and a laptop running Linux connected by an Ethernet cable.

For the server, the radvd.conf file has

# define the ethernet connection
interface eno1
{
AdvSendAdvert on;
MaxRtrAdvInterval 60;
MinDelayBetweenRAs 60;

prefix 2001:ccc::/64
{
# AdvOnLink on;
# AdvAutonomous on;
AdvRouterAddr on;
};

# route 2001:ff99::/64
# {
# # AdvRoutePreference medium;
# # 3600 = 1 hour
# AdvRouteLifetime 3600;
#
# };
};

The key information is

Comments

Data following #

The name of the interface

interface eno1{….};

prefix statement

prefix 2001:ccc::/64{…}

2001:ccc::/64 is the ipv6 address range or, to say it another way, ipv6 addresses with the left 64 bits beginning with 2001:0ccc:0000:0000. In IP V6 this is known as the prefix.

Basically this prefix statement means “this interface is a route to the specified prefix”.

This creates some addresses on the server for the connection.

eno1    inet6 2001:ccc::e02a:943b:3642:1d73/64 scope global temporary dynamic...       
eno1    inet6 2001:ccc::dbf:5c90:61a6:20ae/64 scope global dynamic mngtmpaddr...
eno1    inet6 2001:ccc::c48b:e8f1:495c:5b52/64 scope global temporary dynamic ...       
eno1    inet6 2001:ccc::2d8:61ff:fee9:312a/64 scope global dynamic mngtmpaddr ...

create routes

The prefix statement creates a route on the server to get to the laptop

2001:ccc::/64 dev eno1 proto ra metric 100 pref medium
2001:ccc::/64 dev eno1 proto kernel metric 256 ...

This says that on the server, if there is a request for an address in the range 2001:ccc::/64 send it via device eno1.

route 2001:ff99::/64 statement.

If present this passes information to the remote end of the connection, see below. It says “I (the server) know how to route to 2001:ff99::/64”.

It the routing address does not show up in any ip -6 commands on the server.

I could see the Route Information data flowing across the network within a Router Advertisment packet. But I could not see it being used at the remote end.

I think the best thing to do is ignore the route statement.

Polling

The radvd code periodically sends information along the connection to the other end, at an interval you specify. See radvdump below on how to display it.

At the other end of the connection…

At the remote (laptop) end of the connection, using WiresShark to display the data received, shows a Router Advertisement with

Internet Protocol Version 6, Src: fe80::a2f0:9936:ddfd:95fa, Dst: ff02::1
Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    ...
    ICMPv6 Option (Prefix information : 2001:ccc::/64)
    ICMPv6 Option (Route Information : Medium 2001:ff99::/64)
    ICMPv6 Option (Source link-layer address : 00:d8:61:e9:31:2a)

Where

  • fe80::a2f0:9936:ddfd:95fa is the link-local address on the server machine
  • ff02::1 the multicast address “All nodes” on a link (link-local scope)”
  • Prefix information : 2001:ccc::7/64 the prefix of the IP address 2001:ccc:0:0…
  • Route Information : Medium 2001:ff99::/64 This is from the “route” in the radvd configuration file on the Linux Server. It tells the laptop that this connection knows about routing to 2001:ccc::/64 .

From the route information it dynamically creates a route on the laptop to the server.

2001:ff99::/64 via fe80::a2f0:9936:ddfd:95fa dev enp0s31f6 proto ra metric 100 ...

This creates a route to 2001:ff99::/64 via the IP address fe80::….95fa, from the laptop end of the Ethernet connection with name enp0s31f6 to where-ever the connection goes to (in this example it goes to my server).

If I ping 2001:ffcc::9 from the laptop, it will use this route on its way to the z/OS server.

Connection to z/OS

Within the radvd.conf file is the definition to get to z/OS. This interface looks like an Ethernet (the ip -6 link command gives link/ether). This is for a different radvd configuration file to the earlier example.

interface  tap1
{
   AdvSendAdvert on; 
   MaxRtrAdvInterval 60;
   MinDelayBetweenRAs  60; 
   AdvManagedFlag  on;
   AdvOtherConfigFlag on;
   
   prefix 2001:db8:1::/64
   {
     AdvOnLink on;
     AdvAutonomous on;
     AdvRouterAddr on;
    
   };
   prefix 2001:db8:1::99/128
   {
     AdvOnLink on;
     AdvAutonomous on;
     AdvRouterAddr on;
    
   };

   route 2001:db8::/64
   {
     AdvRoutePreference medium;
     AdvRouteLifetime 3100;
   
   };
};

This says there are IP addresses in the range 2001:db8:1::/64 via this connection. z/OS reads the Router Advertisement and creates a route for them. TSO Netstat route gives

DestIP:   2001:db8:1::/64 
  Gw:     :: 
  Intf:   IFPORTCP6         Refcnt:  0000000000 
  Flgs:   UD                MTU:     9000 

The explicit IP address, with the 128 to specify use the whole value, rather than just the prefix,

 prefix 2001:db8:1::99/128
   {
     AdvOnLink on;
     AdvAutonomous on;
     AdvRouterAddr on;    
   };

creates a route in z/OS; from TSO NETSTAT ROUTE

DestIP:   2001:db8:1::99/128 
  Gw:     :: 
  Intf:   IFPORTCP6         Refcnt:  0000000000 
  Flgs:   UHD               MTU:     9000 

In UHD, the U says the interface is Up, the H says this is for a host (a specific end point), and the D says this is dynamically created.

If you try to ping 2001:db8:1::99 from the server, a Neighbour Solicitation request is sent from the server to z/OS, asking “do you have 2001:db8:1:99?”. My z/OS did not have that defined and so did not respond.

When I defined this address on z/OS TCPIP using the obeyfile

INTERFACE IFPORTCP6  DELETE 
INTERFACE IFPORTCP6 
    DEFINE IPAQENET6 
    CHPIDTYPE OSD 
    PORTNAME PORTCP 
    INTFID 7:7:7:7 
                                             
INTERFACE IFPORTCP6 
    ADDADDR 2001:DB8:1::9 
INTERFACE IFPORTCP6 
    ADDADDR 2001:DB8:9::9 
INTERFACE IFPORTCP6 
    ADDADDR 2001:DB8:1::99 

(And used

  • v tcpip,,sto,ifportcp6
  • v tcpip,,obeyfile,USER.Z24C.TCPPARMS(IFACE61)
  • v tcpip,,sta,ifportcp6

to activate it)

After this, the ping was successful because there was a neighbour solicitation for 2001:db8:1::99, and z/OS replied with Neighbour Advertisement of 2001:db8:1::9 saying I have it.

Create a default route

If you specify AdvDefaultLifetime 0 on the interface, this indicates that the router is not a default router and should not appear on the default router list in the Router Advertiser broadcasts. If the value is non zero, the recipient, can use this connection as a default route, for example there is no existing default route. A statically defined default will be used in preference to a dynamically defined one.

Use radvdump to display what it sent in the RA

You can use sudo radvdump to display what is being sent in the Router Advertiser message broadcast over multicast. It looks just like the radvd.conf file, but with all of the values filled in, so you can see the defaults.

How do I see what’s changing in my Linux network configuration?

I was trying to find out what changes were being made to my IP V6 network.

The short answer is the command

sudo nmcli general logging level DEBUG domains ALL

and reset it to the default with

sudo nmcli general logging level INFO domains ALL

You see the last 5 minutes worth of trace using

journalctl -u NetworkManager -S -5m >nw.txt

See man NetworkManager.conf.

Debian Man gives slightly different information including

ALL : all log domains
DEFAULT : default log domains
DHCP : shortcut for “DHCP4,DHCP6”
IP : shortcut for “IP4,IP6”

ADSL : ADSL device operations
AGENTS : Secret agents operations and communication
AUDIT : Audit records
AUTOIP4 : AutoIP operations
BOND : Bonding operations
BRIDGE : Bridging operations
BT : Bluetooth operations
CONCHECK : Connectivity check
CORE : Core daemon and policy operations
DBUS_PROPS : D-Bus property changes
DCB : Data Center Bridging (DCB) operations
DEVICE : Activation and general interface operations
DHCP4 : DHCP for IPv4
DHCP6 : DHCP for IPv6
DISPATCH : Dispatcher scripts
DNS : Domain Name System related operations
ETHER : Ethernet device operations
FIREWALL : FirewallD related operations
INFINIBAND : InfiniBand device operations
IP4 : IPv4-related operations
IP6 : IPv6-related operations
MB : Mobile broadband operations
NONE : when given by itself logging is disabled
OLPC : OLPC Mesh device operations
PLATFORM : OS (platform) operations
PPP : Point-to-point protocol operations
PROXY : logging messages for proxy handling
RFKILL : RFKill subsystem operations
SETTINGS : Settings/config service operations
SHARING : Connection sharing. With TRACE level log queries for dnsmasq instance
SUPPLICANT : WPA supplicant related operations
SUSPEND : Suspend/resume
SYSTEMD : Messages from internal libsystemd
TEAM : Teaming operations
VLAN : VLAN operations
VPN : Virtual Private Network connections and operations
VPN_PLUGIN : logging messages from VPN plugins
WIFI : Wi-Fi device operations
WIFI_SCAN : Wi-Fi scanning operations
WIMAX : WiMAX device operations

it produces data like

<debug> [747.1872] ndisc-lndp[...,"eno1"]: received router advertisement at 17121
<debug> ndisc[...,"eno1"]: scheduling next now/lifetime check: 163 seconds
<debug> ndisc[...,"eno1"]: neighbor discovery configuration changed [GAR]:
<debug> ndisc[...,"eno1"]:   dhcp-level none
<debug> ndisc[...,"eno1"]:   hop limit      : 64
<debug> ndisc[...,"eno1"]:   gateway fe80::a2f0:9936:ddfd:95fa pref medium exp 179.8153
<debug> ndisc[...,"eno1"]:   gateway fe80::9b07:33a1:aa30:e272 pref medium exp 162.8153
<debug> ndisc[...,"eno1"]:   address 2001:bbb::573e:5c69:2ab3:4ae6 exp 81751.8153
<debug> ndisc[...,"eno1"]:   address 2001:ccc::dbf:5c90:61a6:20ae exp 86399.8153
<debug> ndisc[...,"eno1"]:   route 2001:db8::99/128 via :: pref medium exp 86399.8153
<debug> ndisc[...,"eno1"]:   route 2001:ff99::/64 via fe80::a2f0:9936:ddfd:95fa pref medium exp 3099.8153
<debug> ndisc[...,"eno1"]:   route 2001:ccc::/64 via :: pref medium exp 86399.8153
<debug> ndisc[...,"eno1"]:   route 2001:bbb::/64 via :: pref medium exp 81751.8153
<debug>  platform: (eno1) route: append     IPv6 route: 2001:db8::/80 via :: dev 2 metric 100 mss 0 rt-src ndisc
<debug>  platform: (eno1) signal: route   6   added: 2001:db8::/80 via :: dev 2 metric 100 mss 0 rt-src rt-ra

Is “via” needed when creating a Linux IP route?

To get static routing working I needed a route like one of

# specific destination
sudo ip -6 route add fc:1::9/128 via fc::2 dev enp0s31f6r
sudo ip -6 route add fc:1::9/128  via fc::2 
#range of addresses
sudo ip -6 route add fc:1::/64 via fc::2  dev enp0s31f6
sudo ip -6 route add fc:1::/64 via fc::2 

If I a route without the via

sudo ip -6 route add fc:1::9/128 dev enp0s31f6

then it ignored my static routing and did Neighbor Solicitation; it asked adjacent systems if they had knew about the IP address fc:1::9. This is an IP V6 Neighbour Discovery facility.

There were hints around the internet that if the next hop address is not specified, then the “next hop router” will try to locate the passed address.

So the short answer to the question is: “yes. You should specify it when using static routing”.

Why Linux is not responding – it’s the flaming file-wall!

I could not ping my Linux Server, and could not issue a traceroute command. It turns out the firewall was blocking the traceroute flow.

This blog posts describes how I checked this, and fixed the firewall problem.

Traceroute sends (by default) a UDP packet to a port address in the range 33434-33523. It usually responds with a “timed out” type response. If there is no response then there is a good chance that the packet is being dropped by a firewall.

See Understanding traceroute (or tracerte).

Using wireshark I could see UDP packets going in to my Linux, but there was no corresponding reply being returned.

When traceroute worked I got the out inbound UDP packet, and the outbound response with “destination unreachable” (which looks like a problem but actually shows normal behaviour) as shown in the data below. Wireshark highlights it with a black background, because it thinks it is a problem.

SourceDestinationDst PortportProtocolInfo
2001:db8::22001:db8::73343452119UDP52119 → 33434 Len=24
2001:db8::7 2001:db8::7 33434 52119 ICMPV6 Destination Unreachable (Port unreachable)

When traceroute failed I only got the inbound UDP packet

SourceDestinationDst PortportProtocolInfo
2001:db8::22001:db8::73343452119UDP52119 → 33434 Len=24

If the packets is blocked by a firewall, then the traceroute output will have “*” as the node name.

Useful Fire Wall (ufw) is documented here.

I was on Ubuntu Linux 20.04.

Display the status of the firewall

sudo ufw status verbose

This gave me

Status: active
Logging: off (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
Anywhere                   ALLOW IN    10.1.1.2                  
22/tcp                     ALLOW IN    Anywhere                  
20,21,10000:10100/tcp      ALLOW IN    Anywhere                  
21/tcp                     ALLOW IN    Anywhere                  
20/tcp                     ALLOW IN    Anywhere                  
22/tcp (v6)                ALLOW IN    Anywhere (v6)             
20,21,10000:10100/tcp (v6) ALLOW IN    Anywhere (v6)             
21/tcp (v6)                ALLOW IN    Anywhere (v6)             
20/tcp (v6)                ALLOW IN    Anywhere (v6)         

By default,

  • incoming data is blocked
  • outbound data is allowed
  • routed data is blocked.

Logging is off, and problems are not reported.

The displays shows there are no rules for UDP – so any incoming UDP request is blocked (quietly dropped = dropped without telling anyone).

You may want to issue the command and pipe the output to a file, ufw.txt, to keep a record of the status before you make any changes. If you make any changes, they persist – even across reboot.

Enable logging to see what is being blocked

sudo ufw logging on

Rerun your traceroute or command.

At the bottom of /var/log/ufw I had (this has been reformatted to make it display better)

Nov 28 12:27:43 colinpaice kernel: [ 3317.641508] [UFW BLOCK] IN=enp0s31f6 OUT= MAC=8c:16:45:36:f4:8a:00:d8:61:e9:31:2a:86:dd SRC=2001:0db8:0000:0000:0000:0000:0000:0002 DST=2001:0db8:0000:0000:0000:0000:0000:0007
LEN=80
TC=0
HOPLIMIT=1
FLOWLBL=924186
PROTO=UDP
SPT=48582
DPT=33434
LEN=40

Wireshark gave me

Frame 4: 94 bytes on wire (752 bits), 94 bytes captured (752 bits) on interface enp0s31f6, id 0   
Ethernet II, Src: Micro-St_e9:31:2a (00:d8:61:e9:31:2a), Dst: LCFCHeFe_36:f4:8a (8c:16:45:36:f4:8a)
Internet Protocol Version 6, Src: 2001:db8::2, Dst: 2001:db8::7
    0110 .... = Version: 6
    .... 0000 0000 .... .... .... .... .... = Traffic Class: 0x00 
    .... .... .... 1110 0001 1010 0001 1010 = Flow Label: 0xe1a1a
    Payload Length: 40
    Next Header: UDP (17)
    Hop Limit: 1
    Source: 2001:db8::2
    Destination: 2001:db8::7
User Datagram Protocol, Src Port: 48582, Dst Port: 33434
    Source Port: 48582
    Destination Port: 33434
    Length: 40
    Checksum: 0x6ebd [unverified]
    [Checksum Status: Unverified]
    [Stream index: 0]
    [Timestamps]
Data (32 bytes)

From this, we can see the fields match up

  • flow label (0xe1a1a = 924186)
  • source 2001:db8::2 = 2001:0db8:0000:0000:0000:0000:0000:0002
  • destination 2001:db8::7 = 2001:0db8:0000:0000:0000:0000:0000:0007
  • source port 48582
  • destination port 33434.

Port 33434 is used by traceroute, so this is a good clue this is a traceroute packet.

The reason the record was written to the log is [UFW BLOCK]. The firewall blocked it.

The request came in over interface enp0s31f6.

How to enable it.

You can specify different filters, and granularity of parameters.

For example

  • sudo ufw rule allow log proto udp from 2001:db8::2
  • sudo ufw rule allow in on enp0s31f6 log comment ‘colin-ethernet’
  • sudo ufw rule allow proto udp to 2001:db8::7 port 33434:33523 from 2001:db8::2

Where enp0s31f6 is the name of the ethernet link where the traffic comes from.

When running with either of these, I had in the log file

Nov 28 17:03:12 colinpaice kernel: [19847.112045] 
[UFW ALLOW] 
IN=enp0s31f6 OUT= MAC=8c:16:45:36:f4:8a:00:d8:61:e9:31:2a:86:dd SRC=2001:0db8:0001:0000:0000:0000:0000:0009 
DST=2001:0db8:0000:0000:0000:0000:0000:0007 
LEN=60 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=UDP SPT=33434 DPT=33440 LEN=20

and the traceroute worked.

Note: The comment ‘…’ is an administration aid to give a description. It does not come out in the logs.

Display the rules

sudo ufw status numbered

gave

Status: active

     To                         Action      From
     --                         ------      ----
[ 1] Anywhere                   ALLOW IN    10.1.1.2                  
[ 2] 22/tcp                     ALLOW IN    Anywhere                  
[ 3] 20,21,10000:10100/tcp      ALLOW IN    Anywhere                  
[ 4] 21/tcp                     ALLOW IN    Anywhere                  
[ 5] 20/tcp                     ALLOW IN    Anywhere                  
[ 6] Anywhere on enp0s31f6      ALLOW IN    Anywhere                   (log)
[ 7] 22/tcp (v6)                ALLOW IN    Anywhere (v6)             
[ 8] 20,21,10000:10100/tcp (v6) ALLOW IN    Anywhere (v6)             
[ 9] 21/tcp (v6)                ALLOW IN    Anywhere (v6)             
[10] 20/tcp (v6)                ALLOW IN    Anywhere (v6)             
[11] Anywhere (v6) on enp0s31f6 ALLOW IN    Anywhere (v6)              (log)
[12] 2001:db8::7 33434/udp      ALLOW IN    2001:db8::2                (log)

There is now a rule [12] for udp to 2001:db8::7 port 33434

You can use commands like

sudo ufw delete 6

to delete a row.

Note: Always display before delete. Having deleted the rule 6 – rule 7 now becomes rule 6, etc.

Now that it works…

Any changes to ufw are remembered across reboots.

You may want to turn off the logging, until the next problem

sudo ufw logging off

and remove the log from the fire wall rules, by deleting and re-adding the rule.

sudo ufw rule delete allow log proto udp from 2001:db8::2

sudo ufw rule allow proto udp from 2001:db8::2

Why is Ubuntu is running out of space? It is /var/log/journal/…

Low Disk space on “Filesystem root”

I’ve been getting this message more frequently – and I’ve found out why.

It could be

  • the system journal file
  • snap files in the cache
  • stuff in /tmp

You may get this message during installation of a large set of packages. Packages get unpacked into a temporary file – which is deleted afterwards, so you get a temporary hump in usage.

System journal file

There is a “systemd journal file” with content like

Jul 12 15:50:09 colinpaice rtkit-daemon[1385]: Successfully made thread 2682 of process 2540 owned by ‘1000’ RT at priority 10.
Jul 12 16:44:41 colinpaice rtkit-daemon[1385]: Supervising 5 threads of 3 processes of 1 users.
Jul 12 16:45:01 colinpaice CRON[7075]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 12 16:45:01 colinpaice CRON[7076]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 12 16:45:01 colinpaice CRON[7075]: pam_unix(cron:session): session closed for user root
Jul 12 15:58:32 colinpaice kernel: irq_thread+0xda/0x170

This goes back to when I first installed Ubuntu about 4 years ago, but I think a month’s worth of data would be enough.

You can display the disk space used by using

sudo journalctl –disk-usage

and display the contents of the file using

sudo journalctl -n 50 |less

Note: Without sudo you get the userid’s log size… with sudo you get total log size.

The log file is in /var/log/journal/ and was 1.4 GB in size. The size of this file is controlled by the /etc/systemd/journald.conf configuration file. I edited this file (using sudo gedit /etc/systemd.journald.conf).

  • I uncommented SystemMaxFileSize and gave it a value of 500M.
  • I uncommented SystemMaxFiles and gave it a value of 10

You can either reboot, or use

service systemd-journald restart

to restart the systemd journal.

Although I set the value to 500M, after the journal was restarted – it had size 100MB!

I think 100MB is plenty big enough, and I get a log of disk space back.

Snap files in the cache

sudo du -hs /var/lib/snapd/cache/

gives you the space used.

I then used

sudo rm -r /var/lib/snapd/cache/

Other rubbish

The disk usage analyser gives you a picture of all the space on a file system. Click on “Show Applications” and select Disk Usage Analyser

Why is this Linux slower to download than that one

I have a laptop which is my primary work station, and an under desk server for running my z/OS system on top of the same Linux.

Running “apt update” on the laptop was always faster on the laptop compared to the server. Was this because all traffic for the server was going through my laptop? How do I tell?

The boxes are connected with an Ethernet cable, I had to purchase a wireless dongle for my server, my laptop has a built in wireless adapter.

The linux ifconfig or the ip command gives information about the configuration. For example ip a

eno1: flags=4163 mtu 1500
    inet 10.1.0.3 netmask 255.255.255.0 broadcast 10.1.0.255
    inet6 fe80::.... prefixlen 64 scopeid 0x20
    ether 00:... txqueuelen 1000 (Ethernet)
    RX packets 5136 bytes 1445665 (1.4 MB)
    RX errors 0 dropped 0 overruns 0 frame 0
    TX packets 4933 bytes 1692274 (1.6 MB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
    device interrupt 17 memory 0xb1200000-b1220000  
...
wlxd037450ab7ac: flags=4163 mtu 1500
    inet 192.... netmask 255.255.255.0 broadcast 192....
    inet6 2a00:... prefixlen 64 scopeid 0x0
    inet6 fe80::... prefixlen 64 scopeid 0x20
    inet6 2a00:... prefixlen 64 scopeid 0x0
    ether d0:... txqueuelen 1000 (Ethernet)
    RX packets 42427 bytes 60919847 (60.9 MB)
    RX errors 0 dropped 1 overruns 0 frame 0
    TX packets 25996 bytes 2397812 (2.3 MB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
  • The EtherNet connection eno1 has received 5136 packets and 1.4 MB of data
  • The WireLess connection wlx037450ab7ac has received 42427 packets and 60.9 MB of data.

As I had just done an apt upgrade, the wireless had all of the traffic to download the files, so the traffic was not coming through my laptop.

Once the system was updated, the only traffic flowing was down the Ethernet cable as I used the server from my laptop.

Ping

A ping from each system gave a similar response time.

traceroute

traceroute shows you the hops to the destination.

For example

traceroute abc.xyz.com

To specify the interface you need to run as a superuser.

sudo traceroute abc.xyz.com -i wlxd037450ab7ac

gave

1 bthub.home (192....) 4.654 ms 38.438 ms 38.425 ms
2 * * *
3 * * *
4 31.55.185.184 (31.55.185.184) 75.897 ms 75.890 ms 75.861 ms

If you are not running as a superuser you will get:

setsockopt SO_BINDTODEVICE: Operation not permitted

What else is there to help me?

On z/OS the netstat command gives a lot of information about the session, for example the send window size, the receive windows size etc. This information tends not to be available on other platforms.

On linux there is the ss (Socket Statistics) command.

Example output from ss -t -i included

  • ESTAB 0 0 192.168.1.223:58212 192.0.78.12:
  • https
  • cubic the congestion algorithm name, the default congestion algorithm is “cubic”
  • wscale:9,7 if window scale option is used, this field shows the send scale factor and receive scale factor
  • rto:232
  • rtt:29.236/5.693
  • ato:40 mss:1452
  • pmtu:1500
  • rcvmss:880
  • advmss:1460
  • cwnd:10 congestion window size
  • bytes_sent:68335
  • bytes_acked:68336
  • bytes_received:16334
  • segs_out:276
  • segs_in:202
  • data_segs_out:151
  • data_segs_in:123
  • send 4.0Mbps number of bits sent/time of send.
  • lastsnd:376 how long time since the last packet sent, the unit is millisecond
  • lastrcv:348 how long time since the last packet received, the unit is millisecond
  • lastack:348 how long time since the last packet acknowledged, the unit is millisecond
  • pacing_rate 7.9Mbps
  • delivery_rate 2.6Mbps
  • delivered:152
  • app_limited busy:3700ms
  • rcv_space:14600
  • rcv_ssthresh:64076
  • minrtt:23.773

I did not find most of this information very useful. It is all to easy for a developer(I have done it myself) to provide statistics from information which is readily available, rather than ask what information would be useful to debug problems – then collect and publish that information.

Using CISCO openconnect to tunnel to another system from Linux

I needed to use openconnect from CISCO to be able to logon from my Ubuntu system to someone else’s z/OS system.

This was pretty easy, but understanding some of the under the cover’s bits took a bit of time.

Basic install

  • Use sudo apt install openconnect
  • Download the VPNC script from http://www.infradead.org/ 
  • Create the configuration script
    • I saved the script as vpnc-script.sh
    • Using ls /etc/vpnc showed the directory did not exist. Create it and move the file
    • sudo mkdir /etc/vpnc/
    • sudo mv vpnc-script.sh /etc/vpnc/
    • sudo chmod +x /etc/vpnc/vpnc-script.sh
  • You need information from the owners of the vpn server.
    • vpn userid
    • vpn password
    • name of their system
    • IP address of their internal system
    • tso userid
    • tso password
  • I created a script (openc.sh), where XXXXX is the short userid, and password is your long userid:
    • printf ‘%s’ “password” | sudo openconnect –user=XXXXXX –script=/etc/vpnc/vpnc-script.sh vpn.customer.com
  • When you run openc.sh it prompts for your su password on the machine. The print… means you can store the password in the shell script. If you do not specify it, openconnect will prompt you for it.
  • Once the connection is made you can use ping 10.66.77.88, or x3270 -model 5 10.66.77.88 to access the system, where 10.66.77.88 is the IP address the owner of the vpn server gave you.

x3270

The owner of the vpn server gave me the address of the z/OS machine, my userid and password.

I then used

x3270 -model 5 10.66.77.88 to logon to the system.

Hot key

I like to hot key to my z/OS sessions. I used Ubuntu “Settings”-> Keyboard shortcuts, and added a shortcut

  • name: mvsCust
  • Command: wmctrl -a 10.66.77.88
  • Hot key: Ctrl + H

The wmctl -a says make the window active which has 10.66.77.88 in the window page title.

When I press Ctrl +H it makes the customers x3270 session the active window.

Change the x3270 colours

I wanted to change the screen colours, to distinguish it from other 3270 sessions. See Making x3270 green screens blue or red, or yellow with green bits.

FTP

I had to use SFTP colin@10.66.77.88 to ftp to the remote z/OS system (where colin is my TSO userid).

What happens with openconnect, under the covers.

The handshake has several stages

  • Establish a TLS session using the certificate from the server. Once this has completed, any traffic is encrypted. In my case I used the vpn userid and password. The vpn server can be configured to accept certificates instead of userid and password.
  • The server sends down configuration information from the vpn server’s configuration. For example
    • The IP addresses it supports , such as 10.66.0.0 and netmask 255.255.0.0
    • Any changes to the DNS configuration, so it knows to route 10.66.77.78 via the VPN session.
    • The “banner” such as “Welcome to mycom.com. Users of this system do so at their own risk”.
    • A default domain.
    • Which tunnelling device to use – such as tun0.
    • How many configuration statements.
    • Each set of configuration statements.
    • You can see this information by using the -v option on the openconnect command.
  • Using the information sent from the the vpn server, the openconnect client creates environment variables.
  • The script defined (or defaulted, for example /etc/vpnc/vpnc-script.sh) on the openconnect command is invoked, and it uses these environment variables to manage the ip and dns configuration, changing files like /etc/resolv.conf (the local DNS file).

Oh p*x, I’ve lost my changes

I have been using pax to backup the files in my Unix Services directory and needed to restore a file so I could compare it with the last version (and work out why my updates didn’t work). Unfortunately I managed to overwrite my latest version instead of creating a copy.
I backed up my directory using

pax -W “seqparms=’space=(cyl,(10,10))'” -wzvf “//’COLIN.PAX.PYMQI2′” -x os390 /u/tmp/pymqi2/

This created a data set COLIN.PAX.PYMQI2 with the give space parameters, and os390 format.
(If you do not want all the files listed, use –wzf)

To list the contents of this file use

pax -f “//’COLIN.PAX.PYMQI2′”

To display a subset of the files use

pax -f “//’COLIN.PAX.PYMQI2′” /u/tmp/pymqi2/code

which gave

/u/tmp/pymqi2/code/
/u/tmp/pymqi2/code/pymqi/
/u/tmp/pymqi2/code/pymqi/__init__.py
/u/tmp/pymqi2/code/pymqi/old__init__.old
/u/tmp/pymqi2/code/pymqi/aa

And provide more information using the -v option

drwxrwxrwx 1 COLIN    1000      0 Jan 22 17:04 /u/tmp/pymqi2/code/
drwxr-xr-x 1 COLIN    1000      0 Feb 11 13:10 /u/tmp/pymqi2/code/pymqi/
-rw-r--r-- 1 OMVSKERN 1000 133011 Feb 22 13:15 /u/tmp/pymqi2/code/pymqi/init.py
-rw-r----- 1 COLIN    1000 119592 Feb  3 12:59 /u/tmp/pymqi2/code/pymqi/old__init__.old
-rwx------ 1 OMVSKERN 1000 119565 Jan 22 16:43 /u/tmp/pymqi2/code/pymqi/aa

The whoops

To restore an individual file and overwrite the original I used the -r option.

pax -rf “//’COLIN.PAX.PYMQI2′” /u/tmp/pymqi2/pymqi/__init__.py

I was expecting the file to be restored relative to the directory I was in; No – because I had backed up the files using an absolute path it restored the file to the same place, and so it overwrote my changes to the file. I had changed to a temporary directory, but I had not realised how the command worked.

There are several ways of doing it properly.

Restore with rename

pax -rf “//’COLIN.PAX.PYMQI2′” -i /u/tmp/pymqi2/pymqi/__init__.py

The -i option means rename.

I ran the command and it prompted me to rename it

Rename “/u/tmp/pymqi2/pymqi/__init__.py” as…

/tmp/oldinit.py

Set “do not overwrite”

I could also have used the -k option which prevents the overwriting of existing files.

Rename on restore

I could also have used the rename

pax -rf “//’COLIN.PAX.PYMQI2′” -s#/u/tmp/pymqi2/pymqi#/tmp/# /u/tmp/pymqi2/pymqi/__init__.py

Where the -s#/u/tmp/pymqi2/pymqi#/tmp/# / says use the regular expression to change /u/tmp/pymqi2/pymqi to /tmp and so restore it to a different place. Note: The more obvious -s/abc/xyz/, where / is used as the delimiter, would not work, as there is a ‘/’ in the file path.

All of the above

I could have use all of the options -i -k -s…. .

A better way to backup.

I had specified an absolute directory /u/tmp/pymi2/. If I was in this directory when I did the backup I could have used

pax … -x os390 .

Where the . at the end means from this directory, and so backup a relative directory.

If I list the files I get

pax -f “//’COLIN.PAX.PYMQI2A'” ./aa
./aa

And now if I restore the file…

pax -rf “//’COLIN.PAX.PYMQI2A'” ./aa

It restored the file into my working directory /tmp/aa .

So out of all the good ways of backing up and restoring – I chose the worst one. It only took me about 2 hours to remake all the changes I had lost.