Category: linux

Linux: Why are my cursor keys not working?

I created a new userid, and when I use the cursor keys in a command window it does not work as expected and I get commands like ]]?a

Solution

sudo chsh -s /bin/bash myid

If you use the command adduser you can specify the shell at create time.

Once you have done this you need to logoff and logon again (or just start a new window).

You should also have a .profile script for when you logon. My .profile has

# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.

# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022

# if running bash
if [ -n "$BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
	. "$HOME/.bashrc"
    fi
fi

# set PATH so it includes user's private bin directories
PATH="$HOME/bin:$HOME/.local/bin:$PATH"


JVM_ARGS="-Dcom.sun.management.jmxremote.port=9081 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false   "
# export MQCHLLIB=/var/mqm/qmgrs/QMA/@ipcc
export MQSSLKEYR=/var/mqm/qmgrs/QMB/ssl/key
export SWT_GTK3=0
export SWT_WEBKIT2=0
export CLASSPATH="${JAVA_HOME}/lib/tools.jar:${CLASSPATH}"
export LESS="-I -R"

This . profile invokes .bashrc (if it exists)

My file has

####### added for zPDT #######200725095805
export PATH=/usr/z1090/bin:$PATH
export LD_LIBRARY_PATH=/usr/z1090/bin:$LD_LIBRARY_PATH
export MANPATH=/usr/z1090/man:$MANPATH
ulimit -c unlimited
ulimit -d unlimited
####### end of added for zPDT #######

export LESS="-I -R "
export HISTTIMEFORMAT="%T "
export HISTFILESIZE=5000

How to reduce the chance of screwing up in a Linux window.

I had multiple Linux terminal windows open, doing SSH to different machines. I typed shutdown in the wrong window – and the wrong server shutdown!

I had configured different profiles so I could have a white background, a green background and a yellow background for my different systems, but I had got careless and not used them.

I found a neat way of colouring the windows automatically.

xdotool is a command-line X11 automation tool, which allows you to programmatically press keys. You can use this to set the profile of a terminal window.

To create a profile

From the hamburger options,

  • preferences,
  • profiles +
  • give the profile a name, create
  • select the named profile, it will display customising options
  • colours
  • untick Use colours from system theme
  • Click Text or Background
  • Pick a colour, Select
  • Close the window

Select a profile

Manually

  1. From the hamburger option
    • Select Profile
    • Pick a profile
  2. or Shift+f10
    • r (for Profile)
    • 3 for the third option in the list

Programmatically

xdotool key shift+F10 r 3

This does Shift+10, then select r for Profile, then picks the 3rd option

The clever bit

You can make an alias such as

alias somehost="xdotool key shift+F10 r 3; ssh user@somehost; xdotool key shift+F10 r 2"

or a somehost.sh script

#!/bin/sh
xdotool key shift+F10 r 3
ssh me@10.1.0.5
xdotool key shift+F10 r 2

If you enter the somehost command – it will select the 3rd profile, do the ssh. On exit from ssh it resets it back to the 2nd profile.

Restore files from Linux using Duplicity

Duplicity is a program which manages backup and restore of files on your Linux machine.

What is duplicity?

Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.

Backing up files

You run the backup application, and can specify the directories to backup. You can specify which directories to ignore.

The backups can be stored

  • Google drive
  • Networks server
  • Locally attached drive
  • Local Folder

That’s the easy bit.

What files are backed up?

I have my backups going to an external USB drive /media/colinpaice/UbuntuBackup1/home/Backup2024 on my Linux machine. The files have names like

duplicity-full.20240308T084825Z.vol295.difftar.gpg

The command

duplicity list-current-files file:///media/colinpaice/UbuntuBackup1/home/Backup2024 >files2024

Restore a file

duplicity restore -t 3D –file-to-restore ~/ssl/ssl2/rsaca256.csr file:///media/colinpaice/UbuntuBackup/home/Backup2021 ~/ssl/ssl2/rsa256.csr

This restores a file

  • -t 3D from 3 days ago
  • –file-to-restore ~/ssl/ssl2/rsaca256.csr
  • file:///media/colinpaice/UbuntuBackup/home/Backup2021 from this device
  • ~/ssl/ssl2/rsa256.csr to this file

What next

Duplicity can do much more than this. Ive just provided information on the most basic stuff that I have used. See the products web page or man duplicity

Setting up for PING can be difficult!

  • Setting up ping within one machine is trivial
  • Setting up ping between two machines is relatively easy
  • Setting up ping with 3 machines can be hard to get right and to get working.

Most documentation describes how to set up PING between two machines, and does not mention anything more complex.

Ancient history (well 1980’s)

To understand modern TCPIP networks, it helps to know about the origins of Ethernet.

Originally with Ethernet there was a bus; a very long cable. You plugged your computer into this bus. Each computer on the bus had a unique 48 bit MAC address. You could send a request to another computer on this bus using their Ethernet address. You could send a request to ALL computers on this bus. These days this might be, send to all computers:”does anyone have this IP address…?”

An Ethernet bus is connected to an Ethernet router which can route packets to other routers.

Ethernet has evolved and instead of a long bus shared by all users, you have a switch device, and you plug the Ethernet cable from your computer to the switch. Conceptually it is the same, but with the switch you get much better performance and usability. You still have routers to get between different Ethernet environments. The request send to all computers:”does anyone have this IP address…?” goes to the switch, and the switch sends it to all plugged in computers.

When you start TCPIP, it configures the Ethernet hardware adapter with the addresses is should listen for.

Other terminology and concepts

  • A router is a networking device which forwards packets between different networks. Packets of data get sent from the originator through zero or more routers to the final destination.
  • A gateway has several meanings
    • It can connect different network types, for example act as a protocol translator, it can have a built-in firewall
    • It can route IP traffic between different IP subnets
    • It can loosely mean a router

Usually a router or gateway is a dedicated device or hardware.

You can have a computer act as a router or gateway; it takes data from one interface and routes it to a different interface. The computer can pass the data through a fire wall, or transform it, for example converting internal IP addresses to external IP addresses (NAT translation).

At a concept level, a computer’s network adapter is configured only to listen for packets which match the IP addresses of the interface, and ignore the rest.

If you want a computer to act as a router or gateway, you need to configure the network adapter to listen to all traffic, and to process the traffic. This is important when routing traffic from computer A through computer B to computer C.

A simple one hop ping, first time

I have set up my laptop to talk to a server over Ethernet.

I have configured my laptop using

sudo ip -6 addr add 2001:7::/64 dev enp0s31f6

This says for any IP V6 address starting with 2001:0007:0000:0000 then try sending it down the Ethernet cable with interface name enp0s31f6.

Ping a non existing address

If I try to ping an address say 2001:7::99 a packet is sent to all computers on the Ethernet bus

from myipAddress to everyone on the Ethernet bus, does anyone have IP address 2001:7::99?

There is no reply because no computer on the bus, has the address.

Ping the adjacent box

On the server, the IP address of the Ethernet cable is 2001:7::2.

If I ping this address, there are the following flows

From myipAddress (and myMAC) to everyone on the Ethernet bus, does anyone have IP address 2001:7::2?
From 2001:7::2 to myipAddress (MAC), yes I have that IP address. My Ethernet MAC address is …

My laptop puts the remote IP address and its MAC address into its neighbour cache, then issues the ping request.

The ping request looks in the neighbour cache find the IP address and MAC address and issues.

From myipAddress to 2001:7::2 at macAddress .. ping request.

The second and later times I issue a ping

The ping request looks in the neighbour cache find the IP address and MAC address and issues the ping

From myipAddress to 2001:7::2 at macAddress .. ping request.

So while the neighbour cache still has the IP of the target, and it’s MAC address, the ping can be routed to the next hop.

Setting up a multi hop ping

I have my laptop connected via a physical Ethernet cable to my server. The server is connected to z/OS via a virtual Ethernet connection. The IP address of the z/OS end of the virtual Ethernet cable is 2001:7::4.

The ping 2001:7::4 request on my laptop, does not work (as we saw above), because TCP asked everyone on the Ethernet bus if it has address 2001:7::4 – and no machine replies.

You need to define the link to the server machine as a gateway or router-like device which can handle IP addresses which are not on the Ethernet bus. You define it like

sudo ip -6 route add 2001:7::/64 via 2001:7::2

This says for any traffic with IP address 2001:0007:0000:0000:* send it via address 2001:7::2.

This requires 2001:7::2 to be known about, and so needs the following to be configured first

sudo ip -6 route add 2001:7::2 dev enp0s31f6

for TCPIP to know where to send the traffic to.

The route add 2001:7::2 dev enp0s31f6 command sends a broadcast on the Ethernet bus asking – does anyone have 2001:7::2. My server replies saying yes I have it. This is the same as for the ping above.

In summary

To send traffic on the same Ethernet bus you use

sudo ip -6 route add 2001:7::2 dev enp0s31f6

To route it via a router, switch, or computer acting as a router or switch you need.

sudo ip -6 route add 2001:7::/64 via 2001:7::2 or
sudo ip -6 route add 2001:7::/64 via 2001:7::2 <dev enp0s31f6>

The computer acting as a router or switch may need additional configuration. For example to allow it to route traffic from one Ethernet bus to another Ethernet bus you need to enable packet forwarding. On Linux to enable forwarding for all interfaces use

sysctl -w net.ipv6.conf.all.forwarding=1

On z/OS you use a static route such as

ROUTE 2001:7::/64 2001:7::3 JFPORTCP6

Where 2001:7::/64 is the range of addresses 2001:7::3 is (one of) the addresses of the interface at the remote end of the cabl, and JFPORTCP6 is the interface name. This is similar to the Linux route statement above.

You might need to set up the firewall. On my server I needed

sudo ufw route allow in on eno1 out on tap2

What IP Address does the sender have?

This is where is starts to get more complex.

Every network connection has at least one IP address.

With IP V6

  • Each interface gets an “internal” IP address such as fe80::9b07:33a1:aa30:e272
  • You can allocate an external address using the Linux command sudo ip -6 addr add 2001:7::1 dev enp0s31f6
  • If the interface has one external IP address configured, then this will be used.
  • If the interface has more than one external address configured then the first in the list may be used (not always).
  • If the interface does not have an external IP address, then TCPIP will find one from another interface, or allocate one for the duration of the request, such as 2a00:23c5:978f:9999:210a:1e9b:94a4:c8e. This address comes from my wireless connection

When my laptop is started, only the wireless connection has an IPV6 address. A ping request to 2001:7::4 had the origin IP address of 2a00:23c5:978f:9999:210a:1e9b:94a4:c8e which is the first address in the list for the wireless connection.

You can tell ping which address to use, for example

ping -I 2a00:23c5:978f:9999:cff1:dc13:4fc6:f21b 2007:1::4 (this fails because the server does now know to send the response to the requester)

I defined a new address for the interface using

sudo ip -6 addr rep 2002:7::1 dev enp0s31f6

I could issue ping -I 2002:7::1 2001:7::4, but this failed to get a response, because the back-end and intermediate nodes, did not know how to get the response back to 2002:7::1

Does this sender address matter?

Yes, because the remote end needs to have a definition to be able to send the response back.

I had my laptop connected to a Linux server over Ethernet, which in turn was connected to z/OS over a virtual Ethernet.

On z/OS I could see the ping request arrive, but it could not send the response back because it did not know how to get to 2a00:23c5:978f:9999….

I configured the laptop end of the Ethernet to give it an IP address 2001:7::1

sudo ip -6 addr add 2001:7::1/64 dev enp0s31f6

I configured the server to laptop to have an IP address of 2001:7::2

sudo ip -6 route add 2001:7::1/128 dev eno1

I configured the server to z/OS with an IP address and a route

sudo ip -6 addr add 2001:7::3/128 dev tap2
sudo ip -6 route add 2001:7::4/128 dev tap2

Now when I did the ping, the originator was 2001:7::1.

I configured the z/OS interface to send stuff back

INTERFACE JFPORTCP6 
    DEFINE IPAQENET6 
    CHPIDTYPE OSD 
    PORTNAME PORTC 
                                   
INTERFACE JFPORTCP6 
    ADDADDR 2001:DB8:8::9 
INTERFACE JFPORTCP6 
    ADDADDR 2001:DB8::9 
INTERFACE JFPORTCP6 
    ADDADDR 2001:7::4 
START JFPORTCP6

and the routing

BEGINRoutes 
;     Destination      Gateway                 LinkName  Size 
ROUTE 2001:7::/64      2001:7::3              JFPORTCP6   MTU 5000 
...
ENDRoutes

This says that all traffic with destination address 2001:0007:0000:000…. send to interface JFPORTCP6. This interface is connected to a gateway with the address of the remote end of the Ethernet (so on the server) of 2001:7::3.

The server machine needs to act as a router between the different Ethernet buses. You can display and configure this using

sysctl -a |grep forwarding
sudo sysctl -w net.ipv6.conf.all.forwarding=1

Packet forwarding on z/OS

By default the z/OS interface only listens for packets with one of the IP addresses of the interface. For z/OS to be able to be a router; accept all packets, and route them to other interfaces you need:

  • IPCONFIG DATAGRAMFWD in the TCP/IP Profile
  • PRIROUTER on the Interface definition . This configures the Ethernet adapter (hardware) so If a datagram is received at this device for an unknown IP address, the datagram is routed to this TCP/IP instance.

But normally if you are running z/OS you use a cheaper, physical router, rather than use the z/OS to do your routing. It might only be people like me who run z/OS on their laptop who want to try routing through z/OS.

Why doesn’t ctrl-s work in ISPF edit? – ah it does now.

I had been editing a file, saving it, and finding the changes were not being picked up. Looking back, it was obvious; I was using CTRL-S the familiar Linux command, instead of F3 on ISPF.

I fixed this by configuring X3270 (on Linux).

My file /home/colin/.x3270pro now has

...
x3270.keymap: mine
! Definition of the 'mine' keymap
x3270.keymap.mine: #override \
    Alt<Key>4:          String("\\x00a2")\n\
    Ctrl<Key>backslash: String("\\x00a2")\n\
    <Key>Escape:    Clear()\n\
    <Key>End:        FieldEnd()\n\
    Ctrl<Key>Delete:   EraseEOF()\n\
    Ctrl<Key>Right:    NextWord()\n\
    Ctrl<Key>Left:    PreviousWord()\n\
    Ctrl<Key>Up:    Home()\n\
    <Key>Control_L: Reset()\n\
    <Key>Control_R: Reset()\n\
    <Key>Prior: PF(7)\n\
    <Key>Next: PF(8)\n\
    <Btn3Down>:   PA(1)\n\
    Ctrl<Key>1:   PA(1)\n\
    Ctrl<Key>s:   MoveCursor(3,15) String("save") Enter()\n\

When I started a new X3270 session, Ctrl -S went to the command line, typed save and pressed enter. Job Done ! The numbers are 0 based, so 3 means line 4 on the screen.

This makes life so much easier!

Setting up z/OS for TLS clients

There is a lot of configuration needed when setting up TLS(SSL) between a server and a client. There are many options and it is easy to misconfigure. The diagnostic information you get when the TLS handshake fails is usually insufficient to identify any problems.

You need the following on z/OS:

  • One or more Certificate Authority certificates. You can create and use your own for testing. If you want to work with external sites you’ll need a proper (external) CA, but for validation and proof of concept you can create your own CA. You could set up a top level CA CN=CA,O=MYORG, and another one (signed by CA=CA,O=MYORG), called CN=CA,OU=TEST,O=MYORG. Either or both of the public CA certificates will need to be sent to the clients in imported into their keystore.
  • A private/public key, signed by a CA, (such as signed by CA=CA,OU=TEST,O=MYORG).
  • The private key is associated with a userid.
    • The signing operation takes the data (the public key), does a hash sum calculation on the data, encrypts this hash sum, and stores the encrypted hash value, and CA public certificate with the original data. To check the signature, the receiving application compares the CA with its local copy, if that matches, does the same checksum calculation, decodes the encrypted hash sum – and checks the decrypted and locally calculated values match.
    • A certificate is created using one from a list of algorithms. (For example, Elliptic Curve, RSA). When the certificate is sent to the client, the client needs to support the algorithm. Either end can be configured, for example, to support Elliptic Curve, but not RSA.
  • A keyring to contain your private key(s) – this can also contain CA public certificates of the partners (clients or servers).
  • A “site” keyring (public keystore, or trust ring) which holds the public CA certificates of all the other sites you work with. If you have only one keyring per user or application, you need to update each of them if you need to an a new CA to your environment. Many applications are only designed to work with one keyring. Java applications tend to have a key store(for the private key) and a trust store for the CAs.
  • Some applications can support more than one private certificate on a keyring. The certificate needs to match what the client can support.
  • For certificates which are sent to your server, you need a copy of the CA(s) used to sign the incoming certificate. If you have a copy of the CA, then you can validate any certificate that the CA signed. This means you do not have to have a copy of the public certificate of every client. You just need the CA.
    • Some application need access to just one CA in the chain, other applications need access to all certificates in the CA chain.
  • As part of the TLS handshake
    • the client sends up a list of the valid cipher specs it supports (which algorithms, and size of key)
    • the server sends down a subset of the list of cipher spec to use (from the client’s list)
    • the server can also send down its certificate, which contains information such as the distinguished name CN=zSERVER, OU=TEST, O= MYORG, and host name.
    • the client can validate these names – to make sure the host name in the certificate matches the host, and what it was expecting.
    • if requested, the client can send up its certificate for identification. The server can validate the certificate, and can optionally map it to a userid on the server.
  • A userid can be given permission to read certificate in another user’s keyring. A userid needs a higher level of authority to be able to access the private key in another id’s keyring.

Create the Certificate Authority

//IBMRACF  JOB 1,MSGCLASS=H                               
//S1  EXEC PGM=IKJEFT01,REGION=0M                         
//SYSPRINT DD SYSOUT=*                                    
//SYSTSPRT DD SYSOUT=*                                    
//SYSTSIN DD * 
RACDCERT certauth LIST(label('DOCZOSCA')) 
RACDCERT CERTAUTH DELETE(LABEL('DOCZOSCA'))               
RACDCERT GENCERT  -                                         
  CERTAUTH -                                                
  SUBJECTSDN(CN('DocZosCA')- 
             O('COLIN') -                                   
             OU('CA')) - 
  NOTAFTER(   DATE(2027-07-02  ))-                          
  KEYUSAGE(   CERTSIGN )  -      
  SIZE(2048) -                                              
  WITHLABEL('DOCZOSCA') 
/*
//

This certificate is created against “user” CERTAUTH. Keyusage CERTSIGN means it can be used to sign certificates. “user” CERTAUTH is often displayed internally as “irrcerta”.

Once it has been created the certificate should be connected to every ring that may use it, see below.

Export the CA certificate to a file so, clients can access it 

RACDCERT CERTAUTH EXPORT(LABEL('DOCZOSCA')) -
  DSN('IBMUSER.CERT.DOC.CA.PEM') -
  FORMAT(CERTB64) -
  PASSWORD('password')

The file looks like

-----BEGIN CERTIFICATE-----                                        
MIIDYDCCAkigAwIBAgIBADANBgkqhkiG9w0BAQsFADAwMQ4wDAYDVQQKEwVDT0xJ   
TjELMAkGA1UECxMCQ0ExETAPBgNVBAMTCERvY1pvc0NBMB4XDTIyMTAwOTAwMDAw 
...

This can be sent to the clients, so they can validate certificates sent from the server. This file could be sent using cut and paste, or FTP.

Create the keyring for user START1.

The instructions below lists the ring first – in case you need to know what it was before you deleted it”

RACDCERT LISTRING(TN3270)  ID(START1) 

RACDCERT DELRING(TN3270) ID(START1) 

RACDCERT ADDRING(TN3270) ID(START1)                                                          

RACDCERT LISTRING(TN3270)  ID(START1) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Connect the CA to every keyring that needs to use it

RACDCERT ID(START1) CONNECT(RING(TN3270) - 
                            CERTAUTH LABEL('DOCZOSCA'))

Create a user certificate and sign it on z/OS

This creates a certificate and gets is signed – as one operation. You can create a certificate, export it, sent it off to a remote CA, import it, and add it to a userid.

RACDCERT ID(START1) DELETE(LABEL('NISTECC521')) 
                                                                
RACDCERT ID(START1) GENCERT -                                   
  SUBJECTSDN(CN('10.1.1.2') - 
             O('NISTECC521') -                                  
             OU('SSS')) -                                       
   ALTNAME(IP(10.1.1.2))-                                       
   NISTECC - 
   KEYUSAGE(HANDSHAKE) - 
   SIZE(521) - 
   SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -                      
   WITHLABEL('NISTECC521')     -                                
                                                                
RACDCERT id(START1) ALTER(LABEL('NISTECC521'))TRUST             

RACDCERT ID(START1) CONNECT(RING(TN3270) -                      
                            ID(START1)  -                       
                            DEFAULT  - 
                            LABEL('NISTECC521') )               
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh                    
RACDCERT LIST(LABEL('NISTECC521' )) ID(START1)                  
RACDCERT LISTRING(TN3270)  ID(START1)

This creates a certificate with type Elliptic Curve (NISTECC) with a key size of 521. It is signed with the CA certificate created above.

The ALTNAME, is a field the client can verify that the Source Name in the certificate matches the IP address of the connection.

It is connected to the user’s keyring as the DEFAULT. The default certificate is used if the label of a certificate is not specified when using the keyring.

Give a user access to the keyring

PERMIT START1.TN3270.LST CLASS(RDATALIB)  -    
    ID(COLIN )  ACCESS(UPDATE )                          
SETROPTS RACLIST(RDATALIB) refresh
  • Update access give userid COLIN access to the private key.
  • Read access only gives access to the public keys in the ring.

You would typically give a group of userids access, not just to individual userids.

Import the client’s CA’s used to sign the client certificates

This is the opposite to Export the CA certificate to a file so clients can access it above.

Copy the certificate to z/OS. This can be done using FTP or cut and paste.

Use it!

I used it in AT-TLS

TTLSConnectionAdvancedParms       TNCOonAdvParms 
{ 
 ServerCertificateLabel  NISTECC521
 ...
} 
TTLSSignatureParms                TNESigParms 
{ 
   CLientECurves Any 
} 
TTLSEnvironmentAction                 TNEA 
{ 
  HandshakeRole                       ServerWithClientAuth 
  TTLSKeyringParms 
  { 
    Keyring                   start1/TN3270 
  } 
...
}

Linux ls command timestamps in microseconds is easy-ish

Any of the following work

ls -la --time-style=full-iso ...
ls --full-time ...

Which gave me

-rw-r--r-- 1 root root 1534 2023-01-01 16:46:58.394054373 +0000 group

Where the format is

The TIME_STYLE argument can be full-iso, long-iso, iso, locale, or +FORMAT. FORMAT is interpreted like in date(1).

But during installing/removing a package it touched the file, and I have

-rw-r--r-- 1 root root 3784 2022-12-30 11:14:15.436236905 +0000 passwd
-rw-r--r-- 1 root root 3764 2022-12-30 11:14:15.000000000 +0000 passwd-
and 
-rw-r--r-- 1 root root 1534 2023-01-01 16:46:58.394054373 +0000 group
-rw-r--r-- 1 root root 1523 2022-12-30 11:14:15.000000000 +0000 group-

so the temporary files have .000000 microseconds – so there is something else going on!

You can use

alias lt=’ls -ltr –full-time –color=auto’

to make a command “lt” which is the ls command, plus options.

Why has my ethernet connection stopped connecting?

This morning my Ethernet connection between my two Linux systems stopped working. I could see IPV6 stuff flowing over the network, but Linux did not say connected. Also there was no IPV4 address. It took me almost a day to work out what the problem was. Googling and following the advice may have made it worse!

I also include some useful commands for next time it happens.

The high level problem

It looks like the Network Manager has changed.

A week ago, I had files like

/etc/NetworkManager/system-connections/enp0s31f6

containing the definitions for my Ethernet.

Now Network Manager uses

/etc/NetworkManager/system-connections/BTHub6-9999.nmconnection

and these configuration files were missing configuration data. I have a .nmconnection file going back to November, so something has changed.

Further study shows that the

nmcli connection migrate

converts from old format to .nmconnection files, so perhaps this was done under the covers.

Network manager files

Files in /etc/NetworkManager/system-connections/ must be owned by root and be readably only by root – otherwise NetworkManager will ignore it.

Some of my files had the wrong permissions, and so were ignored.

I used

sudo chmod -R 600 /etc/NetworkManager/system-connections/

and restarted NetworkManager

sudo systemctl restart NetworkManager

and missing files reappeared in Network Manager.

For more information about the files see man nm-settings-keyfile.

The detailed problem

Using Wireshark I could see IPV6 traffic flowing over the connection, so the cable was OK, and some of the definitions were OK.

The ip addr command showed there was an IPV6 address for the connection, but no IPV4 address.

I could not find a log for Network Manager with its error messages, see log below for the messages on syslog.

Looking online, there were suggestions that you delete your existing definition and recreate it, also use nm-connection-editor. This may have been a bad move; it is always better to rename than to delete.

Comparing the definitions currently in use /etc/NetworkManager/system-connections/ with a backup version, I could see that the .nmconnection files were in use.

I used Network Manager to change my Ethernet definitions. Under the IPv4 tab

  • IPv4 method: change from Automatic(DHCP) to Manual
  • Address: Added 10.1.0.2 Netmask 255.255.255.0
  • Route: Added 10.1.0.3 Netmask 255.255.255.255.0 Gateway 10.1.0.2

The route statement says to get to 10.1.0.3 go via 10.1.0.2 .

Once I restarted the connection it became active, and the ip -4 addr command showed it had an IPv4 address.

For the other end of the connection I did the matching changes and the end to end connection burst into life!

For my Ethernet connection my file was 

[connection]
id=Wired connection 1
uuid=ecc4df76-4733-45f5-9b67-9fba9ef2d3bf
type=ethernet
interface-name=enp0s31f6
permissions=
timestamp=1673353909

[ethernet]
mac-address-blacklist=

[ipv4]
address1=10.1.0.2/24

dns-priority=100
dns-search=
method=manual
route1=10.1.0.3/24,10.1.0.2

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

[proxy]

This defines the IP address 10.1.0.2, and a route to 10.1.0.3 via 10.1.0.2 .

Useful commands

Display the devices

nmcli d

nmcli device

gives

DEVICE             TYPE      STATE         CONNECTION         
wlp4s0             wifi      connected     BTHub6-78RQ        
enp0s31f6          ethernet  connected     Wired connection 1 
virbr0             bridge    connected     virbr0  d

so for my en0s31f6 device, the connection file is ‘Wired connection 1’

Display the connection

nmcli c

nmcli connection

NAME                UUID                                  TYPE       DEVICE    
BTHub6-78RQ         fc74c8e0-6f96-4e8b-a8ba-6389abbe3396  wifi       wlp4s0    
Wired connection 1  ecc4df76-4733-45f5-9b67-9fba9ef2d3bf  ethernet   enp0s31f6 
virbr0              386a5a3a-023b-41d9-9138-04202d8dfda6  bridge     virbr0

Display more information

nmcli -f all c |less

Display only some fields

nmcli -f name,device,FILENAME c |less

gives

NAME                DEVICE     FILENAME                                                                    
BTHub6-78RQ         wlp4s0     /etc/NetworkManager/system-connections/BTHub6-78RQ.nmconnection.old2        
Wired connection 1  enp0s31f6  /etc/NetworkManager/system-connections/Wired connection 1.nmconnection.old2 
virbr0              virbr0     /run/NetworkManager/system-connections/virbr0.nmconnection

Displaying trace

When the Ethernet connection work, /var/log/syslog had entries

 NetworkManager[11240]: <info>  [....0612] device (enp0s31f6): Activation: starting connection 'enp0s31f6' (c066ca29-2253-41ef-8e69-2251fb15f7b8)
 NetworkManager[11240]: <info>  [....0617] audit: op="connection-activate" uuid="c066ca29-2253-41ef-8e69-2251fb15f7b8" name="enp0s31f6" pid=2585 uid=1000 result="success"
 NetworkManager[11240]: <info>  [....0636] device (enp0s31f6): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
 NetworkManager[11240]: <info>  [....0678] device (enp0s31f6): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
 NetworkManager[11240]: <info>  [....0718] device (enp0s31f6): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
 avahi-daemon[1123]: Joining mDNS multicast group on interface enp0s31f6.IPv6 with address fe80::9b07:33a1:aa30:e272.
 avahi-daemon[1123]: New relevant interface enp0s31f6.IPv6 for mDNS.
 avahi-daemon[1123]: Registering new address record for fe80::9b07:33a1:aa30:e272 on enp0s31f6.*.
 avahi-daemon[1123]: Joining mDNS multicast group on interface enp0s31f6.IPv4 with address 10.1.0.2.
 avahi-daemon[1123]: New relevant interface enp0s31f6.IPv4 for mDNS.
 avahi-daemon[1123]: Registering new address record for 10.1.0.2 on enp0s31f6.IPv4.
 NetworkManager[11240]: <info>  [....0830] device (enp0s31f6): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
 NetworkManager[11240]: <info>  [....1005] device (enp0s31f6): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
 NetworkManager[11240]: <info>  [....1008] device (enp0s31f6): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
 NetworkManager[11240]: <info>  [....1021] device (enp0s31f6): Activation: successful, device activated.

When the connection was defined as DHCP the trace was 

NetworkManager: <info>  [...] device (enp0s31f6): state change: ip-config -> deactivating (reason 'user-requested', sys-iface-state: 'managed')
NetworkManager: <info>  [...] audit: op="device-disconnect" interface="enp0s31f6" ifindex=2 pid=2585 uid=1000 result="success"
NetworkManager: <info>  [...] device (enp0s31f6): state change: deactivating -> disconnected (reason 'user-requested', sys-iface-state: 'managed')
avahi-daemon: Withdrawing address record for fe80::78e8:9e55:9f3f:768 on enp0s31f6.
avahi-daemon: Leaving mDNS multicast group on interface enp0s31f6.IPv6 with address fe80::78e8:9e55:9f3f:768.
avahi-daemon: Interface enp0s31f6.IPv6 no longer relevant for mDNS.
NetworkManager: <info>  [...] dhcp4 (enp0s31f6): canceled DHCP transaction
NetworkManager: <info>  [...] dhcp4 (enp0s31f6): state changed unknown -> done
NetworkManager: <info>  [...] device (enp0s31f6): Activation: starting connection 'Wired connection 1' (ecc4df76-4733-45f5-9b67-9fba9ef2d3bf)
NetworkManager: <info>  [...] device (enp0s31f6): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
NetworkManager: <info>  [...] device (enp0s31f6): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
NetworkManager: <info>  [...] device (enp0s31f6): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
NetworkManager: <info>  [...] dhcp4 (enp0s31f6): activation: beginning transaction (timeout in 45 seconds)
avahi-daemon: Joining mDNS multicast group on interface enp0s31f6.IPv6 with address fe80::78e8:9e55:9f3f:768.
avahi-daemon: New relevant interface enp0s31f6.IPv6 for mDNS.
avahi-daemon: Registering new address record for fe80::78e8:9e55:9f3f:768 on enp0s31f6.*.

This has entries about DHCP.

My machine did not have a DHCP server installed – so any request for DHCP will fail to get an address.

Backing up on Linux is easy – how do I restore?

I’ve been using deja-dup front end to duplicity to backup my Ubuntu Linux. This has worked fine. The only problems I had were trying to restore files I had not backed up (/etc….) – a Defective End User problem.

Restoring one of these files was a little challenge, with bumps in the road… here’s how I did it. I’ve listed the errors I received while using duplicity.

A high level thought – should you change the repository name every year – such as including the year?

Where am I backing up to (where do I restore from)?

Use the deja-dup application (use the Linux show applications and find backup).

Under Storage location will be location and folder.

If this is an external drive it may be mounted under /mnt/… or /media/.

High level view

duplicity collection-status file:///media/colinpaice/UbuntuBackup/home/Backup2023

gave

Last full backup date: Sun Jan  1 17:02:48 2023
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive directory: /home/colinpaice/.cache/duplicity/52b5c01605d52d0976f1434440a7c11a

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Sun Jan  1 17:02:48 2023
Chain end time: Tue Jan 10 08:19:05 2023
Number of contained backup sets: 10
Total number of contained volumes: 250
 Type of backup set:                            Time:   Number of volumes:
                Full         Sun Jan  1 17:02:48 2023               238
         Incremental         Mon Jan  2 08:25:38 2023                 1
         Incremental         Tue Jan  3 08:50:23 2023                 1
         Incremental         Wed Jan  4 08:53:01 2023                 1
         Incremental         Thu Jan  5 08:22:41 2023                 1
         Incremental         Fri Jan  6 08:36:31 2023                 4
         Incremental         Sat Jan  7 10:23:00 2023                 1
         Incremental         Sun Jan  8 12:16:58 2023                 1
         Incremental         Mon Jan  9 09:02:31 2023                 1
         Incremental         Tue Jan 10 08:19:05 2023                 1
-------------------------
No orphaned or incomplete backup sets found.

List files in the backup

duplicity list-current-files file:///media/colinpaice/UbuntuBackup/home/Backup2023/

I saved the list of files using

duplicity list-current-files file:///media/colinpaice/UbuntuBackup/home/Backup2023 >dup.lst
Synchronising remote metadata to local cache…
GnuPG passphrase for decryption:

This took over a minute (it has to check thousands of files names). I think it scans every “signature” file to build a list. The more signature files – the longer it takes.

Inside dup.lst it had entries like

Tue Aug 14 11:41:33 2018 etc/NetworkManager/system-connections/LG-D855_2292

This says the backed up file was from the given date in 2018.

List files available on a given date

I used

duplicity list-current-files –time 5D file:///media/colinpaice/UbuntuBackup/home/Backup2023 > aa.lst

to get a list of files available 5 days ago.

Restore the file

duplicity restore -t 3D –file-to-restore etc/NetworkManager/system-connections/LG-D855_2292 file:///media/colinpaice/UbuntuBackup/home/Backup2023 /home/colinpaice/secret/abc

where

  • -t ( or –time) specify a time period
  • 3D for three days ago. Other options s, m, h, D, W, M, or Y. 1M3d is 33 days ago.
  • etc/Network…. this is the file I wanted to restore. Note no leading / .
  • file:///media…/home/Backup2023 from this backup location.
  • /home/colinpaice/secret/abc and put the file here. The file/directory must not exist.

Restore a directory

Specify the folder (ending in /) and specify a target directory (/home/colinpaice/secret/abcd). It will create the directory if needed.

If you specify a name like etc/…/* it will report it was not found in archive.

duplicity restore -t 3D –file-to-restore etc/NetworkManager/system-connections/ file:///media/colinpaice/UbuntuBackup/home/Backup2023 /home/colinpaice/secret/abcd

Error messages

gpg: WARNING: unsafe permissions on homedir

This says the contents of your secret directory are publicly available. Change the permissions.

chmod 600 /home/colinpaice/.gnupg/*
chmod 700 /home/colinpaice/.gnupg/

Local and Remote metadata are synchronized, no sync needed.

This is OK. It means the metadata is the same in the local cache as in the remote store.

duplicity.dup_collections.CollectionsError: No signature chains found

You have the wrong backup repository name,

  • it needs ///
  • ls should give a filename like /media/colinpaice/UbuntuBackup/…./duplicity-new-signatures.20230101T170248Z.to.20230102T082538Z.sigtar.gpg

Error ‘[Errno 1] Operation not permitted: b’/home/colinpaice/secret/abc” processing .

After the restore, duplicity was not able to use chown on the file to restore it to the original owner.

GPGError: GPG Failed, see log below:

===== Begin GnuPG log =====
gpg: AES256 encrypted data
gpg: gcry_kdf_derive failed: Invalid data
gpg: encrypted with 1 passphrase
gpg: decryption failed: No secret key
===== End GnuPG log =====

I got this asking for a date before backups were taken. See

duplicity collection-status file:///media/colinpaice/UbuntuBackup/home/Backup2023

I was looking for a backup in 2002 – which was not in the 2003 backup.

What does tso netstat neighbour give you?

The command TSO NETSTAT ND gave me 

Query Neighbor cache for 2001:db8:1:0:8024:bff:fe45:840c 
  IntfName: IFPORTCP6          IntfType: IPAQENET6 
  LinkLayerAddr: 82240B45840C  State: Reachable 
  Type: Router                 AdvDfltRtr: No 

Query Neighbor cache for fe80::8024:bff:fe45:840c 
  IntfName: IFPORTCP6          IntfType: IPAQENET6 
  LinkLayerAddr: 82240B45840C  State: Reachable 
  Type: Router                 AdvDfltRtr: No 

Query Neighbor cache for fe80::9863:1eff:fe13:1408 
  IntfName: JFPORTCP6          IntfType: IPAQENET6 
  LinkLayerAddr: 9A631E131408  State: Reachable 
  Type: Router                 AdvDfltRtr: No

On Linux the

ip -6 addr

command gave me

tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 1000
    inet6 2001:db8:1:0:b0fd:f92b:8362:577b/64 ...
    inet6 2001:db8:1:0:8024:bff:fe45:840c/64 ...
    inet6 fe80::8024:bff:fe45:840c/64 ...

tap2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 1000
    inet6 fe80::9863:1eff:fe13:1408/64 ...

The TSO output means

  • Query Neighbor cache for 2001:db8:1:0:8024:bff:fe45:840c. The address is one of the addresses on the remote end of the connection. There is an entry because some traffic came via the address.
  • IntfName: IFPORTCP6 The z/OS Interface name used to create the defintion
  • IntfType: IPAQENET6 the OSA-Express QDIO interfaces statement
  • LinkLayerAddr: 82240B45840C
  • State: Reachable Other options can include stale, which means z/OS has not heard anything from this address for a while
  • Type: Router
  • AdvDfltRtr: No. The information passed in the Router Advertisement, said this was connection does not Advertise a Default Router(AdvDfltRtr).

From the NETSTAT ND output we can see data has been received from

  • IFPORTCP6:2001:db8:1:0:8024:bff:fe45:840c
  • IFPORTCP6:fe80::8024:bff:fe45:840c
  • JFPORTCP6:fe80::9863:1eff:fe13:1408

To get data to flow down the 2001…. address I had to use

ping -I 2001:db8:1:0:8024:bff:fe45:840c 2001:db8:1::9

Where the -I says use the interface address.

You can get information about bytes processed by interface (not by address) using the TSO NETSTAT DEVLINKS command.