This morning my Ethernet connection between my two Linux systems stopped working. I could see IPV6 stuff flowing over the network, but Linux did not say connected. Also there was no IPV4 address. It took me almost a day to work out what the problem was. Googling and following the advice may have made it worse!
I also include some useful commands for next time it happens.
The high level problem
It looks like the Network Manager has changed.
A week ago, I had files like
/etc/NetworkManager/system-connections/enp0s31f6
containing the definitions for my Ethernet.
Now Network Manager uses
/etc/NetworkManager/system-connections/BTHub6-9999.nmconnection
and these configuration files were missing configuration data. I have a .nmconnection file going back to November, so something has changed.
Further study shows that the
nmcli connection migrate
converts from old format to .nmconnection files, so perhaps this was done under the covers.
Network manager files
Files in /etc/NetworkManager/system-connections/ must be owned by root and be readably only by root – otherwise NetworkManager will ignore it.
Some of my files had the wrong permissions, and so were ignored.
I used
sudo chmod -R 600 /etc/NetworkManager/system-connections/
and restarted NetworkManager
sudo systemctl restart NetworkManager
and missing files reappeared in Network Manager.
For more information about the files see man nm-settings-keyfile.
The detailed problem
Using Wireshark I could see IPV6 traffic flowing over the connection, so the cable was OK, and some of the definitions were OK.
The ip addr command showed there was an IPV6 address for the connection, but no IPV4 address.
I could not find a log for Network Manager with its error messages, see log below for the messages on syslog.
Looking online, there were suggestions that you delete your existing definition and recreate it, also use nm-connection-editor. This may have been a bad move; it is always better to rename than to delete.
Comparing the definitions currently in use /etc/NetworkManager/system-connections/ with a backup version, I could see that the .nmconnection files were in use.
I used Network Manager to change my Ethernet definitions. Under the IPv4 tab
- IPv4 method: change from Automatic(DHCP) to Manual
- Address: Added 10.1.0.2 Netmask 255.255.255.0
- Route: Added 10.1.0.3 Netmask 255.255.255.255.0 Gateway 10.1.0.2
The route statement says to get to 10.1.0.3 go via 10.1.0.2 .
Once I restarted the connection it became active, and the ip -4 addr command showed it had an IPv4 address.
For the other end of the connection I did the matching changes and the end to end connection burst into life!
For my Ethernet connection my file was
[connection] id=Wired connection 1 uuid=ecc4df76-4733-45f5-9b67-9fba9ef2d3bf type=ethernet interface-name=enp0s31f6 permissions= timestamp=1673353909 [ethernet] mac-address-blacklist= [ipv4] address1=10.1.0.2/24 dns-priority=100 dns-search= method=manual route1=10.1.0.3/24,10.1.0.2 [ipv6] addr-gen-mode=stable-privacy dns-search= method=auto [proxy]
This defines the IP address 10.1.0.2, and a route to 10.1.0.3 via 10.1.0.2 .
Useful commands
Display the devices
nmcli d
nmcli device
gives
DEVICE TYPE STATE CONNECTION wlp4s0 wifi connected BTHub6-78RQ enp0s31f6 ethernet connected Wired connection 1 virbr0 bridge connected virbr0 d
so for my en0s31f6 device, the connection file is ‘Wired connection 1’
Display the connection
nmcli c
nmcli connection
NAME UUID TYPE DEVICE BTHub6-78RQ fc74c8e0-6f96-4e8b-a8ba-6389abbe3396 wifi wlp4s0 Wired connection 1 ecc4df76-4733-45f5-9b67-9fba9ef2d3bf ethernet enp0s31f6 virbr0 386a5a3a-023b-41d9-9138-04202d8dfda6 bridge virbr0
Display more information
nmcli -f all c |less
Display only some fields
nmcli -f name,device,FILENAME c |less
gives
NAME DEVICE FILENAME BTHub6-78RQ wlp4s0 /etc/NetworkManager/system-connections/BTHub6-78RQ.nmconnection.old2 Wired connection 1 enp0s31f6 /etc/NetworkManager/system-connections/Wired connection 1.nmconnection.old2 virbr0 virbr0 /run/NetworkManager/system-connections/virbr0.nmconnection
Displaying trace
When the Ethernet connection work, /var/log/syslog had entries
NetworkManager[11240]: <info> [....0612] device (enp0s31f6): Activation: starting connection 'enp0s31f6' (c066ca29-2253-41ef-8e69-2251fb15f7b8) NetworkManager[11240]: <info> [....0617] audit: op="connection-activate" uuid="c066ca29-2253-41ef-8e69-2251fb15f7b8" name="enp0s31f6" pid=2585 uid=1000 result="success" NetworkManager[11240]: <info> [....0636] device (enp0s31f6): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed') NetworkManager[11240]: <info> [....0678] device (enp0s31f6): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') NetworkManager[11240]: <info> [....0718] device (enp0s31f6): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed') avahi-daemon[1123]: Joining mDNS multicast group on interface enp0s31f6.IPv6 with address fe80::9b07:33a1:aa30:e272. avahi-daemon[1123]: New relevant interface enp0s31f6.IPv6 for mDNS. avahi-daemon[1123]: Registering new address record for fe80::9b07:33a1:aa30:e272 on enp0s31f6.*. avahi-daemon[1123]: Joining mDNS multicast group on interface enp0s31f6.IPv4 with address 10.1.0.2. avahi-daemon[1123]: New relevant interface enp0s31f6.IPv4 for mDNS. avahi-daemon[1123]: Registering new address record for 10.1.0.2 on enp0s31f6.IPv4. NetworkManager[11240]: <info> [....0830] device (enp0s31f6): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed') NetworkManager[11240]: <info> [....1005] device (enp0s31f6): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed') NetworkManager[11240]: <info> [....1008] device (enp0s31f6): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed') NetworkManager[11240]: <info> [....1021] device (enp0s31f6): Activation: successful, device activated.
When the connection was defined as DHCP the trace was
NetworkManager: <info> [...] device (enp0s31f6): state change: ip-config -> deactivating (reason 'user-requested', sys-iface-state: 'managed') NetworkManager: <info> [...] audit: op="device-disconnect" interface="enp0s31f6" ifindex=2 pid=2585 uid=1000 result="success" NetworkManager: <info> [...] device (enp0s31f6): state change: deactivating -> disconnected (reason 'user-requested', sys-iface-state: 'managed') avahi-daemon: Withdrawing address record for fe80::78e8:9e55:9f3f:768 on enp0s31f6. avahi-daemon: Leaving mDNS multicast group on interface enp0s31f6.IPv6 with address fe80::78e8:9e55:9f3f:768. avahi-daemon: Interface enp0s31f6.IPv6 no longer relevant for mDNS. NetworkManager: <info> [...] dhcp4 (enp0s31f6): canceled DHCP transaction NetworkManager: <info> [...] dhcp4 (enp0s31f6): state changed unknown -> done NetworkManager: <info> [...] device (enp0s31f6): Activation: starting connection 'Wired connection 1' (ecc4df76-4733-45f5-9b67-9fba9ef2d3bf) NetworkManager: <info> [...] device (enp0s31f6): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed') NetworkManager: <info> [...] device (enp0s31f6): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') NetworkManager: <info> [...] device (enp0s31f6): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed') NetworkManager: <info> [...] dhcp4 (enp0s31f6): activation: beginning transaction (timeout in 45 seconds) avahi-daemon: Joining mDNS multicast group on interface enp0s31f6.IPv6 with address fe80::78e8:9e55:9f3f:768. avahi-daemon: New relevant interface enp0s31f6.IPv6 for mDNS. avahi-daemon: Registering new address record for fe80::78e8:9e55:9f3f:768 on enp0s31f6.*.
This has entries about DHCP.
My machine did not have a DHCP server installed – so any request for DHCP will fail to get an address.
ColinPaice 