How do I trace TCP/IP sockets on z/OS?

I stumbled on this by accident.

In the TCPIP.DATA configuration file you can specify

 TRACE SOCKET

I copied the configuration file, and made the change. I used it in my JCL

//SYSTCPD DD DISP=SHR,DSN=USER.Z24C.TCPPARMS(MYDATA)

Note if you use TRACE SOCKET in the configuration file used by every one – then every one will get their sockets traced – which may not be what you want.

The output to SYSPRINT is like

request = HCreate                                                                         
                                                                                          
EZY3829I  pre   0xe3e2d9c2 00c00001 00010000 00000020 e3c3d7c9 d7404040 00000000 00000000 
EZY3830I        0x00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
EZY3831I        0x00000000 1fa77318 00000000 00000000 00000000 00000080 00000000 00000000 
EZY3832I        0x00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
EZY3833I        0xffff0002 00000000 00000000 40404040 40404040 f18681f7 f68686f8 00000000 
EZY3834I        0x00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
                                                                                          
request = HCreate                                                                         
                                                                                          
EZY3835I  post  0xe3e2d9c2 00c00001 00010000 00000020 e3c3d7c9 d7404040 00000000 00000000 
EZY3830I        0x7f5ec0f0 00010000 00000000 00000000 00000000 00000000 00000000 00000000 
EZY3831I        0x00000000 1fa77318 00000000 00000000 00000000 00000080 00000000 00000000 
EZY3832I        0x00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
EZY3833I        0xffff0002 00000031 00000000 40404040 40404040 f18681f7 f68686f8 00000000 
EZY3834I        0x00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

These messages and the content are not documented – they are for IBM Software Support.

Compiling the TCP/IP samples on z/OS

Communications server (TCPIP) on z/OS provides some samples. I had problems getting these to compile, because the JCL in the documentation was a) wrong and b) about 20 years behind times.

Samples

There are some samples in TCPIP.SEZAINST

  • TCPS: a server which listens on a port
  • TCPC: a client which connects to a server using IP address and port
  • UDPC: C socket UDP client
  • UDPS: C socket UDP server
  • MTCCLNT: C socket Multitasking client
  • MTCSRVR: C socket Multitasking server
  • MTCCSUB: C socket subtask MTCCSUB

The JCL I used is

//COLCOMPI   JOB 1,MSGCLASS=H,COND=(4,LE) 
//S1          JCLLIB ORDER=CBC.SCCNPRC 
// SET LOADLIB=COLIN.LOAD 
// SET LIBPRFX=CEE 
// SET SOURCE=COLIN.C.SOURCE(TCPSORIG) 
//COMPILE  EXEC PROC=EDCCB, 
//       LIBPRFX=&LIBPRFX, 
//       CPARM='OPTFILE(DD:SYSOPTF),LSEARCH(/usr/include/)', 
// BPARM='SIZE=(900K,124K),RENT,LIST,RMODE=ANY,AMODE=31' 
//COMPILE.SYSLIB DD 
//               DD 
//               DD DISP=SHR,DSN=TCPIP.SEZACMAC 
//*              DD DISP=SHR,DSN=TCPIP.SEZANMAC  for IOCTL 
//COMPILE.SYSOPTF DD * 
DEF(_OE_SOCKETS) 
DEF(MVS) 
LIST,SOURCE 
TEST 
RENT ILP32        LO 
INFO(PAR,USE) 
NOMARGINS EXPMAC   SHOWINC XREF 
LANGLVL(EXTENDED) sscom dll 
DEBUG 
/* 
//COMPILE.SYSIN    DD  DISP=SHR,DSN=&SOURCE 
//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB. 
//BIND.SYSLIB  DD DISP=SHR,DSN=TCPIP.SEZARNT1 
//             DD DISP=SHR,DSN=&LIBPRFX..SCEELKED 
//* BIND.GSK     DD DISP=SHR,DSN=SYS1.SIEALNKE 
//* BIND.CSS    DD DISP=SHR,DSN=SYS1.CSSLIB 
//BIND.SYSIN DD * 
  NAME  TCPS(R) 
//START1   EXEC PGM=TCPS,REGION=0M, 
// PARM='4000          ' 
//STEPLIB  DD DISP=SHR,DSN=&LOADLIB 
//SYSERR   DD SYSOUT=*,DCB=(LRECL=200) 
//SYSOUT   DD SYSOUT=*,DCB=(LRECL=200) 
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200) 

Change the source

The samples do not compile with the above JCL. I needed to remove some includes

#include <manifest.h> 
// #include <bsdtypes.h> 
#include <socket.h> 
#include <in.h> 
// #include <netdb.h> 
#include <stdio.h> 

With the original sample I got compiler messages

ERROR CCN3334 CEE.SCEEH.SYS.H(TYPES):66 Identifier dev_t has already been defined on line 98 of “TCPIP.SEZACMAC(BSDTYPES)”.
ERROR CCN3334 CEE.SCEEH.SYS.H(TYPES):77 Identifier gid_t has already been defined on line 101 of “TCPIP.SEZACMAC(BSDTYPES)”.
ERROR CCN3334 CEE.SCEEH.SYS.H(TYPES):162 Identifier uid_t has already been defined on line 100 of “TCPIP.SEZACMAC(BSDTYPES)”.
ERROR CCN3334 CEE.SCEEH.H(NETDB):87 Identifier in_addr has already been defined on line 158 of “TCPIP.SEZACMAC(IN)”.


INFORMATIONAL CCN3409 TCPIP.SEZAINST(TCPS):133 The static variable “ibmcopyr” is defined but never referenced.

I tried many combinations of #define but could not get it to compile, unless I removed the #includes.

Compile problems I stumbled upon

Identifier dev_t has already been defined on line ...                                                     
Identifier gid_t has already been defined on line ...                                                     
Identifier uid_t has already been defined on line ....

This was caused by the wrong libraries in SYSLIB. I needed

  • CEE.SCEEH.H
  • CEE.SCEEH.SYS.H
  • TCPIP.SEZACMAC
  • TCPIP.SEZANMAC

The compile problems were caused by CEE.SCEEH.SYS.H being missing.

Execution problems

I had some strange execution problem when I tried to use AT-TLS within the program.

EDC5000I No error occurred. (errno2=0x05620062)

The errno2 reason from TSO BPXMTEXT 05620062 was

BPXFSOPN 04/27/18
JRNoFileNoCreatFlag: A service tried to open a nonexistent file without O_CREAT

Action: The open service request cannot be processed. Correct the name or the open flags and retry the operation.

Which seems very strange. I have a feeling that this field is not properly initialised and that this value can be ignored.

Sending an email from z/OS

This started off as a question to me about TLS certificates and ended up with a Python solution.

The short version of this blog post is I could send an email to Gmail using Python – but could not use the IBM CSSMTP solution because it does not support authentication. I was also sent a link to Sending Email from z/OS using Java.

As far as I can tell, SMTP authentication does not use client certificates. If you want to use userid and password, make sure the session is encrypted using TLS.

What types of email are there?

Through JES spool

On my system with id S0W1, I can use TSO XMIT S0W1.IBMUSER and send an email.

The recipient (IBMUSER) can use TSO receive to read the email.

You can use XMIT OVERTHER.MYID to send the email over NJE to other system, such as z/OS, z/VM and z/VSE. You say (on z/OS) xmit node.userid da(‘…’). or xmit node.userid . The JES2 configuration has an IP address and port to send it to a remote spool.

JES2 destinations

You can configure destinations on JES2, and have program (“a writer”) process the node name (destination) data.

You can display the defined destinations using the JES2 command

$DDESTID

Example JCL to write to a destination,

//COPY EXEC PGM=IEBGENER 
//SYSPRINT DD SYSOUT=* 
//SYSIN DD DUMMY 
//SYSUT2 DD SYSOUT=(A,MYDEST),SPIN=UNALLOC 
//SYSUT1 DD * 
line1
/*
// 

This writes the data from SYSUT1 to SYOUT2 which goes to nodename MYDEST.

You configure CSSMP and specify ExtWrtName MYDEST, for CSSMTP to read from.

SMTP

There is Simple Mail Transport Protocol. A stream of data is provided such as

MAIL FROM:<ME@MYSYSTEM.COM> 
RCPT TO:<MYBOSS@MYSYSTEM.COMU> 
RCPT TO:<YOU@yoursystem.com> 
DATA 
Blah blah blah. 
QUIT 

which is sent to the mail server over TCPIP (usually port 25 or 587).

SMTP is a popular protocol. It can support encrypted session encryption. The protocol can support different levels of authorisation.

z/OS provides CSSMTP which reads from the spool and sends the data over TCPIP to a server. However CSSMTP does not seems to support the passing of authentication information to the server – and as most mail servers want you to authenticate before sending emails – CSSMTP cannot be used for these. I know this from the documentation because the AUTH command is not listed in the list of supported SMTP commands, and AUTH does not work!

Using Python to send an email.

I used Python to send an email to my Gmail account. The script I used is

#!/usr/bin/python 
import smtplib 
import ssl 
import sys 
from email.mime.text import MIMEText 
host = "173.194.76.108" 
sender = 'colinpaicetest@gmail.com' 
receivers = ['colinpaicemq@gmail.com','colinpaicetest@gmail.com'] 
msg = MIMEText('This is test mail2') 
msg['Subject'] = 'Test mail3' 
msg['From'] = 'colinpaice3@gmail.com' 
msg['To'] ='colinpaicemq@gmail.com , colinpaicetest@gmail.com' 
with smtplib.SMTP(host , 587) as server: 
      # if you need to validate the certificate sent to client.. you need a context
      context=ssl.create_default_context(
                 purpose=ssl.Purpose.CLIENT_AUTH,cafile="pem.pem") 
      server.starttls(context=context) 
      # server.starttls() # for servers that do not send a certificate 
      print(server.esmtp_features) # print info about the connection such as auth types
      server.login('colinpaice3@gmail.com', 'abcpasswordxyz') 
      server.sendmail(sender, receivers, msg.as_string()) 
      print("Successfully sent email") 
  • context=ssl.create_default_context(purpose=ssl.Purpose.CLIENT_AUTH,cafile=”pem.pem”) is used to set up the CA cert for verifying any certificate sent from the server. GMAIL does not send a certificate, so the CA is not needed, and you can use server.starttls()
  • server.starttls(context=context) creates the TLS session. This is handled by PAGENT.
  • server.login(‘colinpaice3@gmail.com’, ‘ZYC123….’) does the authentication
  • server.sendmail(sender, receivers, msg.as_string()) sends the message.

The id in the sender must match the id in the login.

This does not use PAGENT or AT-TLS.

In the TLS handshake 57 cipher specs were sent to the server, and cipher spec TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xC02B) was used.

Setting up CSSMTP

Basic configuration

Even though I could not use CSSMTP because I could not authenticate, I learned a lot on my doomed journey.

In the CSSMTP configuration I had

ExtWrtName    MYDEST             # the external writer name 
TargetServer 
{ 
   TargetIp aaa.bbb.ccc.ddd      # This IP address 
  # TargetName  
   ConnectPort   25              # port to connect to target server 
   Secure        Yes             # Transport Layer Security (TLS) 
} 
...

Each CSSMTP instance reads from one nodeName and sends to one server. If you want to support more nodeNames you need more CSSMTP instances.

Certificates

As part of the TLS handshake, some servers will send down a certificate (to authenticate the server).

If your TLS provider, such as AT-TLS, uses keyrings, you need to import certificate into RACF, and connect the CA certificate to the keyring. Your keyring needs the CA for this certificate. You can get the servers certificate from the network flow, or from an openssl command.

If you are not using keyrings, for example Python uses openssl, you need the certificate in a file in Unix Services.

Getting my z/OS client to talk to a remote server over TLS

I looked into this because someone asked me about using the mail client on z/OS talking to a mail server somewhere in the internet.

My z/OS running on zPDT under Linux did not have a working connection to the outside world.

Configuring Linux to pass the z/OS traffic to the external network.

A ping to the server, got no response back.

Using Wireshark on the wireless connection, I could see my request from z/OS coming via Linux and going to the outside network. The source IP address was 10.1.1.1 – a “local only” address, and so the server could not send a response back to me.

The routing through Linux worked because I had a default route; ip route gave me

default via 192.168.xxx.xxx4 dev wlxd037450ab7ac proto dhcp metric 600 
10.1.0.0/24 dev eno1 proto kernel scope link src 10.1.0.3 metric 100 
10.1.0.0/24 via 10.1.0.3 dev eno1 proto static metric 100 
10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1 

To get the “right” IP address passed to the external network, was surprisingly easy. It was documented in the zPDT book.

I created a script

echo ‘Your firewall must be enabled for this command to be meaningful’
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -F FORWARD
iptables -P FORWARD ACCEPT
iptables -t nat -A POSTROUTING -o wlxd037450ab7ac -j MASQUERADE
# iptables -I INPUT -p tcp --dport 3270 -j ACCEPT

and used sudo to execute it.

I started the firewall

sudo ufw status
sudo ufw enable

I could then successfully ping the address

When I used Wireshark on the connection between Linux and z/OS, the source IP address was 10.1.1.2. When I used Wireshark on the wireless connection between Linux and the external network, the IP address was 192.168.xxx.xxx. This shows the effect of the iptables commands, it changed the 10.1.1.2 to a 192.168.xxx.xxx – and back again on the replies.

Getting the server’s certificate

I configured the AT-TLS definitions, for TLS support, specifying the choice of cipher specs I wanted to be used, and specified the keyring.

My z/OS client connected to the remote server. I could see the “client hello” flow, and the “server hello” response. In the server hello flow were the CA certificates the server supported. The client validates the certificates by checking the client’s key store.

I did not have the matching CA certificate in my keystore, and so my client failed to validate the certificate sent down, and the connection ended.

I could do it properly, and get the certificate from the official source for the server, but this was a test system, and I was happy to trust what was sent down in the handshake.

You can use wireshark or openssl to get the certificate

With openssl

openssl s_client -showcerts -tls1_2 -starttls smtp -connect smtp.gmail.com:587 | sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’

You’ll need to interupt it (Ctrl-C) and paste the data into a file (google.pem)

You can display the contents of the certificate using

openssl x509 -in google.pem -text -noout|less

With Wireshark

display the certificates in the Server Hello flow.

right click on the certificate and use right click, export packet bytes. I created a file user.crt. This is a binary file.

The openssl command (on Linux)

openssl x509 -inform der -in user.crt -text > usercert.pem

displays the file in .pem format.

Put the certificate into the keyring

If you are using something that needs a keyring, you need to import the certificate into RACF and connect it to a keyring. If you are using something that uses openssl, such as Python, you need a .pem file in Unix Services.

I created a file on z/OS (VB) COLIN.USERCERT.PEM, and copied the .pem file into it (between and including the BEGIN CERTIFICATE and END CERTIFICATE lines).

I imported it, and connected it to my keyring using

//IBMRACF  JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=*                                           
//SYSTSPRT DD SYSOUT=*                                           
//SYSTSIN DD *                                                   
RACDCERT CHECKCERT('COLIN.USERCERT.PEM')                         
RACDCERT DELETE  (LABEL('USERTRUST.ORG'))    CERTAUTH            
RACDCERT ADD('COLIN.USERCERT.PEM')        -                      
   CERTAUTH  WITHLABEL('USERTRUST.ORG') TRUST 
RACDCERT ID(START1) CONNECT(RING(TN3270) - 
                            CERTAUTH    - 
                            LABEL('USERTRUST.ORG')) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh                     
                                                                 
RACDCERT LISTRING(TN3270)  ID(START1)                            
/*                                                              

Once I had refreshed my PAGENT job, and restarted my client code, I was able to establish a TLS session.

Client certificate

In the TLS handshake, the mail servers did not send down a request for the client certificate. As no certificate was requested, or sent up to the server, no certificate authentication was done. You need to check your mail server for it authentication process – it may just be userid and password. This information is sent once the TLS session has been established – so it flows encrypted.

z/OS PAGENT error messages and system SSL codes

Message

SYSERR :001: plfm_kernel_init: socket(INET, DGRAM, 0), failed, errno=EDC5112I Resource temporarily unavailable., errno2=74610296

OBJERR :001: init_PEP_and_kernel: Kernel initialization failed for image ‘TCPIP2’,

RACF profile EZB.INITSTACK.*.TCPIP2 CLASS(SERVAUTH) was missing.

//S1 EXEC PGM=IKJEFT01,REGION=0M
//STEPLIB DD DISP=SHR,DSN=SYS1.MIGLIB
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RDEFINE SERVAUTH EZB.INITSTACK..TCPIP2 PERMIT EZB.INITSTACK..TCPIP2 CLASS(SERVAUTH)
ID(START1) ACCESS(READ)
SETROPTS RACLIST(SERVAUTH) REFRESH
/*

WARNING:005: ..pinit_fetch_policy_profile: Ignoring non-image config file

In my image file I had a

TTLSRule
{

}

which is not allowed. They have to be in the TTLSConfig… file.

The error message was

WARNING:005: ..pinit_fetch_policy_profile: Ignoring non-image config file ‘TTLSRule’ statement

402 No SSL cipher specifications.

In some situations the cipher specs default.

You have to specify them for TLS 1.3

RC: Code 515 Initial handshake

Code 515 Key share list is not valid

I got this when trying to use TLS 1.3.

You need

TTLSSignatureParms
{
ClientKeyShareGroups 0023
ClientKeyShareGroups 0024
ClientKeyShareGroups 0025
ClientKeyShareGroups 0030

}

or ServerKeyShareGroup

517 No matches between elliptic curve and key share lists

Need something like

TTLSSignatureParms CPESigParms
{
CLientECurves 0023
CLientECurves 0024
CLientECurves 0025
CLientECurves 0029
ClientKeyShareGroups 0023
ClientKeyShareGroups 0024
ClientKeyShareGroups 0025
ClientKeyShareGroups 0029
}

519 Required ciphers have not been specified

I had TLS 1.3 specified, but no TLS 1.3 cipher specs.

Getting the simplest OSPF network to work.

I struggled (and failed) to get OSPF routing to work with IPV6, so I tried with IP V4. This only took a couple of hours to get working. But I could not find any documentation which had baby steps to show you how it works, and what any output means.

This blog post is getting two Linux machines and z/OS to work with IPV4 and OSPF routing.

Some other blog articles give examples of the commands you can use to explore the configuration, and find what is running where.

What is OSPF

OSPF is a routing protocol where a router knows the topology of the network – rather than just the next hop. As the network changes, the changes are sent to the routers and their picture of the network is updated. OSPF scales to large number of routers.

My configuration

I used the frr (Free Range Routing) package which has routing capabilities for OSPF, OSPF6, RIP etc.

The laptop had

  • ip v4 address 10.1.0.2/24
  • routes
    • 10.1.0.0/24 dev enp0s31f6 proto kernel scope link src 10.1.0.2 metric 100
    • 10.1.0.0/24 via 10.1.0.2 dev enp0s31f6 proto static metric 100
    • 10.1.1.0/24 via 10.1.0.3 dev enp0s31f6
  • ospf router id 1.2.3.4

The server had

  • ip v4 address 10.1.0.3/24
  • routes
    • 10.1.0.0/24 dev eno1 proto kernel scope link src 10.1.0.3 metric 100
    • 10.1.0.0/24 via 10.1.0.3 dev eno1 proto static metric 100
  • ospf router-id 9.2.3.4

The z/OS system has

  • ip v4 address 10.1.1.2
  • routes
    • 10.1.0.0/24 via 10.1.1.1 on ETH1
    • 10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1
  • ospf router-id 10.1.1.2

Laptop frr.conf configuration file

The configuration file is described here.

frr version 7.2.1
frr defaults traditional
hostname laptop
log file /var/log/frr/frr.log
log timestamp precision 6

hostname laptop
service integrated-vtysh-config
...

!
interface enp0s31f6
 description colins ospf
 ip address 10.1.0.2 peer 10.1.0.3/24
 ip ospf area 0.0.0.0

!
router ospf
 ospf router-id 1.2.3.4

line vty

Server frr.conf configuration file

frr version 7.2.1
frr defaults traditional
hostname colin-ThinkCentre-M920s
log file /var/log/frr/frr.log
log timestamp precision 6
hostname Server
service integrated-vtysh-config

interface eno1
 description colins ospf
 ip address 10.1.0.3 peer 10.1.0.2/24
 ip ospf area 0.0.0.0

!
router ospf
 ospf router-id 9.2.3.4
!
line vty

z/OS configuration

TCPIP configuration file – defining ETH1

DEVICE PORTA  MPCIPA 
LINK ETH1  IPAQENET PORTA 
HOME 10.1.1.2 ETH1 
PORT 
   520 UDP OMP2                ; RouteD Server 
BEGINRoutes 
;     Destination   SubnetMask    FirstHop       LinkName  Size 

ROUTE 10.0.0.0    255.0.0.0           =        ETH1 MTU 1492 
ROUTE DEFAULT                     10.1.1.1     ETH1 MTU 1492 
ROUTE 10.1.0.0    255.255.255.0   10.1.1.1     ETH1 MTU 1492 
ROUTE 10.1.1.0    255.255.255.0       =        ETH1 MTU 1492 
ENDRoutes 
ITRACE OFF 
IPCONFIG NODATAGRAMFWD 
UDPCONFIG RESTRICTLOWPORTS 
TCPCONFIG RESTRICTLOWPORTS 
TCPCONFIG TTLS 
START PORTA 

JFPORTCP4 Interface configuration

This is in member USER.Z24C.TCPPARMS(jFACE41)

INTERFACE JFPORTCP4 
    DEFINE IPAQENET 
    CHPIDTYPE OSD 
    IPADDR 10.1.3.2 
    PORTNAME PORT2 

activate and start this using

v tcpip,tcpip,obeyfile,USER.Z24C.TCPPARMS(jFACE41) 

v tcpip,tcpip,sta,jfportcp4

OMPROUTE procedure

//OMPROUTE PROC 
// SET PO='POSIX(ON)' 
//OMPROUTE EXEC PGM=OMPROUTE,REGION=0M,TIME=NOLIMIT, 
// PARM=('&PO.,ENVAR("_CEE_ENVFILE_S=DD:STDENV")/ -6t2 -6d2') 
//OMPCFG DD DISP=SHR,DSN=USER.Z24C.TCPPARMS(&SYSJOBNM) 
//STDENV DD DISP=SHR,DSN=USER.Z24C.TCPPARMS(ENV&SYSJOBNM) 
//SYSPRINT DD SYSOUT=* 
//SYSOUT   DD SYSOUT=* 
//SYSTCPD DD DISP=SHR,DSN=ADCD.Z24C.TCPPARMS(TCPDATA) 
//CEEDUMP  DD SYSOUT=*,DCB=(RECFM=FB,LRECL=132,BLKSIZE=132) 
//  PEND 

and started with

S OMPROUTE,jobname=omp1

USER.Z24C.TCPPARMS(ENV&SYSJOBNM)

RESOLVER_CONFIG=//'ADCD.Z24C.TCPPARMS(TCPDATA)' 
OMPROUTE_DEBUG_FILE=/tmp/logs/omproute.debug 
OMPROUTE_IPV6_DEBUG_FILE=/tmp/logs/omprout6.debug 
OMPROUTE_DEBUG_FILE_CONTROL=1000,5 

OMPROUTE configuration USER.Z24C.TCPPARMS(OMP1)

ospf  RouterID=10.1.1.2; 
                                           
ospf_interface IP_address=10.1.1.2 
      name=ETH1 
      subnet_mask=255.255.255.0 
      ; 
ospf_interface IP_address=10.1.3.2 
      name=JFPORTCP4 
      subnet_mask=255.255.255.0 
      ; 

Startup joblog messages

EZZ7800I OMP1 STARTING
EZZ8171I OMP1 IPV4 OSPF IS USING CONFIGURED ROUTER ID 10.1.1.2 FROM OSPF STATEMENT
EZZ7898I OMP1 INITIALIZATION COMPLETE
EZZ8100I OMP1 SUBAGENT STARTING

OSPF on z/OS, basic commands

This article follows on from getting the simplest example of OSPF working. It gives the z/OS commands to display useful information.

I want to


OMP1

I configured multiple TCPIP subsystems, and each one had an OMPROUTE defined. I used a started task OEMP1, as the OMPROUTE for my base TCPIP.

If you have only one TCPIP subsystem, you can use OMPROUTE as your name.

F OMP1,OSPF,areasum

This displays the area summary.

AREA ID        AUTHENTICATION   #IFCS  #NETS  #RTRS  #BRDRS DEMAND     
0.0.0.0           NONE              2      3      3      0  OFF        

F OMP1,OSPF,EXTERNAL

EZZ7853I AREA LINK STATE DATABASE                        
TYPE LS DESTINATION     LS ORIGINATOR     SEQNO     AGE   XSUM
                # ADVERTISEMENTS:       0                     
                CHECKSUM TOTAL:         0X0                   

F OMP1,ospf,list,areas

“Displays all information concerning configured OSPF areas and their associated ranges.”

 EZZ7832I AREA CONFIGURATION 820 
 AREA ID          AUTYPE          STUB? DEFAULT-COST IMPORT-SUMMARIES? 
 0.0.0.0          0=NONE           NO          N/A           N/A 
                                                                               
 --AREA RANGES-- 
 AREA ID          ADDRESS          MASK             ADVERTISE? 
 0.0.0.0          11.11.0.0        255.255.255.0    YES 

The entry with address 11.11.0.0 comes from the omproute configuration file entry

range ip_address=11.11.0.1 
      subnet_mask=255.255.255.0 
      ; 

F OMP1,ospf,list,ifs

“For each OSPF interface, display the IP address and configured parameters as coded in the
OMPROUTE configuation file”

 EZZ7833I INTERFACE CONFIGURATION 822 
 IP ADDRESS      AREA             COST RTRNS TRDLY PRI HELLO  DEAD DB_EX 
 10.1.3.2        0.0.0.0             1     5     1   1    10    40    40 
 10.1.1.2        0.0.0.0             1     5     1   1    10    40    40 

F OMP1,ospf,list,nbma

“Displays the interface address and polling interval related to interfaces connected to nonbroadcast multiaccess networks.”

 NBMA CONFIGURATION 824 
 INTERFACE ADDR      POLL INTERVAL 
 << NONE CONFIGURED >> 

F OMP1,ospf,list,nbrs

“Displays the configured neighbors on non-broadcast networks”

 NEIGHBOR CONFIGURATION 826 
 NEIGHBOR ADDR     INTERFACE ADDRESS   DR ELIGIBLE? 
 << NONE CONFIGURED >> 

“Displays all virtual links that have been configured with this router as an endpoint.”

F OMP1,ospf,database,areaid=0.0.0.0

EZZ7853I AREA LINK STATE DATABASE                           
TYPE LS DESTINATION     LS ORIGINATOR     SEQNO     AGE   XSUM     
  1  1.2.3.4            1.2.3.4         0X80000013   61  0X3D8D    
  1  9.2.3.4            9.2.3.4         0X8000001A  393  0X5A78    
  1 @10.1.1.2           10.1.1.2        0X8000000D  286  0X9E22    
  2  10.1.0.2           1.2.3.4         0X80000006 1241  0XC35E    
  2  10.1.1.1           9.2.3.4         0X80000003  353  0X8197    
  2 @10.1.1.2           10.1.1.2        0X80000005 3600  0X64BD    
  2  10.1.3.1           9.2.3.4         0X80000003  383  0X6BAB    
  2 @10.1.3.2           10.1.1.2        0X80000005 3600  0X4ED1    

(LS) Type is described here.

  1. Router links advertisement
  2. Network link advertisement
  3. Summary link advertisement
  4. Summary ASBR advertisement
  5. Autonomous System (AS -think entire network) external link.
  • LS ORIGINATOR: Indicates the router that originated the advertisement.
  • LS DESTINATION: Indicates an IP destination (network, subnet, or host).

From the above

TYPE LS DESTINATION     LS ORIGINATOR
  2  10.1.0.2           1.2.3.4        

means router 1.2.3.4 told every one that it has the end of a network link, and its address is 10.1.0.2.

TYPE LS DESTINATION     LS ORIGINATOR      
  1  1.2.3.4            1.2.3.4

says router 1.2.3.4 told every one “here I am, router 1.2.3.4”.

You can use the type and destination in the command:

F OMP1,OSPF,LSA,LSTYPE=…,LSID=…

For example

below.

F OMP1,OSPF,LSA,LSTYPE=1,LSID=1.2.3.4

This allows you to see a lot of information about an individual element of the OSPF database.

LSTYPE=1 is for Router Links Advertisment.

The valid LSID values are given in the output of F OMP1,ospf,database,areaid=0.0.0.0 above.

F OMP1,OSPF,LSA,LSTYPE=1,LSID=9.2.3.4 
EZZ7880I LSA DETAILS  
  LS DESTINATION (ID): 9.2.3.4                     
  LS ORIGINATOR:   9.2.3.4 
  ROUTER TYPE:      (0X00)                         
  # ROUTER IFCS:   3                        
    LINK ID:          10.1.0.2        
    LINK DATA:        10.1.0.3        
    INTERFACE TYPE:   2               
    
    LINK ID:          10.1.1.2        
    LINK DATA:        10.1.1.1        
    INTERFACE TYPE:   2               
   
    LINK ID:          10.1.3.2        
    LINK DATA:        10.1.3.1        
    INTERFACE TYPE:   2 
  • LINK ID: Is the IP address of the remote end
  • LINK DATA: Is the IP address of the router’s end
  • INTERFACE TYPE: 2 is “Network links”.

F OMP1,OSPF,LSA,LSTYPE=2,LSID=10.1.0.3

This allows you to see a lot of information about an individual element of the OSPF database.

LSTYPE=2 is “Network links the set of routers attached to a network”.

The valid LSID values are given in the output of F OMP1,ospf,database,areaid=0.0.0.0 above, with type=2.

F OMP1,OSPF,LSA,LSTYPE=2,LSID=10.1.0.3                     
EZZ7880I LSA DETAILS                                   
LS OPTIONS:      E (0X02)                          
LS TYPE:         2                                 
LS DESTINATION (ID): 10.1.0.3                      
LS ORIGINATOR:   9.2.3.4                           
NETWORK MASK:    255.255.255.0                     
 ATTACHED ROUTER: 1.2.3.4          (100)    
 ATTACHED ROUTER: 9.2.3.4          (100)    

Where (100) is the metric.

F OMP1,ospf,if

 EZZ7849I INTERFACES 832 
 IFC ADDRESS     PHYS         ASSOC. AREA     TYPE   STATE  #NBRS  #ADJS 
 10.1.3.2        JFPORTCP4    0.0.0.0         BRDCST   64      1      1 
 10.1.1.2        ETH1         0.0.0.0         BRDCST   64      1      1 

F OMP1,ospf,neighbor

EZZ7851I NEIGHBOR SUMMARY 834 
 NEIGHBOR ADDR   NEIGHBOR ID     STATE  LSRXL DBSUM LSREQ HSUP IFC 
 10.1.3.1        9.2.3.4           128      0     0     0  OFF JFPORTCP4 
 10.1.1.1        9.2.3.4           128      0     0     0  OFF ETH1 

F OMP1,ospf,routers

EZZ7855I OSPF ROUTERS 836 
DTYPE RTYPE DESTINATION AREA COST NEXT HOP(S)
NONE

F OMP1,ospf,statistics

EZZ7856I OSPF STATISTICS 838 
                 OSPF ROUTER ID:         10.1.1.2 (*OSPF) 
                 EXTERNAL COMPARISON:    TYPE 2 
                 AS BOUNDARY CAPABILITY: NO 
                                                                          
 ATTACHED AREAS:                  1  OSPF PACKETS RCVD:             3336 
 OSPF PACKETS RCVD W/ERRS:        0  TRANSIT NODES ALLOCATED:         84 
 TRANSIT NODES FREED:            78  LS ADV. ALLOCATED:                1 
 LS ADV. FREED:                   1  QUEUE HEADERS ALLOC:             32 
 QUEUE HEADERS AVAIL:            32  MAXIMUM LSA SIZE:               512 
 # DIJKSTRA RUNS:                 4  INCREMENTAL SUMM. UPDATES:        0 
 INCREMENTAL VL UPDATES:          0  MULTICAST PKTS SENT:           3371 
 UNICAST PKTS SENT:               7  LS ADV. AGED OUT:                 1 
 LS ADV. FLUSHED:                 1  PTRS TO INVALID LS ADV:           0 
 INCREMENTAL EXT. UPDATES:        0 

F OMP1,OSPF,LSA,LSTYPE=2,LSID=10.1.0.3

Where

  • LSTYPE=2 is “Network links the set of routers attached to a network”.
  • 10.1.0.3 is an LS destination (from F OMP1,ospf,database,areaid=…) It comes from the frr definition below
interface eno1
   ip address 10.1.0.3 peer 10.1.0.2/24
 

Only addresses on the Server are accepted. Addresses from the Laptop are not valid.

In the command F OMP1,OSPF,LSA,LSTYPE=1,LSID=1.2.3.4, some of the LINK IDs seem to be valid.

F OMP1,OSPF,LSA,LSTYPE=1,LSID=x.x.x.x

This allows you to see a lot of information about an individual element of the OSPF database.

The LSATYPE is described in here. LSTYPE=1 is for Router Links Advertisment.

The LSID is one of the routers, for example in

  • F OMP1,ospf,database,areaid=0.0.0.0, it displays, LS DESTINATION LS ORIGINATOR
  • F OMP1,ospf,neighbor, it displays NEIGHBOR ID
F OMP1,OSPF,LSA,LSTYPE=1,LSID=9.2.3.4 
EZZ7880I LSA DETAILS  
  LS DESTINATION (ID): 9.2.3.4                     
  LS ORIGINATOR:   9.2.3.4 
  ROUTER TYPE:      (0X00)                         
  # ROUTER IFCS:   3                               
     LINK ID:          10.1.0.3               
     LINK DATA:        10.1.0.3               
        INTERFACE TYPE:   2
     LINK ID:          10.1.1.1
     LINK DATA:        10.1.1.1              
        INTERFACE TYPE:   2 
     LINK ID:          10.1.3.1              
     LINK DATA:        10.1.3.1              
        INTERFACE TYPE:   2 

F OMP1,RTTABLE

EZZ7847I ROUTING TABLE 842 
 TYPE   DEST NET         MASK      COST    AGE     NEXT HOP(S) 
                                                                        
 STAT*  10.0.0.0         FF000000  0       16079   10.1.1.2 
  SPF   10.1.0.0         FFFFFF00  101     16071   10.1.1.1         (2) 
  SPF*  10.1.1.0         FFFFFF00  1       16078   ETH1 
  SPF*  10.1.3.0         FFFFFF00  1       16078   JFPORTCP4 
  SPF   11.1.0.2         FFFFFFFF  201     4733    10.1.1.1         (2) 
                        0 NETS DELETED, 3 NETS INACTIVE 

(2) is the number of equal-cost routes to the destination.

D TCPIP,,OMPROUTE,RTTABLE,DEST=10.1.0.0

gives

EZZ7874I ROUTE EXPANSION 105                   
DESTINATION:    10.1.0.0                       
MASK:           255.255.255.0                  
ROUTE TYPE:     SPF                            
DISTANCE:       101                            
AGE:            943                            
NEXT HOP(S):    10.1.1.1          (ETH1)       
                10.1.3.1          (JFPORTCP4)  

OSPF on Linux with frr: the basic commands

This article follows on from getting the simplest example of OSPF working. It gives the frr commands to display useful information.

How to extract useful information

This article is a good introduction in drawing the network based on the information from OSPF.

Filter the output

With the frr show commands you can use regular expressions to filter the output data.

show ip ospf database route | include address|router

gives

laptop# show ip ospf database route  | include address|router
  LS Type: router-LSA
     (Link ID) Designated Router address: 10.1.0.3
     (Link Data) Router Interface address: 10.1.0.2
  LS Type: router-LSA
     (Link ID) Designated Router address: 10.1.0.3
     (Link Data) Router Interface address: 10.1.0.3
     ...

You can also issue the sudo vtysh -c “show ip database route” | …. and use standard Linux facilities like grep, less and sort.

Use JSON

You can display the output in JSON format, for example

show ip ospf route json

gives

Server# show ip ospf route json 
{ "10.1.0.0/24": { "routeType": "N", "cost": 100, "area": "0.0.0.0", "nexthops": [ { "ip": " ", "directly attached to": "eno1" } ] }... }

With JSON you can find out the field names for example “cost” has a value 100.

Options and flags

Many commands give options and flags, such as

Options: 0x2 : *|-|-|-|-|-|E|-
LS Flags: 0x6

I’ve collected some interpretation on these here.

I want to…


frr commands

show ip ospf

 OSPF Routing Process, Router ID: 1.2.3.4
 ...
 Number of areas attached to this router: 1
 Area ID: 0.0.0.0 (Backbone)
   Number of interfaces in this area: Total: 1, Active: 1
   Number of fully adjacent neighbors in this area: 1
   Area has no authentication
   SPF algorithm executed 4 times
   Number of LSA 5
   Number of router LSA 3. Checksum Sum 0x000109da
   Number of network LSA 2. Checksum Sum 0x000139df
   Number of summary LSA 0. Checksum Sum 0x00000000
   ...

There are 3 router Link States, and 2 network Link States; they are displayed below:

show ip ospf database

OSPF Router with ID (1.2.3.4)
  Router Link States (Area 0.0.0.0)
    Link ID         ADV Router      Age  Seq#       CkSum  Link count
    1.2.3.4        1.2.3.4          288 0x80000003 0x15a9 1
    9.2.3.4        9.2.3.4          288 0x80000007 0x56f1 2
    10.1.1.2       10.1.1.2        1078 0x8000001e 0x9d40 1
  Net Link States (Area 0.0.0.0)
    Link ID         ADV Router      Age  Seq#       CkSum
    10.1.0.3       9.2.3.4          289 0x80000001 0x7ba2
    10.1.1.2       10.1.1.2        1082 0x80000003 0xbe3d

show ip ospf database router self-originate

This shows the links attached to this OSPF environment.

OSPF Router with ID (9.2.3.4)
Router Link States (Area 0.0.0.0)
Link State ID: 9.2.3.4 
Number of Links: 2
  Link connected to: Stub Network
  (Link ID) Net: 10.1.0.0
  (Link Data) Network Mask: 255.255.255.0

  Link connected to: a Transit Network
  (Link ID) Designated Router address: 10.1.1.2
  (Link Data) Router Interface address: 10.1.1.1

show ip ospf database router

  OSPF Router with ID (1.2.3.4)
  Router Link States (Area 0.0.0.0)
===================================
  LS age: 387
  Options: 0x2  : *|-|-|-|-|-|E|-
  LS Flags: 0x3  
  Flags: 0x0
  LS Type: router-LSA
  Link State ID: 1.2.3.4 
  Advertising Router: 1.2.3.4
  ...
  Length: 36

  Number of Links: 1

  Link connected to: a Transit Network
    (Link ID) Designated Router address: 10.1.0.3
    (Link Data) Router Interface address: 10.1.0.2
   ...
===================================
  LS Type: router-LSA
  Link State ID: 9.2.3.4 
  Advertising Router: 9.2.3.4
 
 Number of Links: 2
 Link connected to: a Transit Network
  (Link ID) Designated Router address: 10.1.0.3
  (Link Data) Router Interface address: 10.1.0.3

 Link connected to: a Transit Network
  (Link ID) Designated Router address: 10.1.1.2
  (Link Data) Router Interface address: 10.1.1.1
===================================
  LS Type: router-LSA
  Link State ID: 10.1.1.2 
  Advertising Router: 10.1.1.2
  Number of Links: 1
  Link connected to: a Transit Network
   (Link ID) Designated Router address: 10.1.1.2
   (Link Data) Router Interface address: 10.1.1.2 

show ip ospf database router 9.2.3.4

laptop# show ip ospf database router 9.2.3.4
OSPF Router with ID (1.2.3.4)
Router Link States (Area 0.0.0.0)
LS Type: router-LSA
Link State ID: 9.2.3.4 
Advertising Router: 9.2.3.4
Number of Links: 2
  Link connected to: a Transit Network
  (Link ID) Designated Router address: 10.1.0.3
  (Link Data) Router Interface address: 10.1.0.3

  Link connected to: a Transit Network
  (Link ID) Designated Router address: 10.1.1.1
  (Link Data) Router Interface address: 10.1.1.1 

show ip ospf database network

laptop# show ip ospf database network 

  OSPF Router with ID (1.2.3.4)
  Net Link States (Area 0.0.0.0)
  ====   
  LS age:
  LS Type: network-LSA
  Link State ID: 10.1.0.3 (address of Designated Router)
  Advertising Router: 9.2.3.4
 
  Network Mask: /24
    Attached Router: 1.2.3.4
    Attached Router: 9.2.3.4
  ====
  LS age:...
  LS Type: network-LSA
  Link State ID: 10.1.1.2 (address of Designated Router)
  Advertising Router: 10.1.1.2
  Network Mask: /24
    Attached Router: 10.1.1.2
    Attached Router: 9.2.3.4

show ip ospf route

Server# show ip ospf route
============ OSPF network routing table ============
N    10.1.0.0/24           [100] area: 0.0.0.0
                           directly attached to eno1
N    10.1.1.0/24           [10000] area: 0.0.0.0
                           directly attached to tap0
N    10.1.3.0/24           [10000] area: 0.0.0.0
                           directly attached to tap2
N    11.1.0.2/32           [200] area: 0.0.0.0
                           via 10.1.0.2, eno1

Where

  • N is the route type,
    • N, Network, Intra area
    • N IA, network, Inter area
    • D IA, Discard Inter area
  • 10.1.0.0.24 is the IP address
  • [] is the cost
  • 0.0.0.0 is the area

show ip ospf interface enp0s31f6

This command shows the interface on the local system. I’ve displayed what I think is important. There are many more parameters, and it is missing the description from the configuration file!

enp0s31f6 is up
  ... 
  ifindex 2, MTU 1500 bytes, BW 1000 Mbit <UP,BROADCAST,RUNNING,MULTICAST>
  Internet Address 10.1.0.2/24, Broadcast 10.1.0.255, Area 0.0.0.0
  Router ID 1.2.3.4, Network Type BROADCAST, Cost: 100
  Designated Router (ID) 9.2.3.4 Interface Address 10.1.0.3/24
  Backup Designated Router (ID) 1.2.3.4, Interface Address 10.1.0.2
  Neighbor Count is 1, Adjacent neighbor count is 1
  ...

show ip ospf interface traffic

Interface HELLO   DB-Desc LS-Req LS-Update LS-Ack Packets      
          Rx/Tx   Rx/Tx   Rx/Tx  Rx/Tx     Rx/Tx  Queued       
----------------------------------------------------------
enp0s31f6 128/129 4/3     1/1    11/5      4/10   0

show ip ospf router-info

This just shows a setting – or Router Information is disabled on this router.

show ip route

The output below shows there is one OSPF defined route (which has been active for 1 hour 9:51 minutes). (There are other routes defined.)

Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/600] via 192.168.1.254, wlp4s0, 01:10:41
O   10.1.0.0/24 [110/100] is directly connected, enp0s31f6, weight 1, 01:10:41
K * 10.1.0.0/24 [0/100] via 10.1.0.2, enp0s31f6, 01:10:41
C>* 10.1.0.0/24 is directly connected, enp0s31f6, 01:10:41
O   10.1.1.0/24 [110/10100] via 10.1.0.3, enp0s31f6, weight 1, 01:09:51
K>* 10.1.1.0/24 [0/0] via 10.1.0.3, enp0s31f6, 01:10:41
K>* 10.2.1.0/24 [0/0] is directly connected, enp0s31f6, 01:10:41
K>* 10.3.1.0/24 [0/0] via 10.1.0.3, enp0s31f6, 01:10:41
K>* 169.254.0.0/16 [0/1000] is directly connected, virbr0 linkdown, 01:10:41
C>* 192.168.1.0/24 is directly connected, wlp4s0, 01:10:41

show ip rpf

Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

C>  10.1.0.0/24 is directly connected, enp0s31f6, 01:12:22
C>  192.168.1.0/24 is directly connected, wlp4s0, 01:12:22

Options and flags

You get information displayed like

Options: 0x2  : *|-|-|-|-|-|E|-
LS Flags: 0x6  

Where the options are: See Wikipedia.

  • * reserved
  • O – router’s willingness to receive and forward Opaque-LSAs
  • DC – Handling of Demand Circuits
  • EA” : “-“, describes the router’s willingness to receive and forward External Attributes LSAs
  • N/P – if area is NSSA.
  • MC – Multicast datagrams forwarded
  • E – external link advertisements are not flooded into OSPF
  • M/T – Multi-Topology (MT) Routing in OSPF
  • T – router’s TOS capability

and flags are:

  • SELF 0x01
  • SELF_CHECKED 0x02
  • RECEIVED 0x04
  • APPROVED 0x08
  • DISCARD 0x10
  • LOCAL_XLT 0x20
  • PREMATURE_AGE 0x40
  • IN_MAXAGE 0x80

Other information

ip -4 route

colinpaice@colinpaice:~$ ip -4 route
default via 192.168.1.254 dev wlp4s0 proto dhcp metric 600 
10.1.0.0/24 dev enp0s31f6 proto kernel scope link src 10.1.0.2 metric 100 
10.1.0.0/24 via 10.1.0.2 dev enp0s31f6 proto static metric 100 
10.1.1.0/24 via 10.1.0.3 dev enp0s31f6 
10.2.1.0/24 dev enp0s31f6 scope link 
10.3.1.0/24 via 10.1.0.3 dev enp0s31f6 
169.254.0.0/16 dev virbr0 scope link metric 1000 linkdown 
192.168.1.0/24 dev wlp4s0 proto kernel scope link src 192.168.1.222 metric 600 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown 

Understanding what OSPF does from the data flows.

I found that understanding the flows between to OSPF nodes helped me understand OSPF.

I used Wireshark to trace the data sent from my OSPF router with id 9.2.3.4.

There are four basic flows

  1. My router sending configuration information to the remote router
  2. The remote router sending acknowledgments back to my router
  3. The remote router sending configuration information to my router (the same as 1. above, but in the opposite direction)
  4. My router sending acknowledgements back to the remote router (the same as 2., but in the opposite direction).

It looks like a lot of data flowing – but I focused on my router sending information to the remote router.

Background information

Link state information helps others build a map of the configuration. This gives status information about the links.

Each router sends “new” information to the remote end of the connection; for example a Link State Update. The remote end acknowledges these with a Link State Acknowledgement.

While the local router is sending stuff to the remote router, the remote router is sending it’s configuration information to the local router.

Once the configuration information has been exchanged, and the configuration information stabilises, there is still a periodic “Hello Packet” between each router. This is a heartbeat to tell the remote end that the local end is still alive. The “Hello Packet” is sent out typically every 10 seconds. Updates are sent out around the “Hello Packet” time, so changes typically propagate through the network, 10 seconds a hop.

Information is exchanged via Link State Advertisement (LSA) which advertises the state of a link.

  • LSA type 1 is for routers, it contains information about routers
  • LSA type 2 is for networks, it contains information about IP addresses

Stub areas.

If you had all boxes in one big area – every box will know about other boxes. This may not scale well.

You can create areas, for example an area could be a country. Areas are connected together through the backbone area, area 0. An area, such as area1, can have information such as for addresses in area 17, go via the default routing to the backbone, and let the router where area 1 joins the backbone area sort out the routing.

Nodes in area 1need fewer definitions – as the definitions just say “go by the backbone”

Summary

I restarted my laptop, and it joined the network.
It’s configuration was

OSPF router id 1.2.3.4

Somewhere else in the network a node received two flows

  • Flow 1
    • I am router, 9.2.3.4
    • Type 1 Router-LSA. I have 3 direct connections
      • Remote end’s IP address 10.1.1.2, my address 10.1.1.1
      • Remote end’s IP address 10.1.3.2, my address 10.1.3.1
      • Remove end’s IP address 10.1.0.3, my address 10.1.0.3
    • Type 2- Network LSA
      • Attached routers 1.2.3.4 and 9.2.3.4
  • Flow 2
    • I am router 1.2.3.4
    • Type 1 Router LSA
      • I have IP address 10.1.0.0 type stub
      • I have IP address 12.1.0.1 type stub.

If the configuration changes, such as a new address is added to the node, the data broadcast is the current configuration.

Each system supporting OSPF gets the same information and can build up a database of the network, and can make informed routing decisions.

Changing the configuration

Adding an address to a link

I used the command

sudo ip -4 addr add 12.12.0.1 dev enp0s31f6

to add an additional IP address to the Ethernet connection on my laptop. The command

ip -4 addr gave

enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 10.1.0.2/24 brd 10.1.0.255 scope global noprefixroute enp0s31f6
       valid_lft forever preferred_lft forever
    inet 12.13.0.1/32 scope global enp0s31f6
       valid_lft forever preferred_lft forever
    inet 12.14.0.1/32 scope global enp0s31f6
       valid_lft forever preferred_lft forever
    inet 12.12.0.1/32 scope global enp0s31f6
       valid_lft forever preferred_lft foreverd

This cause a flow to the z/OS system, saying “this is all the IP addresses I know about”.

LS Update Packet
 Number of LSAs: 1
 LSA-type 1 (Router-LSA), len 72
  LS Type: Router-LSA (1)
  Link State ID: 1.2.3.4
  Advertising Router: 1.2.3.4
   Number of Links: 4
    Type: Transit  ID: 10.1.0.3        Data: 10.1.0.2        Metric: 100
     Type: Stub     ID: 12.13.0.1      Data: 255.255.255.255 Metric: 100
     Type: Stub     ID: 12.14.0.1      Data: 255.255.255.255 Metric: 100
     Type: Stub     ID: 12.12.0.1      Data: 255.255.255.255 Metric: 100

The transit address was the original address.

The stub address(es) were added manually.

Deleting an address to a link

I used the command

sudo ip -4 addr del 12.12.0.1 dev enp0s31f6

to remove the link I had previously added.

This cause a flow to the z/OS system, saying “this is all the IP addresses I know about” – omitting the address I had just deleted.

LS Update Packet
 Number of LSAs: 1
 LSA-type 1 (Router-LSA), len 72
  LS Type: Router-LSA (1)
  Link State ID: 1.2.3.4
  Advertising Router: 1.2.3.4
   Number of Links: 4
     Type: Transit ID: 10.1.0.3  Data: 10.1.0.2        Metric: 100
     Type: Stub    ID: 12.13.0.1 Data: 255.255.255.255 Metric: 100
     Type: Stub    ID: 12.14.0.1 Data: 255.255.255.255 Metric: 100
     Type: Stub    ID: 12.12.0.1 Data: 255.255.255.255 Metric: 100

One way flows in more detail

The “Hello packet”

  • I have Source OSPF router 9.2.3.4, area 0.0.0.0

DB Description

Source OSPF router 9.2.3.4, area 0.0.0.0

DB Description

“I know about…”

Source ospf router 9.2.3.4, area 0.0.0.0

  • LSA-type 1 (Router-LSA) Link State ID 1.2.3.4 advertising 1.2.3.4
  • LSA-type 1 (Router-LSA) Link State ID 9.2.3.4 advertising 9.2.3.4
  • LSA-type 1 (Router-LSA) Link State ID 10.1.1.2 advertising 10.1.1.2
  • LSA-type 2 (Network-LSA) Link State ID 10.1.0.2 advertising 1.2.3.4
  • LSA-type 2 (Networ-kLSA) Link State ID 10.1.1.2 advertising 10.1.1.2

Link state update

“Here is information about the links and the IP addresses”.

Source router 9.2.3.4, Area 0.0.0.0

  • LSA-type 1 (Router-LSA) Link State ID 1.2.3.4 advertising 1.2.3.4
    • Links: Type Transit ID 10.1.0.2 Data 10.1.0.2 Metric 100
  • LSA-type 1 (Router-LSA) Link State ID 9.2.3.4 advertising 9.2.3.4
    • Links: Type Transit ID 10.1.0.2 Data 10.1.0.3 Metric 100
    • Links: Type Stub ID 10.1.1.0 Data 255.255.255.0 Metric 1000
  • LSA-type 1 (Router-LSA) Link State ID 10.1.1.2 advertising 10.1.1.2
    • Links: Type Transit ID 10.1.1.1 Data 10.1.1.2 Metric 1
  • LSA-type 2 (Network-LSA) Link State ID 10.1.0.2 advertising 1.2.3.4
    • Attached router: 1.2.3.4
    • Attached router 9.2.3.4
  • LSA-type 2 (Network-LSA) Link State ID 10.1.1.2 Advertising 10.1.1.2
    • Attached router 10.1.1.2
    • Attached router 9.2.3.4

DB Description

I have OSPF router 9.2.3.4, Area 0.0.0.0

I support external routing

Link state update (2)

Source router 9.2.3.4, Area 0.0.0.0

Link State Type Router

  • LSA-type 1 (Router-LSA) Link State ID 9.2.3.4 advertising 9.2.3.4
    • Links: Type Transit ID 10.1.0.2 Data 10.1.0.3 Metric 100
    • Links: Type Transit ID 10.1.1.1 Data 10.1.1.1 Metric 1000
  • LSA-type 1 (Router-LSA) Link State ID 10.1.1.1 advertising 9.2.3.4
    • Attached router: 9.2.3.4
    • Attached router: 10.1.1.2

Hello Packet

Periodically (every 10 or so seconds) there is a Hello Packet flow, which acts as a heartbeat to let the remote end the know the local end is still alive.

One minute: Understanding TCPIP routing: Static, RIP, OSPF

This is another blog post in the series “One minute…” which gives the basic concepts of a topic, with enough information so that you can read other documentation, but without going too deeply.

IP networks can range in size from 2 nodes(machines), to millions of nodes(machines), and a packet can go from my machine to any available machines – and it arrives! How does this miracle work?

I’ll work with IP V6 to make it more interesting (and there is already a lot of documentation for IP V4)

I have and old laptop, connected by Ethernet to my new laptop. My new laptop is connected by wireless to my server which is connected to z/OS. I can ping from the old laptop to z/OS.

  • Each machine needs connectivity for example wireless, Ethernet, or both.
  • Each machine has one or more interfaces where the connectivity comes in (think Ethernet port, and Wireless connection). This is sometimes known as a device.
  • Each interface has one or more IP addresses.
  • You can have hardware routers, or can route through software, without a hardware router. A hardware router can do more than route.
  • Each machine can route traffic over an interface (or throw away the packet).
    • If there is only one interface this is easy – all traffic goes down it.
    • If there is more than one interface you can specify which address ranges go to which interface.
    • You can have a default catch-all if none of the definitions match
    • You can have the same address using different interfaces, and the system can exploit metrics to decide which will be used.
    • You can have policy based routing. For example
      • packets from this premier user, going to a specific IP address should use the high performance (and more expensive) interface,
      • people using the free service, use the slower(and cheaper) interface.

Modern routing uses the network topology to manage the routing tables and metrics in each machine.

Static

The administrator defines a table of “if you want get to… then use this interface, the default is to send the packet using this … interface”. For example with z/OS

BEGINRoutes 
;     Destination   SubnetMask    FirstHop    LinkName  Size 
; ROUTE 192.168.0.0 255.255.255.0       =     ETH2 MTU 1492 
ROUTE 10.0.0.0      255.0.0.0           =     ETH1 MTU 1492 
ROUTE DEFAULT                     10.1.1.1    ETH1 MTU 1492 
ROUTE 10.1.0.0      255.255.255.0   10.1.1.1  ETH1 MTU 1492 

ROUTE 2001:db8::/64 fe80::f8b5:3466:aa53:2f56 JFPORTCP2 MTU 5000 
ROUTE fe80::17      HOST =                    IFPORTCP6 MTU 5000 
ROUTE default6      fe80::f8b5:e4ff:fe59:2e51 IFPORTCP6 MTU 5000
                                                                      
ENDRoutes 

Says

  • All traffic for 10.*.*.* goes via interface ETH1.
  • If no rule matches (for IP V4) use the DEFAULT route via ETH1. The remote end of the connection has IP address 10.1.1.1
  • All traffic for IPV6 address 2001:db8:0:* goes via interface JFPORTCP2
  • If no rule matches (for IP V6) use the DEFAULT6 route via IFPORTCP6. The remote end of the connection has IP address fe80::f8b5:e4ff:fe59:2e51.

On Linux the ip route command gave

default via 192.168.1.254 dev wlxd037450ab7ac proto dhcp metric 600 
10.1.0.0/24 dev eno1 proto kernel scope link src 10.1.0.3 metric 100 
10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1 

This says

  • The default is to send any traffic via device wlxd037450ab7ac.
  • Any traffic for 10.1.0.* goes via device eno1
  • Any traffic for 10.1.1.* goes via device tap0.

Routing Information Protocol(RIP)

Manually assigning metrics (priorities) to hint which routes are best, quickly becomes unmanageable when the number of nodes(hosts) increases.

If the 1980’s the first attempt to solve this was using RIP. It uses “hop count” of the destination from the machine as a metric. A route with a small hop count will get selected over a route with a large hop count. Of course this means that each machine needs to know the topology. RIP can support at most 15 hops.

Each node participating in RIP learns about all other nodes participating in RIP.

Every 30 seconds each node sends to adjacent nodes “I know about the following nodes and their route statements”. Given this, eventually all nodes connected to the network will know the complete topology.
For example, from the frr(Free Range Routing) trace on Linux

RIPng update timer expired!
RIPng update routes on interface tap1
  send interface tap1
  SEND response version 1 packet size 144
   2001:db8::/64 metric 1 tag 0
    2001:db8:1::/64 metric 1 tag 0
   2002::/64 metric 2 tag 0
    2002:2::/64 metric 2 tag 0
   2008::/64 metric 3 tag 0
    2009::/64 metric 1 tag 0
    2a00:23c5:978f:6e01::/64 metric 1 tag 0

This says

  • The 30 second timer woke up
  • It sent information to interface tap1
  • 2001:db8::/64 metric 1 this is on my host(1 hop)
  • 2002::/64 metric 2 this is from a router directly connected to me (2 hops).
  • 2008::/64 metric 3 is connected to a router connected to a router directly connected to me (3 hops.)

On z/OS the command F OMP1,RT6TABLE gave me message EZZ7979I . See OMPROUTE IPv6 main routing table for more information

DESTINATION: 2002::/64 
  NEXT HOP: FE80::E42D:73FF:FEB1:1AB8 
  TYPE:  RIP           COST:  3         AGE: 10 
DESTINATION: 2001:DB8::/64 
  NEXT HOP: FE80::E42D:73FF:FEB1:1AB8 
  TYPE:  RIP*          COST:  2         AGE: 0 

This says

  • To get to 2002::/64 go down interface with the IP address FE80::E42D:73FF:FEB1:1AB8.
  • This route has been provided by the RIP code.
  • The destination is 3 hops away (in the information sent from the server it was 2 hops away)

The fields are

  • RIP – Indicates a route that was learned through the IPv6 RIP protocol.
  • * An asterisk (*) after the route type indicates that the route has a directly connected backup.
  • Cost 3 – this route is 3 hops away.
  • Age 10 -Indicates the time that has elapsed since the routing table entry was last refreshed

OSPF (Open Shortest Path First)

OSPF was developed after RIP, as RIP had limitations – the maximum number of hops was 15, and every 30 seconds there was a deluge of information being sent around. The OSPF standard came out in 1998 10 years after RIP.

Using OSPF, when a system starts up it sends to the neighbouring systems “Hello,_ my router id is 9.3.4.66, and I have the following IP addresses and routes.” This information is propagated to all nodes in the OSPF area. When a node receives this information it updates its internal map (database) with this information. Every 10 seconds or so each node sends a “Hello Packet” to the adjacent node to say “I’m still here”. If this packet is not received, then it can broadcast “This node …. is not_responsive/dead”, and all other nodes can then update their maps.

If the configuration changes, for example an IP address is added to an interface, the node’s information is propagated throughout the network. In a stable network, the network traffic is just the “Hello packet” sent to the next node, and any configuration changes propagated.

One of the pieces of information sent out about node’s route is the metric or “cost”. When a node is deciding which interface to route a packet to, OSPF can calculate the overall “cost” and if there are a choice of routes to the destination it can decide which interface gives the best cost.

To make it easier to administer, you can have areas, so you might have an area being the UK, another area being Denmark, and another area being the USA.