How do I find out about my VIPA configuration?

This follows on from setting up VIPA for the Liberty web server to provide High Availability.  I had a few problems setting it up, and this blog post is about some of the commands I used to get it working.

I cover

  1. Is the VIPA active?
  2. Where is it running
  3. Are applications processing requests

Some IP basics.

  1. Every connection has an IP address at each end.  An address looks like 10.3.4.15 or 4 * 8 bit numbers.
  2. My machine has several connections, ethernet, wireless, and a tunnelling connection to z/OS. Each connection has a different IP address.
  3. Packets get routed through the network depending on the destination IP address.  The router has logic like,  packets going to 10.4.5.* go does this connection, packets for 17.2.2.* go down that connection, any other packets – try sending them to down the connection 11.13.6.6.
  4. The router uses a netmask to calculate which connection to use.
    1. A net mask is a string of 1’s followed by 0s.  For example 255.255.255.0 – or 3 * 8 =24 ones.
    2. A router takes a packet IP address and a netmask and logically ands them together, and uses the result to decide where to route the packet.
    3. A connection handling 10.4.1.0 to 10.4.1.255 would have a netmask of 255.255.255.0 (also written /24 bits) a default connection may handle all packets for 10.* with a netmask of 255.0.0.0 or /8.

My scenario

  1. I have my desktop machine running Ubnutu Linux
  2. I have z/OS (called SOW1) running on my desktop using the zPDT.
  3. I have 3 TCPIP images (stacks) running on the z/OS image
    1. TCPIP running as the front end
    2. TCPIP2 running as a backend – this could be on another LPAR
    3. TCPIp3 running as a backend
  4. I have a VIPA defined with address 10.1.3.10

What configuration does Ubuntu have?

There are many commands to display network configuration information on Linux.

What address does Ubuntu have?

ip address gives a lot of information – but I did not use it

What packet routing does my desktop have?

the command ip route gives

  1. 10.1.0.0/24 dev eno1 proto kernel scope link src 10.1.0.3 metric 100
  2. 10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1
  3. 10.1.2.0/24 dev tap1 proto kernel scope link src 10.1.2.1
  4. 10.1.3.0/24 dev tap0 scope link
  5. 10.20.2.4 dev tap0 scope link
  6. 192.168.1.0/24 dev wlxd037450ab7ac proto kernel scope link src 192.168.1.67 metric 600

Bold line(2) shows

  • Traffic for any address between 10.1.1.0 and 10.1.1.255 (remember the netmask /24 means 24 bits or 255.255.255.0) goes  to device(connection) tap0
  • The IP address for the desktop end of the connection is 1.1.1.1

Bold line(4) shows

  • that any traffic 10.1.3.0 to 10.1.3.255 goes to device tap0

The command used to set this up was sudo ip route add 10.1.3.0/24 dev tap0

Bold line(5) shows

  • that traffic to 10.20.2.4 goes to device tap0.

The command used to set this up was sudo ip route add  10.20.2.4 dev tap0

What is the routing for an IP address ?

You can use traceroute command to display which route a packet would take. For example

  • traceroute 10.1.3.10
    • traceroute to 10.1.3.10 (10.1.3.10), 30 hops max, 60 byte packets
    • 1 10.1.3.10 (10.1.3.10) 4.963 ms 4.980 ms 5.887 ms

For a connection that is not defined

traceroute 10.20.2.5 
traceroute to 10.20.2.5 (10.20.2.5), 30 hops max, 60 byte packets
1 bthub.home (192.nnn.1.mmm) 3.170 ms 4.742 ms 6.379 ms
2 * * *

So we can see it went to my bt hub  wireless router.

You can also use the ping command.  On linux there is the -R option for display route.

ping -R 10.1.3.10 
PING 10.1.3.10 (10.1.3.10) 56(124) bytes of data.
64 bytes from 10.1.1.2: icmp_seq=1 ttl=64 time=2.54 ms
NOP
    RR: 10.1.1.1
        10.1.1.2
        10.1.1.1

The request went to 10.1.1.1.  10.1.1.2 caught it, and sent the reply back, via 10.1.1.1

I was looking for my VIPA address, 10.1.3.10, and we can see it got to 10.1.1.2.

For the ping to work, there must be a server processing the ping request.  If there are no applications processing the VIPA, the VIPA is not active, so a ping will fail.

A successful ping to a VIPA address means a packet can get to the LPAR, be processed and  the reply set back.  If the ping does not respond it could be

  1. The VIPA is not active
  2. The VIPA is active and a packet was sent to the LPAR hosting the VIPA, but it could not send a response back due to a set up error.

How to issue change TCPIP configuration on z/OS

You can change the configuration of a TCPIP image using the operator command

V TCPIP,TCPIPn,OBEY,filename

Where

  1. V TCPIP tells z/OS to route this TCPIP
  2. TCPIPn is the name of the TCPIP address space to direct the command to, for example V TCPIP,TCPIP2.  If there is only one TCPIP running you can use V TCPIP,,
  3. OBEY this is the TCP command
  4. filename is the parameter passed to the OBEY command.   The filename containing the commands/configuration to be executed.

How to display information on z/OS

There are three ways of displaying TCPIP information, for example the IP address(es) of the TCP image

  1. The operator command D TCPIP,TCPIP2,NETSTAT,HOME… similar in syntax to the V TCPIP command above
  2. The TSO command NETSTAT HOME TCP TCPIP2
  3. The USS command netstat -h -p tcpip   The commands are similar to but different from Linux commands!

The output is usually similar between the commands.

What is the IP address of my TCPIP image?

From the TSO NETSTAT HOME command

EZZ2350I MVS TCP/IP NETSTAT CS V2R4 TCPIP Name: TCPIP2 17:15:53
EZZ2700I Home address list:
EZZ2701I Address   Link         Flg
EZZ2702I -------   ----         ---
EZZ2703I 10.1.1.3  ETH1         P
EZZ2703I 10.1.2.3  ETHB
EZZ2703I 172.1.1.2 EZASAMEMVS
EZZ2703I 10.1.3.10 VIPL0A01030A I
EZZ2703I 127.0.0.1 LOOPBACK

10.1.1.3  ties up with the information on the desktop which had IP addresses had 10.1.1.1 for device tap0, and 10.1.2.3 ties up with 10.1.2.1 for device tap1.

For the links

  1. I configured link ETH1 and ETHB.
  2. The VIPL0A01030A takes the IP address and converts it to hex so 10.1.3.10 becomes VIPL 0A 01 03 0A
  3. EZASAMEMVS is prefix EZA and “SAME MVS”.   This is generated by TCPIP from the DYNAMIXCF configuration.

What routing is there?

The command TSO command NETSTAT ROUTE TCP TCPIP2 or the USS command netstat -r -p tcpip gives

MVS TCP/IP NETSTAT CS V2R4 TCPIP Name: TCPIP2 16:15:43 
Destination  Gateway  Flags Refcnt     Interface 
----------- -------   ----- ------     --------- 
Default      10.1.1.1 UGS   0000000000 ETH1 
10.0.0.0/8   0.0.0.0  US    0000000000 ETH1 
10.1.1.3/32  0.0.0.0  UH    0000000000 ETH1 
10.1.2.0/24  0.0.0.0  US    0000000000 ETHB 
10.1.2.3/32  0.0.0.0  UH    0000000000 ETHB 
127.0.0.1/32 0.0.0.0  UH    0000000000 LOOPBACK 
172.1.1.1/32 0.0.0.0  UHS   0000000000 EZASAMEMVS 
172.1.1.2/32 0.0.0.0  UH    0000000000 EZASAMEMVS 
172.1.1.3/32 0.0.0.0  UHS   0000000000 EZASAMEMVS

This shows that to get to 10.1.2.0 to10.1.2.255 (with a netmask of /24 or  255.255.255.0) it goes by link(interface) ETHB

What is happening to my VIPA on  z/OS?

On the OSA connection (think ethernet connection)  from the desktop to my z/OS environment there could be several LPARs using the OSA, each with multiple TCP images.

The operator command D TCPIP,TCPIP2,SYSPLEX,VIPADYN issued on any LPAR on any active TCPIP image gives a Sysplex view of the VIPA configuration

11.54.05 STC09473  EZZ8260I SYSPLEX CS V2R4 387                         C  
VIPA DYNAMIC DISPLAY FROM TCPIP    AT S0W1                                 
IPADDR: 10.1.3.10  LINKNAME: VIPL0A01030A                                  
  ORIGIN: VIPADEFINE                                                       
  TCPNAME  MVSNAME  STATUS RANK ADDRESS MASK    NETWORK PREFIX  DIST       
  -------- -------- ------ ---- --------------- --------------- ----       
  TCPIP    S0W1     ACTIVE      255.255.255.0   10.1.3.0        DIST       
  TCPIP2   S0W1     BACKUP 001                                  DEST       
  TCPIP3   S0W1     ACTIVE                                      DEST       
IPADDR: 10.1.4.10                                                          
  TCPNAME  MVSNAME  STATUS RANK ADDRESS MASK    NETWORK PREFIX  DIST       
  -------- -------- ------ ---- --------------- --------------- ----       
  TCPIP3   S0W1     ACTIVE      255.255.255.0   10.1.4.0                   
  TCPIP2   S0W1     MOVING      255.255.255.0   0.0.0.0                    

IPADDR:10.1.3.10

The VIPA 10.1.3.10 was created using a VIPADEFINE.

We see that TCPIP on S0W1 “owns” the VIPA  10.1.3.10 and is responsible for distributing requests.  This image is DISTributing requests to other TCPIP Images.

The DEST means it is a target for connections ( a DESTination)  and has a server processing requests. BOTH means it is a DESTination and  DISTributing connections, and has a server processing them.

IPADDR:10.1.4.10

We can see that TCPIP3 is processing request.   TCPIP2 is not processing requests, it does not have a network prefix.

How are DVIPA connection requests distributed?

You need to ask the TCP that owns the VIPA. In my case, from the previous section, this is TCPIP.

The TSO command NETSTAT VDPT TCP TCPIP  or the USS command netstat -O -p tcpip gives

MVS TCP/IP NETSTAT CS V2R4 TCPIP Name: TCPIP 16:49:42 
Dynamic VIPA Destination Port Table for TCP/IP stacks: 
Dest IPaddr DPort DestXCF Addr Rdy TotalConn  WLM TSR Flg 
----------- ----- ------------ --- ---------  --- --- --- 
10.1.3.10   08443 172.1.1.2    001 0000000005  01 100 
  DistMethod: Roundrobin 
  TCSR: 100 CER: 100 SEF: 100 
  ActConn:   0000000000 
10.1.3.10   08443 172.1.1.3    000 0000000000  01 100 
  DistMethod: Roundrobin 
  TCSR: 100 CER: 100 SEF: 100 
  ActConn:  0000000000

We have a heading showing the TCPIP image name, and we are looking at Dynamic VIPA Destination Port Table for TCP/IP stacks.

When report was generated the application on 172.1.1.2 (TCPIP2) was active, and the application on TCPIP3 had been stopped.

From

Dest IPaddr DPort DestXCF Addr Rdy TotalConn  WLM TSR Flg 
----------- ----- ------------ --- ---------  --- --- --- 
10.1.3.10   08443 172.1.1.2    001 0000000005  01 100 
ActConn: 0000000000

We can see

  • Dest IPaddr: 10.1.3.10 is our VIPA address
  • DPort :08443 is the destination port
  • DestXCF Addr: 172.1.1.2 is where the request is going – we know this is TCPIP2.  It would be good if it could say SOW1.TCPIP2
  • Rdy: 001 there is one active application listening
  • TotalConn: 0000000005 there have been 5 requests to this application
  • ActConn: 0000000000 there are no active connections to this application

As TotalConn is greater than 0, this means there have been connections to the application, so is a good sign to show the set-up is working.

Because the front end TCPIP is distributing the requests using Roundrobin – each TCPIP should get a connection in turn.

When I started the application on TCPIP3, and started another application on TCPIP2.  When I ran a workload I had 10 requests go to TCPIP3 and 10 requests go to TCPIP2.  On TCPIP2 the requests were evenly distributed between the two servers.  It looked like round robin, but I do know know if this was design or chance

How do I know if I have a backup configuration defined?

I set up TCPIP with a VIPABACKUP configuration.   The operator command d tcpip,tcpip,sysplex,vipadyn  gave me

VIPA DYNAMIC DISPLAY FROM TCPIP AT S0W1 
IPADDR: 10.1.3.10 LINKNAME: VIPL0A01030A 
ORIGIN: VIPADEFINE 
TCPNAME  MVSNAME  STATUS RANK ADDRESS MASK    NETWORK PREFIX  DIST 
-------- -------- ------ ---- --------------- --------------- ---- 
TCPIP    S0W1     ACTIVE      255.255.255.0   10.1.3.0        DIST 
TCPIP2   S0W1     BACKUP 001                                  DEST 
TCPIP3   S0W1     ACTIVE                                      DEST

We can see that TCPIP2 is defined as being the backup.

What connections does this TCPIP have

You can use the TSO command NETSTAT ALLCONN TCP TCPIP2 or the USS command  netstat -a -p tcpip2  to show what sessions are active.

MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP2          07:19:15  
User Id  Conn     Local Socket           Foreign Socket         State  
-------  ----     ------------           --------------         -----  
MYSERVER 0000003F 10.1.3.10..8443        0.0.0.0..0             Listen 
MYSERVER 0000003C 10.1.3.10..8443        0.0.0.0..0             Listen 
MYSERVER 0000004B 10.1.3.10..8443        0.0.0.0..0             Listen 

This shows there are 3 instances of MYSERVER running using IP address 10.1.3.10 and port 8443.

There will usually be a lot of output.  You can filter the request by

  • tso netstat allconn tcp tcpip2 (ipaddr 10.1.3.10
  • tso netstat allconn tcp tcpip2 (port 8443
  • uss  netstat -a -p tcpip2 -I 10.1.3.10 
  • uss netstat -a -p tcpip2 -P 8443
  • operator D TCPIP,tcpip2,netstat,allconn,ipaddr=10.1.3.10 
  • operator D TCPIP,tcpip2,netstat,allconn,port=8443

 

What VIPA stuff does this TCPIP have?

USS netstat -v  -p tcpip3 or TSO NETSTAT VIPADYN TCP TCPIP3

MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP3          10:21:04 
Dynamic VIPA: 
  IP Address      AddressMask     Status    Origination     DistStat 
  ----------      -----------     ------    -----------     -------- 
  10.1.3.10       255.255.255.0   Active                    Dest 
    ActTime:      08/30/2020 10:40:10 
  10.1.4.10       255.255.255.0   Active    VIPARange Bind 
    ActTime:      08/30/2020 11:03:05        JobName:        MYSERVER

The 10.1.3.10 VIPA is created using VIPADEFINE.  VIPA 10.1.4.10 was create by means of VIPARANGE.

There may be multiple jobs processing the port. MYSERVER is just one of them.

 

HA Liberty web server – implementing VIPA with distributing connections.

Overview of VIPA solutions

You can implement VIPA, where you give your application its own IP address, across multiple TCPIP images.   This solves the problem of certificates not matching the host IP address.

You can have

  • One TCPIP image processing the connection requests. You have multiple TCPIP images – but only one TCPIP image at a time processes the connections.   If the TCPIP image stops, another can take over.
  • Multiple TCPIP stacks can process connection requests. A front end TCPIP image takes the connection requests and distributes them to TCPIP instances where the application is running.   You can use load balancing across multiple TCPIP images such distributing the connection techniques such as Round Robin, or Hot Standby.  This is based on Sysplex Distributor technology.

This blog post discusses the second case.

To provide background information, I created

Sysplex Distributor background

Sysplex Distributor is like having a router inside your TCPIP on z/OS; it can route traffic transparently to other TCP images in the environment.  The Sysplex Distributor can distribute connection requests for a  VIPA requests to TCPIP images where the application is running.    This set up is called Distributed VIPA (DVIPA).

It took me about 2 weeks to get a Sysplex DVIPA working – about a week understanding the documentation on VIPA, and the other week trying to understand why it didn’t work – and the simple configuration error I had.

I’ll break it down into simple stages which should help you understand the documentation.

The scenario

The scenario I used was going from my laptop, to z/OS running under zPDT on my laptop.  In effect there was an OSA connection between Ubuntu and my z/OS LPAR.  If you do not know what an OSA is, think of it as an Ethernet connection which can plug into multiple z/OS LPARs.

  1. My Ubuntu had an address of 10.1.1.1 over the tunnel connection
  2. I had one LPAR  with three TCPIP images
    1. TCPIP, host address 10.1.1.1 for the primary “front end”
    2. TCPIP2, host address 10.1.1.2 for the backup “front end” and where a server instance was running
    3. TCPIP3, host address 10.1.1.3 with a server instance running.
  3. I used a VIPA address of 10.1.3.10

The steps are

  1. Connect from Ubuntu to the z/OS
  2. Configure the LPAR(s)
  3. Define the VIPA configuration
  4. Define the “routing” to where the server was running.
  5. Getting the server to use the VIPA
  6. Commands to see what is going on (or not as the case may be)

Connect from Ubuntu to the z/OS

I had an existing tunnel connection from Ubuntu to z/OS.

I used the ip route command

sudo ip r add 10.1.3.10 link tap0

to define the route to 10.1.3.10 via the tunnelling device tap0 .  This looks like an OSA connection into z/OS.

Configure the LPAR(s)

The Sysplex Router uses XCF communications between LPARs and TCPIP images on the LPARs.

You configure each TCPIP with a statement

IFCONFIG DYNAMICXCF 172.1.2.x

My “frontend” TCP/IP had  IFCONFIG DYNAMICXCF 172.1.2.1, the other two TCP/IP images had 172.1.2.2 and 172.1.2.3.

The 172.1.2.x address can be any address not used by your enterprise.  It is internal to the Sysplex configuration.

Define the VIPA configuration

You define the configuration once, in the front end TCPIP.  It is visible from the other TCP/IP images because the information is shared via the DYNAMICXCF.

You define the VIPA in the front end TCP/IP image with a VIPADEFINE netmask address.  I used

VIPADYNAMIC
  VIPADEFINE 255.255.255.0 10.1.3.10
...
ENDVIPADYNAMIC

You can define VIPABACKUP in another TCPIP image, so if the main front end TCP/IP is not available then a backup can take the traffic and distribute it to the other TCP/IP stacks.

When the main front end TCP/IP image is restarted, you can have it take back the routing.

Define the “routing” to where the server was running.

You can define a variety of ways of routing the work

  • A Hot Standby – where the input to the front end TCP/IP image is routed to a single “backend” application’s TCP/IP image.  If this fails, the work is routed to a running backup application.
  • A Round Robin – where requests are routed in turn to each TCPIP with an active application.
  • Routing depending on WLM or other load characteristics.

This routing is done by the VIPADISTRIBUTE command in the front end TCP/IP image.

The definitions for the front end TCPIP

IPCONFIG SYSPLEXROUTING 
    DYNAMICXCF 172.1.1.1 255.255.255.0 3 

VIPADYNAMIC 
   VIPADEFINE 255.255.255.0 10.1.3.10 

   VIPADISTRIBUTE DEFINE DISTM ROUNDROBIN 10.1 .3.10 PORT 8443 
      DESTIP 
         172.1.1.2 
         172.1.1.3 
ENDVIPADYNAMIC 

This routes connection requests to the TCPIP images with the DYNAMICXCF of 172.1.1.2 (TCPIP2) and 172.1.1.3 (TCPIP3)

The definitions for TCPIP2

IPCONFIG SYSPLEXROUTING 
DYNAMICXCF 172.1.1.2 255.255.255.0 3 

The definitions for TCIP3

IPCONFIG SYSPLEXROUTING 
DYNAMICXCF 172.1.1.3 255.255.255.0 3

Use of VIPABACKUP

If the backup TCPIP front end image it used, it can have its own VIPADISTRIBUTE statement, or just use the same statement shared from the main front end TCP/IP image.  It is better to have the VIPADISTIBUTE statements, for the case when the backup TCPIP is started before the front end TCPIP.   The backup needs the VIPADISTRIBUTE statements. (These statements can be put into a PDS, and included using the INCLUDE dataset(member) statement in both primary and backup environments.)

To define TCPIP2 as a backup I used

VIPADYNAMIC 
    VIPABACKUP MOVEABLE IMMEDIATE 255.255.255.0 10.1.3.10 
    VIPADISTRIBUTE DEFINE DISTM ROUNDROBIN 10.1.3.10 PORT 8443 
        DESTIP 
        172.1.1.2 
        172.1.1.3 
ENDVIPADYNAMIC 

Getting the server to use the VIPA

The TCP/IP images hosting the applications just have the IFCONFIG DYNAMICXCF aa.bb.cc.dd statement.  They do not have any VIPADYNAMIC … ENDVIPADYNAMIC statements unless they are the main or backup front end TCP/IP images.

The application can connect using the VIPA address, for example create the SSLSOCKET Listener passing the VIPA address.   You can also configure TCP/IP so when a port is used, it binds to a particular IP address for example

PORT 8443 BIND 10.1.3.10

So an application using port 8443 to listen, will get IP address 10.1.3.10 – which in my case is a VIPA address.

You can use

PORT
9443 TCP * SHAREPORT BIND 10.1.3.7

to allow the port to be shared by many applications on a TCPIP Instance.

How are the connections distributed?

The VIPADISTRIBUTE  has many routing options. I used Hot Standby and Round Robin.

With RoundRobin, I had

  • the front end TCPIP
  • TCPIP2 with two Liberty servers
  • TCPIP3 with  one Liberty server

I ran some workload and found that the server on TCPIP3 had half the requests, and each of the two servers on TCPIP2 had a quarter of the overall requests.  This shows the routing is done at the TCPIP level – not the number of servers.

HA Liberty web server – implementing VIPA using the simpler VIPARANGE technology

Overview of VIPA solutions

You can implement VIPA, where you give your application its own IP address, across multiple TCPIP images.   This solves the problem of certificates not matching the host IP address.

  • One TCPIP image processes the connection requests. You have multiple TCPIP images – but only one TCPIP image at a time processes the connections.   If the TCPIP image stops, another can take over.
  • Multiple TCPIP stacks can process connection requests. This uses Sysplex Distributor;  a front end TCPIP image takes the connection requests and distributes them to TCPIP instances where the application is running.   You can use load balancing such as Round Robin, or Hot Standby.

This blog post discusses the first case.

To provide background information, I created

Using VIPARANGE configuration

The technique uses the VIPARANGE configuration statement.

The concept is that many LPARs can be attached to an OSA adapter, one, just one,  TCPIP stack (I dont know which of the available images) takes the connection requests and passes them on to the application on that TCPIP image.

You allocate a range of TCPIP address for your applications, with the same network prefix, for example 9.4.6.x   Allocate a host id to a Liberty, for example 9.4.6.7.   The Liberty instance keeps this address for whereever it runs.  You configure your routers so that  9.4.5.* are routed to the OSA adapter.

For each TCPIP image where you want to run Liberty, add to the  TCPIP startup configuration (or to an OBEY file)

VIPADYNAMIC 
   VIPARANGE DEFINE 255.255.255.0 9.4.6.7 
ENDVIPADYNAMIC

The 255.255.255.0 is the  subnet mask.  If your organisation uses a different subnet mask, it affects the IP addresses used.

These instructions say that they are defining a range of IP addresses on this LPAR, for  9.4.6.1 to 9.4.6.254

If an application connects to TCPIP, and the bind specifies a value in this range (9.4.6.1 to 9.4.6.254) then it is considered a VIPA address.  If the application used 9.4.6.7 this would count as a VIPA address.

When your application (Liberty) connects to TCP and uses an address in the VIPARANGE,  the TCPIP instance will create a dynamic IP address.   When I started my server application,   I got a z/OS console message

EZD1205I DYNAMIC VIPA 9.4.6.7 WAS CREATED USING BIND BY jobname ON TCPIP2.

When I shut down the server I got

EZD1298I DYNAMIC VIPA  9.4.6.7 DELETED FROM TCPIP2
EZD1207I DYNAMIC VIPA 9.4.6.7 WAS DELETED USING CLOSE API BY jobname ON TCPIP2

If the VIPA address is active on more than one TCPIP image, just one image will get all of the requests.  If you stop this image, another TCPIP image can take over.

If you have a different server using the same IP address, but a different port number, because they use the same IP address, the same LPAR will process the requests.

With VIPAROUTE you do not get connections distributed to more than one TCPIP image.

In your browser use  9.4.6.7:9443 address, the network router, routes this to the OSA, a TCPIP captures this and passes it to the application (Liberty).   As part of the handshake Liberty sends down its certificate, which has a SAN of  9.4.6.7 which matches the IP address, so this works.

On another day, when a different z/OS image is capturing the VIPA address connections,  the TCPIP address is 9.4.6.7 as before, so this matches the SAN in the certificate.

Testing it

In a test I used “ping -R 9.4.6.7 ” to the VIPA address.
This reported it was sent to TCPIP stack with 10.1.1.2. When I shut this TCPIP image down, ping reported the request was sent to 10.1.1.3.  It did this with no manual intervention.

 

HA Liberty web server – introduction to using VIPA to provide high availability connectivity.

I struggled to set up Liberty to provide a Highly Available solution – if I shut down one TCPIP instance, I want to access Liberty through another TCPIP instance.  In principle it is easy; but there is just a little problem when you are using certificates on the z/OS image.  I’ll use names rather than IP addresses in the discussion below.

The problem

Take the simple scenario where have your Liberty instance running on z/OS image with IP address MVS1 port 9443.

Your web browser uses MVS1:9443/ibmmq/console to initiate the sign-on.  As part of the handshake Liberty sends down its certificate with the Subject Alternative Name(SAN) of MVS1.    The browser checks that this SAN is the same as where the Liberty instance is running and, as it matches, the logon proceeds.

You want to shut down that LPAR, and run the work on another LPAR,  MVS2.    The Liberty instance starts up, binds to TCPIP and waits for requests.   The web browser connects to MVS2, and sends down the certificate with the SAN of MVS1.  As MVS1 does not match MVS2, the browser complains saying that someone could be stealing your information.

You could have a certificate for each LPAR, but this is additional administration overhead.

You also have the problem of your browser getting to the different IP address.  This could be a different URL, update to the DNS server, a change of router configuration, or change your work station to direct the request to a different place.

The solution

This has been solved using Virtual IP Addressing or VIPA.   In simple terms, give the Web Server its own IP address, which can move around between different LPARs in a Sysplex.

This area has a lot of new, complex jargon.  You have Static VIPAs, Dynamic VIPAs, and Distributed VIPAs .  The TCP documentation is not bad, but it focuses on TCP/IP, not how it will be used.   The documentation has example configurations, but one configuration covers many scenarios.  I was looking for  a simple, getting started example.

There are a couple of ways of setting up the configuration.

  • The simplest scenario, use VIPARANGE.
    • Liberty has its own IP address which can be activated on different TCP images
    • Once set up, when the web server binds to TCPIP, the IP address is created on the TCPIP image.
    • When the VIPA is active, the TCPIP image will listen for the request.  I had two TCPIP images listening on the same OSA connection.   The connections went to one TCPIP.   When I stopped that TCPIP, the connections automatically went to the other TCPIP.  When I stopped both TCPIP the client got “No route to host (Host unreachable)”.
    • You can have a web server with the same IP address running on different TCPIP images at the same time (with different configuration files).
  • Use Sysplex Distributer.   This has a front end IP which takes connection requests and distributes them to TCPIP images.   It can do this using “Hot Standby”, “round robin” and other techniques.  If the front end is shut down, you can configure other TCPIP images to take over.

How it works – TCPIP (Background knowledge needed to understand how to configure a VIPA)

When I was a child, I had a series of books called “How it works…”, for example “How it works – the motor car”, “How it works – Nuclear Physics.  As I’ve been working on making Liberty highy available, and using Virtual IP Addresses (VIPAs), I’ve realised that I had had holes in my knowledge about TCPIP.    There are many books about how TCP/IP works, but they do not provide the information in an easily digested format  – and often went too deep too quickly.   So this blog post is my view on what you need to know to understand VIPAs etc.

I’ll only consider TCPIP V4.

Topics covered

  • IP connections
  • Subnet mask
  • How applications use TCPIP
  • How applications can bind to a specific IP address and port
  • On z/OS
    • How to issue TCPIP commands on z/OS
    • What is the IP address of my TCPIP image?
    • What routing is there on my TCPIP image
  • On Linux
    • What IP address does my Ubuntu machine have
    • What routing is there on my Ubuntu machine
    • What is the routing for a particular IP address ?

Some IP basics.

  1. Every connection has an IP address at each end.  An address looks like 10.3.4.15 or 4 * 8 bit numbers.
  2. You can use a name instead of a number, so you could have MVS1.SSS.COM.  To covert this to an IP address you call a Domain Name Server(DNS).  You pass it MVS1.SSS.COM and get back 10.3.4.15.
  3. My machine has several connections (logic bits of wire connected to the back of the machine), Ethernet, wireless, and a tunnelling connection to z/OS. Each connection has a different IP address.
  4. Packets get routed through the network depending on the destination IP address.  The router has logic like,  packets going to 10.4.5.* go does this connection, packets for 17.2.2.* go down that connection, any other packets – try sending them to down the connection 11.13.6.6.
  5. The router uses a netmask to calculate which connection to use.
    1. A net mask is a string of 1’s followed by 0s.  For example 255.255.255.0 – or 3 * 8 =24 ones.
    2. A router takes a packet IP address and a netmask and logically ands them together, and uses the result to decide where to route the packet.
    3. A connection handling 10.4.1.0 to 10.4.1.255 would have a netmask of 255.255.255.0 (also written /24 bits) a default connection may handle all packets for 10.* with a netmask of 255.0.0.0 or /8.
  6. Multiple z/OS LPARs can be attached to an OSA Adapter (think if it as Ethernet with more function), they can all be listening for an IP address – only one LPAR will get the data.  If that LPAR goes down, another LPAR will get the data.

How applications use TCPIP

You have network connection (for example wireless) which connect your machine to another machine.  On each machine applications use a port.

When your application talks to another application it establishes a session with the IP address:port.

Applications including web servers, web browsers, 3270 emulators and FTP connect.

If your application is a server it may bind to a specific port, if not your application can say give me any free port.  A port can be set up, so it is shared, so two servers can listen to connection requests on it.  Only one will get the connection request.

A server application can say I am interested in traffic on port 9443 – coming in over a specific IP address, or coming in over any address.

How does an application specify a bind value .

A Java application can issue a request for a specific port and IP address.

ServerSocket listener = factory.createServerSocket(port,1,host )

where port is 9443 and host is “10.1.3.7”

You can also configure this in the TCPIP parameters

PORT
9443 TCP * SHAREPORT BIND 10.1.3.7

You can also control which applications can use specify which ports by using the SAF resname and the RACF profile

EZB.PORTACCESS.sysname.tcpname.resname.

Changing TCPIP configuration on z/OS

The startup configuration for a TCPIP instance is in the JCL PROFILE  ddname,  or a file like TCPIP.PROFILE.

You can change the configuration of a TCPIP image using the operator command

V TCPIP,TCPIPn,OBEY,filename

Where

  1. V TCPIP tells z/OS to route this TCPIP
  2. TCPIPn is the name of the TCPIP address space to direct the command to, for example V TCPIP,TCPIP2.  If there is only one TCPIP running you can use V TCPIP,,
  3. OBEY this is the TCP command
  4. filename is the parameter passed to the OBEY command.   The filename containing the commands/configuration to be executed.

How to display information on z/OS

There are three ways of displaying TCPIP information, for example the IP address(es) of the TCP image

  1. The operator command D TCPIP,TCPIP2,NETSTAT,HOME
  2. The TSO command NETSTAT HOME TCP TCPIP2
  3. The USS command netstat -h -p tcpip   The commands are similar to, but different from Linux commands!

The output is usually similar between the commands.

What is the IP address of my z/OS TCPIP image?

From the TSO NETSTAT HOME command

EZZ2350I MVS TCP/IP NETSTAT CS V2R4 TCPIP Name: TCPIP2 17:15:53
EZZ2700I Home address list:
EZZ2701I Address   Link         Flg
EZZ2702I -------   ----         ---
EZZ2703I 10.1.1.3  ETH1         P
EZZ2703I 10.1.2.3  ETHB
EZZ2703I 172.1.1.2 EZASAMEMVS
EZZ2703I 10.1.3.10 VIPL0A01030A I
EZZ2703I 127.0.0.1 LOOPBACK

For the links

  1. I configured link ETH1 and ETHB.
  2. The VIPL0A01030A takes the VIPA IP address and converts it to hex so 10.1.3.10 becomes VIPL 0A 01 03 0A
  3. EZASAMEMVS is prefix EZA and “SAME MVS”.   This is generated by TCPIP from the DYNAMIXCF configuration.
  4. You always get a LOOPBACK address at 127.0.0.1

What routing is there on z/OZ TCPIP?

The command TSO command NETSTAT ROUTE TCP TCPIP2 or the USS command netstat -r -p tcpip gives

MVS TCP/IP NETSTAT CS V2R4 TCPIP Name: TCPIP2 16:15:43 
Destination  Gateway  Flags Refcnt     Interface 
----------- -------   ----- ------     --------- 
Default      10.1.1.1 UGS   0000000000 ETH1 
10.0.0.0/8   0.0.0.0  US    0000000000 ETH1 
10.1.1.3/32  0.0.0.0  UH    0000000000 ETH1 
10.1.2.0/24  0.0.0.0  US    0000000000 ETHB 
10.1.2.3/32  0.0.0.0  UH    0000000000 ETHB 
127.0.0.1/32 0.0.0.0  UH    0000000000 LOOPBACK 
172.1.1.1/32 0.0.0.0  UHS   0000000000 EZASAMEMVS 
172.1.1.2/32 0.0.0.0  UH    0000000000 EZASAMEMVS 
172.1.1.3/32 0.0.0.0  UHS   0000000000 EZASAMEMVS

This shows that to get to 10.1.2.0 to10.1.2.255 (with a netmask of /24 or  255.255.255.0) it goes by link(interface) ETHB.

What configuration does Ubuntu have?

There are many commands to display network configuration information on Linux, for example ip and the older, superseded command, ifconfig.

What address does Ubuntu have?

ip address gives a lot of information – but I did not use it

What packet routing does my desktop have?

The command ip route gives

  1. 10.1.0.0/24 dev eno1 proto kernel scope link src 10.1.0.3 metric 100
  2. 10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1
  3. 10.1.2.0/24 dev tap1 proto kernel scope link src 10.1.2.1
  4. 10.1.3.0/24 dev tap0 scope link
  5. 10.20.2.4 dev tap0 scope link
  6. 192.168.1.0/24 dev wlxd037450ab7ac proto kernel scope link src 192.168.1.67 metric 600

Bold line(2) shows

  • Traffic for any address between 10.1.1.0 and 10.1.1.255 (remember the netmask /24 means 24 bits or 255.255.255.0) goes  to device(connection) tap0
  • The IP address for the desktop end of the connection is 1.1.1.1

Bold line(4) shows

  • that any traffic 10.1.3.0 to 10.1.3.255 goes to device tap0

The command used to set this up was sudo ip route add 10.1.3.0/24 dev tap0

Bold line(5) shows

  • that traffic to 10.20.2.4 goes to device tap0.

The command used to set this up was sudo ip route add  10.20.2.4 dev tap0

What is the routing for a particular IP address ?

You can use traceroute command to display which route a packet would take. For example

  • traceroute 10.1.3.10
    • traceroute to 10.1.3.10 (10.1.3.10), 30 hops max, 60 byte packets
      1 colins machine(10.1.1.1) 30 ms !H 30 ms !H 30 ms !H

This shows the route to 10.1.3.10 went to the connection with IP address 10.1.1.1

For a connection that is not defined

traceroute 10.20.2.5 
traceroute to 10.20.2.5 (10.20.2.5), 30 hops max, 60 byte packets
1 bthub.home (192.nnn.1.mmm) 3.170 ms 4.742 ms 6.379 ms
2 * * *

So we can see it went to my bt hub  wireless router.

You can also use the ping command.  On linux there is the -R option for display route.

ping -R 10.1.3.10 
PING 10.1.3.10 (10.1.3.10) 56(124) bytes of data.
64 bytes from 10.1.1.2: icmp_seq=1 ttl=64 time=2.54 ms
NOP
    RR: 10.1.1.1
        10.1.1.2
        10.1.1.1

The request went to 10.1.1.1.  10.1.1.2 caught it, and sent the reply back, via 10.1.1.1

I was looking for my VIPA address, 10.1.3.10, and we can see it got to 10.1.1.2.

For the ping to work, there must be a server processing the ping request.  If there are no applications processing the VIPA, the VIPA is not active, so a ping will fail.

A successful ping to a VIPA address means a packet can get to the LPAR, be processed and  the reply set back.  If the ping does not respond it could be

  1. The VIPA is not active
  2. The VIPA is active and a packet was sent to the LPAR hosting the VIPA, but it could not send a response back due to a set up error.