Accessing lots of data efficiently using C binary tree functions.

A binary tree is an efficient way of storing data in memory (in a tree!) The C run time library has a set of functions that allow you to create and manage data in a binary tree, but it is not that easy to set up.

Core concepts

Each element, or node, in the tree has two children and a pointer to the record. Each child can be null, or the address of another child. The left child contains elements which are less than the parent, the right node contains elements which are greater than the parent node. To find an element in the tree, you start at the top, or the root node, and compare the value of the nodes value, with the element you are looking for and work down the tree until you find a matching record or you hit the bottom of the tree. You provide a compare function takes two records and returns:

  • -1 if the first record is greater than the second record
  • 0 if the two records are the same item
  • 1 if the first record is less than the second record.

The comparison could be a simple string compare, or comparing data in a structure for example

typedef struct {
char pString[5]; // to the message text
int count ;
 int date;
} MyNode;

int Node_compare(const void *MyNode1, const void *MyNode2) {
  MyNode * p1, *p2;
int rc;
 // check the dates
  rc = p1-> date - p2 -> date;
  if (rc == 0 )
 // check the string
rc = strcmp(p1->pString, p2->pString);
  // return the result
  return rc;

}

You need a root for the tree

void *ROOT = NULL;

Some C I do not understand

In some of the routines you need code like

char * buffer = * (char **) in;
//char * buffer = (char *) in;

With my knowledge of C, these two statements look the same. Only the first one works.

I have code

void print_Node(const void *ptr, ...) {
...
 MyNode p = *(MyNode**) ptr;

There may be a smarter way of referencing the passed data, but the definition of “MyNode p” and using “* (Mynode **)” on the passed in pointer, works.

Referencing structures

It took me a while to work out the correct type definitions to get it to work. For example

int Node_compare(const void *, const void *);
MyNode * pMyNode;
void * p;
p = (void *) pMyNode;
MyNode * q;
/* see if it is in the tree already */
q = *(MyNode **) tsearch(p, (void **) &ROOT, Node_Compare);

The various functions expect pointer defined with type (void *).

Find an element

To look for for an element you use the tfind (tree find) function. The parameters are

  • a pointer to the record (or string) partially initialised with the values used by the compare function.
  • the root of the tree
  • the compare function, described above.

It returns the record, or a null.

To use this function you have to create a record containing the values the compare function can use it. This record can be in automatic storage, or you can use malloc() to allocate storage for it.

typedef struct {
char pString[5]; // to the message text
int count ;
 int date;
} MyNode;
MyNode tempNode;
memcpy(&tempNode.String("WXYZ"),sizeof("WXYZ");
tempNode.date = ....

You do not need to initialise tempNode.count, as this is not used in the comparison.

Adding an element

There is no tadd() function as such, there is a tsearch which provides “look for this node, and add it if it was not found“.

The parameters are

  • a pointer to the record. 
  • the root of the tree
  • the compare function, described above.

it returns the address of a record. If it is the one you passed in – then it was added. It it returns a different node – an existing record was returned.

See below for a more detailed description.

Deleting a node

You pass the standard three parameters in.

Walking the tree.

I’ve typically used a binary tree to store lots of information, such as a list of error messages and the count of occurrences, then, at the end of the job, print them all out in sequence. This is called walking the tree.

You use

twalk(root, print_function);

This calls the print_function for every element in the tree.

Planning for the tree

If you plan to walk the tree and print the elements in order, then the compare function needs to be written so the data is in the right order.

For example, I want to report the number of error messages, and the count of the messages, by day.

With my structure

typedef struct {
char pString[5]; // to the message text
int count ;
 int date;
} MyNode;

If I use

int Node_compare(const void *MyNode1, const void *MyNode2) {
  MyNode * p1, *p2;
int rc;
 // check the dates
  rc = p1-> date - p2 -> date;
  if (rc == 0 )
 // check the string
rc = strcmp(p1->pString, p2->pString);
  return rc;

}

then the data will be sorted by date, then message within date order

If I use

int Node_compare(const void *MyNode1, const void *MyNode2) {
  MyNode * p1, *p2;
int rc;
 // check the string
rc = strcmp(p1->pString, p2->pString);
  if (rc == 0 )

  // check the dates
  rc = p1-> date - p2 -> date;
  
  return rc;

}

it will report in message sequence, then by date within message.

Adding nodes to the tree.

You need to malloc storage for each node you intend to add, because the node may be stored within the tree.

MyNode  * pTode;
/* allocate our Node - we cant use automatic storage as it may */
/* be added to the tree - so must not be deleted */
pNode = (MyNode *) malloc(sizeof(MyNode));
if (pNode == 0)
{
perror("Malloc for Node failed ");
return 8;
}

/* initialise it */
strcpy(&pNode->pString[0],"ABCD") ;
pNode -> date = date();
pNode->count = 1;

void * p;
p = (void *) pNode;

MyNode * qreturned;
/* see if it is in the tree already */
qreturned = *(MyNode **) tsearch(p, (void **) &ROOT,Node_compare);

if (qreturned == p)
{
/* it didnt exist before - thus it was added */
   /* possibly do something   */
     initialise the remained of the record
}

else /* it did exist before so we need to update the count field */
{
   qreturned->count += 1; Update the values
   /* release the storage we dont need */
free(pNode);
}

All data must be self contained within the node, or to permanently allocate storage (with malloc()). If it references something in automatic storage, then when the automatic storage is reused, your pointer will point to invalid data.

Print the tree

twalk(ROOT, print_Node); // ROOT not &ROOT because of no updates, and pass the function
//
// which invokes
//
void print_Node(const void *ptr, VISIT order, int level) {
 if (order == leaf || order == postorder) {
  MyNode p = *(MyNode**) ptr;
  printf("Msg %4.4s date %d count %d\n",
        p->pString, p->date,p-> count);
  // level is how far down the tree the node is. Root = 0
  }
}

Putting it together

I wanted to collect information from RACF records and be able to refer back to the records. I treated the records as char *

The essence of the code is

Compare

#define LRECORD 294
int compare207(const void *MyNode1, const void *MyNode2) {
  int rc = memcmp( ((char * ) Mynode1) + 1
             ((char * ) Mynode2) + 1,261)
return rc;
}

Add a record

void add207(char * pRACF) 
{
void * p = malloc(LRECORD);
 ...
memcpy(p ,pRACF,294);
qreturned = *(char **) tsearch(p, (void **) &ROOT207,
compare207);
if (qreturned == p)
{
// it was added
}
else /* it did exist before so we need to update the count field */
{
//it existed - so update the fields);
  // qreturned -> ....
}

Find a record

char * find207(char * pIn) 
{
char * * pOut;
pOut =*(char **) tfind( (const void *) pIn,
(void **)&ROOT207,
compare207);
return pOut;
}

Walk the tree and print a record

void tree207print(const void *in    , VISIT order, int level) { 
char * pBuffer = * (char **) in;

 if (order == leaf || order == postorder)
{
  printf("Data %s\n",pBbuffer -> .... );
 }
}
void twalk207() 
{
twalk(ROOT207, tree207print);
}

CS IP filtering: writing a program to use Network management interfaces

You can write a program to get network management information from TCPIP by using the Network Management Interface API interface. It may be just as easy to use the IPSEC -f display command, and post process the output.

The Network Management Interface (nmi) is documented here (and links from this page) but it is not very easy to follow and get working. Some of the facts are there, but a lot is left to poor end user to determine how it works.

Overall the program is pretty simple

  • Connect to the server
  • Receive the init flow (the header) from the server
  • Send a request to the server
  • Receive the data from the server – this is where it gets a little tricker.

The make file

It took me a while to get my program to compile. Here is the make file I used

# Our first Makefile 
cparmsa= -Wc,"SSCOM,DEBUG,DEF(MVS,_ALL_SOURCE),UNDEF(_OPEN_DEFAULT,_NO_PROTO)
cparmsb= ,SO,SHOW,LIST(client.lst),XREF,ILP32,DLL,SKIPS(HIDE),LANGLVL(EXTENDED)"
syslib= -I'/usr/include' -I'/usr/include/sys'
all: main
parts = client.o
main: $(parts)
cc -o client $(parts)

%.o: %.c
cc -c -o $@ $(syslib) $(cparmsa)$(cparmsb) -V $<
clean:
rm *.o

It took time to find I needed UNDEF(_OPEN_DEFAULT,_NO_PROTO) . _NO_PROTO prevents function prototypes from being generated. As I wanted function prototypes, I had to UNDEF it!

My C program is was in file client.c

Include files

#define _OPEN_SYS_SOCK_IPV6 
#define _XOPEN_SOURCE_EXTENDED 1
#define _OPEN_SYS_SOCK_EXT
#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <netdb.h>
#include <ezbnmsec.h>
#include <sys/un.h>
#include <stdarg.h>
#include <iconv.h>
#include <netdb.h>
#include <errno.h>
#include <string.h>
#include <stdlib.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <unistd.h>
#include <//'COLIN.C.SOURCE(PRINTHEX)'> 

Connect to the server

This is a standard TCP/IP send/receive application using AF_UNIX and a socket stream.

int    sd=-1, rc; 
struct sockaddr_un serveraddr;
sd = socket(AF_UNIX, SOCK_STREAM, 0);
if (sd < 0)
{
perror("socket() failed");
break;
}
memset(&serveraddr, 0, sizeof(serveraddr));
serveraddr.sun_family = AF_UNIX;
#define SERVER_PATH "/var/sock/ipsecmgmt"
strcpy(serveraddr.sun_path, SERVER_PATH);

/********************************************************************/
/* Use the connect() function to establish a connection to the */
/* server. */
/********************************************************************/
rc = connect(sd, (struct sockaddr *)&serveraddr, SUN_LEN(&serveraddr));
if (rc < 0)
{

perror("connect() failed");
break;
}

My nmi request structure

I used

struct     { 
struct _NMsecMessageHdr header;
struct _NMsecRecordHdr record;
struct _NMsecSecDesc section;
struct _NMsecInFilter filter ;
} nmi;

I could use this to send a request to IKED, to receive data from the response to the initial connect.

Get the initial data from the server

After the connect request, the server sends back an init data stream into the nmi structure.

if (recv(sd,&nmi.header, sizeof(nmi.header), 0) < 0) 
{
tcperror("Recv()");
exit(6);
}
printf("Return code:%d\n",nmi.header.NMsMRc);
if (nmi.header.NMsMRc != 0)
{
printf("Reason code:%d\n",nmi.header.NMsMRsn);
printf("Message type:%hd\n",nmi.header.NMsMType);
printf("NMI version :%hd\n",nmi.header.NMsMVersion);
}
printf("==============\n");

Setting up input structure

I initialised the nmi structure using

      memset(&nmi.filter,0,sizeof(nmi.filter));       
      nmi.header.NMsMIdent = NMsec_MSGIDENT ;
nmi.header.NMsMHdrLength = sizeof(nmi.header);
nmi.header.NMsMVersion = NMsec_VERSION2;
nmi.header.NMsMMsgLength = sizeof(nmi);

// output record descriptor
memset(&nmi.header.NMsMOutRec ,0,sizeof(nmi.header.NMsMOutRec));

// input record descriptor
nmi.header.NMsMInRec.NMsIROffset = offsetof(nmi,record) ;
nmi.header.NMsMInRec.NMsIRRsvd1 = 0;
nmi.header.NMsMInRec.NMsIRNumber = 0; // start with none

      // need tcpip address space padded with blanks
  memset(&nmi.header.NMsMTarget,' ',sizeof(nmi.header.NMsMTarget));
  memcpy(&nmi.header.NMsMTarget,"TCPIP",5);

// clear the record
memset(&nmi.record,0,sizeof(nmi.record));

// record definition
nmi.record.NMsRIdent = NMsec_RECIDENT;
nmi.record.NMsRLength= sizeof(nmi.record) + sizeof(nmi.section) + sizeof(nmi.spare);
nmi.record.NMsRNumCascadeSecDesc = 0;
nmi.record.NMsRNumSecDesc = 1; // 1 section

// section
nmi.section.NMsSOffset = sizeof(nmi.spare ); // offset from start of record
nmi.section.NMsSLength = sizeof(nmi.spare);
nmi.section.NMsSNumber = 0;
//
// specify the request
//
// nmi.header.NMsMType = NMsec_GET_IPFLTCURR;
// nmi.header.NMsMType = NMsec_GET_SUMMARY;
// nmi.header.NMsMType = NMsec_GET_STACKINFO;
// nmi.header.NMsMType = NMsec_GET_IPFLTDEFAULT ;
// nmi.header.NMsMType = NMsec_GET_IPFLTCURR ;
// nmi.header.NMsMType = NMsec_GET_IPFLTPOLICY ;

Build the request

We were passed in the request type ( option 1 to 6) and an optional filter name in pFilterName;

If we have a filter name, we need to set the number of headers, number of sections, and number of filters – they were all set to 0.



// process the input parameters
int type = NMsec_GET_SUMMARY ;// preset this
char * pFilterValue = "";   // no filter

if (argc >= 2 ) type = atoi(argv[1]);
if (argc >= 3 ) pFilterValue = argv[2];


nmi.header.NMsMType = type;
// we can have an optional filter for these requests
if ((nmi.header.NMsMType == NMsec_GET_IPFLTCURR
||nmi.header.NMsMType == NMsec_GET_IPFLTDEFAULT
||nmi.header.NMsMType == NMsec_GET_IPFLTDEFAULT )
&& strlen(pFilterValue) > 0
)
{
if ( strlen(pFilterValue) > sizeof(nmi.filter.NMsFltObjName))
printf("Filter name %s is too long \n",pFilterValue);
else
{
// set up the filter
memset(&nmi.filter.NMsFltObjName,' ',sizeof(nmi.filter.NMsFltObjName));
memcpy(&nmi.filter.NMsFltObjName,pFilterValue,strlen(pFilterValue));
nmi.header.NMsMInRec.NMsIRNumber = 1; // one header record
nmi.record.NMsRNumSecDesc = 1; // 1 section
nmi.filter.NMsFltFlagObjNam = 1; // one filter
}
}

and send it

if (send(sd,&nmi       , sizeof(nmi       ), 0) < 0) 
{
printf("send failed\n");
tcperror("Send()");
exit(5);
}
printf("send done\n");

Receiving the response

This gets a little more complex. You cannot send an nmi structure larger than about 10KB, but the data returned can be much larger than this!

You need to peek at the data in the buffer, and allocate a buffer big enough for the returned data, then receive the data

Initial set up for receive

struct inData  { 
struct _NMsecMessageHdr header;
struct _NMsecRecordHdr record;
struct _NMsecSecDesc section;
char spare[1024 ];
} ;


struct inData * pInData;
char * pInBuffer;
int lInBuffer =1024;
pInBuffer = malloc(lInBuffer);
if (pInBuffer == 0)
{
printf("Malloc failed\n");
exit(12);
}
pInData = (struct inData * ) pInBuffer;

Use the MSG_PEEK option of recv.

int lRecv = 0; 
lRecv = recv(sd,pInBuffer,lInBuffer, MSG_PEEK);
if (lRecv < 0) // there was a problem
{
printf("recv failed\n");
tcperror("Recv()");
exit(6);
}
// look at the length of the data from the peek buffer
// if it is larger than our buffer reallocate buffer
if (pInData -> header. NMsMMsgLength > lInBuffer)
{
printf("Buffer not big enough - making it %d\n",pInData -> header. NMsMMsgLength);
lInBuffer = pInData -> header. NMsMMsgLength; // use the received length
free(pInBuffer);
pInBuffer = malloc(lInBuffer);
if (pInBuffer == 0)
{
printf("Malloc failed\n");
exit(12);
}
pInData = (struct inData * ) pInBuffer;
}

and now the real receive

// now do the real receive 
lRecv = recv(sd,pInBuffer,lInBuffer , 0 );
if (lRecv < 0)
{
printf("rec failed\n");
tcperror("Recv()");
exit(6);
}
printf("Length received %d\n",lRecv);
// set up defensive programming
char * pEnd;
pEnd = pInBuffer + lRecv ;

Check it worked

printf("Return code:%d\n",pInData->header.NMsMRc); 
printf("Reason code:%d\n",pInData->header.NMsMRsn);
printf("Message type:%hd\n",pInData->header.NMsMType);

if (InData->header.NMsMRc > 0)
{
// error handling
}

Process the data

printf("Output total  %d actual %d\n",pInData->header.NMsMOutRec.NMsORTotal 
  , pInData->header.NMsMOutRec.NMsORNumber);

char * pCurr; // current position in the output buffer.
struct _NMsecRecordHdr * pRecord ;
struct _NMsecSecDesc * pSection;
int nRecords = pInData->header.NMsMOutRec.NMsORNumber;
printf("nRecords %d\n",nRecords);
// output record descriptor locates the first output record
pRecord = (struct _NMsecRecordHdr *)
(pInBuffer + pInData->header.NMsMOutRec.NMsOROffset);
for (int rLoop = 0;rLoop <nRecords;rLoop ++)
{
pCurr = (char *) pRecord;
// defensive programming
if ( pCurr > pEnd)
{
    printf(" Record ran off the end of the buffer \n");
  leave = 1;
  break;
}
printf("Record number %d\n",rLoop);
  // section follows the header
  pCurr += sizeof(pInData->record); // size of header
  pSection = (struct _NMsecSecDesc * ) pCurr;
  // for number of sections defined in the record header
 for (unsigned short sLoop = 0; sLoop < pRecord -> NMsRNumSecDesc ; sLoop ++)
 {
char * pData = (char *) pRecord + pSection-> NMsSOffset;
int lData = pSection-> NMsSLength;
// each section can have multiple data items
for (int sdLoop = 0; sdLoop < pSection-> NMsSNumber; sdLoop ++)
{
.... process the data
pData += lData;
} // for (int sdLoop = 0; sdLoop < pSection-> NMsSNumber; sdLoop ++)
pSection ++; // move to the next section
} // for (unsigned short sLoop = 0; sLoop < pRecord -> NMsRNumSecDesc ; sLoop ++)
 // move to the next record by jumping the length of the record
pRecord = (struct _NMsecRecordHdr *)((char *) pRecord + pRecord -> NMsRLength);
} // for (int rLoop = 0;rLoop <nRecords;rLoop ++)

Process the data

printf("Record:%d Section:%d part %d \n",rLoop,sLoop,sdLoop); 
if( pInData->header.NMsMType == NMsec_GET_IPFLTCURR )
{
printFilter(pData);
// printHex(stdout,pData,lData);
}
else
if( pInData->header.NMsMType == NMsec_GET_IPFLTDEFAULT) printFilter(pData);
else
if( pInData->header.NMsMType == NMsec_GET_IPFLTPOLICY ) printFilter(pData);
else
if( pInData->header.NMsMType == NMsec_GET_IPFLTCURR ) printFilter(pData);
else

if( pInData->header.NMsMType == NMsec_GET_SUMMARY ) printStatistics(pData);
else
if( pInData->header.NMsMType == NMsec_GET_STACKINFO ) printStackInfo(pData);
else
printHex(stdout,pData,lData);

Display the data

Process the data for example

void printStackInfo(char * pData) 
{
struct _NMsecStack * pStack = (struct _NMsecStack *) pData ;
printf(" Stack name %24.24s\n",pStack -> NMsStackName);
printf(" Configured filters %n",pStack -> NMsStackFilterCount ) ;
printf(" Defensive filters %d\n",pStack -> NMsStackDefFltCount ) ;
}

In

void printStatistics(char * pData) 
{
struct _NMsecStatistics * pStats = (struct _NMsecStatistics *) pData ;
printf(" filter packets denied %lld\n",pStats -> NMsStatFilterDeny);
printf(" filter packets discarded mismatch %lld\n",pStats -> NMsStatFilterMismatch) ;
printf(" filter packets discarded match %lld\n",pStats -> NMsStatFilterMatch ) ;
}

You need to use %lld – as the numbers are 64 bit integers.

These numbers are reset if the rule is altered.

Using TCPIP Port with SAF

I had configured my TCPIP to have

PORT 
...
3023 TCP * SAF VERIFY ; COLIN PAICE

This says

  • 3023 is the port number
  • TCP for when the request is TCP ( and not UDP)
  • * Any job
  • SAF check with the SAF interface
  • VERIFY. This is part of the RACF resource name.

I used X3270 to provide TLS sessions into z/OS. When TN3270 started (or was refreshed), I got

EZZ6035I TN3270 DEBUG CONFIG EXCEPTION
 LINE: N/A MOD: EZBTMCVU
 RCODE: 8020-00 Initialization of the Telnet Port failed.
 PARM1: 0000102B PARM2: 3023 PARM3: 00000000
EZZ6038I TN3270 COMMAND OBEYFILE COMPLETE
EZZ6035I TN3270 DEBUG TASK EXCEPTION 766
 TASK: MAIN MOD: EZBTZMST
 RCODE: 1018-01 The Port task has ended in error.
 PARM1: 0000102B PARM2: 00000BCF PARM3: 00000000

I could not find parm1 (0000102B) defined anywhere.

I defined the SERVAUTH profile EZB.PORTACCESS.*.*.* with fail WARN, and I got a message

ICH408I USER(TCPIP ) GROUP(OMVSGRP ) NAME(TCPIP
EZB.PORTACCESS.S0W1.TCPIP.VERIFY CL(SERVAUTH)
 WARNING: INSUFFICIENT AUTHORITY – TEMPORARY ACCESS ALLOWED
 FROM EZB.PORTACCESS...* (G)
 ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

Where

  • SOW1 is my z/OS system
  • TCPIP is my TCPIP image name
  • VERIFY – is the value in the TCPIP configuration file

With the RACF definition I did not get the error message.

I then defined a proper profile for it.

CS IP Filtering: Trying to use FTP

With my FTP, the initial connection worked on port 21, then it switched to use a different port for the file transfer. These were not set up on my system. 

On my FTP client, I received

229 Entering Extended Passive Mode (|||1028|)

which means it was trying to use port 1028. This was not configured, and so the transfer failed.

You have to configure FTP to use a range of ports, and then configure the Policy Agent with these ports.

In the FTPD JCL I have

//FTPD   PROC MODULE='FTPD',PARMS='' 
//FTPD EXEC PGM=&MODULE,REGION=4096K,TIME=NOLIMIT,
// PARM='POSIX(ON) ALL31(ON)/&PARMS'
//CEEDUMP DD SYSOUT=*
//SYSFTPD DD DISP=SHR,DSN=TCPIP.FTP.DATA
//* SYSTCPD explicitly identifieS which file iS to be
//* uSed to obtain the parameterS defined by TCPIP.DATA.
//* The SYSTCPD DD Statement Should be placed in the JCL of
//* the Server. The file can be any Sequential data Set,
//*SYSTCPD DD DISP=SHR,DSN=TCPIP.SEZAINST(TCPDATA)
//SYSTCPD DD DISP=SHR,DSN=ADCD.&SYSVER..TCPPARMS(TCPDATA)

Within the SYSFTPD dataset you need to configure the PASSIVEDATAPORTS statement.

See setting up FTP Daemon on z/OS.

Setting up FTP Server on z/OS

I had problems using the FTP server on z/OS once I had configured IP Filtering (any IP addresses and ports which are not in the configuration are dropped).

Once an FTP client has established contact using port 21, the transfer switches to a different port. You can control which port range is used.

Setting up the FTP server.

If you start the FTP Daemon (S FTPD) it starts up another procedure (FTPD1) and the initial job ends.

You need to configure the SYSLOGD daemon to capture any output from the FTP task.

In /etc/syslog.conf I set up

*.FTPD*.*.*         /var/log/FTPD.%Y.%m.%d 

so any output from jobs with a name FTPD* will go into the specified file.

In my file I had

EZYFT46E Error in dd:SYSFTPD=TCPIP.SEZAINST(FTPSDATA) file: line 1283 near column 1.
EZY2642E Unknown keyword: PASSIVEDATAPORTS(8000,8100)

Configuring ports

To limit which ports FTP uses you need to specify PASSIVEDATAPORTS

PASSIVEDATAPORTS (8000,8100)

with a blank between the keyword and the (.

You also need to tell TCPIP that the port range is reserved for TCPIP’s use for example

PORTRANGE
... 50000 100 TCP AUTHPORT

Where AUTHPORT Indicates that all ports in the port range are not available for use by any user except FTP, and only when FTP is configured to use PASSIVEDATAPORTS. AUTHPORT is valid only with the TCP protocol.

When you try to transfer a file you get a message

ftp> get ‘…’ … local: …: ‘…’
229 Entering Extended Passive Mode (|||8061|)

where 8061 is the port which was used.

IPSEC definitons for IP Filtering

For my very restrictive access from my laptop to z/OS (and no other devices) I used

IpFilterRule FTPnI21 
{
IpSourceAddrGroupRef zGroup
IpSourceAddr 10.1.0.2
IpDestAddr 10.1.1.2
IpGenericFilterActionRef permitlog
IpService
{
Protocol Tcp
DestinationPortRange 21
Direction inbound
Routing local
}
IpService
{
Protocol Tcp
DestinationPortRange 8000-8100
Direction inbound
Routing local
}
}

and

IpFilterRule FTPO21 
{
IpSourceAddr 10.1.1.2
IpDestAddr 10.1.0.2
IpGenericFilterActionRef permitlog
IpService
{
Protocol Tcp
SourcePortRange 21
Direction outbound
Routing local
}
IpService
{
Protocol Tcp
SourcePortRange 8000-8100
Direction outbound
Routing local
}
}

CS IP filtering: defining dynamic rules for IP traffic

To see all my blog posts on IP filtering see here.

With Communications Server(CS) on z/OS you can manage IP traffic in your TCPIP image. You can allow or deny a IP packet.

You can define

  • Default rules to allow traffic, in TCPIP Profile
  • To define a set of named rules using Policy agent. These can allow or deny access
  • Individual dynamic rules which can be automated, for example external monitors, using DMD

Dynamic rules

You can use the Unix command ipsec -F to add,delete or modify individual rules. These rules might be generated by an external monitoring scheme, which can say to devices

“we seem to have a problem – use this rule to stop traffic from this destination while the CS people investigate and update the rules in the policy agent”

an example is

ipsec -F add srcip 10.1.0.2 destip all prot icmp dir inbound mode block log yes loglimit 1 lifetime 30 -N COLIN1 -p TCPIP

This defines a rule called COLIN1 on TCPIP address space TCPIP. Note you can specify a global rule for all TCPIP address spaces, by specifying -G instead of -p TCPIP

  • Source IP address 10.1.0.2
  • Destination IP address all
  • Only ICMP prototols
  • Direction inbound
  • Mode block. You can use Simulate to allow the packet – but to write an event to the log. This is useful when building rules.
  • Log – write it to the syslogd log
  • Loglimit – this logs at an average rate of 1 event per 5 minutes – so you can avoid flooding the logs.
  • Lifetime – this rule expires after 30 minutes.

Display all dynamic rules

ipsec -F display -p TCPIP

and delete it

ipsec -F delete -N COLIN1 -p TCPIP

When you add a rule, you get information In the log such as

EZD1723I Defensive filter added: 11/21/2023 08:58:10.86 filter rule= COLIN1 ext= 1 
sipaddr= 10.1.0.2 / 32 dipaddr= 0.0.0.0 / 0 proto= icmp(1) type= all code= all fragmentsonly= no dir= inbound routing= local mode= block log= yes lifetime= 30 
userid= IBMUSER global= no loglimit= 1

Log limited output

If you specify loglimit:

The value specifies the limit of the average rate of filter-match messages generated in a 5-minute interval for a defensive filter.

You get a trace message like

EZD0838I Defensive filter packet would have been denied messages limited: 
11/21/2023 13:04:22.92 filter_rule= COLIN2 filter_ext= 1 
filter_sipaddr= 10.1.0.2 / 32  filter_dipaddr= 0.0.0.0 / 0 
filter_proto= icmp(1) type= 8 code= all  filter_fragmentsonly= no filter_dir= inbound filter_routing= local suppressed_count= 274 

This shows there were 274 suppressed messages

EZD0838I:This message is issued when limiting of filter match messages was requested for a defensive filter and at least one “packet would have been denied” message (EZD1722I) for the defensive filter was suppressed during the preceding five minutes.

Suppressed counts: The number of “packet would have been denied” messages (EZD1722I) for the defensive filter that were suppressed during the preceding five minutes.

CS IP filtering: understanding logged messages

To see all my blog posts on IP filtering see here.

If you are using Communications Server IP filtering to allow/deny IP traffic to destinations, you need the TMDR daemon running. TMDR writes messages to the syslogd which can write them to a file

The trace output is like

EZD0815I Packet denied by policy: 11/17/2023 13:09:17.00 filter rule= Colin ext= 2 sipaddr=10.1.0.2 dipaddr= 10.1.1.2 proto= icmp(1) type= 8 code= 0 -=
Interface= 10.1.1.2 (I) secclass= 255 dest= local len= 84 vpnaction= N/A
tunnelID= N/A ifcname= ETH1 fragment= N

The request

  • came from 10.1.0.2 ( the Source IP ADDRress)
  • going to 10.1.1.2 the (Destination IP ADDRess
  • The protocol was icmp. Within the IP protocol an icmp packet has type 1.
  • Type is Echo request (used to called ping), and icmp packet with type 8. See here.
  • The name of the z/OS interface definition ETH1. See NETSTAT DEVLINKS
  • The IP address of the interface 10.1.1.2
  • The packet was inbound(I)
  • There was a keyword with no value -= … this is useful when parsing the traffic, because it shows a null object.

Using the Unix System Services command

ipsec -p TCPIP -f display > a

This gave

...
FilterName:                   Colin 
FilterNameExtension:          1 
...
Direction:                    Outbound 
...
FilterName:                   Colin 
FilterNameExtension:          2 
...
Direction:                    Inbound 

From filter rule= Colin ext= 2 we can see this was inbound.

With an input file

IpFilterPolicy 
{ 
  PreDecap off 
  FilterLogging on 
  IpFilterLogImplicit yes 
  AllowOnDemand yes 
  IpFilterRule Colin 
  { 
    IpSourceAddr 10.1.1.2 
    IpDestAddr   All 
    IpService 
    { 
      Protocol all 
      Direction bidirectional 
      Routing local 
    } 
    IpService 
    { 
#  PING  ICMP type 8 
      Protocol icmp Type 8 
      Direction bidirectional 
      Routing local 
    } 
    IpGenericFilterActionRef permit 
    IpService 
    { 
# 
      Protocol icmp Type 9 
      Direction bidirectional 
      Routing local 
    } 
} 

With Direction bidirectional it creates two rules, one in bound, and one outbound.

When PING was performed – this was rule 4 – from the bold text.

CS IP filtering: understanding the display output of the ipsec -f display command

To see all my blog posts on IP filtering see here.

The list below are the fields I found interesting ( mainly because they were not “n/a”)

FilterName: COLIN1

The name comes from

  • COLIN1 comes from a defensive rule like ipsec -F add … -N COLIN1, or from the name of an Policy rule IpFilterRule.
  • SYSDEFAULTDENYRULE comes from the default default rule
  • SYSDEFAULTRULE.1 comes from the default rule .1 means the first rule in the file


FilterNameExtension: 1

There can be more than one rule from a definition. For example a Direction bidirectional will create an inbound and an outbound rule. Within an IpFilterRule there can be multiple IpService definitions, each potentially with Direction bidirectional. The extension will increase for each rule.

Type: Defensive

It can be Generic if created by Policy Agent.

DefensiveType: Stack

For Type: Defensive it can be Global or Stack. For other type it is n/a.

State: Active

This always seems active (in the current list)

Action: Defensive Block

This can be Defensive Block, Deny, or Permit.

Scope: Local

Direction: Inbound

Or outbound.


SecurityClass: 0

You can classify interfaces with a security class of 1 to 255. 0 Means not specified.

Logging: All

Indicates the logging that is to be performed when the filter is invoked. Possible values are:

  • All: A log entry is generated if data traffic is permitted or denied.
  • Permit: A log entry is generated only when data traffic is permitted.
  • Deny: A log entry is generated only when data traffic is denied.
  • None: No log entries are generated from this filter.

LogLimit: 1

Used to summarise the number of times defensive rules are logged.

Protocol: ICMP(1)

For example UDP, TCP, igmp.

SourceAddress…: 10.1.0.2

There are several fields which define the source address:

SourceAddress, such as 10.1.1.2

  • Source Address: such as 10.1.1.2
  • SourceAddressPrefix, 24 as in IpSourceAddr 10.1.1.2/24
  • SourceAddressRange, 11.2.1.2 as in IpSourceAddr 10.1.1.2-11.2.1.2
  • SourceAddressGranularity, used in dynamic tunnels and dynamic VPNs.

A value of 0.0.0.0 is all.

SourcePort: and SourcePortrange

For example SourcePortRange 600 604 gives SourcePort 600 and SourcePortRange 604.

DestAddress: 0.0.0.0

See SourceAddress above.

DestPort…:

See SourcePort above

CreateTime: 2023/11/22 17:57:26

The time it was first activated (the time the TCPIP stack created its entry).

UpdateTime: 2023/11/22 17:57:26

The time it was last changed within the TCPIP stack.

DiscardAction: Silent

Indicates the discard action for packets that are discarded as a result of this filter rule. Possible values are:

  • Silent: Packets are discarded silently.
  • ICMP: When a packet is discarded, an ICMP or ICMPv6 error is sent to the origin of the discarded packet to indicate that the packet was administratively prohibited.


FilterMatches: 0

For a defensive filter the number of times this rule has been hit.

LifetimeExpires: 2023/11/22 18:27:26

For a defensive filter the time this rule will expire.


CS IP filtering: configuring TMDR – Traffic Management Daemon Regulation

To see all my blog posts on IP filtering see here.

If you are using Communications Server IP filtering to allow/deny IP traffic to destinations, you need the TRMD daemon running. With IP filtering you can log when access is allowed/denied.

The TRMD is needed to write the log events.

TRMD started task JCL

//TRMD      PROC  OPT='-P TCPIP -d 1' 
//* 
//*   IBM Communications Server for z/OS 
//*   SMP/E distribution name: EZATRMDP 
//TRMD   EXEC PGM=EZATRMD,REGION=4096K,TIME=NOLIMIT, 
//      PARM=('ENVAR("_CEE_ENVFILE_S=DD:STDENV")/&OPT') 
//STDENV   DD * 
RESOLVER_CONFIG=//'ADCD.Z24C.TCPPARMS(TCPDATA)' 
export TZ=GMT0 
/* 
//SYSPRINT DD SYSOUT=*,DCB=(RECFM=F,LRECL=80,BLKSIZE=80) 
//SYSIN    DD DUMMY 
//SYSERR   DD SYSOUT=* 
//SYSOUT   DD SYSOUT=*,DCB=(RECFM=F,LRECL=80,BLKSIZE=80) 
//CEEDUMP  DD SYSOUT=*,DCB=(RECFM=FB,LRECL=132,BLKSIZE=132) 

I do not know what authority it needs to run. It runs as userid START1 on my ADCD system.

When you start it, it starts up a job with name TMDR1, and the “started task” TMDR ends.

You stop it using P TMDR1.

Logging

The syslogd daemon needs to be running.

In my /etc/syslog.conf file I have

*.TRMD1.*.info      /var/log/TRMD1I.%Y.%m.%d 
*.TRMD1.*.debug     /var/log/TRMD1D.%Y.%m.%d 

Both of these write the access/denied messages – you would typically just have one of these.