Processing PCF with Java

I started writing a C program to process PCF data, to do a better job than the provided samples.  For example create a date-time object for tools like ElasticSearch and Splunk, and interpret numeric values as strings.   I found I was writing more and more code to fill in some of the holes in the samples.    I decided to use Java, as there are classes called PCFMessage, and code to allow me to iterate over the PCF elements.
I struggled with processing Monitoring messages, but I’ll create a separate post for the misery of this.

I also struggled with the accounting trace, and have blogged the specific information here.

I’ll give the key bits of knowledge, and let any interested reader fill in the blanks – for example adding try…catch statements around the code.

MQ provides a PCF Agent…

But it does not work for getting PCF messages.   It allows you to send a PCF request, and it returns the result.  This does not work for just getting PCF messages.

Connect to MQ

mq = new MQQueueManager(data.qm, parms);

Access the queue

queue = mq.accessQueue(data.queue,MQConstants.MQOO_BROWSE  | MQConstants.MQOO_FAIL_IF_QUIESCING);

Get the message from MQ

MQGetMessageOptions gmo = new MQGetMessageOptions();

gmo.waitInterval = data.waitTime;
gmo.options = MQConstants.MQGMO_WAIT
+ MQConstants.MQGMO_PROPERTIES_IN_HANDLE
// + MQConstants.MQGMO_BROWSE_NEXT
;
msg = new MQMessage();
queue.get(msg, gmo);

Display any message properties

The MQ PCF code does not handle messages with a MQRFH2 and MQPCF headers.

Use gmo.options = … + MQConstants.MQGMO_PROPERTIES_IN_HANDLE to convert the RFH2 to message properties.

Process the properties.

Enumeration props = null;
props = msg.getPropertyNames("%");
if (props != null) {
 while (props.hasMoreElements()) {
   String propName = (String) props.nextElement();
   Object propObject = msg.getObjectProperty(propName);
   System.out.println(propName, propObject.toString() );
}

There is an MQHeaderList class, and iterator, but these did not seem very reliable. Even after I deleted a the MQRFH2 header, it still existed, and the PCF classes did not work.

Make it a PCF’able message

PCFMessage message = new PCFMessage(msg);

Display all of the elements in the PCF message

Enumeration<?> e = message.getParameters();
while( e.hasMoreElements()) {
   PCFParameter p = (PCFParameter)e.nextElement();
   if (p.getType() == MQConstants.MQCFT_GROUP )doGroup((MQCFGR) p);
   else {
     int parm   p.getParameter());  // number
     String tp = p.getParameterName();  // string
     String value = p.getStringValue() );  // values
     System.out.println(tp +":" + value); 
   }
}

Or you can get the specific fields from the message, for example message.getStringParameterValue(MQConstants.MQCAMO_START_DATE);

Process a group of elements

void doGroup(MQCFGR sgroup)
{
  Enumeration<?> f = sgroup.getParameters();
  while(f.hasMoreElements()){
     PCFParameter q = (PCFParameter)f.nextElement();
     if (p.getType() == MQConstants.MQCFT_GROUP )doGroup((MQCFGR) q);
     else System.out.println("q.getParameterName() +":" + q.getStringValue() );
}

And use similar logic for processing the fields as before.

Processing lists

Many elements have a list of items;  a list of integers, or  a list of 64 bit integers(long).   The array has an element:0 for Non Persistent and element:1 for Persistent;

I found it easiest to convert integers to longs, so I had to write fewer methods.

long[] longs = {0,0};

if (q.getType() == MQConstants.MQCFT_INTEGER64_LIST)
{ 
  MQCFIL64 i64l = ( MQCFIL64) q;
  longs = i64l.getValues(); 
}
else if (q.getType() == MQConstants.MQCFT_INTEGER_LIST)
{ 
  MQCFIL il = ( MQCFIL) q;
  int[] ii = il.getValues();
  longs = new long[ii.length];  // should be if size 2
  for (int iil = 0;iil < ii.length;iil++)
    longs[iil]= ii[iil];
}

Providing summary data

With fields like the number of puts, the data is an array of two values, Non Persistent, and Persistent.

It makes later processing much easier if you create

System.out.println("Puts_Total:" + (longs[0]  + longs[1]);
System.out.println("Puts_NP:" + long[0]); 
System.out.println("Puts_P:"  + long[1]);

There are verbs that produce data for Non Persistent and Persistent messages BROWSE, GET, PUT, PUT, PUT1 and different categories; number of requests, the maximum message size, the minimum message size, and the total bytes put or got.

MQIAMO_xxxs, MQIAMO_xxx_MAX_BYTES , MQIAMO_xxx_MIN_BYTES and MQIAMO64_xxx_BYTES

For MAX values the calculation is easy

REAL_GET_MAX_BYTES =max(longs[0],long[1]);

For minimum the calculation is more tricky, you need to consider if any messages were put with that persistence.  For example

If you have PUT_MIN_BYTES = [50,0] and PUTS[1,0] then the minimum size is 50.
If you have PUT_MIN_BYTES = [50,0] and PUTS[1,1] then the minimum size is 0.

public long P_NP_MIN( int[] values,int[]count ) {
  long min;
  if ( count[0] > 0 && count[1]> 0 )
  {
    if (value[1]< value[0]) min = value[1];
    else min = value[0];
  }
  else if (count[0] == 0 ) 
     min = value[1];
  else // count[0] > 0 and count[1] = 0
     min = value[0];
  return min;
}

then use it

long REAL_GET_MIN = P_NP_MIN(message.getIntListParameterValue(MQConstants. MQIAMO_GET_MIN_BYTES),
                             message.getIntListParameterValue(MQConstants. MQIAMO_GETS)
                           )      

Convert date and time into DateTime;

In some records, such as statistics and accounting, you have start dates and time, and end date and time.
It is useful to calculate the duration, and provide a dateTime value for tools like Splunk and ElasticSearch.
For example, fields MQConstants.MQCAMO_START_DATE and MQConstants.MQCAMO_START_TIME.

Some dates have “.” in the date, and sometimes it is “:”‘.

long doDateTime(PCFMessage m , 
               String label, 
               int date,  // eg MQConstants.MQCAMO_START_DATE 
               int time) // eg MQConstants.MQCAMO_START_TIME
  {
  Calendar cal = Calendar.getInstance();
  SimpleDateFormat sdfParse = new SimpleDateFormat("YYYY-MM-ddHH.mm.ss", Locale.ENGLISH);
  SimpleDateFormat sdfFormat = new SimpleDateFormat("YYYY-MM-dd'T'HH:mm:ss'Z'", Locale.ENGLISH);
  long timeMS = 0;
  try {
     // get the values from the message
     String startDate = m.getStringParameterValue(date);
     String startTime = m.getStringParameterValue(time);
     startTime = startTime.replace(":",".") ; // in case some times have ":" in the value
     String dateTime = startDate +startTime;  // make the dateTime value
     cal.setTime(sdfParse.parse(dateTime));   // Parse the combined value
     Date dateTimeVal = cal.getTime(); 
     System.out.println(label+":" + sdfFormat.format(dateTimeVal)); // print formatted time
     timeMS = dateTimeVal.getTime(); // number of milliseconds
    } catch (PCFException e) {
	// TODO Auto-generated catch block
	e.printStackTrace();
   } catch (ParseException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
   }
   return timeMS;
}

You need to substring the first 8 characters of the ActivityTrace OperationDate, as it is padded with nulls.

Some times are in format HH.mm.ss, and so the SimpleDateFormat has ‘.’ in the time parsing statement.  The ActivityTrace OperationTime has HH:mm:ss, so  the doDateTime routine  uses ..replace(“:”,”.”) .

Calculating the duration of statistics and accounting records.

You can then do

long startTime =
doDateTime(m,"StartDateTime",MQConstants.MQCAMO_START_DATE,
                             MQConstants.MQCAMO_START_TIME);
long endTime =
doDateTime(m,"EndDateTime",MQConstants.MQCAMO_END_DATE, 
                           MQConstants.MQCAMO_END_TIME);
long durationInMilliseconds = endTime-startTime;
if (durationInMilliseconds == 0) durationInMilliseconds = 1;

And use this to calculate rates, such as messages put per second, as part of the output of the program.

Doing calculations on the data

If you are iterating over the parameters, you need to be careful about data from multiple parameters.

For example within a queue accounting record you get fields

MQIAMO_OPENS
MQCAMO_OPEN_DATE
MQCAMO_OPEN_TIME
MQIAMO_CLOSES
MQCAMO_CLOSE_DATE
MQCAMO_CLOSE_TIME
MQIAMO_PUTS
MQIAMO_PUTS_FAILED
MQIAMO_PUT1S
MQIAMO_PUT1S_FAILED
MQIAMO64_PUT_BYTES
MQIAMO_PUT_MIN_BYTES
MQIAMO_PUT_MAX_BYTES

So if you want to display the number of puts failed as a percentage of all puts you need logic like

switch (p.getParameter()) {
  case MQConstants.MQIAMO_PUTS:
   puts = (int[] ) p.getValue();
   break;

case MQConstants.MQIAMO_PUTS_FAILED:
   int PutsfailedPC =  100 * (int) p.getValue()/ (put[0]+puts[1]);
break;

The values in a PCF message could in theory be in any order, though I expect a defect would be raised if the order changed.
To be bullet proof – you could iterate over the elements, and store them in a hashmap, then check the fields you want to use exist, then extract the values and use them.

Special cases…

I hit some “special cases” – what I would call bugs, in some of the processing.

Many strings have trailing nulls instead of trailing blanks (as documented), so you should to trim nulls off every string value.  Here is some code from Gwydion

String trimTrailingNulls(String s) {
  int stringLength = s.length();
  int i;
  for (i = stringLength; i > 0; i--) {
     if (s.charAt(i - 1) != (char) 0) {
       break;
     }
  }
  if (i != stringLength) {
    return s.substring(0, i);
  }
  return s;
}

Of course in C, printing a string with a trailing null is not a problem.

 

When I’ve got my code tested, and handled all of the strange data I’ll put my program up on github.