Using enclaves in a C program.

On z/OS enclaves allow you to set the priority of business transactions within your program, and to record the CPU used by the threads involved in the transaction – even if they are in a different address space.

Many of the WLM functions are assembler macros which require supervisor state. There are C run time functions which do some of the WLM functions. There are also Java methods which invoke the C run time functions.

Not all of the WLM functions are available in the C run time environment. You can set up enclaves, but the application cannot query information about the enclave, such as total CPU used, or the WLM parameters.

Minimal C program

The minimal program is below, and the key WLM functions are explained afterwards.

#include <sys/__wlm.h>
int main(void) { 
  wlmetok_t  enclavetoken; 
  server_classify_t   classify; 
  long rc; 
  // 
  //  Connect to work manager 
  // 
  unsigned int  connectToken = ConnectWorkMgr("JES","SM3");
  classify = __server_classify_create( ); 
  // pass the connection token to the classify
  //  This is needed but not documented 
  rc = __server_classify(classify, 
                         _SERVER_CLASSIFY_CONNTKN, 
                        (char *) connectToken 
       ); 
  rc = __server_classify(classify, 
                         _SERVER_CLASSIFY_TRANSACTION_NAME, 
                         "TCI2" 
  for ( int loop=0;loop < 1000 ;loop++) 
  { 
    rc= CreateWorkUnit(&enclavetoken, 
                       classify, 
                       NULL, 
                       "COLINS"   ); 
    rc = JoinWorkUnit(& enclavetoken); 
 
    // do some work to burn some CPU 
    for ( int i = 0;i< 100000 ;i++) 
    { 
      double xx = i/0.5; 
      double yy = xx * xx; 
    } 
    rc = LeaveWorkUnit(&enclavetoken); 
    rc = DeleteWorkUnit(&enclavetoken);   
    rc = DisconnectServer(&connectToken);
}
 

What are they key functions?

unsigned int connectToken = ConnectWorkMgr(“JES”,”SM3″); 

This creates a connection to WLM, and uses the subsystem JES, and subsystem name of SM3.  Note: On my system it is JES, not JES2.   The WLM dialogs, option 6. Classification Rules list the subsystems available.  You can browse a subsystem type and see the available definitions.  I had

         -------Qualifier--------                 -------Class--------  
Action   Type Name     Start        Service     Report   
 ____  1 SI   SM3      ___          TCI1SC      THRU
 ____  2   TN  TCI3    ___          TCI1SC      TCI3
 ____  2   TN  TCI2    ___          TCI1SC      TCI2

server_classify_t  classify = __server_classify_create( );

CreateWorkUnit, the function used  to create an independent enclave (business transaction), needs to be able to classify the transaction to determine what service class (priority) to give the enclave.  This request sets up the classify control block.

rc = __server_classify(classify, _SERVER_CLASSIFY_CONNTKN, (char *)&connectToken );

The documentation does not tell you to pass the connection token.  If you omit this step the CreateWorkUnit fails with error code errno2=0x0330083B.

The __server_classify expects a char * as the value, so you have to use (char *) & connectionToken.

rc = __server_classify(classify, _SERVER_CLASSIFY_TRANSACTION_NAME, “TCI2” );

This tells WLM about the transaction we want to use.  TRANSACTION_NAME matches up with TN above in the WLM definitions.  This says the business transaction is called TCI2.  There are other criteria such as userid, plan or LU. See here for the list.   The list is incomplete, as it does not support classifiers like Client IP address which is available with the assembler macros.

rc= CreateWorkUnit(&enclavetoken, classify,  NULL, “COLINS” );

This uses the classification parameter defined above, to create the independent enclave, and return the enclave token. 

The documentation for CreateWorkUnit says you need to pass the arrival time,  Address of a doubleword (unsigned long long) field that contains the arrival time of the work request  in STCK format.  I created a small assembler function which just returned a STCK value to my C program.  However I passed NULL and it seemed to produce the correct values  – I think CreateWorkUnit does a STCK for you.

You pass in the name of the function(“COLIN”).  The only place I had seen this is if you use the QueryWorkUnitClassification() to extract the classify information.  For example QueryWorkUnitClassification gave a control block with non empty fields   _ecdtrxn[8]=TCI2 , _ecdsubt[4]=JES , _ecdfcn[8]=COLIN  , _ecdsubn[8]=SM3 . This function does not return the report class or service class.

rc = JoinWorkUnit(& enclavetoken);

This cause any work this TCB does to be recorded against the enclave.

rc =LeaveWorkUnit(&enclavetoken);

This stop work being be recorded against the enclave.  Subsequent work gets charged to the home address space.

rc= DeleteWorkUnit(& enclavetoken);

The business transaction has finished.  Information about the response time and CPU used are stored.

rc =  DisconnectServer(&connectToken);

This disconnects from WLM.

Using a subtask

I had a different thread n the program which did some work for the transaction. Using the enclave token, this work can be recorded against the transaction using

// The enclave token is passed with the request
rc = JoinWorkUnit(&enclaveToken); 
do some work...
rc = LeaveWorkUnit(&enclaveToken);  

This would be useful if you are using a connect pool to connect to MQ or DB2 subsystem. You have a pool of threads which have done the expensive connect with a particular userid, and the thread is used to execute the MQ or DB2 subsystem as that userid.

Other function available

Other functions available

Dependent (address space) enclave

The above discussion was for an business transaction, know in the publications as an Independent enclave.   An address space can have a Dependent enclave where the CPU is recorded as “Dependent Enclave” within the address space.  You use the function ContinueWorkUnit(&enclave) to return the enclave token.    You then use JoinWorkUnit and LeaveWorkUnit as before.  I can not see why you might want to use this.

Display the classification

You can use the QueryWorkUnitClassification to return a structure for the classification.

Reset the classification.

If you want to classify a different transaction, you can use server_classify_init() to reset the structure.

Set up a server

You can set up a server where your application puts work onto WLM queues, and other threads can get work.   This is an advanced topic which I have not looked into.

Make your enclave visible across the sysplex

You can use ExportWorkUnit  and ImportWorkUnit to have your enclave be visible in the sysplex.

Query what systems in the sysplex are running in goal mode.

You can use QueryMetrics() to obtain the systems in the sysplex that are in goal mode. This includes  available CPU capacity and resource constraint status. 

What is not available to the C interface

One reason why I was investigating enclaves was to understand the enclave data in the SMF 30 records.  There is an assembler macro IWMEQTME which returns the CPU, ZIIP and ZAPP times, used by the independent enclaves.  Unfortunately this requires supervisor state.    I wrote some assembler code to extract this and display the data.  Another complication is that the IWLM macros are AMODE 31 – so it did not work with my 64 bit C program.