Understanding OSGI data

My original problem was to find the MQ configuration data used by applications running under Liberty on z/OS. I eventually found I could display all the configuration data from Liberty. This provided me with more data than I wanted to know. However this uses an interface with no authentication on authorization, and it can stop parts of Liberty.

I used the bundles command, collected the information in a file, and wrote a python script to process this file. This put each jar’s information in its own file, so it is readable.

This post describes what I see in it – It is not documented, so I may be wrong.

Within the data are

Within the file are two section

  • Registered services – services this jar provides
  • Services in use – services which this jar uses.

These contain dictionaries of  service properties. For example

{javax.jms.Queue
javax.jms.Destination
}=
{
component.id=416
config.displayId=jmsQueue[default-0]
config.id=com.ibm.ws.jca.adminObject.supertype[extends-0]
osgi.jndi.service.name=jms/stockRequestQueue
properties.0.CCSID=1208
properties.0.baseQueueManagerName=
properties.0.baseQueueName=STOCK_REQUEST
...
service.id=795
}

and

{javax.jms.Queue
javax.jms.Destination
}=
{osgi.jndi.service.name=jms/stockResponseQueue
properties.0.baseQueueManagerName=
properties.0.baseQueueName=STOCK_RESPONSE
...
service.id=796
}

This shows there are multiple object of type javax.jms.Queue/javax.jms.Destination.  One for  a queue called STOCK_REQUEST, the other for STOCK_RESPONSE

The object type com.ibm.ws.jca.service.AdminObjectService has almost the same information as the javax.jms.Queue object.    They have a different service.id, so they are different services.

I used grep to find all instances of STOCK_REQUEST.  There were 3 entries in 2 jar files

  • jar com.ibm.ws.app.manager_1.1.41 cl200620200528-0414:
    • {com.ibm.wsspi.application.lifecycle.ApplicationRecycleComponent,
      com.ibm.wsspi.resource.ResourceFactory,
      com.ibm.ws.jca.service.AdminObjectService
      }=
    • config.displayId=jmsQueue[default-0],
      jndiName=jms/stockRequestQueue,
      properties.0.baseQueueName=STOCK_REQUEST
  • jar com.ibm.ws.jca_1.0.41 cl200620200528-0414:
    • {com.ibm.wsspi.application.lifecycle.ApplicationRecycleComponent
      com.ibm.wsspi.resource.ResourceFactory
      com.ibm.ws.jca.service.AdminObjectService
      }=
    •  config.displayId=jmsQueue[default-0]
      jndiName=jms/stockRequestQueue
      properties.0.baseQueueName=STOCK_REQUEST
      service.id=795
  • jar com.ibm.ws.jca_1.0.41 cl200620200528-0414:
    • {javax.jms.Queue
      javax.jms.Destination
      }=
    •  config.displayId=jmsQueue[default-0]
      osgi.jndi.service.name=jms/stockRequestQueue
      properties.0.baseQueueName=STOCK_REQUEST
      service.id=796

We can see the same data is in multiple places – it should all be similar.

MQ pool size

I was interested in the information about MQ connection pools.  I found this in com.ibm.ws.jca.cm_1.1.41.cl200620200528-0414.

  • agedTimeout=-1
  • component.id=491
  • component.name=com.ibm.ws.jca.connectionManager
  • config.displayId=connectionManager[ConMgr1]
  • config.id=com.ibm.ws.jca.connectionManager[ConMgr1]
  • config.overrides=true
  • config.source=file
  • connectionTimeout=30
  • enableSharingForDirectLookups=true
  • id=ConMgr1
  • maxIdleTime=1800
  • maxPoolSize=5
  • purgePolicy=EntirePool
  • reapTime=180

See here for what they mean.

My python code for taking the output from “bundles” and create individual files

import sys
 fIn = open("/home/colinpaice/Documents/osgi.log", "r")
 for x in range(4): # ignore the top 4 lines
    line = fIn.readline()
    if not line: break
 outputFile = 0
 for x in range(100000): # 
    line = fIn.readline()
    if not line: break
    if line[0:5] == "osgi>": break 
    if line[0] != ' ': # new file 
      if outputFile != 0: 
         outputFile.close()
      line1 = line.split();
      fn = 'output/' + line1[0]
      outputFile = open(fn,'w')
      outputFile.write(line) 
    elif line[0:5] == "  Id=": 
        v = line.split('"')
        outputFile.write(v[0] +"\n" ) 
        if len(v) > 1 :
            outputFile.write("  "+ '"'+v[1] +'"\n')  # services
    elif line[0:5] == '    {':
        p = line[5:-2].split("}={")
        p0 = p[0].split(", ")
        pad = "     {"        
        for ip0 in p0:
            outputFile.write(pad + ip0 +"\n")
            pad = "      "  
        outputFile.write("     }="+"\n")        
        p1 = p[1].split(", ")
        c = {}  # dictionary of the keyword=values
        for ip1 in p1:
             v = ip1.split("=")
             if ( len(v) ==1  ):  # no k=value, just value
                value = value + " " + v[0] # append it to the previous one
                if value.endswith("]"): # if it is the last one 
                                        # update the dict
                   c[kw] = value
             else: 
                 kw = v[0]
                 value = v[1]              
                 c[kw] = value 
        pad = "     {"  # only the first one has  {...          
        for key in sorted(c.keys()): # print alphabetically                
           outputFile.write(pad + key+"="+c[key] +"\n")
           pad = "      "
        outputFile.write("     }" +"\n")              
    else:       
        outputFile.write(line  ) 

Using OSGI to display information about the Liberty configurations. Useful commands

This post gives some of the commands you can issue to OSGI in Liberty.

I set up access to Liberty OSGi using

<featureManager>
 <feature>osgiConsole-1.0</feature> 
</featureManager>

osgi.console=10.1.3.10:5471. in bootstrap.properties file, and restarted Liberty.

Once it was active I used

telnet 10.1.3.10 5471 |tee -i osgi.log

to issue commands to OSGI and save the output in the osgi.log file.

A subset of the display commands

help
list the available commands (there are many)
help lb
explain the lb command
disconnect
you have to type out the complete command
lb
list the bundles. This produces a list with

  • ID – a number
  • State – Active
  • Name JMS Connection Factory
lb jca
list bundles with jca in the description.
bundles
list all information about all bundles.
bundles 99
Give more information on the bundle with id 99
headers 99
lists the information from the manifest about the specified bundle id.
jndilist
list the high level jndi tree elements.
jndilist com/ibm
List the element under this tree.
jndilist jms
The JMS definitions

  • cf1: com.ibm.mq.connector.outbound.ConnectionFactoryImpl: com.ibm.mq.connector.outbound.ConnectionFactoryImpl…
  • stockRequestQueue: com.ibm.mq.connector.outbound.MQQueueProxy: com.ibm.mq.connector.outbound.MQQueueProxy…
  • stockResponseQueue: com.ibm.mq.connector.outbound.MQQueueProxy: com.ibm.mq.connector.outbound.MQQueueProxy…
ls
lists the files in the directory on disk.
cat name
lists the contents of the named file from disk – see I said this was a dangerous command.

More than you ever wanted to know about your Liberty configuration – overview

It all started with wanting to look at the MQ configuration within MQWEB and z/OS Connect. I could not find a way of displaying the Liberty configuration information. I could look at the *.xml files, but this did not give me the information which is not in the configuration files. I thought about writing a Liberty feature which printed out the configuration, but then I stumbled across OSGI and the ability to query configuration information from outside of Liberty.

What is OSGI ?

The OSGi specification describes a modular system and a service platform for the Java programming language that implements a complete and dynamic component model, something that does not exist in standalone Java/VM environments. Applications or components, coming in the form of bundles for deployment, can be remotely installed, started, stopped, updated, and uninstalled without requiring a reboot; management of Java packages/classes is specified in great detail. Application life cycle management is implemented via APIs that allow for remote downloading of management policies. The service registry allows bundles to detect the addition of new services, or the removal of services, and adapt accordingly.

Wikipedia

OSGI uses the term “bundle”. It looks like a bundle is a jar file, and the facilities within it. Listing one bundle had com.ibm.ws.ssl_1.3.41.cl200620200528-0414 [72] in it.

  • com.ibm.ws.ssl_1.3.41 is a jar file com.ibm.ws.ssl_1.3.41.jar
  • many files had cl200620200528-0414
  • [72] this was for bundle 72.

How does it work?

Each program package(jar file) contains a manifest.mf file. For example

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: MyService bundle
Bundle-SymbolicName: com.sample.myservice
Bundle-Version: 1.0.0
Bundle-Activator: com.sample.myservice.Activator
Import-Package:  org.apache.commons.logging;version="1.0.4"
Export-Package:  com.sample.myservice.api;version="1.0.0"

This gives information on

  • Information about your package (including its version number).
  • The entry point for the bundle.
  • What services it provide to other packages (com.sample.myservice.api).
  • What services it needs from other packages in order for it to work (com.sample.myservice.api).

OSGI sorts out all of these dependencies, provides access to the configuration in the *.xml files and additional configuration data hidden within the packages.

How can I use it to display configuration?

This is not very well documented. I think it is dangerous, as the osgi port allows unauthenticated access to start and stop services. I have not been able to restrict it to authorised users, or control what users have access to.

I found the following….

You need to specify in your Liberty *.xml files

<featureManager> 
<feature>osgiConsole-1.0</feature>
</featureManager>

and put

osgi.console=ip:port 

in the bootstrap.properties file. For example osgi.console=10.1.3.10:5471 . I found a reference to osgi.console.ssh=ip:port, but this port was not opened during liberty startup.

If you specify osgi.console=5471, this uses the localhost 127.0.0.1, so you can only access it from the local machine. For example TSO TELNET 127.0.0.1 5471. Telnet from TSO is not very usable – it is better to use a telnet session from off z/OS, for example Linux.

From Linux I used:

telnet 10.1.3.10 5471 |tee -i osgi.log

This allowed me to issue interactive commands, and have the output written to the osgi.log file.

There are a variety of commands to display, start stop etc.

The list bundle command lb gave over 200 entries like

177|Active | 12|com.ibm.ws.messaging.jmsspec.common... 
178|Active | 12|com.ibm.websphere.javaee.jms.2.0 ... 
179|Active | 12|WMQ Resource Adapter ... 

The command bundle 179 gave information about the WMQ Resource Adapter above.

I used the bundles command to list all information about all bundles. I then used some python script to process the log file and write information about each bundle to its own file. From the jca file I could see com.ibm.ws.jca.service.ConnectionFactoryService has properties

jndiName=jms/cf1
properties.0.CCSID=819
properties.0.cleanupLevel=SAFE
properties.0.cloneSupport=DISABLED
properties.0.config.referenceType=com.ibm.ws.jca.properties.wmqJms.jmsConnectionFactory
properties.0.connectionfactory-interface=javax.jms.ConnectionFactory
properties.0.failIfQuiesce=true
properties.0.headerCompression=NONE
properties.0.managedconnectionfactory-class=com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl
properties.0.messageCompression=NONE
properties.0.messageSelection=CLIENT
properties.0.port=1414
properties.0.providerVersion=unspecified
properties.0.pubAckInterval=25
properties.0.queueManager=CSQ9
properties.0.rescanInterval=5000
properties.0.resourceAdapterConfig.id=wmqJms
properties.0.shareConvAllowed=true
properties.0.sparseSubscriptions=false
properties.0.sslResetCount=0
properties.0.statusRefreshInterval=60000
properties.0.subscriptionStore=BROKER
properties.0.targetClientMatching=true
properties.0.transportType=BINDINGS
properties.0.wildcardFormat=TOPIC

and I could see the key properties for the queue manager connection.

For my MQ queue I could see

jndiName=jms/stockRequestQueue
properties.0.CCSID=1208
properties.0.adminobject-class=com.ibm.mq.connector.outbound.MQQueueProxy
properties.0.adminobject-interface=javax.jms.Queue
properties.0.arbitraryProperties=
properties.0.baseQueueManagerName=
properties.0.baseQueueName=STOCK_REQUEST
properties.0.config.referenceType=com.ibm.ws.jca.properties.wmqJms.jmsQueue
properties.0.encoding=NATIVE
properties.0.expiry=APP
properties.0.failIfQuiesce=true
properties.0.persistence=APP
properties.0.priority=APP
properties.0.putAsyncAllowed=DESTINATION
properties.0.readAheadAllowed=DESTINATION
properties.0.readAheadClosePolicy=ALL
properties.0.receiveConversion=CLIENT_MSG
properties.0.resourceAdapterConfig.id=wmqJms
properties.0.targetClient=MQ

and finally I found the connection pool information I was looking for:

connectionTimeout=30
id=ConMgr1
maxIdleTime=1800
maxPoolSize=5
purgePolicy=EntirePool
reapTime=180

Phew – what a lot of work to get this information.