Tag: rest

Error connecting to JMX endpoint: Unsupported protocol: rest

I got this error message when I was trying to use JMX into the WLP web server, when using the restConnector-2.0 interface in Liberty.

The documentation was not that helpful.  Oneproblem was I suffered from the curse of cut and paste, and used a Windows environment variable %JAVA_HOME%  instead of using the Linux $JAVA_HOME.  Another problem was caused documentation saying add the jar to the class path, and then the class path was ignored.

True documentation.

When you use the  -jar option, the JAR file is the source of all user classes, and other user class path settings are ignored.  See here.

Unhelpful documentation

Configuring secure JMX connection to Liberty   says

jconsole -J-Djava.class.path=...;%WLP_HOME%/clients/restConnector.jar

This was for Windows – on Linux it would be $WLP_HOME – except on my system $WLP_HOME was not set.

jconsole

Some of the jar files are in $JAVA_HOME, you can use the environment variable, or specify it yourself.  Note %JAVA_HOME% is windows, so be careful when you use cut and paste.

  • jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:/opt/mqm/web/clients/restConnector.jar
  • jconsole -J-Djava.class.path=/usr/lib/jvm/java-8-openjdk-amd64/lib/jconsole.jar:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/opt/mqm/web/clients/restConnector.jar

If the window displays “Secure connection failed” restart jconsole and use the -debug option.   For me this gave  “java.io.IOException jmx.remote.credentials not provided. Set to a String[2] {user,password}”.  I entered my userid and password, and this connected successfully.

I could not connect using my .jar file.

I was using JMXQuery to extract the data.  I was getting the Error connecting to JMX endpoint: Unsupported protocol: rest message.
Adding the jar file to my class path did not help, as the class path is ignored when using java the -jar parameter.

How to fix it

There are two ways of fixing this.

  1. Put the required jar file in the extensions path, not the class path
  2. Use the java -classpath…  instead of specifying java -jar

1. You need to have the jar for in the extensions path, not in the class path.

See How classes are found  in the Java documentation.  It says

  • Bootstrap classes – Classes that comprise the Java platform, including the classes in rt.jar and several other important jar files.
  • Extension classes – Classes that use the Java Extension mechanism. These are bundled as .jar files located in the extensions directory.
  • User classes – Classes defined by developers and third parties that do not take advantage of the extension mechanism. You identify the location of these classes using the -classpath option on the command line (the preferred method) or by using the CLASSPATH environment variable.

There is a system property java.ext.dirs  which gives the location of the Extension classes. On my system this was

/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/java/packages/lib/ext

  • The /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext: is java dependent. You should not put your files in this directory
  • /usr/java/packages/lib/ext This is for “user” extensions.

The directory /usr/java/packages/lib/ext did not exist on my system, so I had to do, create it, copy the web connection jar file to it, and grant permissions on it.

  • sudo mkdir -p /usr/java/packages/lib/ext
  • sudo cp /opt/mqm/web/clients/restConnector.jar /usr/java/packages/lib/ext/
  • sudo chmod 444 /usr/java/packages/lib/ext/restConnector.jar

2. Use a -classpath – not a -jar

In a .jar file there is a META-INF/MANIFEST.MF file which includes information on the entry point.

Manifest-Version: 1.0
Main-Class: src.Client

Instead of using the -jar option to point to the jar, you can use the -classpath to point to the jar and explicitly specify the entry point.  For example

java -cp ./Client.jar:/opt/mqm/web/clients/restConnector.jar  
src/Client
service:jmx:rest://localhost:9443/IBMJMXConnectorREST
  • java -cp ./Client.jar:/opt/mqm/web/clients/restConnector.jar –  use the classpath option, and specify the needed jar files.  Client.jar is my program.  /opt/mqm/web/clients/restConnector.jar  is the Liberty provided jar.
  • src/Client – the “entry point” class to use
  • service:jmx:rest://localhost:9443/IBMJMXConnectorREST – the url to use.
  • note the absence of a -jar option.

In the manifest of the Client.jar file it had Main-Class: src.Client.   This is for the the src/Client.java source file.   This ties up with the src/Client.class as seen when you use the command jar -tvf Client.jar .

Python and mq REST api

I found cURL a good way of using the mq REST API, but I wanted to do more.  cURL depends on a package called libcurl, which can be used by other languages.

Python seemed the next obvious place to look.

As I have found out, using digital certificates for authentication is hard to set up, and using signed certificates is even harder.  As I had done the hard work of setting up the certificates before I tried curl and Python, the curl and Python experience was pretty easy.

I looked at using the Python “request” package.   This allows you to specify most of the parameters that libcurl needs, except it does not allow you to specify the password for the user’s keystore.

I then looked at the Python package pycurl package.    This is a slightly lower level API, but got it working in an hour or so.
My whole program is below.

During the testing I got various errors, such as “77”.  These are documented here. 

The messages were clear, for example

CURLE_SSL_CACERT_BADFILE (77) Problem with reading the SSL CA cert (path? access rights?).

Which was enough to tell me where to look.

All the things you can do with curl, you can do with pycurl.

 

# program - based on code in http://pycurl.io/docs/latest/quickstart.html
import sys
import pycurl

from io import BytesIO

# header_function take from http://pycurl.io/docs/latest/quickstart.html
headers = {}
def header_function(header_line):
# HTTP standard specifies that headers are encoded in iso-8859-1.

header_line = header_line.decode('iso-8859-1')

# Header lines include the first status line (HTTP/1.x ...).
# We are going to ignore all lines that don't have a colon in them.
# This will botch headers that are split on multiple lines...
if ':' not in header_line:
  return

# Break the header line into header name and value.
name, value = header_line.split(':', 1)
print("header",name,value)

home = "/home/colinpaice/ssl/ssl2/"
ca=home+"cacert.pem"
cert=home+"testuser.pem"
key=home+"testuser.key.pem"
cookie=home+"cookie.jar.txt"
url="https://127.0.0.1:9443/ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=type"

buffer = BytesIO()
c = pycurl.Curl()
print("C=",c)
try:
  # see option names here https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
  # PycURL option names are derived from libcurl
  # option names by removing the CURLOPT_ prefix. 
  c.setopt(c.URL, url) 
  c.setopt(c.WRITEDATA, buffer) 
  c.setopt(pycurl.CAINFO, ca) 
  c.setopt(pycurl.CAPATH, "") 
  c.setopt(pycurl.SSLKEY, key) 
  c.setopt(pycurl.SSLCERT, cert) 
  c.setopt(pycurl.SSL_ENABLE_ALPN,1)
  c.setopt(pycurl.HTTP_VERSION,pycurl.CURL_HTTP_VERSION_2_0)
  c.setopt(pycurl.COOKIE,cookie) 
  c.setopt(pycurl.COOKIEJAR,cookie) 
  c.setopt(pycurl.SSLKEYPASSWD , "password") 
  c.setopt(c.HEADERFUNCTION, header_function)  
# c.setopt(c.VERBOSE, True)
  c.perform() 
  c.close()
except Exception as e: 
  print("exception :",e ) 
finally: 
  print("done") 
body = buffer.getvalue() # Body is a byte string. 
# We have to know the encoding in order to print it to a text file 
# such as standard output. 
print(body.decode('iso-8859-1'))

 

Getting MQConsole (brower interface to administer MQ via mqweb) working

It was a new year, as I sat in my basement cave while the gale force winds blow around the house, I thought I would try to use the new MQWeb and MQConsole, and see how it stands up to “the Paice treatement”.    The MQWeb allows you to administer MQ from a web browser, or a rest interface (for example using CURL or Python).  This technology has been around for a few years now.  I know it is being enhanced every few months through the continuous delivery channel.

The installation and getting started reminded me of an old car belonging to my father.  The car was not easy to get started (it had a starting handle!), but once it was started it worked pretty well.

Getting it up and running in a test sandbox took about 1 hour.   It took me about two week to get mqweb set up properly using digital certificates, and to document how I did it.  Being security related, there must be a team which tries to make it as hard as possible to diagnose problems so as not to provide useful information to a hacker.  It also took a while to  work out how to use mqweb  in an enterprise where you have multiple machines and have to support many users.  It also feels a bit buggy and some of it was frustrating, but as it is being continuously improved, I am sure it will get better.

Ive written some blog posts

I had MQ 9.1.3 running on my laptop running Ubuntu 18.04.

Getting it installed and up and running.

Initially I followed the  9.1 instructions here.   After lots of clicking and guessing I got to this page which gave me some instructions (but they were not very helpful). There are various mistakes on the page such as var/mqm/web should be /var/mqm/web.  I ignored the instructions and simply used sudo apt install /home/colinpaice/…/ibmmq-web_9.1.3.0_amd64.deb to install it.

The configuration file /opt/mqm/web/mq/samp/configuration/basic_registry.xml has predefined userids and the configuration is suitable to have an initial look at the MQWEB.

I used

cp /opt/mqm/web/mq/samp/configuration/basic_registry.xml 
/var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml

to copy the configuration file.

Starting and stopping the mqweb

The strmqweb command failed for me.   This was strange  as commands like strmqm works.  This is because there is a symbolic link /usr/bin/strmqm which points to /opt/mqm/bin/strmqm, but no link for the mqweb commands.

See here  which explains there is a /usr/bin/strmqm → /opt/mqm/bin/strmqm , but not for the mqweb stuff. I think this is an IBM Whoops.

I created these myself using

sudo ln -s /opt/mqm/bin/dspmqweb /usr/bin/dspmqweb
sudo ln -s /opt/mqm/bin/endmqweb /usr/bin/endmqweb
sudo ln -s /opt/mqm/bin/setmqweb /usr/bin/setmqweb
sudo ln -s /opt/mqm/bin/strmqweb /usr/bin/strmqweb

The configuration file is deep down a directory tree.

I created  a symbolic link to the file using

ln -s /var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml web.xml

so I can do  gedit ~/web.xml

and if you forget where the file really is, use ls -l web.xml

I used the strmqweb command to start the mqweb server.

I used dspmqweb and got

MQWB1124I: Server ‘mqweb’ is running.

MQWB1123E: The status of the mqweb server applications cannot be determined.  A request was made to read the status of the deployed mqweb server applications, however the data appears corrupt. This may indicate that there is already an mqweb server started on this system, probably related to another IBM MQ instance.

The MQWB1123E message only happened occasionally – I think it is a timing problem and can be ignored.

I stopped the mqweb instance using endmqweb

Log files

There is a file /var/mqm/web/installations/Installation1/servers/mqweb/logs/console.log  which has audit type statement in it.

There is a file /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log which has more messages (including time stamps).   This file is more useful.

I defined a symbolic link to this file, to make debugging easier.

ln -s/var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log messages.log

When the strmqweb command is issued,

  • it deletes the previous console.log
  • it rename the messages.log to a file with a time stamp in the file name
  • it deletes any other message logs files.

After starting and stopping the web server several times the only files I had were

  • messages_20.01.05_15.18.50.0.log
  • messages.log
  • console.log

You may want to put the strmqweb command in a shell script which saves away any message and console files.

The command dspmqweb gives output like

MQWB1124I: Server 'mqweb' is running.
URLS:
  https://localhost:9443/ibmmq/rest/v1/
  https://localhost:9443/ibmmq/console/

This tells you which URL you need to use.

Note: port 9443 is the default port for WebSphere Liberty Profile.  If it is in use you will have to configure a different port.

First logon

I logged on to Firefox browser using the address https://localhost:9443/ibmmq/console/ .  Make sure you the https in https:… .  If you use http: without the https, the logon fails with message “The connection was reset”.

Using https:… gave me big error screen and

Warning: Potential Security Risk Ahead 
localhost:9443 uses an invalid security certificate.
The certificate is not trusted because it is self-signed.
Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT

While  you are exploring the mqconsole, you can accept this.  To fix it properly is a big piece of work.  See my other blog posts.

I signed on using userid mqadmin and password mqadmin and it showed the queue managers.

Select the row of an active queue manager. The table header changes to give options.  Select properties to display the queue manager properties.

The queue manager attributes do not refresh in real time

You have to go back to the queue manager table and re-display the data.   This is not a big problem as the attributes do not typically change frequently.  I noticed this when I changed an accounting parameter, and the attribute page did not show the change.

Adding widgets to the dashboard.

There are two ways of adding widgets for MQ objects.

  1. From the list of queue manages, select a queue manager, then on the title like, click on the “…”  (more actions) button and select “Add new dashboard tab”. This creates a dashboard with all of the MQ Objects defined, MQ Queues, client connections, MCA connections, listeners etc. You can select and delete widgets you do not need.
  2. Click on the “Add widget” button.

It may be quicker and easier to use the first option to add all widgets and delete the widgets you do not need.

Create more tabs

At the top of the browser window next to the “tab”,  click on the “+”. This defines a new dashboard, use the add widget button to select the widget you want to define.

Each userid has their own dashboard (tab layout and widget)

See the next topic if you want people to have the same dashboard.

Export the dashboard for enterprise deploy or backup

At the top of the screen is an icon with three vertical dots for dashboard settings. You can export the dashboard and widgets to a JSON file.

  • You can change the queue manager names and import it on another queue manager.  This is useful to enterprise users who have to support many queue managers in a similar environment.  Being a JSON file you can process the JSON to change queue manager names.  I could not find a way of importing it except from a web page.  This make it challenging to deploy automatically.
  • You can have another user import it, so they get the same dashboard.  If it changes, they have to import it manually.
  • You may want to export your dashboard every week and back it up.

Using the widgets

I clicked on the “Queue on …”  widget.

I clicked on the “Queue depth” column for queues, and it quickly sorted the queue depth.

I could see I had a total of 33 non system queues. By clicking on the settings wheel, I could select “show system objects”.

If you select the settings wheel, you can select a different queue manager.  By changing this you could have one tab showing queues on different queue managers on the machine, and another tab showing channels on different queue managers on the machine.  You could also have a tab per queue manager, and have queues and channels for one queue manager on that tab.

I could refresh a widget by using the refresh icon.

There is a search box at the top of each widget. It searches for the value in any column. So typing in 003 gave me queue CP00003 and DEEPQ with depth 1000003.
At the bottom of the widget it said Total: 90 Filtered:2

If you select a row, the search box changes and give you a list of actions.

  • Delete queue
  • Properties
  • Put message
  • Browse message
  • More actions → Manage authority records
  • 1 item selected
  • Deselect

You can select all the objects in a widget by typing “a” , or to deselect using “shift a”.  Note: it selects all items – not just the filtered items. For example I typed “a” and the header line said “33 objects selected”. At the bottom of the widget is said total 33 filtered 8.  So be careful if you were thinking of doing bulk changes on all objects.

I was unable to select more than one object, using the cursor keys.

It was easy to delete widgets by selecting the X icon.

You can move the widgets around by grabbing the title line and dragging it.

If you hover on the title line of a widget, a pencil icon appears which allows you to rename the widget.

You can control how many widgets are displayed per line by clicking on the down arrow in the tab (at the top of the page) and selecting how many columns to use.  This is a not very smart.

  • I selected 5 column layout.
  • It did not reflow the widgets automatically.  Each line had 2 widgets and lots of space to the right.  I could drag a widget to the top line.  If I then went to 2 column layout, and back to 5 column layout – I got back to two widgets per line
  • If you select an item, the search box becomes a list of icons.  With a narrow widget, you only get the as many icons as fit in the space, for example you do not get the  “…” (more action) icon.
  • The formatting within a table is not very smart. I had a truncated queue name SYSTEM.ADMIN.CH and lots of space for the queue depth. I think the data is displayed in a table and the columns are the same width, and not changeable.

It may be better to have no more than 2 or 3 widgets per line.

Using operating system security.

The basic mqweb configuration file used hard coded userids mqadmin with password mqadmin. This is not very secure.

You can use the operating system userids and passwords using a different configuration file

I used

  • cp /opt/mqm/web/mq/samp/configuration/local_os_registry.xml  /var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml
  • chmod o+w /var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml
    • to give me update access to the file.

I changed my file to have

<enterpriseApplication id="com.ibm.mq.console">
  <application-bnd>
    <security-role name="MQWebAdmin">
      <user name="colinpaice" realm="defaultRealm"/>
    </security-role>

    <security-role name="MQWebAdminRO">
      <group name="test"/>
    </security-role>
   </application-bnd>
</enterpriseApplication>

Notes.

  • The realm=”defaultRealm” is to do with Jave Enterprise Edition security. Just specify it.
  • Each security-role name section must be unique. I specified <security-role name=”MQWebAdminRO”>… twice. Only the last one was used, I was hoping it would be cumulative.
  • You can specify multiple <user …> or <group… > lines.

See here  and here  for pointers to the IBM documentation.

Managing mqwebuser.xml

You can include files into the mqwebuser.xml files using the xml

<include optional="true" location="pathname/filename"/>
or
<include optional="true" location="url"/>

You can put groups of definitions in one file and have them included.

For example in the file payroll.xml have

<group name="mqsysprog"/>
<group name="payroll"/>

For each of the configuration files for the payroll queue managers have

<security-role name="MQWebAdmin">
  <include optional="true" location="payroll.xml"/> </security-role> <security-role name="MQWebAdminRO"> <group name="test"/> </security-role>

How do I check what role I have?

At the top right of your browser window is a porthole with a circle in it. Click on this, and then click on “about”. It gave me

Principal:colinpaice - Administrator (Password Authentication)
A different userid gave
Principal:testuser - Read-Only Administrator (Password Authentication)

Can I have the logon time out?

Yes, you set a time out value using the ltpaExpiration value. See here.

Use dspmqweb properties -a|grep ltpaEx  and note the ltpaExpiration value.

Use  setmqweb properties -k ltpaExpiration -v time    to set the time in minutes.

Note:

  • After you are logged on for this time period, your session is cancelled and you have to logon again, this happens whether the session is busy or idle.
  • The setmqweb command updates the mqwebuser.xml file on disk. If you were editing the file you will need to reload the file from disk and reapply the changes.
  • The above setmqweb command added <variable name=”ltpaExpiration” value=”10″/> to the mqwebuser.xml file. You could just update the file yourself and avoid this concurrent  update problem.

There is one timeout value for all users, so if you have a screen displaying charts from mqweb, this will also time out.

If you are using certificates to provide authentication

  • your session will be dropped, and automatically reconnected.
  • you cannot logoff – you have to drop the browser tab
  • in the top right of your page the icon will be a black circle with a which “i” in it.  If you are not using certificates this will be a porthole with a circle in it.