How to accept a certificate if I do not have their CA?

Someone contacted me saying that they had a z/OS application which was trying to access a remote server using TLS, but the connection was failing because the incoming certificate could not be validated.

Or or to put it another way, a certificate and its CA arrived at my system; how do I get the CA into my keyring to allow validation.

One way of getting the certificate is to use a browser. I used chrome.

  • enter the URL and press enter. It should display a page with a padlock at the front of the URL
  • click on this padlock
  • it should say “connection is secure”
  • click on “certificate is valid”, it displays certificate information
  • click on details. It should list the certificate and any CA certificates
  • click on the CA of interest
  • click Export and save the file (as base 64 encoded ASCII)
  • this will create a file containing —–BEGIN CERTIFICATE—– … —–END CERTIFICATE—–
  • check that this CA certificate is valid, and does not belong to a bad guy
  • upload this to a sequential file on z/OS
  • “check” it
//COLRACFI JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT CHECKCERT('COLIN.IMPORT.CA.PEM') 
  • Add it to RACF database
RACDCERT CERTAUTH     ADD(''COLIN.IMPORT.CA.PEM') - 
  WITHLABEL('MYBANK.CA') TRUST 
  • connect it to the keyring, CICSID/CICSRING
RACDCERT   CONNECT(CERTAUTH    LABEL('MYBANK.CA') - 
  RING(CICSRING) USAGE(CERTAUTH)) ID(CICSID) 

How to debug a failing AT-TLS connection

AT-TLS provide TLS connection to applications, without changing the application. The IP calls are intercepted and the TLS work is done under the covers.

I was trying to debug a mail server from z/OS to Linux, and could not see any error messages.

In my Policy agent configuration I had

TTLSRule                         CSSMTPRule 
{ 
   RemotePortRange               25 
   Direction                     Outbound 
   TTLSGroupActionRef            CSSMTPGroup 
   TTLSEnvironmentActionRef      CSSMTPEnvironment 
} 
TTLSGroupAction                  CSSMTPGroup 
{ 
   TTLSEnabled                   On 
} 
TTLSEnvironmentAction            CSSMTPEnvironment 
{ 
   HandshakeRole Client 
   TTLSKeyRingParms 
   { 
      Keyring                   START1/MQRING
   } 
    } 
    TTLSEnvironmentAdvancedParms 
    { 
       ApplicationControlled      On 
    } 
 } 

I changed only the Keyring, and left the remainder unchanged. You can specify a TRACE statement, but it looks like it defaults to 2 (Errors are traced to syslogd. The messages are issued with syslogd priority code err).

When I started CSSMTP I got messages

EZD1819I CSSMTP UNABLE TO ESTABLISH A TLS CONNECTION TO TARGET SERVER 10.1.0.2                                                                
EZD1820E CSSMTP NO TARGET SERVER IS CAPABLE OF RECEIVING MAIL AT THIS TIME 

Which were not very helpful.
I had the syslogd daemon running, and it put its output in /var/log. In several TCPIP.* files I had an error message

EZD1286I TTLS Error GRPID: 00000007 ENVID: 00000002 CONNID: 00000036
LOCAL: 10.1.1.2..1032 REMOTE: 10.1.0.2..25 JOBNAME: CSSMTP USERID:
START1 RULE: CSSMTPRule RC: 417 Initial Handshake 0000000000000000
0000005011421D10 0000000000000000

Where the RC is described (0 to 4999) here and 5000 and over described here

If the syslogd daemon is not running, the messages come out on syslog.

Updating a web page from a server – sending a complete HTML document.

When using a web server as a back-end to an HTML page, for example filling in a form, the result from the back-end can be

  1. “Redirect to a new page”.
  2. A stream of HTML. This is an HTML document where the “boilerplate” or constant data, is intermixed with the variable data included inline. The HTML is displayed, and replaces the previous page.
  3. A stream of data containing data such as changed fields. The requesting page can use this stream to update its page.

Sending a complete HTML document

Using a python back-end program I can write an entire (and complete) HTML document.

#!/usr/bin/python3
import cgitb
import cgi

cgitb.enable()

print("Content-Type: application/json")
print("ColinHeader: Value")
print("ColinHeader2: Value")
# indicate end of headers
print()
# now the html page
print("<!doctype html>")
print('"<html lang="en">')
print("<head>"
print("<body>")
print('<form id="target"  action="cgi-bin/first.py" enctype="multipart/form-data"  >')
print('<label for="email">Enter your email: </label>')
print('<input id=email name="email" value="test@example.com" title="colins email address">')
print('<label for="password">Enter your pw: </label>')
print('<input id=password name="password" value="pw">')
print('<input type="submit">')print("</form>") print("</body>") print("</html>")

Where

  • print(‘<label for=”email”>Enter your email: </label>’)
    is constant text.
  • print(‘<input id=email name=”email” value=”test@example.com” title=”colins email address”>’)
    is the variable data putting text@example.com into the field.
  • print(‘<label for=”password”>Enter your pw: </label>’)
    is constant text.
  • print(‘<input id=password name=”password” value=”pw”>’)
    is variable text, putting pw into the field.

This is a terrible way of doing it. Presentation (red text) is mixed up with data (green text). This was a no-no about 40 years ago! You might have several similar pages, and any updates would have to be made to all pages. Using pages in national different languages is hard to implement; you need to have a multiple program, one for each language.

Updating a web page from a server – sending back just the updates

When using a web server as a back-end to an HTML page, for example filling in a form, the result from the back-end can be

  1. “Redirect to a new page”.
  2. A stream of HTML. This is an HTML document where the “boilerplate” or constant data, is intermixed with the variable data included inline. The HTML is displayed, and replaces the previous page.
  3. A stream of data containing data such as changed fields. The requesting page can use this stream to update its page.

multiple program, one for each language.

Sending back just updates.

I could have my back-end server program send back just changed elements.

#!/usr/bin/python3
import cgitb
import cgi

cgitb.enable()

print("Content-Type: application/json")
print("ColinHeader: Value")
print("ColinHeader2: Value")
# indicate end of headers
print()
#now the data
print('<p id=email>New-mail</p>')
print('<p id=pw>*********</p>')

Note: This does not send an HTML document, it just sends changed fields.

Where

  • print(‘<p id=email>New-mail</p>‘) id=email is the field in the requester page to be updated
  • print(‘<p id=pw>*********</p>‘) id=pw is the field in the requestor page to be updated.

The front page needs to be smarter, and be able to process these fields. See fetch in HTML sending stuff to the back-end server

In the example below it updates the content of the field with id=”passed”, so that the information returned is only displayed. It does not update the fields. This is described below.

<p id="errorField">No Errors yet</p>

<form id="target"  action="cgi-bin/first.py" enctype="multipart/form-data"  >
  <label for="email">Enter your email: </label>
  <input id=email    name="email" value="test@example.com" title="colins email address">
  <label for="password">Enter your pw: </label>

  <input id=password name="password" value="pw">
  <input type="submit">
  <input type="text" onblur = "check(this)" >
</form>
<p id="passed">old info</p>

<script>
document.forms["target"].addEventListener('submit', (event) => {
  event.preventDefault();
  fetch("cgi-bin/first.py", {
      method: 'POST',
      body: new URLSearchParams(new FormData(event.target)) // event.target is the form
  }).then((response) => { 
       
      if (!response.ok) {
        throw new Error(`HTTP error! Status: ${response.status}`);
      }
        return (response.text());
    })
    .then(function(result) 
    {
      // put the entire response into an exist field.
      document.getElementById("passed").innerHTML = result;     

   }).catch((error) => {
      console.log(error);// TODO handle error
      return(error);
  });
});
</script>

where

.then(function(result) 
    {
    }

is passed the body of the html as a string. The value result is available within the function.

 document.getElementById("passed").innerHTML = result;

This updates <p id=”passed”>old info</p> and gives it the content in result ( the whole data stream).

Parsing a returned document

If you are are returning a whole HTML document <!doctype>…</html> you can use DOMparser to parse the document.

var doc2 = new DOMParser().parseFromString(result, "text/xml");
let x = doc2.getElementById("... 

I was using rexx and it was returning lines of data – not a whole HTML document, and so the DOMParser complained that there was an incomplete document.

Parsing data (not necessarily a document)

By adding the text to an element already in the front page you can use the page’s parser to process the data.

For example add the returned text to a field in the front page

document.getElementById("passed").innerHTML = result;

This will display all of the returned data in the field, and it will be visible ( unless the field is hidden). You can remove elements – see below.

You can now access this data using standard navigation.

My Python program had

print('<p id="z" style="color:red">incolinsz red</p>')
print('<p id="z">colins2</p>')
print('<p id="z">colins3</p>')

print('<p id="update" update="email">Updated@email</p>')
print('<p id="update" update="password">newpw</p>')

print('<p id="focus" which="email"/>')

I want to

  • Display the elements with id=”z”
  • Give a field a focus.
  • Update the fields in the document from the data, where id=”update”, and update=”…” is the name of the field on the page.

Display the elements with id=”z”

let e = document.getElementById("errorField");
e.innerHTML = "";

let  myzs = q.querySelectorAll("#z");
console.log("size of myzs" + myzs.length);
Array.from(myzs).forEach((element) => {
      console.log(element.innerHTML )
      e.innerHTML += element.outerHTML;
}); 

Where

  • let e = document.getElementById(“errorField”); Locate the field where we want to store the error elements
  • e.innerHTML = “”; Clear this
  • let myzs = q.querySelectorAll(“#z”); Get a list of all the elements with id=”z”
  • console.log(“size of myzs” + myzs.length); Debugging – display how many we have
  • Array.from(myzs).forEach((element) => For each one. We need an array to use forEach
  • {
    • console.log(element.innerHTML ) Display the content
    • e.innerHTML += element.outerHTML; Build up the field by concatenating the elements. Include the html, such as styling.
  • });

Note: If element.innerHTML is used, then only the value is used. If outerHTML is used, the formatting is also copied across, for example the text <p id=”z” style=”color:red”>incolinsz red</p> is displayed in red.

Set the focus

This is an example of passing data across the interface. The Javascript looks for the first element like print(‘<p id=”focus” which=”email”/>’) with an id of focus. It uses the “which” attribute to pass a field name.

The Javascript code is

let focus=q.querySelectorAll("#focus");
if (focus.length > 0)
{
  let f0 = focus[0].getAttribute("which");
  let d =  document.getElementById(f0);
  d.focus()
  d.setAttribute('style', 'color: red');
}
  • let focus=q.querySelectorAll(“#focus”); Get all the elements with id=”focus”
  • if (focus.length > 0) If we had at least one…
  • {
    • let f0 = focus[0].getAttribute(“which”); Use the first one found, and extract the value of the “which=” attribute.
    • let d = document.getElementById(f0); Look for the element with the same name in the main document
    • if (d !== null) if it was found
    • {
    • d.focus() Set the focus
    • d.setAttribute(‘style’, ‘color: red’); As it is hard to see if the focus was set, change the colour of the field as well.
    • }
  • }

Updates the fields in the document

The Javascript is

let  updates = q.querySelectorAll("#update");
Array.from(updates).forEach((element) => {
   console.log(element.innerHTML )
   let f0 = element.getAttribute("update");
   let e = document.getElementById(f0);
   if (e !== null)
   {
     e.value = element.innerHTML;     
   }
   else 
   {
     console.log("element not found:" + f0)
    }
}); 

Where

  • let updates = q.querySelectorAll(“#update”); Select all elements which have id=”update”
  • Array.from(updates).forEach((element) => { We need an array to be able to use forEach
    • console.log(element.innerHTML ) Display it
    • let f0 = element.getAttribute(“update”); Get the value of the “update=…” attribute for the individual element
    • let e = document.getElementById(f0); Locate the element in the main document
    • if (e !== null) If it was found
      • {
        • e.value = element.innerHTML; Set the value to what was passed in
      • }
      • else Log we have a problem – and ignore it
      • {
      • console.log(“element not found:” + f0)
      • }
  • });

Removing elements

If you want to display a subset of the elements, you may want to delete some elements.

Array.from(myzs).forEach((element) => {
  x = element
  x.parentNode.removeChild(x);    
}); 

For the elements above with id=”z” the code does

  • Array.from(myzs).forEach((element) => { Create an array so you can use forEach
    • x = element save it
    • x.parentNode.removeChild(x); Delete it
  • });

Create a new field

If you want to create a new field and insert the data you can do

// append data to an existing field.
var d = document.createElement('div');
d.setAttribute("id", "Special");
var myPara = document.getElementById("passed");
myPara.appendChild(d);
d.innerHTML = result;

This does

  • // append data to an existing field.
  • var d = document.createElement(‘div’); create a div section
  • d.setAttribute(“id”, “Special”); and give it a name
  • var myPara = document.getElementById(“passed”); locate the element in the document with the id “passed“.
  • myPara.appendChild(d); Attach our new field to the item.
  • d.innerHTML = result; Sets the content to what was passed back.

You will see the data appear on the web page.

Note: if you execute the page more than one, you will get more and more data

Updating a web page from a server – redirect to a new page.

When using a web server as a back-end to an HTML page, for example filling in a form, the result from the back-end can be

  1. “Redirect to a new page”.
  2. A stream of HTML. This is an HTML document where the “boilerplate” or constant data, is intermixed with the variable data included inline. The HTML is displayed, and replaces the previous page.
  3. A stream of data containing data such as changed fields. The requesting page can use this stream to update its page.

Redirect to a new page

The server can send back status meaning redirect

#!/usr/bin/python3
import cgitb
import cgi
import sys, os, io

cgitb.enable(display=0, logdir="/home/colinpaice/first.log")
print('Status: 303 See Other2')
print("Content-Type: text/html")
// the url and any parmeters
print("Location: ../c3.html?colin=p1&error=p2")
# needs blank line to say end of headers
print()

This causes page c3.html to be displayed. The page can process the URL and process the options passed on the url.

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<HTML lang="en">
<HEAD>
<script>
//  this function is  invoked on page load 
function js_onload_code (e){
  var url = document.location.href;
  // split the parameters from the url  base
  var p = url.split('?')[1]
  if (p)
  {
    var params = p.split('&');
    if (params)
    {
      var l = params.length;
      // display what was passed in
      alert("colin:"+url); 
      for (var i = 0 ; i < l; i++) 
      {
        // split it into kw"="value
        tmp = params[i].split('=');
        document.write(tmp[0] + "=" + tmp[1] +"!<br>");
        }
      } // if params
    } // if
}
// this causes the above script to be executed.
window.onload= js_onload_code ();
   
</script>
<BODY  >
<P>This is page c3.html
</BODY>
</HTML>

This displayed a page with url http://localhost/c3.html?colin=p1&error=p2

colin=p1!
error=p2!

This is page c3.html

If you repeatedly refresh this page you will get more and more data! It is better to use a field, then clear it before adding the information.

Collecting a wire-shark trace with TLS active

With TLS, the data transfer is encrypted. With TLS 1.3, part of the handshake is encrypted. In Wireshark this shows up as Encrypted Application Data.

You can configure wireshark to decrypt the data flow . You need to set an environment variable, and start the application from the terminal window, to pick up the environment variable.

export SSLKEYLOGFILE=$HOME/sslkeylog.log
google-chrome

This will cause the application( the google-chrome browser) to start and to write the TLS key data to the file.

Configure Wireshark to use this file:

  • Edit
  • Preferences
  • Expand the Protocols twistie
  • Scroll down to TLS ( typing T gets you near it)
  • Enter the Master-Secret log file name mine is /home/colinpaice/sslkeylog.log

Now, if you run the test you should get the data decrypted

Transport Layer Security
 TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
   Content Type: Change Cipher Spec (20)
   Change Cipher Spec Message
 TLSv1.3 Record Layer: Alert (Level: Fatal, Description: Decrypt Error)
  Opaque Type: Application Data (23)
    [Content Type: Alert (21)]
    Alert Message
      Level: Fatal (2)
      Description: Decrypt Error (51)

HTML sending stuff to the back-end server.

The simple and most common way of sending HTML data to a back-end server is using an anchor link

<a href="./echo.rexx">echo</a> 

This invokes the back-end program and displays the output. You have no control over the headers or what data is sent.

You can do more sophisticated stuff using the JavaScript fetch function, for example

  • specify different headers
  • change the data sent to the server, for example specify which data you want sent to the server, perhaps only send changed values rather than all input values
  • send the data to a different URL.

A quick aside on reading the JavaScript documentation and use of the arrow function

A lot of the documentation on JavaScript uses the arrow function, which can be useful, but I find distracting.

You can use a function such as mysubmit which takes an event:

function mysubmit(event)
{
console.log(event);
}
document.forms["target"].addEventListener('submit', mysubmit ) 

When the submit event occurs, “mysubmit” is invoked, and passed the event;

This can also be written, using inline functions as

document.forms["target"].addEventListener('submit', mysubmit(event) => { console.log(event;...} )
or
document.forms["target"].addEventListener('submit', (event) => { console.log(event;...} ) 

I have also seen it written as

function mysubmit(zevent)
{
console.log(zevent)
}
document.forms["target"].addEventListener('submit', (event) => mysubmit(event) ) 

At first glance this looks it could be simplified. However this is useful if you wanted to pass additional parameters to the function.

function mysubmit(zevent,a)
{
// a will have the value abc
console.log(zevent)
}
document.forms["target"].addEventListener('submit', (event) => mysubmit(event,'abc') ) 

I find the first example, document.forms[“target”].addEventListener(‘submit’, mysubmit ) easier to follow; and easier to write as you have to worry about matching the ending “})”s.

Using fetch

I used an HTML page with a form, so I could process the data. The fetch function sends the data to the specified URL, and waits for the response. You can specified headers and body and you get back a response object.

Setup

You need to associate a function with the “submit” of the form

 document.forms["target"].addEventListener('submit', mysubmit ) 

where mysubmit is the name of a callback function, for example

function submit(event) 
{ 
  event.preventDefault(); 
  displayChanged(event) 
  fetch("./echo.rexx", { 
      method: 'POST', 
      body: new URLSearchParams(new FormData(event.target)) // event.target is the form 
    })
    .then( check   )
    .then( zesult  ) 
    .catch(mycatch ) 
} 

Where

  • function submit(event) { defines the function, the parameter variable is called event.
  • event.preventDefault(); disables normal behaviour and lets me manage it.
  • displayChanged(event) my processing of the data before sending the data, see below.
  • fetch(“./echo.rexx”, { This sends the request to the backed “./echo.rexx”.
    • method: ‘POST’,
    • body: new URLSearchParams(new FormData(event.target)) Fetch has two parameters, the URL and a set of options, such as body, a set of headers{} etc. In this instance the second parameter is {method:”POST”, body: …. }.
  • }) // end of fetch
  • .then(check) this waits until the fetch finishes, see below
  • .then(zesult( ) see below
  • .catch( mycatch ) any exception
  • } // end of function

The .then processing is about using promises. Basically the .then has a wait for the response and return something to the next .then statement.

Check

This waits for the completion of the request.

function check(response) 
{ 
  if (!response.ok) { 
       throw new Error(`HTTP error! Status: ${response.status}`); 
  } 
  return response.text() 
 } 
  • It checks the “ok” flag in the data, if it is not OK, then throw an exception. The variable response.status has an error message.
  • Extract the body of the response and pass it on.

zesult

This waits for the completion of the previous “return response.text()”

function zesult(text) 
{ 
//console.log(text ); 
  let e = document.getElementById("errorField"); 
  e.innerHTML = text;                                                                     
 return (text) 
} 

and replaces the value of the piece of HTML with id=”errorfield”. For example

<p id="errorField" style="color:red ">No Errors yet</p> 

I do not know why two “.then”s are needed. It looks like it started another asynchronous process to extract the body from the response. In “check” if you display “response” it gives you some data. If you display “response.body” is gives a in-flight promise.

If you are tracing this, and you do not get into your code, check you have specified event.preventDefault(); before you do the fetch.

Looking at what data is being sent

Formdata allows you to work with the keyword=value data as generated by a form. You cannot get the data for other HTML elements outside of a form.

In my submit function I have displayChanged(event)

function displayChanged(event) 
{
// display the parameters
...
// Display what has changed
... 
}

Display the parameters

function displayChanged(event) 
{ 
  // display all the data 
  // extract into an iterable object  
 let f = new FormData(event.target)  ; 
 let data = ""
 for ([key, value] of f) { 
    data += ", " + key + ":" + value 
 } 
 console.log(data) 
...
}

For the data in my form, this displayed

,i1:initial value, text1:test@example.com, email:test@example.com, password:pw18,password25:pw25

where the bold names are the input field names.

Display what has changed

There is no direct way of processing just the changed variables, but you can fake it.

I had an input fields defined with an onchange call back.

<input id="password" name="password" value="pw18" onchange="change(this)" >
<script> 
function change(o){ 
  if  (o!= undefined) 
     o.setAttribute("changed","yes"); 
} 
</script>

If the field is changed, then the onchange -> change function is called. This sets an attribute “changed” to “yes”.

In the submit processing there

function displayChanged(event) 
{ 
...
  // display what has changed
  let er = event.target; 
  Array.from(er).forEach((r)  => 
  { 
    let id = r.getAttribute("id") 
    let ch = r.getAttribute("changed") 
    console.log( "id " + id + " changed " + ch ) 
  }) 
...
}

When I changed the two password fields, the console log had

id text1 changed null
id email1 changed null
id password21 changed yes
id password2 changed yes

Changing the parameters passed to the server.

You pass the parameters to the server using

fetch("./echo.rexx", { 
    method: 'POST', 
    body: new URLSearchParams(new FormData(event.target)) 

You can create a new instance of FromData, add keyword/value parameters to it, and pass it through to the back-end.

Above I explained how you can identify the “changed” data. You could take these elements and add to a new FormData, and pass those through to the server.

JavaScript: selecting an element

When writing some JavaScript to update elements in a page, I got confused selecting which elements to process. Below is my little crib sheet to help me remember.

Naming of parts.

You can have some HTML like

<div id="colin" name="request" class="myclass">requestor<div>

This division element can be referred to by the id=”colin”. It has an attribute name with a value of request. It has formatting parameters defined in the style sheet class=”myclass”.

If you have an element myDiv, you can use

let myclass=myDiv.getAttribute("class");
myDiv.setAttribute("class", "newclass");
myDiv.setAttribute("id", "newid");
myDiv.setAttribute("name", "newname");
myDiv.setAttribute("newattribute", "new");
myDiv.removeAttribute("name");

The id,class and any attribute look the same, but are slightly different when used to select elements.

Structure

You can have a low level element like a paragraph <p>..</p>

You can have more complex objects like a table <table>…</table> which can have table rows <tr>…</tr>, which can have table cells <td>…</td> which can have other elements such as <p>…</p>.

A division <div..>/div> can be used around text or an element to give it properties, such as a formatting or giving a word an id.

The high level structure is a document. You can create and add element to another element. You can remove elements. If you remove a table element, all of its children, <tr> and<td> are removed as well.

You can have elements which are not part of the document. You can add them to the document.

Locating an element in JavaScript

Originally there was just getElementById:

document.getElementById('newid');

This searches within document and its children for the first element with id=”newid”. If you have more than one element with id=”newid” only the first will be returned.

With a document

<!doctype html>
<html lang="en">
<head>
</head>
<body>
<p id="passed" class="myclass"></p>
<p id="p1" name="Colin">Colin</p>
<p id="p1" name="Jo">Jo</> 
</body>
</html>
var q= document.getElementById('passed');

returns an object called p#passed. This means it is a <p>aragraph object with id=”passed”.

It was quickly found that getElementById was not flexible enough, and so

querySelectorAll and querySelector were created. These are much more flexible.

  • let p=querySelector(“p”) returns the first element of type <p>.
  • let pall=querySelectorAll(“p”) returns a NodeList of all the <p> elements. Each element in the node list represents a <p> element. In my example HTML pall has a length of 3, because there are 3 <p> elements.
  • let p=querySelector(“#p1”) returns the first node with id=”p1″. It returns an element with “p#p1” saying this is a <p> element.
  • let pnl=querySelectorAll(“#p1”) with the above HTML returns a NodeList of length 2 where:
    • pnl[0] is the first node with id=”p1″. It returns an element with “p#p1”
    • pnl[1] is the second node with id=”p1″. It returns an element with “p#p1”.
  • let pnl=querySelectorAll(“p.myclass”) returns all “p” elements with class=”myclass”.
  • let pnl=querySelectorAll(“.myclass”) returns all elements with class=”myclass” regardless of type.
  • let nl=querySelectorAll(“[name=’colin’]”) returns all elements with attribute name=”colin”. With the above example data it returns a nodeList with one element which is a p#p1.

These selectors can be combined so for example you could ask for all <p> nodes within a <table> with id=”mytab” which have name=”colin”.

Finding data in a subtree

Within my HTML page I have

<table id="ztable" frame="box" rules="rows"> 
<tr id="template"> 
   <td id="col0" name="serial"><div><p>data</p></div></td> 
   <td id="col1" name="requestor"><div>requestor<div></td>    
</tr> 
<tr id="tr1"></tr>
</table> 

<table id="table2"> 
<tr id="tr1"></tr>
</table>

I can find the first table, and extract the rows out of it.

let tab = document.getElementById('ztable');
let tr = tab.querySelectorAll("tr");

This locates the table with id=”ztable” then searches within this for all <tr> tags. It returns a node list of length 2. It does not return the row in the second table(table2).

Not found?

Am I blind or is there no good documentation on how to write a web server application?

There is a lot of documentation on the internet on how to create a web page, for example using Java script or CSS to do clever things on a page. There is a lot of documentation on how to write a back-end application, with the choice of more than 10 different languages (Python, PHP, Java etc). But I could not find any good documentation on “start here”, how to get the back-end server to update a field on the web page, or what authorisation choices there are.

This blog post tries to cover some of the holes. It is what I have learned – it may not be the best; it may even be wrong, but I hope it helps.

If there is a good source of information, please tell me, and I’ll reference it.

What does a back-end do?

The back-end responds to a request such as

c3.html?colin=p1&error=p2&name=Colin%20Paice”

Where

  • c3.html is the name of the URL invoking the server. This could be page c3.html, or “myPython.py” an application script written in python, or “server”, a label which maps to an html page or a application name.
  • ? this divides the URL name from any parameters .
  • colin=p1. An HTML field with id called colin, has a value of “p1”
  • & a separator between arguments.
  • name=Colin%20Paice, an HTML field with id “name” has a value “Colin Paice”. Special characters like blank and & are replaced by their ASCII hex equivalent, so %20 is a blank.

Some requests pass data via STDIN, and the back-end application has to read from this stream to get the data.

The back-end can reply with a

  • A redirect to a different page (with parameters).
  • An HTML document where “constant” data such as “Please enter your Surname” is intermixed with “variable” data such as “Jones”. To update one field on the page, you have to send the whole document.
  • Other data, for example JSON, or HTML like <p id=”error” field=”firstName”> Invalid value </p> and the front end page can take this data and process it – for example to display the data, or highlight a value.

The request can come from an HTML page or as a REST request, where the application creates the string like c3.html?colin=p1&error=p2&name=Colin%20Paice” and uses an application such as CURL to send the request to the server, then parses the response.

As a developer for the back-end, I do not want to send the whole HTML document back – as I may not have it! I just want to send back a response to the request, for example field-1 has an invalid value, field-2 has an invalid length, the value of field-3 is not in the data base.

If you have multiple servers then a request may not go back to the same server because of load balancing.

Using TLS, it may do a new handshake for every request. TLS can cache session information for a short period to avoid the full handshake.

With some back-ends a request coming in starts a new thread, so there is a noticeable delay (half a second) when sending a request to a back-end. It is faster to use a static HTML home page than to call the back-end server to say “display the home page”.

State is not usually stored the back end, so you may want to return state information to the web page, and have it as a hidden, read only input field, which is sent to the back-end on the next request; or use cookies.

You need to decide what headers you need. You may want to use Cross-origin resource sharing (CORS) for protection.

You need to determine if the back-end can support the services. For example some z/OS services can easily be done using Rexx, but not using Python or PHP.

Basic design questions

What authorisation is needed?

  • Do you want to ensure the users use TLS to establish the session.
  • Do you want the server to send a certificate to the client, so the client authenticates the server.
  • Do you want the client to send a certificate to the server to authenticate the client, or make it optional?
  • Does the end user need to enter a userid and password. Do you have access to the enterprise wide identity system to be able to validate the userid or password, or will you just use the operating system’s checking.
  • You can have the back-end lookup the certificate and get back a userid.
  • Do you want the back-end to execute with the user’s authorisation, or the same authentication for all users.
  • Which ids are authorised to use what facilities, for example scripts, web pages, authorised operating system facilities, database tables etc..

What checking do you need to perform?

You may have checking on a web page, for example check a value is a number, or a string of a certain length. You also need to do the same checks in your back- end, as people may use a REST API to access your back-end, and no checking will have been done. With in-page checking, you get a response much faster than if the request goes to the back-end to validate

How do you transfer information from the front end to the back-end?

You can use cookies to store information in the clients front end. You need to consider Secure cookies and HttpOnly cookies, and should consider Same-site cookies.

You can have hidden input fields and read-only fields on a web page. These get passed to the back-end. A page called “developer.html” might have

<input type=”hidden” name=”action” value=”colinhidden”>
<input type=”text” name=”org” id=”name” value=”myorg.com” readonly=”readonly”>

<input type=”text” name=”role” id=”name” value=”Developer” readonly=”readonly”>

to pass action=”colinhidden” without the end user seeing it, and passing org=”myorg.com” with the user being able to see it, but not change it, and to pass “role= developer”. If you are using a REST API, you can create whatever values you like, as long as they are syntactically correct.

How do you transfer information from the back end to the front-end?

  • Using redirect does not work very well – it may be doable – but not very elegant.
  • Send a whole dynamically built HTML page – does not score well on the isolation of data and boilerplate.
  • Sending data back for the front end page to process. This seems to be the best solution.

The front end page can use javaScript to process the returned data. If the back-end has sent back:

<div id=”error” field=”name”> Invalid name specified>/div>
<div id=”error” field=”date”> Date is in the past</div>
<div id=”focus” which=”Surname”>

The JavaScript can do

  • Select all elements with a particular id, such as id=”error” and display them.
  • Take action over specific fields, for example for id=”error” then change the matching fields value to be red.
  • Take an action such as make the Surname field the focus.

You need to agree the protocol any records, and attributes, and what to do if there is a problem.

Why doesn’t ctrl-s work in ISPF edit? – ah it does now.

I had been editing a file, saving it, and finding the changes were not being picked up. Looking back, it was obvious; I was using CTRL-S the familiar Linux command, instead of F3 on ISPF.

I fixed this by configuring X3270 (on Linux).

My file /home/colin/.x3270pro now has

...
x3270.keymap: mine
! Definition of the 'mine' keymap
x3270.keymap.mine: #override \
    Alt<Key>4:          String("\\x00a2")\n\
    Ctrl<Key>backslash: String("\\x00a2")\n\
    <Key>Escape:    Clear()\n\
    <Key>End:        FieldEnd()\n\
    Ctrl<Key>Delete:   EraseEOF()\n\
    Ctrl<Key>Right:    NextWord()\n\
    Ctrl<Key>Left:    PreviousWord()\n\
    Ctrl<Key>Up:    Home()\n\
    <Key>Control_L: Reset()\n\
    <Key>Control_R: Reset()\n\
    <Key>Prior: PF(7)\n\
    <Key>Next: PF(8)\n\
    <Btn3Down>:   PA(1)\n\
    Ctrl<Key>1:   PA(1)\n\
    Ctrl<Key>s:   MoveCursor(3,15) String("save") Enter()\n\

When I started a new X3270 session, Ctrl -S went to the command line, typed save and pressed enter. Job Done ! The numbers are 0 based, so 3 means line 4 on the screen.

This makes life so much easier!