Updating a web page from a server – sending a complete HTML document.

When using a web server as a back-end to an HTML page, for example filling in a form, the result from the back-end can be

  1. “Redirect to a new page”.
  2. A stream of HTML. This is an HTML document where the “boilerplate” or constant data, is intermixed with the variable data included inline. The HTML is displayed, and replaces the previous page.
  3. A stream of data containing data such as changed fields. The requesting page can use this stream to update its page.

Sending a complete HTML document

Using a python back-end program I can write an entire (and complete) HTML document.

#!/usr/bin/python3
import cgitb
import cgi

cgitb.enable()

print("Content-Type: application/json")
print("ColinHeader: Value")
print("ColinHeader2: Value")
# indicate end of headers
print()
# now the html page
print("<!doctype html>")
print('"<html lang="en">')
print("<head>"
print("<body>")
print('<form id="target"  action="cgi-bin/first.py" enctype="multipart/form-data"  >')
print('<label for="email">Enter your email: </label>')
print('<input id=email name="email" value="test@example.com" title="colins email address">')
print('<label for="password">Enter your pw: </label>')
print('<input id=password name="password" value="pw">')
print('<input type="submit">')print("</form>") print("</body>") print("</html>")

Where

  • print(‘<label for=”email”>Enter your email: </label>’)
    is constant text.
  • print(‘<input id=email name=”email” value=”test@example.com” title=”colins email address”>’)
    is the variable data putting text@example.com into the field.
  • print(‘<label for=”password”>Enter your pw: </label>’)
    is constant text.
  • print(‘<input id=password name=”password” value=”pw”>’)
    is variable text, putting pw into the field.

This is a terrible way of doing it. Presentation (red text) is mixed up with data (green text). This was a no-no about 40 years ago! You might have several similar pages, and any updates would have to be made to all pages. Using pages in national different languages is hard to implement; you need to have a multiple program, one for each language.

Updating a web page from a server – sending back just the updates

When using a web server as a back-end to an HTML page, for example filling in a form, the result from the back-end can be

  1. “Redirect to a new page”.
  2. A stream of HTML. This is an HTML document where the “boilerplate” or constant data, is intermixed with the variable data included inline. The HTML is displayed, and replaces the previous page.
  3. A stream of data containing data such as changed fields. The requesting page can use this stream to update its page.

multiple program, one for each language.

Sending back just updates.

I could have my back-end server program send back just changed elements.

#!/usr/bin/python3
import cgitb
import cgi

cgitb.enable()

print("Content-Type: application/json")
print("ColinHeader: Value")
print("ColinHeader2: Value")
# indicate end of headers
print()
#now the data
print('<p id=email>New-mail</p>')
print('<p id=pw>*********</p>')

Note: This does not send an HTML document, it just sends changed fields.

Where

  • print(‘<p id=email>New-mail</p>‘) id=email is the field in the requester page to be updated
  • print(‘<p id=pw>*********</p>‘) id=pw is the field in the requestor page to be updated.

The front page needs to be smarter, and be able to process these fields. See fetch in HTML sending stuff to the back-end server

In the example below it updates the content of the field with id=”passed”, so that the information returned is only displayed. It does not update the fields. This is described below.

<p id="errorField">No Errors yet</p>

<form id="target"  action="cgi-bin/first.py" enctype="multipart/form-data"  >
  <label for="email">Enter your email: </label>
  <input id=email    name="email" value="test@example.com" title="colins email address">
  <label for="password">Enter your pw: </label>

  <input id=password name="password" value="pw">
  <input type="submit">
  <input type="text" onblur = "check(this)" >
</form>
<p id="passed">old info</p>

<script>
document.forms["target"].addEventListener('submit', (event) => {
  event.preventDefault();
  fetch("cgi-bin/first.py", {
      method: 'POST',
      body: new URLSearchParams(new FormData(event.target)) // event.target is the form
  }).then((response) => { 
       
      if (!response.ok) {
        throw new Error(`HTTP error! Status: ${response.status}`);
      }
        return (response.text());
    })
    .then(function(result) 
    {
      // put the entire response into an exist field.
      document.getElementById("passed").innerHTML = result;     

   }).catch((error) => {
      console.log(error);// TODO handle error
      return(error);
  });
});
</script>

where

.then(function(result) 
    {
    }

is passed the body of the html as a string. The value result is available within the function.

 document.getElementById("passed").innerHTML = result;

This updates <p id=”passed”>old info</p> and gives it the content in result ( the whole data stream).

Parsing a returned document

If you are are returning a whole HTML document <!doctype>…</html> you can use DOMparser to parse the document.

var doc2 = new DOMParser().parseFromString(result, "text/xml");
let x = doc2.getElementById("... 

I was using rexx and it was returning lines of data – not a whole HTML document, and so the DOMParser complained that there was an incomplete document.

Parsing data (not necessarily a document)

By adding the text to an element already in the front page you can use the page’s parser to process the data.

For example add the returned text to a field in the front page

document.getElementById("passed").innerHTML = result;

This will display all of the returned data in the field, and it will be visible ( unless the field is hidden). You can remove elements – see below.

You can now access this data using standard navigation.

My Python program had

print('<p id="z" style="color:red">incolinsz red</p>')
print('<p id="z">colins2</p>')
print('<p id="z">colins3</p>')

print('<p id="update" update="email">Updated@email</p>')
print('<p id="update" update="password">newpw</p>')

print('<p id="focus" which="email"/>')

I want to

  • Display the elements with id=”z”
  • Give a field a focus.
  • Update the fields in the document from the data, where id=”update”, and update=”…” is the name of the field on the page.

Display the elements with id=”z”

let e = document.getElementById("errorField");
e.innerHTML = "";

let  myzs = q.querySelectorAll("#z");
console.log("size of myzs" + myzs.length);
Array.from(myzs).forEach((element) => {
      console.log(element.innerHTML )
      e.innerHTML += element.outerHTML;
}); 

Where

  • let e = document.getElementById(“errorField”); Locate the field where we want to store the error elements
  • e.innerHTML = “”; Clear this
  • let myzs = q.querySelectorAll(“#z”); Get a list of all the elements with id=”z”
  • console.log(“size of myzs” + myzs.length); Debugging – display how many we have
  • Array.from(myzs).forEach((element) => For each one. We need an array to use forEach
  • {
    • console.log(element.innerHTML ) Display the content
    • e.innerHTML += element.outerHTML; Build up the field by concatenating the elements. Include the html, such as styling.
  • });

Note: If element.innerHTML is used, then only the value is used. If outerHTML is used, the formatting is also copied across, for example the text <p id=”z” style=”color:red”>incolinsz red</p> is displayed in red.

Set the focus

This is an example of passing data across the interface. The Javascript looks for the first element like print(‘<p id=”focus” which=”email”/>’) with an id of focus. It uses the “which” attribute to pass a field name.

The Javascript code is

let focus=q.querySelectorAll("#focus");
if (focus.length > 0)
{
  let f0 = focus[0].getAttribute("which");
  let d =  document.getElementById(f0);
  d.focus()
  d.setAttribute('style', 'color: red');
}
  • let focus=q.querySelectorAll(“#focus”); Get all the elements with id=”focus”
  • if (focus.length > 0) If we had at least one…
  • {
    • let f0 = focus[0].getAttribute(“which”); Use the first one found, and extract the value of the “which=” attribute.
    • let d = document.getElementById(f0); Look for the element with the same name in the main document
    • if (d !== null) if it was found
    • {
    • d.focus() Set the focus
    • d.setAttribute(‘style’, ‘color: red’); As it is hard to see if the focus was set, change the colour of the field as well.
    • }
  • }

Updates the fields in the document

The Javascript is

let  updates = q.querySelectorAll("#update");
Array.from(updates).forEach((element) => {
   console.log(element.innerHTML )
   let f0 = element.getAttribute("update");
   let e = document.getElementById(f0);
   if (e !== null)
   {
     e.value = element.innerHTML;     
   }
   else 
   {
     console.log("element not found:" + f0)
    }
}); 

Where

  • let updates = q.querySelectorAll(“#update”); Select all elements which have id=”update”
  • Array.from(updates).forEach((element) => { We need an array to be able to use forEach
    • console.log(element.innerHTML ) Display it
    • let f0 = element.getAttribute(“update”); Get the value of the “update=…” attribute for the individual element
    • let e = document.getElementById(f0); Locate the element in the main document
    • if (e !== null) If it was found
      • {
        • e.value = element.innerHTML; Set the value to what was passed in
      • }
      • else Log we have a problem – and ignore it
      • {
      • console.log(“element not found:” + f0)
      • }
  • });

Removing elements

If you want to display a subset of the elements, you may want to delete some elements.

Array.from(myzs).forEach((element) => {
  x = element
  x.parentNode.removeChild(x);    
}); 

For the elements above with id=”z” the code does

  • Array.from(myzs).forEach((element) => { Create an array so you can use forEach
    • x = element save it
    • x.parentNode.removeChild(x); Delete it
  • });

Create a new field

If you want to create a new field and insert the data you can do

// append data to an existing field.
var d = document.createElement('div');
d.setAttribute("id", "Special");
var myPara = document.getElementById("passed");
myPara.appendChild(d);
d.innerHTML = result;

This does

  • // append data to an existing field.
  • var d = document.createElement(‘div’); create a div section
  • d.setAttribute(“id”, “Special”); and give it a name
  • var myPara = document.getElementById(“passed”); locate the element in the document with the id “passed“.
  • myPara.appendChild(d); Attach our new field to the item.
  • d.innerHTML = result; Sets the content to what was passed back.

You will see the data appear on the web page.

Note: if you execute the page more than one, you will get more and more data

Updating a web page from a server – redirect to a new page.

When using a web server as a back-end to an HTML page, for example filling in a form, the result from the back-end can be

  1. “Redirect to a new page”.
  2. A stream of HTML. This is an HTML document where the “boilerplate” or constant data, is intermixed with the variable data included inline. The HTML is displayed, and replaces the previous page.
  3. A stream of data containing data such as changed fields. The requesting page can use this stream to update its page.

Redirect to a new page

The server can send back status meaning redirect

#!/usr/bin/python3
import cgitb
import cgi
import sys, os, io

cgitb.enable(display=0, logdir="/home/colinpaice/first.log")
print('Status: 303 See Other2')
print("Content-Type: text/html")
// the url and any parmeters
print("Location: ../c3.html?colin=p1&error=p2")
# needs blank line to say end of headers
print()

This causes page c3.html to be displayed. The page can process the URL and process the options passed on the url.

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<HTML lang="en">
<HEAD>
<script>
//  this function is  invoked on page load 
function js_onload_code (e){
  var url = document.location.href;
  // split the parameters from the url  base
  var p = url.split('?')[1]
  if (p)
  {
    var params = p.split('&');
    if (params)
    {
      var l = params.length;
      // display what was passed in
      alert("colin:"+url); 
      for (var i = 0 ; i < l; i++) 
      {
        // split it into kw"="value
        tmp = params[i].split('=');
        document.write(tmp[0] + "=" + tmp[1] +"!<br>");
        }
      } // if params
    } // if
}
// this causes the above script to be executed.
window.onload= js_onload_code ();
   
</script>
<BODY  >
<P>This is page c3.html
</BODY>
</HTML>

This displayed a page with url http://localhost/c3.html?colin=p1&error=p2

colin=p1!
error=p2!

This is page c3.html

If you repeatedly refresh this page you will get more and more data! It is better to use a field, then clear it before adding the information.

HTML sending stuff to the back-end server.

The simple and most common way of sending HTML data to a back-end server is using an anchor link

<a href="./echo.rexx">echo</a> 

This invokes the back-end program and displays the output. You have no control over the headers or what data is sent.

You can do more sophisticated stuff using the JavaScript fetch function, for example

  • specify different headers
  • change the data sent to the server, for example specify which data you want sent to the server, perhaps only send changed values rather than all input values
  • send the data to a different URL.

A quick aside on reading the JavaScript documentation and use of the arrow function

A lot of the documentation on JavaScript uses the arrow function, which can be useful, but I find distracting.

You can use a function such as mysubmit which takes an event:

function mysubmit(event)
{
console.log(event);
}
document.forms["target"].addEventListener('submit', mysubmit ) 

When the submit event occurs, “mysubmit” is invoked, and passed the event;

This can also be written, using inline functions as

document.forms["target"].addEventListener('submit', mysubmit(event) => { console.log(event;...} )
or
document.forms["target"].addEventListener('submit', (event) => { console.log(event;...} ) 

I have also seen it written as

function mysubmit(zevent)
{
console.log(zevent)
}
document.forms["target"].addEventListener('submit', (event) => mysubmit(event) ) 

At first glance this looks it could be simplified. However this is useful if you wanted to pass additional parameters to the function.

function mysubmit(zevent,a)
{
// a will have the value abc
console.log(zevent)
}
document.forms["target"].addEventListener('submit', (event) => mysubmit(event,'abc') ) 

I find the first example, document.forms[“target”].addEventListener(‘submit’, mysubmit ) easier to follow; and easier to write as you have to worry about matching the ending “})”s.

Using fetch

I used an HTML page with a form, so I could process the data. The fetch function sends the data to the specified URL, and waits for the response. You can specified headers and body and you get back a response object.

Setup

You need to associate a function with the “submit” of the form

 document.forms["target"].addEventListener('submit', mysubmit ) 

where mysubmit is the name of a callback function, for example

function submit(event) 
{ 
  event.preventDefault(); 
  displayChanged(event) 
  fetch("./echo.rexx", { 
      method: 'POST', 
      body: new URLSearchParams(new FormData(event.target)) // event.target is the form 
    })
    .then( check   )
    .then( zesult  ) 
    .catch(mycatch ) 
} 

Where

  • function submit(event) { defines the function, the parameter variable is called event.
  • event.preventDefault(); disables normal behaviour and lets me manage it.
  • displayChanged(event) my processing of the data before sending the data, see below.
  • fetch(“./echo.rexx”, { This sends the request to the backed “./echo.rexx”.
    • method: ‘POST’,
    • body: new URLSearchParams(new FormData(event.target)) Fetch has two parameters, the URL and a set of options, such as body, a set of headers{} etc. In this instance the second parameter is {method:”POST”, body: …. }.
  • }) // end of fetch
  • .then(check) this waits until the fetch finishes, see below
  • .then(zesult( ) see below
  • .catch( mycatch ) any exception
  • } // end of function

The .then processing is about using promises. Basically the .then has a wait for the response and return something to the next .then statement.

Check

This waits for the completion of the request.

function check(response) 
{ 
  if (!response.ok) { 
       throw new Error(`HTTP error! Status: ${response.status}`); 
  } 
  return response.text() 
 } 
  • It checks the “ok” flag in the data, if it is not OK, then throw an exception. The variable response.status has an error message.
  • Extract the body of the response and pass it on.

zesult

This waits for the completion of the previous “return response.text()”

function zesult(text) 
{ 
//console.log(text ); 
  let e = document.getElementById("errorField"); 
  e.innerHTML = text;                                                                     
 return (text) 
} 

and replaces the value of the piece of HTML with id=”errorfield”. For example

<p id="errorField" style="color:red ">No Errors yet</p> 

I do not know why two “.then”s are needed. It looks like it started another asynchronous process to extract the body from the response. In “check” if you display “response” it gives you some data. If you display “response.body” is gives a in-flight promise.

If you are tracing this, and you do not get into your code, check you have specified event.preventDefault(); before you do the fetch.

Looking at what data is being sent

Formdata allows you to work with the keyword=value data as generated by a form. You cannot get the data for other HTML elements outside of a form.

In my submit function I have displayChanged(event)

function displayChanged(event) 
{
// display the parameters
...
// Display what has changed
... 
}

Display the parameters

function displayChanged(event) 
{ 
  // display all the data 
  // extract into an iterable object  
 let f = new FormData(event.target)  ; 
 let data = ""
 for ([key, value] of f) { 
    data += ", " + key + ":" + value 
 } 
 console.log(data) 
...
}

For the data in my form, this displayed

,i1:initial value, text1:test@example.com, email:test@example.com, password:pw18,password25:pw25

where the bold names are the input field names.

Display what has changed

There is no direct way of processing just the changed variables, but you can fake it.

I had an input fields defined with an onchange call back.

<input id="password" name="password" value="pw18" onchange="change(this)" >
<script> 
function change(o){ 
  if  (o!= undefined) 
     o.setAttribute("changed","yes"); 
} 
</script>

If the field is changed, then the onchange -> change function is called. This sets an attribute “changed” to “yes”.

In the submit processing there

function displayChanged(event) 
{ 
...
  // display what has changed
  let er = event.target; 
  Array.from(er).forEach((r)  => 
  { 
    let id = r.getAttribute("id") 
    let ch = r.getAttribute("changed") 
    console.log( "id " + id + " changed " + ch ) 
  }) 
...
}

When I changed the two password fields, the console log had

id text1 changed null
id email1 changed null
id password21 changed yes
id password2 changed yes

Changing the parameters passed to the server.

You pass the parameters to the server using

fetch("./echo.rexx", { 
    method: 'POST', 
    body: new URLSearchParams(new FormData(event.target)) 

You can create a new instance of FromData, add keyword/value parameters to it, and pass it through to the back-end.

Above I explained how you can identify the “changed” data. You could take these elements and add to a new FormData, and pass those through to the server.

JavaScript: selecting an element

When writing some JavaScript to update elements in a page, I got confused selecting which elements to process. Below is my little crib sheet to help me remember.

Naming of parts.

You can have some HTML like

<div id="colin" name="request" class="myclass">requestor<div>

This division element can be referred to by the id=”colin”. It has an attribute name with a value of request. It has formatting parameters defined in the style sheet class=”myclass”.

If you have an element myDiv, you can use

let myclass=myDiv.getAttribute("class");
myDiv.setAttribute("class", "newclass");
myDiv.setAttribute("id", "newid");
myDiv.setAttribute("name", "newname");
myDiv.setAttribute("newattribute", "new");
myDiv.removeAttribute("name");

The id,class and any attribute look the same, but are slightly different when used to select elements.

Structure

You can have a low level element like a paragraph <p>..</p>

You can have more complex objects like a table <table>…</table> which can have table rows <tr>…</tr>, which can have table cells <td>…</td> which can have other elements such as <p>…</p>.

A division <div..>/div> can be used around text or an element to give it properties, such as a formatting or giving a word an id.

The high level structure is a document. You can create and add element to another element. You can remove elements. If you remove a table element, all of its children, <tr> and<td> are removed as well.

You can have elements which are not part of the document. You can add them to the document.

Locating an element in JavaScript

Originally there was just getElementById:

document.getElementById('newid');

This searches within document and its children for the first element with id=”newid”. If you have more than one element with id=”newid” only the first will be returned.

With a document

<!doctype html>
<html lang="en">
<head>
</head>
<body>
<p id="passed" class="myclass"></p>
<p id="p1" name="Colin">Colin</p>
<p id="p1" name="Jo">Jo</> 
</body>
</html>
var q= document.getElementById('passed');

returns an object called p#passed. This means it is a <p>aragraph object with id=”passed”.

It was quickly found that getElementById was not flexible enough, and so

querySelectorAll and querySelector were created. These are much more flexible.

  • let p=querySelector(“p”) returns the first element of type <p>.
  • let pall=querySelectorAll(“p”) returns a NodeList of all the <p> elements. Each element in the node list represents a <p> element. In my example HTML pall has a length of 3, because there are 3 <p> elements.
  • let p=querySelector(“#p1”) returns the first node with id=”p1″. It returns an element with “p#p1” saying this is a <p> element.
  • let pnl=querySelectorAll(“#p1”) with the above HTML returns a NodeList of length 2 where:
    • pnl[0] is the first node with id=”p1″. It returns an element with “p#p1”
    • pnl[1] is the second node with id=”p1″. It returns an element with “p#p1”.
  • let pnl=querySelectorAll(“p.myclass”) returns all “p” elements with class=”myclass”.
  • let pnl=querySelectorAll(“.myclass”) returns all elements with class=”myclass” regardless of type.
  • let nl=querySelectorAll(“[name=’colin’]”) returns all elements with attribute name=”colin”. With the above example data it returns a nodeList with one element which is a p#p1.

These selectors can be combined so for example you could ask for all <p> nodes within a <table> with id=”mytab” which have name=”colin”.

Finding data in a subtree

Within my HTML page I have

<table id="ztable" frame="box" rules="rows"> 
<tr id="template"> 
   <td id="col0" name="serial"><div><p>data</p></div></td> 
   <td id="col1" name="requestor"><div>requestor<div></td>    
</tr> 
<tr id="tr1"></tr>
</table> 

<table id="table2"> 
<tr id="tr1"></tr>
</table>

I can find the first table, and extract the rows out of it.

let tab = document.getElementById('ztable');
let tr = tab.querySelectorAll("tr");

This locates the table with id=”ztable” then searches within this for all <tr> tags. It returns a node list of length 2. It does not return the row in the second table(table2).

Not found?