Using PyArg_ParseTupleAndKeywords to parse data in Python external functions.

I was failing to use PyArg_ParseTupleAndKeywords succcessfully in an external Python function. It took about a day to get it to work. Below are some of the lessons learned in using this facility.

Background

When Python calls an external function (written in C) you can pass parameters, and keywords. For example a function taking one positional parameter, and keywords a,b,c,d could be used:

rc = zconsole.put( 12345 , d = 7, b = 10 )

rc = zconsole.put( 12345 , b=10, d= 7)

Where

• 12345 is a positional parameter,
• keyword d is set to the value 7
• keyword b is set to the value 10
• the keyword values a and c are not set, and the values are unchanged.

My C program had

static PyObject *console_put(PyObject *self, PyObject *args, PyObject *keywds ) {

int a = 0;
int b = 0;
int c = 0;
int d = 0;
int pos0 = 0;
int pos1 = 0;
static char *kwlist[] = {“pos0″,”pos1,”,”a”,”b”,”c”,”d”,NULL};
if (!PyArg_ParseTupleAndKeywords(args, keywds, “i|i$iiii”, kwlist,
&pos0 ,
&pos1 ,
&a , // i
&b , // i
&c , // i
&d )) // i
{
return NULL;
}

The kwlist array contains the keywords, so it includes a,b,c,d. It must also include labels for the positional parameters (I was missing this for the first day!). pos0 is for the first positional parameter. There is also an optional second position parameter pos1. The list ends with a NULL.

The PyArg_ParseTupleAndKeywords function takes

• args passed into your function
• keywds passed into your function
• the format string “i|i$iiii”. This says
  • i – a required positional integer
  • | – following parameters are optional
  • i -an optional positional integer
  • $ – the following are keywords
  • iiii – 4 integers corresponding to the keywords in {“pos0″,”pos1,”,“a”,”b”,”c”,”d”,NULL};

The message

SystemError: more argument specifiers than keyword list entries (remaining format:’i’)

means the number of elements in the format string does not match the number of elements in the keywords array. I got the above error when I removed “pos0″,”pos1” from the list.

Processing strings

With well defined values like integers have a known length. Strings can have different sizes.

In the format you define s#,

• s – return a pointer to the string
• # – give the length of the string.

Py_ssize_t lMsg;
char * p;
static char kwlist[] = {“pos0″,”pos1″,”a”,”b”,”c”,”d”,NULL};
if (!PyArg_ParseTupleAndKeywords(args, keywds, “i|i$s#iii”, kwlist,
&pos0 ,
&pos1 ,
&p , // s this variable is a string
&lMsg , // this is the size of the string the ‘#’ above
&b , // i this variable is int
&c , // i this variable is an int
&d )) // i
{ return NULL; }
printf(“lMsg=%d\n”,lMsg);
printf(“String=%*.*s\n”,lMsg,lMsg,p);
printf(“pos0=%d\n”,pos0);

The code

rc = zconsole.put( 12345 , a = “abcde”, b = 10 )

gave

lMsg=5
String=abcde
pos0=12345

…

Setting up the function to use keywords

You need to set a flag to say that keywords are being used. For example

static struct PyMethodDef console_methods[] = {
{“put”, (PyCFunction)console_put,METH_KEYWORDS | METH_VARARGS, console_put_doc},
{NULL, (PyCFunction)NULL, 0, NULL} /* sentinel */
};

I want to pass a variable number of data items

You can configure PyArg_ParseTupleAndKeywords with a list containing optional fields, but how do you pass multiple fields in?

You can say parse the data, and return a field as a python object.

PyObject *rv = NULL;

if (!PyArg_ParseTupleAndKeywords(args, keywds, “O|….”, kwlist,
&rv ,
….

rc = PyTuple_Check(rv );
if (rc == 1 ) // it is a tuple
{
size = PyTuple_Size(rv );
printf(“Tuple size %d\n”,size);
}

The “O” code says return the object, rather than the string.

You can now do your own checks on the code for example get the type of the object.

• rc = PyList_Check(t );
• rc = PyLong_Check( t)
• rc = PyBytes_Check(t );
• rc = PyByteArray_Check(t);
• rc = PyBool_Check(t);
• rc = PyFloat_Check(t);
• rc = PyFunction_Check(t) ;
• rc = PySet_Check(t );
• rc = PyTuple_Check(t );
• rc = PyDict_Check(t );
• rc = PyUnicode_Check(t);
• rc = PyComplex_Check(t);

There is no “getType” function as such see here.

Once you know it’s type you can use Pyxxx_Size to get the size, or number of elements in the object.

If the data is a tuple of 3 bits of data

rc = zconsole.put((“a123456789″,”b123″,”c44” ) )

PyTuple_Check(rv) will return 1, and you can get the number of elements using PyTuple_Size which returns 3.

You can use PyArg_ParseTuple to dig into the objects you have retrieved, for example, for the object rv obtained above:

if (!PyArg_ParseTuple(rv, “s#|s#”,
&p1 , // message text
&lMsg1 , //i
&p2 , // message text
&lMsg2))
{
return NULL;
}
printf(“p1 %.s\n”,lMsg1,lMsg1,p1);
printf(“p2 %.s\n”,lMsg2,lMsg2,p2);

Parsing a fixed size tuple

You can use ‘(…)’ within the format string to process data in a tuple, so you do not need to dig into the object

if (!PyArg_ParseTupleAndKeywords(args, keywds, “(s#s#)|$iiiii”, kwlist,
&p1 , // string1
&lMsg1 , //i
&p2 , // string2
&lMsg2, //i
…

Getting the type of an object

There is no “getType” function as such.

You can use

int type;
type = PyType_GetFlags(Py_TYPE(rv ));
printf(“TYPE = %8.8x\n”,type);

The values of type are listed here. A constant is defined as

#define Py_TPFLAGS_TUPLE_SUBCLASS (1UL << 26)
etc

which is 0x04000000. I found it easier to use the pyTuple_Check() function than decode the fields.