The array.array type is an excellent type for storing a large amount of
"native" elements, such as integers, chars, doubles, etc., without
involving the heavy machinery of numpy. It's both blazingly fast and
reasonably efficient with memory. The one thing missing from the array
module is the ability to directly access array values from C.
This might seem superfluous, as it's perfectly possible to manipulate
array contents from Python/C using PyObject_CallMethod and friends. The
problem is that it requires the native values to be marshalled to Python
objects, only to be immediately converted back to native values by the
array code. This can be a problem when, for example, a numeric array
needs to be filled with contents, such as in this hypothetical example:
/* error checking and refcounting subtleties omitted for brevity */
PyObject *load_data(Source *src)
{
PyObject *array_type = get_array_type();
PyObject *array = PyObject_CallFunction(array_type, "c", 'd');
PyObject *append = PyObect_GetAttrString(array, "append");
while (!source_done(src)) {
double num = source_next(src);
PyObject *f = PyFloat_FromDouble(num);
PyObject *ret = PyObject_CallFunctionObjArgs(append, f, NULL);
if (!ret)
return NULL;
Py_DECREF(ret);
Py_DECREF(f);
}
Py_DECREF(array_type);
return array;
}
The inner loop must convert each C double to a Python Float, only for
the array to immediately extract the double back from the Float and
store it into the underlying array of C doubles. This may seem like a
nitpick, but it turns out that more than half of the time of this
function is spent creating and deleting those short-lived floating-point
objects.
Float creation is already well-optimized, so opportunities for speedup
lie elsewhere. The array object exposes a writable buffer, which can be
used to store values directly. For test purposes I created a faster
"append" specialized for doubles, defined like this:
int array_append(PyObject *array, PyObject *appendfun, double val)
{
PyObject *ret;
double *buf;
Py_ssize_t bufsize;
static PyObject *zero;
if (!zero)
zero = PyFloat_FromDouble(0);
// append dummy zero value, created only once
ret = PyObject_CallFunctionObjArgs(appendfun, zero, NULL);
if (!ret)
return -1;
Py_DECREF(ret);
// append the element directly at the end of the C buffer
PyObject_AsWriteBuffer(array, (void **) &buf, &bufsize));
buf[bufsize / sizeof(double) - 1] = val;
return 0;
}
This hack actually speeds up array creation by a significant percentage
(30-40% in my case, and that's for code that was producing the values by
parsing a large text file).
It turns out that an even faster method of creating an array is by using
the fromstring() method. fromstring() requires an actual string, not a
buffer, so in C++ I created an std::vector<double> with a contiguous
array of doubles, passed that array to PyString_FromStringAndSize, and
called array.fromstring with the resulting string. Despite all the
unnecessary copying, the result was much faster than either of the
previous versions.
Would it be possible for the array module to define a C interface for
the most frequent operations on array objects, such as appending an
item, and getting/setting an item? Failing that, could we at least make
fromstring() accept an arbitrary read buffer, not just an actual string?
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
