Re: releasing memory to malloc
The workaround I went with made use of the shelve module and calls to gc.collect() to cap the memory consumed by the Python allocator. It was a bit intrusive but it got the job done. Would a method in the gc module that released memory to malloc be something that could get added to Python? Or are there some reasons why allowing that would be a bad idea? Regards, Iker P.S. This may be a repeat of an earlier message - it seems that google groups may have discarded my earlier post. [EMAIL PROTECTED] wrote: > Is there any way to get Python to release memory back to the > C allocator? I'm currently running a script that goes through > the following steps: > > 1) Creates a very large number of Python objects to produce > a relatively small data structure that sits in a C extension. > The Python objects consume quite a bit of memory. > > 2) Releases all the Python objects. > > 3) Invokes a function of said C extension for further > processing. This step needs as much memory as possible. > > I'd like step 2 to return memory to the C allocator so that it > is available to the extension in step 3 (which uses malloc). > > Regards, > Iker Arizmendi -- http://mail.python.org/mailman/listinfo/python-list
Re: releasing memory to malloc
The workaround I've settled for uses the shelve module and calls to gc.collect() to put a cap on the amount of memory the Python allocator consumes. A bit more intrusive but it gets the job done. Would a gc method that released memory to malloc be something worth adding to Python? Or are there reasons why allowing this is a bad idea? Iker [EMAIL PROTECTED] wrote: > Is there any way to get Python to release memory back to the > C allocator? I'm currently running a script that goes through > the following steps: > > 1) Creates a very large number of Python objects to produce > a relatively small data structure that sits in a C extension. > The Python objects consume quite a bit of memory. > > 2) Releases all the Python objects. > > 3) Invokes a function of said C extension for further > processing. This step needs as much memory as possible. > > I'd like step 2 to return memory to the C allocator so that it > is available to the extension in step 3 (which uses malloc). > > Regards, > Iker Arizmendi -- http://mail.python.org/mailman/listinfo/python-list
Re: releasing memory to malloc
I happen to have the code for the C library in question, but I don't think this is the way to go in general. If there's a way to get Python to give memory back to the C allocator I can avoid touching the library at all. Regards, Iker John Machin wrote: > [EMAIL PROTECTED] wrote: > > I can, but the extension is only a thin wrapper around a general > > purpose C library which is also used independently of Python. > > > > So change the library to use xmalloc etc and add something like this to > the .h file: > > #ifdef PYMEM > #define xmalloc PyMem_Malloc > etc > #else > #define xmalloc malloc > etc > #endif -- http://mail.python.org/mailman/listinfo/python-list
Re: releasing memory to malloc
I can, but the extension is only a thin wrapper around a general purpose C library which is also used independently of Python. Iker Gabriel Genellina wrote: > Can you modify the C source? If you can, use the Python memory > allocation functions PyMem_Malloc/PyMem_Realloc/PyMem_Free. > > > > Gabriel Genellina > Softlab SRL > > > > > > __ > Preguntá. Respondé. Descubrí. > Todo lo que querías saber, y lo que ni imaginabas, > está en Yahoo! Respuestas (Beta). > ¡Probalo ya! > http://www.yahoo.com.ar/respuestas -- http://mail.python.org/mailman/listinfo/python-list
releasing memory to malloc
Is there any way to get Python to release memory back to the C allocator? I'm currently running a script that goes through the following steps: 1) Creates a very large number of Python objects to produce a relatively small data structure that sits in a C extension. The Python objects consume quite a bit of memory. 2) Releases all the Python objects. 3) Invokes a function of said C extension for further processing. This step needs as much memory as possible. I'd like step 2 to return memory to the C allocator so that it is available to the extension in step 3 (which uses malloc). Regards, Iker Arizmendi -- http://mail.python.org/mailman/listinfo/python-list
Re: popen and exit code on Windows
Thanks for the reply Giovanni. Is this behaviour of the close method on Windows documented anywhere? Regards, Iker -- http://mail.python.org/mailman/listinfo/python-list
popen and exit code on Windows
On UNIX one can use popen* to get a pipe for reading, a pipe for writing, and the exit code of the child process via a call to close() on the last pipe. Is there any way, in principle, to simulate such behaviour on Windows? Some googling reveals that direct use of the popen* functions on Windows will not do the trick, but are there indirect ways? Regards, Iker -- http://mail.python.org/mailman/listinfo/python-list
Adding "proxy" functions to a type
Hello all.
Is there a convenient scheme within a C extension to add
methods to a type in such a way as to allow me to
transparently add a "proxy" around them? For example:
typedef PyObject* (*PyMethodCall)(PyObject*, PyObject*);
PyObject* middleMan(PyObject* self, PyObject* args) {
printf("Before call to wrapped funct\n");
PyMethodCall actualFunc = getRealFunction(...);
PyObject* retval = actualFunc(self, args);
printf("After call to wrapped func\n");
return retval;
}
void addMethod(PyTypeObject* t, PyMethodCall* m,
char* name, char* doc) {
// code to forward calls to "middleMan" while putting the
// pointer to "m" somewhere convenient
...
}
My current solution is cumbersome and involves adding a
special field to the PyObject associated with my type, a
custom tp_getattro function and the "middleMan" function:
struct MyPyObj {
PyObject_HEAD
PyMethodCall realfunc;
};
// when adding the method def for "m" instead of pointing it to
// the given function, I point it to the middle man and save
// the "m" function somewhere I can find it later.
void addMethod(PyTypeObject* t, PyMethodCall m,
char* name, char* doc) {
PyMethodDef* def = allocPyMethodDef(t, name);
def->ml_name = name;
def->ml_doc = doc;
def->ml_meth = middleMan;
def->ml_flags = METH_VARARGS;
saveTargetFunction(name, m);
// note I add these here so that documentation is
// available within the interpreter
PyObject* methobj = PyDescr_NewMethod(t, def);
PyDict_SetItemString(t->tp_dict, def->ml_name, methobj);
Py_DECREF(methobj);
}
// when the interpreter does a lookup on an instance of my
// type I set the "realfunc" member of my PyObject and return
// a bound method (which will call into middleMan).
PyObject* customGetaAttro(PyObject* self, PyObject* name) {
MyPyObj* rself = (MyPyObj*)self;
rself->realfunc = findSavedTargetFunc(name);
PyMethodDef* mdef = getMethodDef(self->ob_type, name);
return PyCFunction_New(def, self);
}
// finally, when the middle man is called it extracts the "real"
// function from self and calls that.
PyObject* middleMan(PyObject* self, PyObject* args) {
MyPyObj* rself = (MyPyObj*)(self);
printf("pre call\n");
PyObject* rv = rself->realfunc(rself->obj, args);
printf("post call\n");
rself->realfunc = 0;
return rv;
}
The problem here is that this doesn't work for static functions which
lack a self argument, or for module level functions.
Is there a better way?
Thanks,
Iker Arizmendi
--
http://mail.python.org/mailman/listinfo/python-list
Inheriting socket handles on Windows
Hello all.
I'm trying to get a server process to do the following
on both Linux and Windows:
1) Create a socket and bind it to some port number.
2) Use spawnl to create other processes that will
then listen and accept on that socket.
On Linux I've managed to do that by using command line
args to pass the FD number of the listening socket to
the spawned processes. Eg:
# SERVER
#
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', ))
s.listen(5)
exe = "childscript"
fds = str(s.fileno())
pid = os.spawnl(os.P_NOWAIT, exe, exe, fds))
print "process spawned"
time.sleep(5)
# CLIENT
# where fds was received from the parent
#
fds = int(sys.argv[1])
s = socket.fromfd(fds, socket.AF_INET, socket.SOCK_STREAM)
s.listen(5)
print "child listening..."
However, this fails on Windows as it doesn't have the fromfd
function. Aren't WinSock handles inherited by default? And if
so, is there some other way to perform a listen()/accept() in
this case?
Regards,
Iker Arizmendi
--
http://mail.python.org/mailman/listinfo/python-list
