Re: Calling python from C with OpenMP
Dennis Lee Bieber writes: > It's been tried -- but the non-GIL implementations tend to be > slower at everything else. Has Micropython been compared? CPython needs the GIL because of its frequent twiddling of reference counts. Without the GIL, multi-threaded CPython would have to acquire and release a lock whenever it touched a refcount, which slows things down badly. MicroPython uses a tracing garbage collector instead of refcounts, so there's no issue of having to lock refcounts all the time. It's fairly common in such systems to stop all the user threads during GC, but they can happily run in parallel the rest of the time. Come to think of it, I don't know if MicroPython currently supports threads at all! But its implementation style (i.e. no refcounts) is more parallelism-friendly than CPython's. -- https://mail.python.org/mailman/listinfo/python-list
Re: Calling python from C with OpenMP
On Friday, May 13, 2016 at 7:12:33 PM UTC+2, MRAB wrote: > Every PyGILState_Ensure call must be matched with a PyGILState_Release > call. The way it's currently written, it won't call PyGILState_Release > if ret is NULL. Yeah, that's tiny bug, however it is not the main problem... > However, I don't think you'll gain much here because you can gain from > multi-threading only if the threads can run in parallel. You need to > hold the GIL while making Python calls, and only 1 thread can hold the > GIL at any time. In addition to the fact that it still does not work, what you here point out, is the main problem. I actually think I'm giving up for now. Maybe I'm able to solve this later (when python does not use a GIL anymore...?) Thanks to Sturla and MRAB anyway! -Øystein -- https://mail.python.org/mailman/listinfo/python-list
Re: Calling python from C with OpenMP
On 2016-05-13 17:22, Øystein Schønning-Johansen wrote:
On Friday, May 13, 2016 at 2:04:53 AM UTC+2, Sturla Molden wrote:
You must own the GIL before you can safely use the Python C API, object
creation and refcounting in particular. Use the "Simplified GIL API" to
grab the GIL and release it when you are done.
I've now read about the GIL and it looks like I am in deep problems.
I've added the GILState lock to the threaded loop like this:
#pragma omp parallel for
for( int i = 0; i < 10; i++ ){
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
PyObject *ret = PyObject_CallMethod( mult_obj, "do_multiply", "i", i );
if( !ret ){
printf("Cannot call 'do_multiply'\n");
continue;
}
printf("The value calculated in Python was: %3d\n", (int)
PyLong_AsLong(ret));
Py_DECREF(ret);
PyGILState_Release(gstate);
}
but still no success. Have I done it right?
regs,
-Øystein
Every PyGILState_Ensure call must be matched with a PyGILState_Release
call. The way it's currently written, it won't call PyGILState_Release
if ret is NULL.
However, I don't think you'll gain much here because you can gain from
multi-threading only if the threads can run in parallel. You need to
hold the GIL while making Python calls, and only 1 thread can hold the
GIL at any time.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Calling python from C with OpenMP
On Friday, May 13, 2016 at 2:04:53 AM UTC+2, Sturla Molden wrote:
> You must own the GIL before you can safely use the Python C API, object
> creation and refcounting in particular. Use the "Simplified GIL API" to
> grab the GIL and release it when you are done.
I've now read about the GIL and it looks like I am in deep problems.
I've added the GILState lock to the threaded loop like this:
#pragma omp parallel for
for( int i = 0; i < 10; i++ ){
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
PyObject *ret = PyObject_CallMethod( mult_obj, "do_multiply", "i", i );
if( !ret ){
printf("Cannot call 'do_multiply'\n");
continue;
}
printf("The value calculated in Python was: %3d\n", (int)
PyLong_AsLong(ret));
Py_DECREF(ret);
PyGILState_Release(gstate);
}
but still no success. Have I done it right?
regs,
-Øystein
--
https://mail.python.org/mailman/listinfo/python-list
Re: Calling python from C with OpenMP
wrote: > Second and most important question: When I run this code it sometimes > segementation faults, and sometimes some threads run normal and some > other threads says "Cannot call 'do_multiply'". Sometimes I get the > message: Fatal Python error: GC object already tracked. And some times it > even runs normally... > I understand there is some kind of race condition here, where python > tries to refer to some memory that has been already released. But how can > I avoid this? What am I doing wrong? (or, less likely, is this a bug?) You must own the GIL before you can safely use the Python C API, object creation and refcounting in particular. Use the "Simplified GIL API" to grab the GIL and release it when you are done. -- https://mail.python.org/mailman/listinfo/python-list
Calling python from C with OpenMP
Hi,
I have a framework written in C and I need to call Python from that framework.
I have written the code, and it runs fine, however when I recompile with OpenMP
enabled, I get segmentation faults and some times an error message:
Fatal Python error: GC object already tracked
I'm able to reconstruct the bug with this simple code:
/* main.c */
#include
#include
int main()
{
wchar_t *program = Py_DecodeLocale( "My_problem", NULL );
if( !program ){
fprintf(stderr, "cannot decode python program name\n");
return -1;
}
Py_SetProgramName( program );
Py_Initialize();
PyObject *sys = PyImport_ImportModule("sys");
PyObject *path = PyObject_GetAttrString(sys, "path");
PyList_Append(path, PyUnicode_FromString("."));
PyObject *module_filename = PyUnicode_FromString( "multiplier" );
if(!module_filename){
printf("Cannot create python module multiplier.\n");
return -1;
}
PyObject *module = PyImport_Import( module_filename );
if(!module){
printf("Cannot create python.\n");
return -1;
}
PyObject *mult_obj = PyObject_CallMethod( module ,"multiplier", "i", 7);
if(!mult_obj){
printf("Cannot create python multiplier class instance\n");
return -1;
}
Py_DECREF( module );
Py_DECREF( module_filename );
Py_DECREF( path );
Py_DECREF( sys );
/* Up to now we have actually done:
* >>> import multiplier
* >>> mult_obj = multipier.multiplier(7)
*/
/* lets try something like:
* >>> for x in range(10):
* ... printf(mult_obj.do_multiply(x))
*/
#pragma omp parallel for
for( int i = 0; i < 10; i++ ){
PyObject *ret = PyObject_CallMethod( mult_obj, "do_multiply", "i", i );
if( !ret ){
printf("Cannot call 'do_multiply'\n");
continue;
}
printf("The value calculated in Python was: %3d\n", (int)
PyLong_AsLong(ret));
Py_DECREF(ret);
}
Py_DECREF(mult_obj);
Py_Finalize();
return 0;
}
Compile with:
gcc -std=gnu99 -O3 -Wall -Wextra -fopenmp `pkg-config --cflags --libs python3`
-lgomp main.c -o main
Then you need the python code:
# multiplier.py
class multiplier(object):
def __init__(self, factor):
self.factor = factor
def do_multiply(self, x):
return self.factor * x
First question: Does my C code leak memory? Valgrind says it does, but the
memory footprint of the executable is stable while looping?
Second and most important question: When I run this code it sometimes
segementation faults, and sometimes some threads run normal and some other
threads says "Cannot call 'do_multiply'". Sometimes I get the message: Fatal
Python error: GC object already tracked. And some times it even runs normally...
I understand there is some kind of race condition here, where python tries to
refer to some memory that has been already released. But how can I avoid this?
What am I doing wrong? (or, less likely, is this a bug?)
Maybe needless to say, but the code works when compiled w/o OpenMP.
Using Python 3.5.1
Thanks,
-Øystein
--
https://mail.python.org/mailman/listinfo/python-list
Re: calling python from C#...
Python for .NET http://www.zope.org/Members/Brian/PythonNet -- http://mail.python.org/mailman/listinfo/python-list
Re: calling python from C#...
I actually find Python for .NET most convenient http://www.zope.org/Members/Brian/PythonNet I did not try py2exe with these applications, so I cannot comment. -- http://mail.python.org/mailman/listinfo/python-list
Re: calling python from C#...
There is IronPython which compiles to .NET. And there was another project bridging the .NET runtime with the standard Python interpreter of which I forgot the name. Jens -- http://mail.python.org/mailman/listinfo/python-list
Re: calling python from C#...
I use python for .NET in some applications... I've always used it for using c# code from python (but i had some problems using it with py2exe and win2000). I'll try using it the other way (c# calling python). I also thought about trying ironpython, but it's too young for my work projects... The COM aproach seemed more stable to me what do u think? -- http://mail.python.org/mailman/listinfo/python-list
Re: calling python from C#...
Writing COM Servers is not hard http://www.python.org/windows/win32com/QuickStartServerCom.html IronPython is the other way. Choose COM Server approach if you are using the a lot of standard library functions (as I recall IronPython is not complete here, yet). Definitely choose this approach if you are using C extensions to Python. IronPython is good for simpler Python tasks at this moment. Expect this to change in the future. Of course, there is always XML-RPC, SOAP, CORBA etc for language independence if you don't mind Client/Server setup. You can also write code in C# and drive it from Python. Check Python for .NET. -- http://mail.python.org/mailman/listinfo/python-list
calling python from C#...
Hi everybody, I need write an application in c#, but i wnat to use some functions that i have already written in python. Is there an "easy" way to call my python code from c#? I must use COM or there are other ways? And, if COM is the only way, where can i find some tutorials or examples? I've never wrote com server and i really don't know where and how to start tnx Fabio -- http://mail.python.org/mailman/listinfo/python-list
Re: segfault when calling Python from C thread
Fredrik Lundh wrote:
> Greg Chapman wrote:
>
> > Your callback function needs to hold the Python GIL (and have a
> > vaild threadstate) before it calls any Python C-API functions.
> > Change the last part of it to:
> >
> >PyGILState_STATE state;
> >
> >/* ... */
> >
> >/* Time to call the callback */
> >
> >state = PyGILState_Ensure();
> >
> >arglist = Py_BuildValue("(s)", str);
> >result = PyEval_CallObject(my_callback, arglist);
> >Py_DECREF(arglist);
> >if(result == NULL)
> >return;
> >Py_DECREF(result);
> >
> >PyGILState_Release(state);
> > }
>
> you might wish to make sure you release the GIL even if the callback
> raises an exception...
>
>
Argh, thanks for catching that. You probably put that too politely
though (in case anyone sees this who might think that is optional): one
should absolutely make sure all code paths which call PyGILState_Ensure
have a matching call to PyGILState_Release.
---
Greg Chapman
--
http://mail.python.org/mailman/listinfo/python-list
Re: segfault when calling Python from C thread
Greg Chapman wrote:
> Your callback function needs to hold the Python GIL (and have a vaild
> threadstate) before it calls any Python C-API functions. Change the
> last part of it to:
>
>PyGILState_STATE state;
>
>/* ... */
>
>/* Time to call the callback */
>
>state = PyGILState_Ensure();
>
>arglist = Py_BuildValue("(s)", str);
>result = PyEval_CallObject(my_callback, arglist);
>Py_DECREF(arglist);
>if(result == NULL)
>return;
>Py_DECREF(result);
>
>PyGILState_Release(state);
> }
you might wish to make sure you release the GIL even if the callback
raises an exception...
--
http://mail.python.org/mailman/listinfo/python-list
Re: segfault when calling Python from C thread
Travis Berg wrote:
>
> I'm running into a problem when trying to perform a callback to a
> Python function from a C extension. Specifically, the callback is
> being made by a pthread that seems to cause the problem. If I call
> the callback from the parent process, it works fine. The PyObject is
> static, and holds the same value in both Parent and thread, so I'm at
> a loss as to what would be different in the pthread from the parent
> that would cause a segfault on the callback. The machine specifics
> are an x86 intel processor with RedHat linux.
>
>
> /* calling callback */
> void callback(char * str) {
> PyObject *arglist;
> PyObject *result;
> if(str == NULL)
> return;
>
> if(my_callback == NULL) {
> printf("no callback function provided, returning...\n");
> return;
> }
>
> /* Time to call the callback */
> arglist = Py_BuildValue("(s)", str);
> result = PyEval_CallObject(my_callback, arglist);
> Py_DECREF(arglist);
> if(result == NULL)
> return;
> Py_DECREF(result);
> }
Your callback function needs to hold the Python GIL (and have a vaild
threadstate) before it calls any Python C-API functions. Change the
last part of it to:
PyGILState_STATE state;
/* ... */
/* Time to call the callback */
state = PyGILState_Ensure();
arglist = Py_BuildValue("(s)", str);
result = PyEval_CallObject(my_callback, arglist);
Py_DECREF(arglist);
if(result == NULL)
return;
Py_DECREF(result);
PyGILState_Release(state);
}
Also, somewhere in your main thread you should call PyEval_InitThreads
before any of the callback threads execute. (This call is made
automatically if you are creating new threads using Python's thread
module, but if the new threads are created by some C code, you need to
call it yourself.)
---
Greg Chapman
--
http://mail.python.org/mailman/listinfo/python-list
segfault when calling Python from C thread
I'm running into a problem when trying to perform a callback to a Python
function from a C extension. Specifically, the callback is being made by
a pthread that seems to cause the problem. If I call the callback from
the parent process, it works fine. The PyObject is static, and holds the
same value in both Parent and thread, so I'm at a loss as to what would be
different in the pthread from the parent that would cause a segfault on
the callback. The machine specifics are an x86 intel processor with
RedHat linux.
Here is some clips of the C callback code showing how I'm storing the
callback, then what the actual callback function is like. Any ideas?
The function being called is a simply to display the string of text, and
execution never seems to reach back to the Python code at all.
Thanks,
Travis B.
/* callback function to the Python code */
static PyObject * my_callback = NULL;
/* setting callback function */
static PyObject * my_set_callback(PyObject *dummy, PyObject *args)
{
PyObject *result = NULL;
PyObject *temp;
PyObject *arglist;
if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {
if (!PyCallable_Check(temp)) {
PyErr_SetString(PyExc_TypeError,
"parameter must be callable");
return NULL;
}
Py_XINCREF(temp); /* Add a reference to new callback */
Py_XDECREF(my_callback); /* Dispose of previous callback */
my_callback = temp; /* Remember new callback */
/* return "None" */
Py_INCREF(Py_None);
result = Py_None;
}
return result;
}
/* calling callback */
void callback(char * str) {
PyObject *arglist;
PyObject *result;
if(str == NULL)
return;
if(my_callback == NULL) {
printf("no callback function provided, returning...\n");
return;
}
/* Time to call the callback */
arglist = Py_BuildValue("(s)", str);
result = PyEval_CallObject(my_callback, arglist);
Py_DECREF(arglist);
if(result == NULL)
return;
Py_DECREF(result);
}
--
http://mail.python.org/mailman/listinfo/python-list
