Re: python C-api and thread

[Barry Scott via Python-list]

> On 6 Aug 2024, at 07:11, aotto1968 via Python-list  
> wrote:
> 
> I know but I use a thread like a process because the "conversation" between 
> the threads is done by my
> software. a Thread is usually faster to startup (thread-pool) this mean for 
> high-load this is
> significant faster even than fork.

using processes means you are more robust and can survive a process crash.
using async, assuming you do enough I/O, will out perform threads.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python C-api and thread

[aotto1968 via Python-list]

On 06.08.24 04:34, Grant Edwards wrote:

On 2024-08-05, aotto1968 via Python-list  wrote:


Is it possible to run two completely independent Python interpreters
in one process, each using a thread?

By independent, I mean that no data is shared between the
interpreters and thus the C API can be used without any other
"lock/GIL" etc.


No, not using any OS I've ever seen. The usual definition of "threads"
is that they share data, and the definition of "processes" is that
processes don't share data.

How exactly does what you're trying to do differ from runnig two
Python interpreters in two processes?

--
Grant





I know but I use a thread like a process because the "conversation" between the 
threads is done by my
software. a Thread is usually faster to startup (thread-pool) this mean for 
high-load this is
significant faster even than fork.
--
https://mail.python.org/mailman/listinfo/python-list

Re: python C-api and thread (Posting On Python-List Prohibited)

[aotto1968 via Python-list]

On 06.08.24 02:32, Lawrence D'Oliveiro wrote:

On Mon, 5 Aug 2024 23:19:14 +0200, aotto1968 wrote:


Is it possible to run two completely independent Python interpreters in
one process, each using a thread?

By independent, I mean that no data is shared between the interpreters
and thus the C API can be used without any other "lock/GIL" etc.


Seems like yes
.


→ I think that could be a solution.
--
https://mail.python.org/mailman/listinfo/python-list

Re: python C-api and thread

[Chris Angelico via Python-list]

On Tue, 6 Aug 2024 at 08:48, aotto1968 via Python-list
 wrote:
>
> hi,
>
> Is it possible to run two completely independent Python interpreters in one 
> process, each using a thread?
>
> By independent, I mean that no data is shared between the interpreters and 
> thus the C API can be used without any other
> "lock/GIL" etc.
>

You're probably thinking of subinterpreters:

https://peps.python.org/pep-0734/

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python C-api and thread

[Grant Edwards via Python-list]

On 2024-08-05, aotto1968 via Python-list  wrote:

> Is it possible to run two completely independent Python interpreters
> in one process, each using a thread?
>
> By independent, I mean that no data is shared between the
> interpreters and thus the C API can be used without any other
> "lock/GIL" etc.

No, not using any OS I've ever seen. The usual definition of "threads"
is that they share data, and the definition of "processes" is that
processes don't share data.

How exactly does what you're trying to do differ from runnig two
Python interpreters in two processes?

--
Grant



-- 
https://mail.python.org/mailman/listinfo/python-list

python C-api and thread

[aotto1968 via Python-list]

hi,

Is it possible to run two completely independent Python interpreters in one 
process, each using a thread?

By independent, I mean that no data is shared between the interpreters and thus the C API can be used without any other 
"lock/GIL" etc.


mfg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: how to mark a type as subclass of another type

[Dieter Maurer]

Marco Sulla wrote at 2021-11-2 13:43 +0100:
>I already added the address of the type to tp_base, but it does not work.

It worked for me in `dm.incrementalsearch`.

Maybe, you need a special value for `tp_flags`.

Look at the examples.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: how to mark a type as subclass of another type

[Marco Sulla]

*ahem* evidently I didn't check the right package. it works like a charme :D

On Tue, 2 Nov 2021 at 13:43, Marco Sulla  wrote:
>
> I already added the address of the type to tp_base, but it does not work.
>
> On Mon, 1 Nov 2021 at 17:18, Dieter Maurer  wrote:
> >
> > Marco Sulla wrote at 2021-10-31 23:59 +0100:
> > >I have two types declared as
> > >
> > >PyTypeObject PyX_Type = {
> > >PyVarObject_HEAD_INIT(&PyType_Type, 0)
> > >
> > >etc.
> > >
> > >How can I mark one of the types as subclass of the other one? I tried
> > >to use tp_base but it didn't work.
> >
> > Read the "Python/C Api" documentation. Watch out for `tp_base`.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: how to mark a type as subclass of another type

[Marco Sulla]

I already added the address of the type to tp_base, but it does not work.

On Mon, 1 Nov 2021 at 17:18, Dieter Maurer  wrote:
>
> Marco Sulla wrote at 2021-10-31 23:59 +0100:
> >I have two types declared as
> >
> >PyTypeObject PyX_Type = {
> >PyVarObject_HEAD_INIT(&PyType_Type, 0)
> >
> >etc.
> >
> >How can I mark one of the types as subclass of the other one? I tried
> >to use tp_base but it didn't work.
>
> Read the "Python/C Api" documentation. Watch out for `tp_base`.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: how to mark a type as subclass of another type

[Dieter Maurer]

Marco Sulla wrote at 2021-10-31 23:59 +0100:
>I have two types declared as
>
>PyTypeObject PyX_Type = {
>PyVarObject_HEAD_INIT(&PyType_Type, 0)
>
>etc.
>
>How can I mark one of the types as subclass of the other one? I tried
>to use tp_base but it didn't work.

Read the "Python/C Api" documentation. Watch out for `tp_base`.
-- 
https://mail.python.org/mailman/listinfo/python-list

Python C API: how to mark a type as subclass of another type

[Marco Sulla]

I have two types declared as

PyTypeObject PyX_Type = {
PyVarObject_HEAD_INIT(&PyType_Type, 0)

etc.

How can I mark one of the types as subclass of the other one? I tried
to use tp_base but it didn't work.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: How to debug reference leak?

[Chris Angelico]

On Wed, Sep 28, 2016 at 9:25 PM, Gregory Ewing
 wrote:
> Chris Angelico wrote:
>>
>> If you've Py_DECREFed it and then peek into its internals, you're
>> aiming a gun at your foot.
>
>
> That's true. A safer way would be to look at the refcount
> *before* decreffing and verify that it's what you expect.

Exactly, which is presumably what dieter meant by a refcount of 1
being the one you're holding. A refcount of *2* indicates another
reference somewhere.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: How to debug reference leak?

[Gregory Ewing]

Chris Angelico wrote:

If you've Py_DECREFed it and then peek into its internals, you're
aiming a gun at your foot.


That's true. A safer way would be to look at the refcount
*before* decreffing and verify that it's what you expect.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: How to debug reference leak?

[Chris Angelico]

On Wed, Sep 28, 2016 at 6:56 PM, Gregory Ewing
 wrote:
> dieter wrote:
>>
>> dl l  writes:
>>
>>> When I debug in C++, I see the reference count of a PyObject is 1.
>
>>> How can I find out where is referencing this object?
>>
>>
>> Likely, it is the reference, you are holding:
>
>
> Unless you've just Py_DECREFed it, expecting it to go
> away, and the recfcount is still 1, in which case there's
> still another reference somewhere else.

If you've Py_DECREFed it and then peek into its internals, you're
aiming a gun at your foot. In theory, the decref could have been
followed by some other operation (or a context switch) that allocated
another object and happened to use the same address (not as unlikely
as you might think, given that some object types use free lists).
Always have a reference to something before you mess with it.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: How to debug reference leak?

[Gregory Ewing]

dieter wrote:

dl l  writes:


When I debug in C++, I see the reference count of a PyObject is 1.

>> How can I find out where is referencing this object?


Likely, it is the reference, you are holding:


Unless you've just Py_DECREFed it, expecting it to go
away, and the recfcount is still 1, in which case there's
still another reference somewhere else.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: How to debug reference leak?

[dieter]

dl l  writes:

> When I debug in C++, I see the reference count of a PyObject is 1. I don't
> know where is referencing this object. How can I find out where is
> referencing this object?

Likely, it is the reference, you are holding: typically, whenever
you can access a Python object, this object has at least reference count 1
(because it must lie on at least one access path -- the one used by you).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: How to debug reference leak?

[dieter]

dl l  writes:

> Thanks for reply. Is there any function in C to get the reference objects
> of a object? I want to debug where are referencing the object.

Depending on your understanding of "reference objects", this would
be "gc.get_referents" or "gc.get_referrers".

Of course, those are not "C" functions but Python functions in the module
"gc". However, (with some effort) you can use them in "C" as
well.

Most types (at least the standard container types) will have
"gc" support functions which allow to determine the "referents"
of a corresponding object.

Note: those "gc" functions are implemented in "C". But, almost surely,
they are declared "static", i.e. not globally exposed (in order
not to pollute the global symbol namespace).


When I want to use Python functions in C, I write a bit of "Cython" source
and use "Cython" to compile it into "C". Then, I copy and paste
this "C" code into my own "C" application to have the functions
available there.


Important note for debugging:
(Almost) all Python functions expect that they are called only when the
active thread holds the GIL ("Global Interpreter Lock").
Calling Python functions during debugging may violate this restriction
and this can lead to abnormal behaviour.

In single thread applications, the real danger is likely not too large.
In multi thread applications, I have already seen "SIGSEGV"s caused
by this.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: How to debug reference leak?

[dl l]

When I debug in C++, I see the reference count of a PyObject is 1. I don't
know where is referencing this object. How can I find out where is
referencing this object?

2016-09-27 15:47 GMT+08:00 dl l :

> Thanks for reply. Is there any function in C to get the reference objects
> of a object? I want to debug where are referencing the object.
>
> 2016-09-27 15:01 GMT+08:00 dieter :
>
>> dl l  writes:
>> > I want to check the references of an object. Any way to get the
>> references
>> > of an object with Python C API? Like: gc.get_referrs(), is there similar
>> > API in C lib?
>>
>> "gc" is a module. You can import and access modules from the C API.
>> Thus, you can use "gc.get_referers" from "C" code.
>>
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: How to debug reference leak?

[dl l]

Thanks for reply. Is there any function in C to get the reference objects
of a object? I want to debug where are referencing the object.

2016-09-27 15:01 GMT+08:00 dieter :

> dl l  writes:
> > I want to check the references of an object. Any way to get the
> references
> > of an object with Python C API? Like: gc.get_referrs(), is there similar
> > API in C lib?
>
> "gc" is a module. You can import and access modules from the C API.
> Thus, you can use "gc.get_referers" from "C" code.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: How to debug reference leak?

[dieter]

dl l  writes:
> I want to check the references of an object. Any way to get the references
> of an object with Python C API? Like: gc.get_referrs(), is there similar
> API in C lib?

"gc" is a module. You can import and access modules from the C API.
Thus, you can use "gc.get_referers" from "C" code.

-- 
https://mail.python.org/mailman/listinfo/python-list

Python C API: How to debug reference leak?

[dl l]

I want to check the references of an object. Any way to get the references
of an object with Python C API? Like: gc.get_referrs(), is there similar
API in C lib?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C-API: how to define nested classes?

[Stefan Behnel]

Serge WEINSTOCK, 16.05.2013 10:55:
> I'm currently writing a C extension module for python using the "raw" C-API. 
> I would like to be able to define "nested classes" like in the following 
> python code
> 
> 
> class A:
> class B:
> def __init__(self):
> self.i = 2
> def __init__(self):
> self.b = A.B()
> a = A()
> b = A.B()
> print(a.b.i)
> print(b.i)
> 
> 
> How can I create such nested class with the C-API?

Assuming you really mean Python classes and not extension types, you might
get away with defining them separately and then assigning

   A.B = B

Recent Python versions added a __qualname__ attribute for classes that you
might want to fix up in this case.

If you need better compatibility, consider writing your code in Cython.

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python C-API: how to define nested classes?

[88888 Dihedral]

Serge WEINSTOCK於 2013年5月16日星期四UTC+8下午4時55分07秒寫道：
> Hi,
> 
>  
> 
> I'm currently writing a C extension module for python using the "raw" C-API. 
> I would like to be able to define "nested classes" like in the following 
> python code
> 
>  
> 
> 
> 
> class A:
> 
>     class B:
> 
>     def __init__(self):
> 
>     self.i = 2
> 
>     def __init__(self):
> 
>     self.b = A.B()
> 
> a = A()
> 
> b = A.B()
> 
> print(a.b.i)
> 
> print(b.i)
> 
> 
> 
>  
> 
> How can I create such nested class with the C-API?
> 
>  
> 
> Serge WEINSTOCK
> 
>  
> 
> 
> 
> 
> ___
> 
> This e-mail may contain confidential and/or privileged information. If you 
> are not the intended recipient (or have received this e-mail in error) please 
> notify the sender immediately and delete this e-mail. Any unauthorised 
> copying, disclosure or distribution of the material in this e-mail is 
> prohibited.
> 
> 
> Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for 
> additional disclosures.

The nested class is just defining the scope of the nested 
class object that can be  used solely in all outer proper 
classes.

It is not very useful in a language which supports 
dynamical run-time attribute binding in the instance level.

in the outer 
-- 
http://mail.python.org/mailman/listinfo/python-list

Python C-API: how to define nested classes?

[Serge WEINSTOCK]

Hi,

I'm currently writing a C extension module for python using the "raw" C-API. I 
would like to be able to define "nested classes" like in the following python 
code


class A:
class B:
def __init__(self):
self.i = 2
def __init__(self):
self.b = A.B()
a = A()
b = A.B()
print(a.b.i)
print(b.i)


How can I create such nested class with the C-API?

Serge WEINSTOCK


___
This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorised copying, 
disclosure or distribution of the material in this e-mail is prohibited.

Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for additional 
disclosures.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Executing .pyc using python c api

[88888 Dihedral]

On Tuesday, November 29, 2011 5:02:31 PM UTC+8, Ulrich Eckhardt wrote:
> Am 29.11.2011 08:34, schrieb Mrinalini Kulkarni:
> > I need to run .pyc files using python c api. if i do PyImport_Import it
> > executes the script. However, i need to pass a set of variables and
> > their values which will be accessed from within the script. How can this
> > be done.
> 
> I don't think what you want is possible, due to a think-o in your 
> design. Let me explain...
> Firstly, .pyc files are basically the same as .py files, only in a 
> different presentation. Then, PyImport_Import is basically the same as 
> using "import" in a Python program. Now, and that is where your fault 
> lies, importing a module actually means executing that module! For 
> example, the definition of a function is code that when executed will 
> cause a function to be created and attached to the current scope with 
> the according name. This is what makes it so easy to implement local 
> functions that are parametrized by arguments to the outer function. 
> Still, a function is not something that is "static", like in C or Java, 
> but rather the result of executing its function definition.
> 
> Now, how to get around this? The specialty about the import is that the 
> __name__ attribute is not set to "__main__", upon which many scripts 
> already react. So, in order to "prevent execution" (in the sense that 
> you probably mean), you simply wrap the according code in a function. 
> The function definition will then be executed, giving you a function 
> that you can call with the according parameters, but the function itself 
> will not be executed automatically. If you want that to happen when 
> executing the .pyc file directly, check the content of __name__ and call 
> the function if it is "__main__".
> 
> Note that another approach would be introspection, traversing through 
> the namespaces to find out those parameters, but I would consider this 
> solution as hackish if the one above is feasible.
> 
> Good luck!
> 
> Uli

Please use psyco and pyrex and C or whatever that can read saved results in a 
file, or just learn how to replace a hash or a sort in  python's build in 
library of better speed, don't do reference overheads in 
those c type variables that won't overflow and underflow  and used by other 
objects in python. Not trivial but well documented to cheer for a race!

 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Executing .pyc using python c api

[Ulrich Eckhardt]

Am 29.11.2011 08:34, schrieb Mrinalini Kulkarni:

I need to run .pyc files using python c api. if i do PyImport_Import it
executes the script. However, i need to pass a set of variables and
their values which will be accessed from within the script. How can this
be done.


I don't think what you want is possible, due to a think-o in your 
design. Let me explain...
Firstly, .pyc files are basically the same as .py files, only in a 
different presentation. Then, PyImport_Import is basically the same as 
using "import" in a Python program. Now, and that is where your fault 
lies, importing a module actually means executing that module! For 
example, the definition of a function is code that when executed will 
cause a function to be created and attached to the current scope with 
the according name. This is what makes it so easy to implement local 
functions that are parametrized by arguments to the outer function. 
Still, a function is not something that is "static", like in C or Java, 
but rather the result of executing its function definition.


Now, how to get around this? The specialty about the import is that the 
__name__ attribute is not set to "__main__", upon which many scripts 
already react. So, in order to "prevent execution" (in the sense that 
you probably mean), you simply wrap the according code in a function. 
The function definition will then be executed, giving you a function 
that you can call with the according parameters, but the function itself 
will not be executed automatically. If you want that to happen when 
executing the .pyc file directly, check the content of __name__ and call 
the function if it is "__main__".


Note that another approach would be introspection, traversing through 
the namespaces to find out those parameters, but I would consider this 
solution as hackish if the one above is feasible.


Good luck!

Uli
--
http://mail.python.org/mailman/listinfo/python-list

Re: Executing .pyc using python c api

[Stefan Behnel]

Mrinalini Kulkarni, 29.11.2011 08:34:

I need to run .pyc files using python c api. if i do PyImport_Import it
executes the script. However, i need to pass a set of variables and their
values which will be accessed from within the script. How can this be done.


Assuming you have the source as well, you should change your script to be 
usable as an importable module as well as an executable script. Here is how:


http://docs.python.org/tutorial/modules.html#executing-modules-as-scripts

If you do not have the source, however, you are a bit out of luck. In that 
case, what so you mean by "pass a set of variables"? Do you mean command 
line arguments, or do you want to pass in global names that the module can 
use? In the first case, you may need to modify sys.argv. If the latter, you 
could change the globals() of the imported module. However, both are clumsy 
solutions, and if possible at all, you should change the module source code 
instead.


Stefan

--
http://mail.python.org/mailman/listinfo/python-list

Executing .pyc using python c api

[Mrinalini Kulkarni]

Hi

I need to run .pyc files using python c api. if i do PyImport_Import it 
executes the script. However, i need to pass a set of variables and 
their values which will be accessed from within the script. How can this 
be done.


thanks,

--
http://mail.python.org/mailman/listinfo/python-list

Re: How to dynamically create a derived type in the Python C-API

[Ethan Furman]

The question is on StackOverflow if you want to answer it directly:
http://stackoverflow.com/questions/8066438

~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list

How to dynamically create a derived type in the Python C-API

[Ethan Furman]

Asking on behalf of Sven Marnach:

-
Assume we have the type Noddy as defined in the tutorial on writing C 
extension modules for Python. Now we want to create a derived type, 
overwriting only the __new__() method of Noddy.


Currently I use the following approach (error checking stripped for 
readability):


PyTypeObject *BrownNoddyType =
(PyTypeObject *)PyType_Type.tp_alloc(&PyType_Type, 0);
BrownNoddyType->tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE;
BrownNoddyType->tp_name = "noddy.BrownNoddy";
BrownNoddyType->tp_doc = "BrownNoddy objects";
BrownNoddyType->tp_base = &NoddyType;
BrownNoddyType->tp_new = BrownNoddy_new;
PyType_Ready(BrownNoddyType);

This works, but I'm not sure if it is The Right Way To Do It. I would 
have expected that I have to set the Py_TPFLAGS_HEAPTYPE flag, too, 
because I dynamically allocate the type object on the heap, but doing so 
leads to a segfault in the interpreter.


I also thought about explicitly calling type() using PyObject_Call() or 
similar, but I discarded the idea. I would need to wrap the function 
BrownNoddy_new() in a Python function object and create a dictionary 
mapping __new__ to this function object, which seems silly.


What is the best way to go about this? Is my approach correct? Is there 
an interface function I missed?

-

~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list

Python - C api related - Concurrent Script Execution

[Silver Interactive]

Dear All,

I have developed a C++ server application which handles requests coming in
from my web server. The server application is capable of handling multiple
users at a time. Each user has a session object. Each user also has some
variables associated with it. These variables are maintained in the c++ side
as python dictionaries as they will be processed by executing scripts by the
function PyRun_SimpleString() with each request. What i realised half-way
through the implementation is that I will probably not be able to execute
python scripts in parallel from the c++ end. Is this thinking correct? One
user makes a request and the c++ code calls the python script which has to
fetch information from a remote server (takes some seconds), but I cannot
have other users script execution wait till this first script returns. Am i
missing a point here. Please let me know if there is a way out of this as I
am stuck big time.

Do let me know if you require any more information.

Help will be appreciated. Thanks!!!

Regards
Diana
-- 
http://mail.python.org/mailman/listinfo/python-list

need to extend this C code so that it initiates this python script (python C api)

[aregee]

hi i need help with extending this http://paste.pound-python.org/show/8918/
c code so that it can initiate this python script 
http://paste.pound-python.org/show/8917/
and export file path here..

btw I tried to run following python code :

#include  //changed this to python2.7/Python.h
didn't work the I changed it to Python.h
  // Error msg : fatal
error: Python.h : no file or directory compilation terminated
 //python-dev and
python2.7-dev is already installed.

void process_expression(char* filename,
int num,
char** exp)
{
FILE*   exp_file;

// Initialize a global variable for
// display of expression results
PyRun_SimpleString("x = 0");

// Open and execute the file of
// functions to be made available
// to user expressions
exp_file = fopen(filename, "r");
PyRun_SimpleFile(exp_file, exp);

// Iterate through the expressions
// and execute them
while(num--) {
PyRun_SimpleString(*exp++);
PyRun_SimpleString("print x");
}
}

int main(int argc, char** argv)
{
Py_Initialize();

if(argc != 3) {
printf("Usage: %s FILENAME EXPRESSION+\n");
return 1;
}
process_expression(argv[1], argc - 1, argv + 2);
return 0;
}


thanks
aregee
Ar
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python/C API Inheritance

[Stefan Behnel]

Евгений Почитаев, 02.02.2011 18:21:

May be someone do class inheritance in Python/C API, I know how create
superclass/subclass. But how I can initialize superclass from subclass
initializer?

struct SuperClass {
 PyObject_HEAD;
 //...
};

struct SubClass {
 PyObject_HEAD;


You need to insert the super class struct here, not a plain new object struct.

In case you want to avoid stumbling over problems like this and instead 
concentrate on getting functionality implemented, take a look at Cython.


http://cython.org

Stefan

--
http://mail.python.org/mailman/listinfo/python-list

Python/C API Inheritance

[Евгений Почитаев]

Hi to all.
May be someone do class inheritance in Python/C API, I know how create
superclass/subclass. But how I can initialize superclass from subclass
initializer?

struct SuperClass {
PyObject_HEAD;
//...
};

struct SubClass {
PyObject_HEAD;
//...
};

static int SubClassInit(SubClass *self, PyObject *args, PyObject
*kwds)
{
// how I should initialize super class?
}

PyTypeObject SuperClassType = {
   //...
};

PyTypeObject SubClassType = {
PyObject_HEAD_INIT(NULL)
0, /* ob_size*/
//...
SubClassInit, /* tp_init */
//...
NULL, /* tp_getset */
&SuperClassType, /* tp_base */
NULL, /* tp_dict */
//...
};

Another question how I can access to superclass from subclass(I need
get access to SuperClass C struct members)?

Thanks.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python/c api

[Stefan Behnel]

Tony, 14.10.2010 20:04:

is the python/c api extensively used? and what world-famous software
use it? thanks!


The Sage math system makes heavy use of it, mostly through Cython.

And the Python standard library has lots of modules that use it, e.g. zlib 
and sqlite3. You can expect that most packages on PyPI that are said to be 
written in other languages than Python use the C-API in one way or another.


Stefan

--
http://mail.python.org/mailman/listinfo/python-list

Re: python/c api

[Diez B. Roggisch]

alex23  writes:

> On Oct 15, 5:53 am, de...@web.de (Diez B. Roggisch) wrote:
>> For example Ableton Live, an audio sequencer.
>
> I _have_ Live and I didn't realise this :O Thanks!

Well, it's not a feature for end-users, it's used internally for some
midi controller mapping stuff. Our "API" so to say is Max4Live.

But Python is used, through Boost::Python.

Diez
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python/c api

[Lawrence D'Oliveiro]

In message , Diez B. Roggisch wrote:

> ... and a lot of embedding python into a software.

Let me mention some notable Free Software that does this: Blender, GIMP and 
Scribus, among ones I’ve messed about with recently. Makes an amazing amount 
of power available.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python/c api

[alex23]

On Oct 15, 5:53 am, de...@web.de (Diez B. Roggisch) wrote:
> For example Ableton Live, an audio sequencer.

I _have_ Live and I didn't realise this :O Thanks!

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python/c api

[Diez B. Roggisch]

Tony  writes:

> hi,
>
> is the python/c api extensively used? and what world-famous software
> use it? thanks!

It is, for a lot of extensions for python, and a lot of embedding python
into a software. For example Ableton Live, an audio sequencer. Arc GIS
has it, and the Eve Online. Many more do, I guess.

Diez
-- 
http://mail.python.org/mailman/listinfo/python-list

python/c api

[Tony]

hi,

is the python/c api extensively used? and what world-famous software
use it? thanks!

tony
-- 
http://mail.python.org/mailman/listinfo/python-list

Creating exception class with custom fields in Python/C API

[ty ty]

Hello, list!

I'm writing a wrapper for C-library. When something goes wrong in that library, 
i can get error details. And i want to assign them to fields of my own 
exception class.

For this purpose, i looked throught Modules/_ctypes/_ctypes.c (in python source 
tree) and implemented same things. Briefly:

 define PyObject * Error in header file, 
 write init and other necessary functions 
 assign them to PyMethodDef array
 initialize class's dict with methods above (in function create_error)
 create new exception with no base class and with dict (in function 
create_error)
 call to create_error and assign Error class to module (PyModule_AddObject)
 throw error with PyErr_SetObject(Error, tpl);, where tpl is tuple with error 
details, which are assigned to Error's fields in init function
All of this is consistent with what i saw in Modules/_ctypes/_ctypes.c, i 
think. But it doesn't work: when i call PyErr_SetObject(Error, tpl);, Python's 
runtime raises error:
TypeError: unbound method __init__() must be called with Error instance as 
first argument (got str instance instead)

And PyObject_Print(Error, stdout, 0); returns:
{'__init__':  ...}

It's item for init function in PyMethodDef array:
{"__init__", myerror_init, METH_VARARGS, "initialize error"}

and it's function's signature:
static PyObject * myerror_init(PyObject * self, PyObject *args)

(python version -- 2.6.4)

Why methods are unbound? And what i've missed? Or what is the right and 
pythonic way to define exception with custom class attributes?

Thanks.

(crosspost from stackoverflow: 
http://stackoverflow.com/questions/3118617/creating-exception-class-with-custom-fields-in-python-c-api
 )
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Extension modules and common routines in Python/C API

[Stefan Behnel]

ty ty, 28.06.2010 08:16:

I'm writing wrapper for C library. This library consist of several
parts. And i want split my extension package into different extension
modules. I think, this is the right way ;-)


Depends. If it's somewhat large or deals with sufficiently distinct 
functionality, it might be worth doing.




But, there are some common
parts that exist in extension package, get_library_version, Error, and
so on. I've decided to create additional module, named core, where these
routines and variables are defined.


Is that supposed to be used by Python code or internally by your modules? 
If it's the latter, I'd just write a plain C module and link it into each 
of the extension modules that use it. Otherwise, a common utility module 
might be ok, but I don't know anything about your real code that would 
suggest either way. You may also consider taking both approaches, i.e. 
write a linked-in module and a public Python wrapper around it.




And when get_library_version, for
example, is used by programmer, Error, in opposite, is used by routines
in another modules. As i mentioned in ( above mail), i use create_error
for adding new exception class with neccessary fields. Now i call
create_error in initcore function, which initialize core module. But, if
i don't import package.core, and only import package.module, when
package.module.function fails , python's runtime throws error


Obviously. Just make sure you always import your core module first. If it 
defines something as basic as the main error class (assuming that's what 
you meant with "Error"), I'd make that the very first thing in the init 
function.


You might also want to take a look at Cython. It's a Python dialect that 
makes writing extension modules easy, so if you have a lot of C code to 
wrap, it'll help you get it done substantially faster than in plain C code.


Stefan

--
http://mail.python.org/mailman/listinfo/python-list

Extension modules and common routines in Python/C API

[ty ty]

Hi, folks! 

I'm writing wrapper for C library. This library consist of several parts. And i 
want split my extension package into different extension modules. I think, this 
is the right way ;-) But, there are some common parts that exist in extension 
package, get_library_version, Error, and so on. I've decided to create 
additional module, named core, where these routines and variables are defined. 
And when get_library_version, for example, is used by programmer, Error, in 
opposite, is used by routines in another modules. As i mentioned in ( above 
mail), i use create_error for adding new exception class with neccessary 
fields. Now i call create_error in initcore function, which initialize core 
module. But, if i don't import package.core, and only import package.module, 
when package.module.function fails, python's runtime throws error:
SystemError: error return without exception set
because, Error is not properly initialize in that point. 

And, if i add:

if(!Error) 
  create_error(); 
in init functions of all my modules, create_error runs several times (for each 
imported module). I think, i get a different Error every time. I can't check 
this because of above issue.

So, my questions is: how can i organize this pattern? Should i add import 
package.core in __init__.py in package dir? Or should i create subclass 
exception of Error in every module?
What is the right and pythonic way for doing that? :) 

Thanks!

(crosspost from 
http://stackoverflow.com/questions/3119026/extension-modules-and-common-routines-in-python-c-api
 )
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API, building a module

[MRAB]

P.F.C. wrote:

I see what your saying.

Is there another way of declaring (at least) the docstring apart from 
the actual array?(it would get messy otherwise)


I had also tried defining the variables as static but got the same 
result (I thought static meant in code memory instead of the heap...?)


I am new to C so I may just be going about this all wrong...


The standard way of providing documentation strings is with PyDoc_STRVAR:

const int ge_test_args = METH_NOARGS;   //The flag 
for this function
PyDoc_STRVAR(ge_test_doc, "Test\nwill print \"test\"");//The 
docstring

static PyMethodDef ge_methods[]={
  {"test", NULL, 0, ge_test_doc},//simply replacing the 
flag and the docstring with a constant variable

  {NULL,NULL}
};



-- Maranatha!
-
PFC aka Fezzik aka GAB


On Fri, Nov 27, 2009 at 10:25 PM, MRAB > wrote:


P.F.C. wrote:

Hello, I'm new to the mailing list but have been using python
for a while.

I am now attempting to embed the interpreter into a C
application but am having an issue when building a module
exposing my application's functions to python.
I do not know if this is the right place to ask for help with
it, if it isn't then please let me know where to go.

the problem I'm having is with making a PyMethodDef array
When I make the array like this it works:

static PyMethodDef ge_methods[]={
 {"test",ge_test,METH_NOARGS,"Test returns 123L"},
 {NULL,NULL}
};

but as soon as I try to build the array from constant variables
like this:

const int ge_test_args = METH_NOARGS;  
//The flag for this function
const char* ge_test_doc = "Test\nwill print \"test\"";  
 //The docstring

static PyMethodDef ge_methods[]={
  {"test",ge_test, ge_test_args, ge_test_doc},  
 //simply replacing the flag and the docstring with a constant

variable
  {NULL,NULL}
};

the compiler then gives the following errors:
./test1.c:74: error: initializer element is not constant
./test1.c:74: error: (near initialization for
‘ge_methods[0].ml_flags’)
./test1.c:74: error: initializer element is not constant
./test1.c:74: error: (near initialization for
‘ge_methods[0].ml_doc’)

I'm using the gcc compiler

This may well be because of my lack of understanding the C
language but I was hoping someone could help me out, or at least
point me in the right direction

I also posted about this at
http://talk.christiandevs.com/viewtopic.php?f=13&t=2521

>

I think it's because C 'const' objects aren't true constants, but are
more like read-only variables; you can initialise them in the
declaration but not assign to them otherwise. Thus what you're actually
trying to do is initialise from a variable, not a constant.



--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API, building a module

[MRAB]

P.F.C. wrote:

Hello, I'm new to the mailing list but have been using python for a while.

I am now attempting to embed the interpreter into a C application but am 
having an issue when building a module exposing my application's 
functions to python.
I do not know if this is the right place to ask for help with it, if it 
isn't then please let me know where to go.


the problem I'm having is with making a PyMethodDef array
When I make the array like this it works:

static PyMethodDef ge_methods[]={
  {"test",ge_test,METH_NOARGS,"Test returns 123L"},
  {NULL,NULL}
};

but as soon as I try to build the array from constant variables like this:

const int ge_test_args = METH_NOARGS;   //The flag 
for this function
const char* ge_test_doc = "Test\nwill print \"test\"";//The 
docstring

static PyMethodDef ge_methods[]={
   {"test",ge_test, ge_test_args, ge_test_doc},//simply 
replacing the flag and the docstring with a constant variable

   {NULL,NULL}
};

the compiler then gives the following errors:
./test1.c:74: error: initializer element is not constant
./test1.c:74: error: (near initialization for ‘ge_methods[0].ml_flags’)
./test1.c:74: error: initializer element is not constant
./test1.c:74: error: (near initialization for ‘ge_methods[0].ml_doc’)

I'm using the gcc compiler

This may well be because of my lack of understanding the C language but 
I was hoping someone could help me out, or at least point me in the 
right direction


I also posted about this at 
http://talk.christiandevs.com/viewtopic.php?f=13&t=2521 




I think it's because C 'const' objects aren't true constants, but are
more like read-only variables; you can initialise them in the
declaration but not assign to them otherwise. Thus what you're actually
trying to do is initialise from a variable, not a constant.
--
http://mail.python.org/mailman/listinfo/python-list

Python C API, building a module

[P.F.C.]

Hello, I'm new to the mailing list but have been using python for a while.

I am now attempting to embed the interpreter into a C application but am
having an issue when building a module exposing my application's functions
to python.
I do not know if this is the right place to ask for help with it, if it
isn't then please let me know where to go.

the problem I'm having is with making a PyMethodDef array
When I make the array like this it works:

static PyMethodDef ge_methods[]={
  {"test",ge_test,METH_NOARGS,"Test returns 123L"},
  {NULL,NULL}
};

but as soon as I try to build the array from constant variables like this:

const int ge_test_args = METH_NOARGS;   //The flag for
this function
const char* ge_test_doc = "Test\nwill print \"test\"";//The
docstring
static PyMethodDef ge_methods[]={
   {"test",ge_test, ge_test_args, ge_test_doc},//simply
replacing the flag and the docstring with a constant variable
   {NULL,NULL}
};

the compiler then gives the following errors:
./test1.c:74: error: initializer element is not constant
./test1.c:74: error: (near initialization for ‘ge_methods[0].ml_flags’)
./test1.c:74: error: initializer element is not constant
./test1.c:74: error: (near initialization for ‘ge_methods[0].ml_doc’)

I'm using the gcc compiler

This may well be because of my lack of understanding the C language but I
was hoping someone could help me out, or at least point me in the right
direction

I also posted about this at
http://talk.christiandevs.com/viewtopic.php?f=13&t=2521

Thank you for any help


-- Maranatha!
-
PFC aka Fezzik aka GAB
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API and references

[Gabriel Genellina]

En Thu, 12 Nov 2009 06:23:54 -0300, lallous  escribió:

Everytime I use PyObject_SetAttrString(obj, attr_name, py_val) and I  
don't need the reference to py_val I should decrement the reference  
after this call?


If you own a reference to py_val, and you don't need it anymore, you must
decrement it. It doesn't matter if you call PyObject_SetAttrString or
whatever, except when the called function says it "steals" a reference.


So for example:

PyObject *py_val = PyInt_FromLong(5)
PyObject_SetAttrString(py_obj, "val", py_val);
Py_DECREF(py_val)

Right?


Yes, because PyInt_FromLong returns a new reference, and you own it.


If so, take sysmodule.c:

if (PyObject_SetAttrString(builtins, "_", Py_None) != 0)
return NULL;

Shouldn't they also call Py_DECREF(Py_None) ?


No, because the reference count of Py_None was not incremented previously;
the code doesn't own a reference to Py_None at that time. It's not the
same as the example above.

Same logic applies to PyDict_SetItemString() and the reference should be  
decrement after setting the item (ofcourse if the value is not needed).


Yes, same as your first example.

--
Gabriel Genellina

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API and references

[Mark Dickinson]

On Nov 12, 9:23 am, "lallous"  wrote:
> Hello,
>
> Everytime I use PyObject_SetAttrString(obj, attr_name, py_val) and I don't
> need the reference to py_val I should decrement the reference after this
> call?

Not necessarily:  it depends where py_val came from.  I find the
'ownership' model described in the docs quite helpful:

http://docs.python.org/c-api/intro.html#reference-count-details

It's probably better to read the docs, but here's a short version:

If you create a reference to some Python object in a function in your
code, you then 'own' that reference, and you're responsible for
getting rid of it by the time the function exits.  There are various
ways this can happen:  you can return the reference, thereby
transferring ownership to the calling function; you can explicitly
discard the reference by calling Py_DECREF; or you can transfer
ownership by calling a function that 'steals' the reference (most
functions in the C-API don't steal references, but rather borrow them
for the duration of the function call).

>
> So for example:
>
> PyObject *py_val = PyInt_FromLong(5)
> PyObject_SetAttrString(py_obj, "val", py_val);
> Py_DECREF(py_val)
>
> Right?

Yes.  Here you've created the reference in the first place, so you
should Py_DECREF it before you exit.  Right after the
PyObject_SetAttrString call is a good place to do this.

>
> If so, take sysmodule.c:
>
>         if (PyObject_SetAttrString(builtins, "_", Py_None) != 0)
>                 return NULL;
>
> Shouldn't they also call Py_DECREF(Py_None) ?

No, I don't think so.  I assume you're looking at the sys_displayhook
function?  This function doesn't create new references to Py_None
(well, except when it's about to return Py_None to the caller), so at
this point it doesn't own any reference to Py_None:  it's not
responsible for decrementing the reference count.

> Same logic applies to PyDict_SetItemString()

Yes.


--
Mark
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API and references

[lallous]

Hello Daniel,

Thanks for the reply.



Everytime I use PyObject_SetAttrString(obj, attr_name, py_val) and I 
don't

need the reference to py_val I should decrement the reference after this
call?


 It really depends on /how/ the object is created. If the
method used to create *py_val* increases the reference count
on the object and another function any other function is
used to increase the reference count, you should use Py_DECREF
or Py_XDECREF.



So for example:

PyObject *py_val = PyInt_FromLong(5)
PyObject_SetAttrString(py_obj, "val", py_val);
Py_DECREF(py_val)

Right?


 In this case is right. *PyInt_FromLong()* returns a new
reference: 'Return value: New reference.', which is increasing
the reference count and PyObject_SetAttrString does it twice,
then you have a reference count of two and you need to decrement
the initial reference counting of the object, or you will have
a memory leak.



[quote]
int PyObject_SetAttr(PyObject *o, PyObject *attr_name, PyObject *v)
Set the value of the attribute named attr_name, for object o, to the value 
v. Returns -1 on failure. This is the equivalent of the Python statement 
o.attr_name = v.

int PyObject_SetAttrString(PyObject *o, const char *attr_name, PyObject *v)
Set the value of the attribute named attr_name, for object o, to the value 
v. Returns -1 on failure. This is the equivalent of the Python statement 
o.attr_name = v.

[/quote]

Looking at the documentation, should I have understood that the passed value 
reference will be incremented and that I should decrement it if I don't need 
it?


Or I should have understood just because of the fact that whenever we have x 
= b (be it from the C api in a form of SetAttr()) then we should know that 
b's reference will be incremented. ?


Because, before this discussion I did not know I should decrease the 
reference after SetAttr()




If so, take sysmodule.c:

if (PyObject_SetAttrString(builtins, "_", Py_None) != 0)
return NULL;

Shouldn't they also call Py_DECREF(Py_None) ?


 No, I think that Py_None do not needs to decrease the
reference count...



None is an object like other objects. I think its reference must be taken 
into consideration too, for instance why there is the convenience macro: 
Py_RETURN_NONE ?


--
Elias 


--
http://mail.python.org/mailman/listinfo/python-list

Python C API and references

[lallous]

Hello,

Everytime I use PyObject_SetAttrString(obj, attr_name, py_val) and I don't 
need the reference to py_val I should decrement the reference after this 
call?


So for example:

PyObject *py_val = PyInt_FromLong(5)
PyObject_SetAttrString(py_obj, "val", py_val);
Py_DECREF(py_val)

Right?

If so, take sysmodule.c:

if (PyObject_SetAttrString(builtins, "_", Py_None) != 0)
return NULL;

Shouldn't they also call Py_DECREF(Py_None) ?

Same logic applies to PyDict_SetItemString() and the reference should be 
decrement after setting the item (ofcourse if the value is not needed).


--
Elias 


--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C api: create a new object class

[samwyse]

On Nov 10, 1:09 pm, "lallous"  wrote:
> Hello
>
> I have 3 questions, hope someone can help:
>
> 1)
> How can I create an instance class in Python, currently I do:
>
> class empty:
>   pass
>
> Then anytime I want that class (which I treat like a dictionary):
>
> o = empty()
> o.myattr = 1
> etc
>
> Is there is a one line syntax to instantiate an instance?

I think that you want this:

class c(object):
def __init__(self, **kwds):
self.__dict__ = kwds

x = c(a=1, b=2)
print x.a, x.b
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python C api: create a new object class

[Benjamin Peterson]

lallous  lgwm.org> writes:
> Is there is a one line syntax to instantiate an instance?

You can't instantiate an instance; it's already instantiated.

> 
> Any other ways than this:
> o = new.classobj('object', (), {})

class x: pass

> How can I, similarly, create an object "o" in C api:

Use PyObject_CallFunction(PyType_Type, [arguments])


> Given a PyObject* is there is a way to tell if one can call 
> PyObject_SetAttrString() on that object w/o getting an error?

No.




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python C api: create a new object class

[Martin v. Löwis]

> How can I create an instance class in Python, currently I do:
> 
> class empty:
>  pass
> 
> Then anytime I want that class (which I treat like a dictionary):
> 
> o = empty()
> o.myattr = 1
> etc
> 
> Is there is a one line syntax to instantiate an instance?
> 
> Any other ways than this:
> o = new.classobj('object', (), {})

Most certainly:

o = empty(1) # or: o = empty(1, etc)

This requires you to define

class empty:
  def __init__(self, myattr, etc):
self.myattr = myattr
etc

> 2)
> 
> How can I, similarly, create an object "o" in C api:
> 
> PyObject *o = what_to_call()

o = PyObject_CallFunction(pointer_to_class_object, "")

> 3)
> 
> Given a PyObject* is there is a way to tell if one can call
> PyObject_SetAttrString() on that object w/o getting an error?
> 
> For example, before calling a PyObject* one can see if it is callable,
> but can I test if an object supports setattr?
> 
> (from C api)

You could check whether the object supports setattr at all, but that
would be pretty useless, since most objects do.

What you want to test (would it support setting "myattr" to the specific
value, at this point) is impossible to test: the object may give you
an exception on every third call only (or only if the value is not
an integer, etc). So just call SetAttr, and clear any exception you
get that you don't want to get.

Regards,
Martin
-- 
http://mail.python.org/mailman/listinfo/python-list

Python C api: create a new object class

[lallous]

Hello

I have 3 questions, hope someone can help:

1)
How can I create an instance class in Python, currently I do:

class empty:
 pass

Then anytime I want that class (which I treat like a dictionary):

o = empty()
o.myattr = 1
etc

Is there is a one line syntax to instantiate an instance?

Any other ways than this:
o = new.classobj('object', (), {})

2)

How can I, similarly, create an object "o" in C api:

PyObject *o = what_to_call()


PyObject_SetAttrString(o, "attrname", py_val)

...

One way I found was first calling PyClass_New() (to create an empty class) 
and then passing the return value to PyInstance_NewRaw to get a new instance 
of that empty class.


Can I achieve the same otherwise?

3)

Given a PyObject* is there is a way to tell if one can call 
PyObject_SetAttrString() on that object w/o getting an error?


For example, before calling a PyObject* one can see if it is callable, but 
can I test if an object supports setattr?


(from C api)

Thank you,

Elias 


--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C/API Problem

[Gabriel Genellina]

En Sat, 12 Sep 2009 07:37:18 -0300, Gianfranco Murador  
 escribió:



Ok, I solved the previous error changing the second argument , but i
have another question. Does PyNode_Compile function store the object
code in the file passed as argument? And it will be execute by python?
I mean, it works if i type 'python prova.pyc'?


No; the filename you pass in is just to set the co_filename attribute of  
code objects, and such things.


--
Gabriel Genellina

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C/API Problem

[Gianfranco Murador]

Ok, I solved the previous error changing the second argument , but i
have another question. Does PyNode_Compile function store the object
code in the file passed as argument? And it will be execute by python?
I mean, it works if i type 'python prova.pyc'?
Thank.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python C/API Problem

[Gianfranco Murador]

Yes, i've done some debugging and the error is on
PyParser_SimpleParseString(..) call. I'll try to change the second
arguments..
Thank to all.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python C/API Problem

[Gabriel Genellina]

En Fri, 11 Sep 2009 15:10:45 -0300, Gianfranco Murador  
 escribió:



int main(int argc, char *argv[]) {
Py_Initialize();

struct _node *node = PyParser_SimpleParseString("from time import
time,ctime\n"
"print 'Today 
is',ctime(time())\n",0);
if(node == NULL)
{
printf("Errore nel parsing");
}else{
PyNode_Compile(node, "./prova.pyc");
PyNode_Free(node);
}

Py_Finalize();
return 0;
}



I compile the file without errors, but when i launch the executable i
have a Segmentation Fault. I'm using the shared library of python
2.5.2 under linux.. Any ideas? It's clear that i do some mistakes, but


The second argument to PyParser_SimpleParseString must be one of  
Py_eval_input, Py_file_input or Py_single_input - not 0.
Anyway, if this is the actual cause of your segfault, seems that a  
validation should be added somewhere...


--
Gabriel Genellina

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C/API Problem

[Philip Semanchuk]

On Sep 11, 2009, at 2:10 PM, Gianfranco Murador wrote:


Hi to all python fans,
i'm trying to run this C source file:
[code]

#include 
#include 
#include 
#include 
#include 


int main(int argc, char *argv[]) {
Py_Initialize();

struct _node *node = PyParser_SimpleParseString("from time import
time,ctime\n"
"print 'Today 
is',ctime(time())\n",0);
if(node == NULL)
{
printf("Errore nel parsing");
}else{
PyNode_Compile(node, "./prova.pyc");
PyNode_Free(node);
}

Py_Finalize();
return 0;
}

[/code]

I compile the file without errors, but when i launch the executable i
have a Segmentation Fault. I'm using the shared library of python
2.5.2 under linux.. Any ideas? It's clear that i do some mistakes, but
where?


Hi G,
With some basic debugging you should at least be able to figure out on  
which line the segfault happens. Have you done that?



bye
Philip
--
http://mail.python.org/mailman/listinfo/python-list

Python C/API Problem

[Gianfranco Murador]

Hi to all python fans,
i'm trying to run this C source file:
[code]

#include 
#include 
#include 
#include 
#include 


int main(int argc, char *argv[]) {
Py_Initialize();

struct _node *node = PyParser_SimpleParseString("from time import
time,ctime\n"
"print 'Today 
is',ctime(time())\n",0);
if(node == NULL)
{
printf("Errore nel parsing");
}else{
PyNode_Compile(node, "./prova.pyc");
PyNode_Free(node);
}

Py_Finalize();
return 0;
}

[/code]

I compile the file without errors, but when i launch the executable i
have a Segmentation Fault. I'm using the shared library of python
2.5.2 under linux.. Any ideas? It's clear that i do some mistakes, but
where?
Thanks.
greetings, G.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Wolfgang Strobl]

Andreas Otto  wrote
about his attempts to install and run Cython:

>  5. and start to build the hello world example
>
>I changed:  print "Hello World"
>to: print("Hello World")-> this is V3

AFAIK Cython doesn't support Python 3, yet. See
http://trac.cython.org/cython_trac/ticket/234

I suggest staying with Python 2.x.  What C compiler are you going to
use, btw.?


-- 
Wir danken für die Beachtung aller Sicherheitsbestimmungen
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Stefan Behnel]

Andreas Otto wrote:
>   just my first step in Cython
> 
>   1. download Cython-0.11.1
> 
>   2. read INSTALL.txt
> 
> <
> (1) Run the setup.py script in this directory
> as follows:
> 
> python setup.py install
> 
> This will install the Pyrex package
> into your Python system.
> <
> 
> Question 1: Why you wall it "Pyrex" package ?

That's a left-over. We fixed the name almost everywhere by now, but that
slipped through.

That shows that the installation instructions haven't been touched in years
(although they should work in general).


>   3. this happen
> 
>> python ./setup.py install
> Traceback (most recent call last):
>   File "./setup.py", line 5, in 
> from Cython.Compiler.Version import version
>   File "/home/dev1usr/src/Cython-0.11.1/Cython/__init__.py", line 2, in
> 
> from Shadow import *
> ImportError: No module named Shadow

Relative imports have changed in Py3, so as you write:

>   4. I use a thread enabled python V3 

That's the core problem. :) Like many other Python packages, Cython doesn't
currently run in Py3. You can run it in any Py2 version starting from 2.3,
and the code it generates will run on 2.3 through 3.1a1. But the Cython
compiler itself requires 2.x. This is being fixed up as we speak, and the
2to3 conversion tool works pretty well on the source by now, so there's a
fair chance that Cython 0.12 will run on Py3.

Stefan
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Andreas Otto]

Andreas Otto wrote:

> alex23 wrote:
>> Did you unpack the Cython archive correctly? Is there a Shadow.py in
>> your src/Cython-0.11.1/Cython/ folder?
> 
> yes

dev1...@linux02:~/ext/x86_64-suse-linux/thread/bin/Cython-0.11.1> ls -al
Cython/Shadow.py
-rw-r--r-- 1 dev1usr users 4130  3. Apr 10:52 Cython/Shadow.py


mfg

  Andreas Otto
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Andreas Otto]

alex23 wrote:

> On Apr 17, 4:22 pm, Andreas Otto  wrote:
>> Question 1: Why you wall it "Pyrex" package ?
> 
> From the first paragraph on the Cython site: "Cython is based on the
> well-known Pyrex, but supports more cutting edge functionality and
> optimizations."
> 
>> >python ./setup.py install
>>
>> Traceback (most recent call last):
>> File "./setup.py", line 5, in 
>> from Cython.Compiler.Version import version
>> File "/home/dev1usr/src/Cython-0.11.1/Cython/__init__.py", line 2, in
>> 
>> from Shadow import *
>> ImportError: No module named Shadow
> 
> Did you unpack the Cython archive correctly? Is there a Shadow.py in
> your src/Cython-0.11.1/Cython/ folder?

yes

> 
>> 4. than I try the second part from INSTALL.txt
>> (2) If you prefer not to modify your Python
>> installation, arrange for the directory
>> containing this file (INSTALL.txt) to be in
>> your PYTHONPATH. On unix, also put the bin
>> directory on your PATH.
>>
>> Traceback (most recent call last):
>> File "setup.py", line 3, in 
>> from Cython.Distutils import build_ext
>> ImportError: No module named Cython.Distutils
> 
> Did you follow the 2nd set of instructions & modify both your
> PYTHONPATH & PATH env vars? Have you confirmed they're correct? This
> is the sort of error I'd expect if those weren't set up properly.

I put the the "Cython-0.11.1" into the PATH and don't modify the
PYTHONPATH

the INSTALL.txt say: On unix, *also* put the bin directory on your PATH

but I think I found the reason I have to set the PYTHONPATH to this
directory *and* setup the PATH. I understand *also* as something like "or"

but now the following error happen:


> PYTHONPATH=~/ext/x86_64-suse-linux/thread/bin/Cython-0.11.1/
PATH=$PATH:~/ext/x86_64-suse-linux/thread/bin/Cython-0.11.1/
python ./setup.py build_ext --inplace
Traceback (most recent call last):
  File "./setup.py", line 3, in 
from Cython.Distutils import build_ext
 
File 
"/home/dev1usr/ext/x86_64-suse-linux/thread/bin/Cython-0.11.1/Cython/__init__.py",
line 2, in 
from Shadow import *
ImportError: No module named Shadow


> 
>> 7. a little bit to much errors for the first step, isn't it ?
> 
> Yes, however none of these happened when I just installed Cython to
> the XP, OSX & Ubuntu boxen I'm working on, so this isn't usual
> behaviour.
> 
> What is your setup? What OS?

SuSE 10.3, I have compiled python on my own and install it in:
~/ext/x86_64-suse-linux/thread


--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[alex23]

On Apr 17, 4:22 pm, Andreas Otto  wrote:
>         Question 1: Why you wall it "Pyrex" package ?

>From the first paragraph on the Cython site: "Cython is based on the
well-known Pyrex, but supports more cutting edge functionality and
optimizations."

> >python ./setup.py install
>
> Traceback (most recent call last):
>   File "./setup.py", line 5, in 
>     from Cython.Compiler.Version import version
>   File "/home/dev1usr/src/Cython-0.11.1/Cython/__init__.py", line 2, in
> 
>     from Shadow import *
> ImportError: No module named Shadow

Did you unpack the Cython archive correctly? Is there a Shadow.py in
your src/Cython-0.11.1/Cython/ folder?

>   4. than I try the second part from INSTALL.txt
> (2) If you prefer not to modify your Python
>     installation, arrange for the directory
>     containing this file (INSTALL.txt) to be in
>     your PYTHONPATH. On unix, also put the bin
>     directory on your PATH.
>
> Traceback (most recent call last):
>   File "setup.py", line 3, in 
>     from Cython.Distutils import build_ext
> ImportError: No module named Cython.Distutils

Did you follow the 2nd set of instructions & modify both your
PYTHONPATH & PATH env vars? Have you confirmed they're correct? This
is the sort of error I'd expect if those weren't set up properly.

>   7. a little bit to much errors for the first step, isn't it ?

Yes, however none of these happened when I just installed Cython to
the XP, OSX & Ubuntu boxen I'm working on, so this isn't usual
behaviour.

What is your setup? What OS?
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Andreas Otto]

Hi,

  just my first step in Cython

  1. download Cython-0.11.1

  2. read INSTALL.txt

<
(1) Run the setup.py script in this directory
as follows:

python setup.py install

This will install the Pyrex package
into your Python system.
<

Question 1: Why you wall it "Pyrex" package ?

  3. this happen

>
>python ./setup.py install
Traceback (most recent call last):
  File "./setup.py", line 5, in 
from Cython.Compiler.Version import version
  File "/home/dev1usr/src/Cython-0.11.1/Cython/__init__.py", line 2, in

from Shadow import *
ImportError: No module named Shadow

  4. I use a thread enabled python V3 
<

Question 2: What I have to do ?

  4. than I try the second part from INSTALL.txt

>
(2) If you prefer not to modify your Python
installation, arrange for the directory
containing this file (INSTALL.txt) to be in
your PYTHONPATH. On unix, also put the bin
directory on your PATH.
<

  5. and start to build the hello world example

I changed:  print "Hello World"
to: print("Hello World")-> this is V3

  6. but got the following error

> cat setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension("pymsgque", ["helloworld.pyx"])]
)

> python setup.py build_ext --inplace   
Traceback (most recent call last):
  File "setup.py", line 3, in 
from Cython.Distutils import build_ext
ImportError: No module named Cython.Distutils

  7. a little bit to much errors for the first step, isn't it ?


mfg

  Andreas Otto
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Stefan Behnel]

Andreas Otto wrote:
>   the problem with such kind of framework is usually
>   that you start with the easy stuff and than (after a couple
>   of days/weeks) you come to the difficult stuff and you
>   have to figure out that this kind of problem does not
>   fit into the tool.

That is a very common problem with so-called "wrapper generators", such as
SWIG, PyBindGen, sip, and some others. They get you to a 1:1 wrapper
quickly, but as soon as you try to abstract your wrapper from the
underlying C API, you start reaching the limits of the tools almost as
quickly (or at least have a hard time pushing the tool the way you want).


>   stuff what I do is:
> 
> 1. create objects on the fly as connection handle
> 2. do callbacks from C to Python
> 3. create and delete threads or manage to continue
> work after an fork 
> 4. is server mode start an event-loop and never come
> back

Not uncommon for a C library wrapper at all. I assume that the "server
mode" event-loop calls back into Python from time to time? You will sooo
love the "with gil" function feature in Cython...


>   what I want:
> 
> I want to use this tool but still be able to intermix
> parts of my "C" helper code with the python code
> 
>   question:
> 
> it is possible to write C and python code into the 
> same file ?

I'm glad you asked :)

That's basically what the Cython language is all about. Think of it as a
programming language that is almost Python, but at the same time allows you
to work with C data types and do direct calls into C code. All of that gets
translated into very fast C code that you can easily hand-tune to your
needs. You basically have all the freedom of the Python language with all
the freedom of the C language.

Stefan
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Stefan Behnel]

Andreas Otto wrote:
>  I want to make a language binding for an existing C library
> 
> http://libmsgque.sourceforge.net
> 
>   is this possible ?

Quoting the third paragraph on Cython's homepage (i.e. the link I posted):

"""
This makes Cython the ideal language for wrapping external C libraries, and
for fast C modules that speed up the execution of Python code.
"""

If you want a fast wrapper module that is easy to write and maintain, while
having all the freedom to design the Python-level API the way you want, I
don't think there is any better way than Cython.

Stefan
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Diez B. Roggisch]

it is possible to write C and python code into the 
same file ?


Not as such.

And JNI is an atrocity, btw.

But what you can do (if you have a pure C-API, no C++) is to completely 
ditch the C from the equation and go for ctypes. This allows you to 
easily wrap the C-functions in pure python, and then write a simple 
layour for OO-goodness around it.


Callbacks from C to python work without a hitch, and generally it's 
really cool to work with ctypes.


You can also try & use Cython, a Python-like language that makes 
bridging between C and Python easier. I personally don't have any 
experience with it (only it's predecessor Pyrex, which worked well for 
me) - but to be honest: as long as you aren't in there for speed (Cython 
can make things faster, if you restrict yourself to the subset the 
language supports), ctypes is the easiest thing to go for. No compiler, 
no hassle.


Diez
--
http://mail.python.org/mailman/listinfo/python-list

Re: Interrupt Python C API

[googler . 1 . webmaster]

On 16 Apr., 11:08, Piet van Oostrum  wrote:
> > googler.1.webmas...@spamgourmet.com (g1w) wrote:
> >g1w> hi, yes, thats true, Alan Touring told us, so it would be nice to let
> >g1w> the user abort it.
> >g1w> Is there a chance for windows, too?
>
> I don't know. I have no access to Python on Windows. Maybe there is
> setitimer support on Windows. Or maybe you can use the threading.Timer
> object, like this:
>
> import signal, os
> import threading
>
> signalcode = signal.SIGALRM # on Windows, choose one that exists there.
>
> class AlarmError(Exception):
>     pass
>
> def interrupt():
>     os.kill(os.getpid(), signalcode)
>
> def execute(command, timeout):
>     threading.Timer(timeout, interrupt).start()
>     try:
>         exec(command)
>     except AlarmError, e:
>         print e
>         print 'Aborted "%s"' % (command,)
>     print "continue work"
>
> print "The everlasting command"
> execute("while 1: pass", 10)
> print "The End"
>
> --
> Piet van Oostrum 
> URL:http://pietvanoostrum.com[PGP 8DAE142BE17999C4]
> Private email: p...@vanoostrum.org

thx, i will check... :) merci
--
http://mail.python.org/mailman/listinfo/python-list

Re: Interrupt Python C API

[Piet van Oostrum]

> googler.1.webmas...@spamgourmet.com (g1w) wrote:

>g1w> hi, yes, thats true, Alan Touring told us, so it would be nice to let
>g1w> the user abort it.

>g1w> Is there a chance for windows, too?

I don't know. I have no access to Python on Windows. Maybe there is
setitimer support on Windows. Or maybe you can use the threading.Timer
object, like this:

import signal, os
import threading

signalcode = signal.SIGALRM # on Windows, choose one that exists there.

class AlarmError(Exception):
pass

def interrupt():
os.kill(os.getpid(), signalcode)

def execute(command, timeout):
threading.Timer(timeout, interrupt).start()
try:
exec(command)
except AlarmError, e:
print e
print 'Aborted "%s"' % (command,)
print "continue work"

print "The everlasting command"
execute("while 1: pass", 10)
print "The End"

-- 
Piet van Oostrum 
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Andreas Otto]

Yes, you are right ...

  I read more and found the doc about this ...

  the problem I have is something more tricky ...

  I allready have an extension written for java
  and the easyest thing would be use this as template
  and replace the java specific calls with python calls ...

  but the python C API is much more complex than java JNI.

  I don't understand why because the task is allways the 
  same

  the problem is that i have a C api but don't want to
  use a 1 to 1 translation of the C functions to python.

  java generate the stub with java.h and you have to fill
  the stub with code even if it is just a function call
  of a C api function with near the same name.

  for me as user I would prefer the exact same way in python
  to:   1. write a python wrapper class
2. declare all external function with "native"
3. call python_h to create the necessary C header
4. and let the user bill the body
5. this require to have a clean and small language
native interface (JNI fit on one web page 
that's it)

  but i have helper functions which , for example, create 
  threads and python module have to attach to this thread.

  the problem with such kind of framework is usually
  that you start with the easy stuff and than (after a couple
  of days/weeks) you come to the difficult stuff and you
  have to figure out that this kind of problem does not
  fit into the tool.

  stuff what I do is:

1. create objects on the fly as connection handle
2. do callbacks from C to Python
3. create and delete threads or manage to continue
work after an fork 
4. is server mode start an event-loop and never come
back


  what I want:

I want to use this tool but still be able to intermix
parts of my "C" helper code with the python code

  question:

it is possible to write C and python code into the 
same file ?


--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Ben Kaplan]

On Apr 16, 2009, at 1:53 AM, Andreas Otto   
wrote:



Hi,

I want to make a language binding for an existing C library

   http://libmsgque.sourceforge.net

 is this possible ?
--


Not only is itpossible, it's pretty common. All of the major GUI  
toolkits do this. Look at www.swig.org if you don't want to write it  
all manually.





http://mail.python.org/mailman/listinfo/python-list

--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Andreas Otto]

Hi,

 I want to make a language binding for an existing C library

http://libmsgque.sourceforge.net

  is this possible ?
--
http://mail.python.org/mailman/listinfo/python-list

Re: Interrupt Python C API

[googler . 1 . webmaster]

hi, yes, thats true, Alan Touring told us, so it would be nice to let
the user abort it.

Is there a chance for windows, too?
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question to python C API

[Stefan Behnel]

Andreas Otto wrote:
>   I have the following question ...
> 
>   I write a custom "*.init" method and expect a variable number or arguments

What's a "*.init" method? Do you mean SomeType.__init__() ?


>   This are my questions:
> 
>   1.I need something like a for loop to analyse this arguments
> For now I try the "PyArg_ParseTupleAndKeywords" but i missing 
> somehing 
> like an Object-Array return value as "format" 
> 
>   2.can i combine variable args with keywords ?
> because it will fit into my spec to use both together
> 
>   3.I want to retrieve a bool object as int value, where is a format "O"
> and "O!" for a type object  HOWTO put the type into the function 
> from
> above

You might want to take a look at Cython. It allows you to write C
extensions in a Python-like language, so all of the above become trivial, e.g.

cdef class MyExtensionType(object):
def __init__(self, bint some_c_bool):
 print type(self) is MyExtensionType
 print some_c_bool == 1

Cython will translate this to efficient C code for you.

http://cython.org/

Stefan
--
http://mail.python.org/mailman/listinfo/python-list

Question to python C API

[Andreas Otto]

Hi,

  I have the following question ...

  I write a custom "*.init" method and expect a variable number or arguments

  This are my questions:

  1.I need something like a for loop to analyse this arguments
For now I try the "PyArg_ParseTupleAndKeywords" but i missing somehing 
like an Object-Array return value as "format" 

  2.can i combine variable args with keywords ?
because it will fit into my spec to use both together

  3.I want to retrieve a bool object as int value, where is a format "O"
and "O!" for a type object  HOWTO put the type into the function 
from
above
  
  
mfg

  Andreas Otto
--
http://mail.python.org/mailman/listinfo/python-list

Re: Interrupt Python C API

[Piet van Oostrum]

> googler.1.webmas...@spamgourmet.com (g) wrote:

>g> Hi,
>g> I just have a design problem and don't know how to solve it. I call a
>g> function which
>g> executes a simple "PyRun_String(...)" command.

>g> imagine the script while 1: pass is executed so the app would hang. Is
>g> there any chance
>g> to break out this PyRun_String-function? I just searched the forums
>g> for that stuff
>g> but these information are very rare.

>g> Thanks for any suggestions.

I think PyRun_String is similar to exec. If you are on a Unix system you
can use the alarm signal. Here is a pure Python example, but I suspect
you can translate this to C in a straightforward manner.

import signal

class AlarmError(Exception):
pass

def handler(signum, frame):
raise AlarmError, "command lasts too long"

signal.signal(signal.SIGALRM, handler)

def execute(command, timeout):
signal.alarm(timeout); 
try:
exec(command)
except AlarmError, e:
print e
print 'Aborted "%s"' % (command,)
print "continue work"

print "The everlasting command"
execute("while 1: pass", 10)
print "The End"


-- 
Piet van Oostrum 
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
--
http://mail.python.org/mailman/listinfo/python-list

Re: Interrupt Python C API

[Tim Roberts]

googler.1.webmas...@spamgourmet.com wrote:
>
>I just have a design problem and don't know how to solve it. I call a
>function which
>executes a simple "PyRun_String(...)" command.
>
>imagine the script while 1: pass is executed so the app would hang. Is
>there any chance to break out this PyRun_String-function?

How would you know that it was an infinite loop, rather than just a
long-running script?  Alan Turing taught us that you can't tell whether an
arbitrary script will terminate.
-- 
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.
--
http://mail.python.org/mailman/listinfo/python-list

Interrupt Python C API

[googler . 1 . webmaster]

Hi,

I just have a design problem and don't know how to solve it. I call a
function which
executes a simple "PyRun_String(...)" command.

imagine the script while 1: pass is executed so the app would hang. Is
there any chance
to break out this PyRun_String-function? I just searched the forums
for that stuff
but these information are very rare.

Thanks for any suggestions.

Bye.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API String Memory Consumption

[Hrvoje Niksic]

Carl Banks  writes:

> On Apr 9, 11:23 pm, Hrvoje Niksic  wrote:
>> a...@pythoncraft.com (Aahz) writes:
>> > BTW, note that if you're using Python 2.x, range(100) will cause
>> > a "leak" because ints are never freed.  Instead, use xrange().
>>
>> Note that using xrange() won't help with that particular problem.
>
> I think it will because with xrange the integers will not all have
> to exist at one time, so Python doesn't have to increase the size of
> the integer pool to a million.

Good catch!  I stand corrected.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API String Memory Consumption

[Carl Banks]

On Apr 9, 11:23 pm, Hrvoje Niksic  wrote:
> a...@pythoncraft.com (Aahz) writes:
> > BTW, note that if you're using Python 2.x, range(100) will cause
> > a "leak" because ints are never freed.  Instead, use xrange().
>
> Note that using xrange() won't help with that particular problem.

I think it will because with xrange the integers will not all have to
exist at one time, so Python doesn't have to increase the size of the
integer pool to a million.


Carl Banks
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API String Memory Consumption

[Hrvoje Niksic]

a...@pythoncraft.com (Aahz) writes:

> BTW, note that if you're using Python 2.x, range(100) will cause
> a "leak" because ints are never freed.  Instead, use xrange().

Note that using xrange() won't help with that particular problem.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API String Memory Consumption

[Aahz]

In article <41d95468-8f7a-4647-83e1-6df147744...@u8g2000yqn.googlegroups.com>,
k3xji   wrote:
>
>When I run the following function, I seem to have a mem leak, a 20 mb
>of memory
>is allocated and is not freed. Here is the code I run:
>
 import esauth
 for i in range(100):
>
>... ss = esauth.penc('sumer')
>...
 for i in range(100):
>
>... ss = esauth.penc('sumer')
>...

BTW, note that if you're using Python 2.x, range(100) will cause a
"leak" because ints are never freed.  Instead, use xrange().
-- 
Aahz (a...@pythoncraft.com)   <*> http://www.pythoncraft.com/

Why is this newsgroup different from all other newsgroups?
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API String Memory Consumption

[Floris Bruynooghe]

On Apr 7, 2:10 pm, John Machin  wrote:
> On Apr 7, 9:19 pm, MRAB  wrote:
>
>
>
> > k3xji wrote:
> > > Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
> > > the problem resolved. However, I really cannot understand why the
> > > first version does not work. Here is the latest code that has no
> > > problems at all:
>
> > > static PyObject *
> > > penc(PyObject *self, PyObject *args)
> > > {
> > >    PyObject * result = NULL;
> > >    unsigned char *s= NULL;
> > >    unsigned char *buf = NULL;
> > >    unsigned int v,len,i = 0;
>
> > >    if (!PyArg_ParseTuple(args, "s#", &s, &len))
> > >         return NULL;
>
> > >    buf = (unsigned char *) PyMem_Malloc(len);
> > >    if (buf == NULL) {
> > >            PyErr_NoMemory();
> > >            return NULL;
> > >    }
>
> > >         /* string manipulation. */
>
> > >    result = PyString_FromStringAndSize((char *)buf, len);
> > >    PyMem_Free(buf);
> > >    return result;
> > > }

I assume you're doing a memcpy() somewhere in there...  This is also
safer then your first version since the python string can contain an
embeded \0 and the strdup() of the first version would not copy that.
But maybe you're sure your input doesn't have NULLs in them so it
might be fine.

>
> > In general I'd say don't mix your memory allocators. I don't know
> > whether CPython implements PyMem_Malloc using malloc,
>
> The fantastic manual (http://docs.python.org/c-api/
> memory.html#overview) says: """the C allocator and the Python memory
> manager ... implement different algorithms and operate on different
> heaps""".
>
> > but it's better to
> > stick with CPython's memory allocators when writing for CPython.
>
> for the reasons given in the last paragraph of the above reference.

That document explictly says you're allowed to use malloc() and free()
in extensions.  There is nothing wrong with allocating things on
different heaps, I've done and seen it many times and never had
trouble.

Why the original problem ocurred I don't understand either tough.

Regards
Floris
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API String Memory Consumption

[Benjamin Peterson]

Carl Banks  gmail.com> writes:
> However, Python apparently does leak a reference if passed a Unicode
> object; PyArg_ParseTuple automatically creates an encoded string but
> never decrefs it.  (That might be necessary evil to preserve
> compatibility, though.  PyString_AS_STRING does it too.)

Unicode objects cache a copy of themselves as default encoded strings. It is
deallocated when the unicode object its self is.




--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API String Memory Consumption

[John Machin]

On Apr 7, 9:19 pm, MRAB  wrote:
> k3xji wrote:
> > Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
> > the problem resolved. However, I really cannot understand why the
> > first version does not work. Here is the latest code that has no
> > problems at all:
>
> > static PyObject *
> > penc(PyObject *self, PyObject *args)
> > {
> >    PyObject * result = NULL;
> >    unsigned char *s= NULL;
> >    unsigned char *buf = NULL;
> >    unsigned int v,len,i = 0;
>
> >    if (!PyArg_ParseTuple(args, "s#", &s, &len))
> >         return NULL;
>
> >    buf = (unsigned char *) PyMem_Malloc(len);
> >    if (buf == NULL) {
> >            PyErr_NoMemory();
> >            return NULL;
> >    }
>
> >         /* string manipulation. */
>
> >    result = PyString_FromStringAndSize((char *)buf, len);
> >    PyMem_Free(buf);
> >    return result;
> > }
>
> In general I'd say don't mix your memory allocators. I don't know
> whether CPython implements PyMem_Malloc using malloc,

The fantastic manual (http://docs.python.org/c-api/
memory.html#overview) says: """the C allocator and the Python memory
manager ... implement different algorithms and operate on different
heaps""".

> but it's better to
> stick with CPython's memory allocators when writing for CPython.

for the reasons given in the last paragraph of the above reference.

HTH,
John
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API String Memory Consumption

[MRAB]

k3xji wrote:

Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
the problem resolved. However, I really cannot understand why the
first version does not work. Here is the latest code that has no
problems at all:

static PyObject *
penc(PyObject *self, PyObject *args)
{
PyObject * result = NULL;
unsigned char *s= NULL;
unsigned char *buf = NULL;
unsigned int v,len,i = 0;

if (!PyArg_ParseTuple(args, "s#", &s, &len))
return NULL;

buf = (unsigned char *) PyMem_Malloc(len);
if (buf == NULL) {
PyErr_NoMemory();
return NULL;
}

/* string manipulation. */

result = PyString_FromStringAndSize((char *)buf, len);
PyMem_Free(buf);
return result;
}


In general I'd say don't mix your memory allocators. I don't know
whether CPython implements PyMem_Malloc using malloc, but it's better to
stick with CPython's memory allocators when writing for CPython.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API String Memory Consumption

[k3xji]

Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
the problem resolved. However, I really cannot understand why the
first version does not work. Here is the latest code that has no
problems at all:

static PyObject *
penc(PyObject *self, PyObject *args)
{
PyObject * result = NULL;
unsigned char *s= NULL;
unsigned char *buf = NULL;
unsigned int v,len,i = 0;

if (!PyArg_ParseTuple(args, "s#", &s, &len))
return NULL;

buf = (unsigned char *) PyMem_Malloc(len);
if (buf == NULL) {
PyErr_NoMemory();
return NULL;
}

/* string manipulation. */

result = PyString_FromStringAndSize((char *)buf, len);
PyMem_Free(buf);
return result;
}

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API String Memory Consumption

[Carl Banks]

On Apr 7, 12:01 am, k3xji  wrote:
> When I run the following function, I seem to have a mem leak, a 20 mb
> of memory
> is allocated and is not freed. Here is the code I run:
>
> >>> import esauth
> >>> for i in range(100):
>
> ...     ss = esauth.penc('sumer')
> ...
>
> >>> for i in range(100):
>
> ...     ss = esauth.penc('sumer')
> ...
>
> And here is the penc() function.
>
> static PyObject *
> penc(PyObject *self, PyObject *args)
> {
>         unsigned char *s= NULL;
>         unsigned char *buf = NULL;
>         PyObject * result = NULL;
>         unsigned int v,len,i = 0;
>
>         if (!PyArg_ParseTuple(args, "s#", &s, &len))
>         return NULL;
>
>         buf = strdup(s);
>         if (!buf) {
>                 PyErr_SetString(PyExc_MemoryError,
>                         "Out of memory: strdup failed");
>                 return NULL;
>         }
>
>         /*string manipulation*/
>
>         result = PyString_FromString(buf);
>         free(buf);
>         return result;
>
> }
>
> Am I doing something wrong?


It might just be an unfortunate case where malloc keeps allocating
memory higher and higher on the heap even though it frees all the
memory.  And since it doesn't give it back to the OS, it runs out.

However, Python apparently does leak a reference if passed a Unicode
object; PyArg_ParseTuple automatically creates an encoded string but
never decrefs it.  (That might be necessary evil to preserve
compatibility, though.  PyString_AS_STRING does it too.)


Carl Banks
--
http://mail.python.org/mailman/listinfo/python-list

Python C API String Memory Consumption

[k3xji]

When I run the following function, I seem to have a mem leak, a 20 mb
of memory
is allocated and is not freed. Here is the code I run:

>>> import esauth
>>> for i in range(100):

... ss = esauth.penc('sumer')
...
>>> for i in range(100):

... ss = esauth.penc('sumer')
...

And here is the penc() function.

static PyObject *
penc(PyObject *self, PyObject *args)
{
unsigned char *s= NULL;
unsigned char *buf = NULL;
PyObject * result = NULL;
unsigned int v,len,i = 0;

if (!PyArg_ParseTuple(args, "s#", &s, &len))
return NULL;

buf = strdup(s);
if (!buf) {
PyErr_SetString(PyExc_MemoryError,
"Out of memory: strdup failed");
return NULL;
}

/*string manipulation*/

result = PyString_FromString(buf);
free(buf);
return result;

}

Am I doing something wrong?

Thanks,
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C-API Object Allocation

[Stefan Behnel]

> On Feb 21, 2009, at 10:01 AM, William Newbery wrote:
>> Ive been learning the C-API lately so I can write python extensions
>> for some of my c++ stuff.

First thing that comes to (my) mind is that you probably do not want to
write code against the bare C-API. Instead, give Cython a try.

http://cython.org

Stefan
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C-API Object Allocation

[Philip Semanchuk]

On Feb 21, 2009, at 10:01 AM, William Newbery wrote:

Ive been learning the C-API lately so I can write python extensions  
for some of my c++ stuff.


I want to use the new and delete operators for creating and  
destroying my objects.


The problem is python seems to break it into several stages. tp_new,  
tp_init and tp_alloc for creation and tp_del, tp_free and tp_dealloc  
for destruction. However c++ just has new which allocates and fully  
constructs the object and delete which destructs and deallocates the  
object.


Which of the python tp_* methods do I need to provide and what must  
they do to be compatible with python subclassing.


Also I want to be able to create the object directly in c++ eg  
"PyObject *obj = new MyExtensionObject(args);"


Hi William,
You can't use C++ new and delete to create Python objects. Not if you  
intend to use them in Python, anyway. Python has its own memory  
management, reference counting, etc. and C++ knows nothing about that.  
You could probably cook up an ugly hack that might work, but you'd be  
swimming against the tide.


If you want to create Python objects, you need ask Python to do it.  
The function PyObject_New() is probably what you want:

http://docs.python.org/c-api/allocation.html


FYI, there are lists specifically for the Python C API and C++ issues.  
I don't mean to chase you away from here; plenty of people ask C API  
questions here. But the other groups are helpful to know about and,  
having just learned about them myself I'm trying to spread the word.  
The lists are here:

http://mail.python.org/mailman/listinfo


Cheers
Philip
--
http://mail.python.org/mailman/listinfo/python-list

Python C-API Object Allocation

[William Newbery]

Ive been learning the C-API lately so I can write python extensions for some of 
my c++ stuff.

I want to use the new and delete operators for creating and destroying my 
objects.

The problem is python seems to break it into several stages. tp_new, tp_init 
and tp_alloc for creation and tp_del, tp_free and tp_dealloc for destruction. 
However c++ just has new which allocates and fully constructs the object and 
delete which destructs and deallocates the object.


Which of the python tp_* methods do I need to provide and what must they do to 
be compatible with python subclassing.

Also I want to be able to create the object directly in c++ eg "PyObject *obj = 
new MyExtensionObject(args);"



_
Love Hotmail?  Check out the new services from Windows Live! 
http://clk.atdmt.com/UKM/go/132630768/direct/01/--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API (PyObject_CallMethod clear object)

[Gabriel Genellina]

En Mon, 26 Jan 2009 08:47:31 -0200,   
escribió:

On 26 Jan., 03:25, "Gabriel Genellina"  wrote:
En Sun, 25 Jan 2009 23:46:01 -0200,  
 escribió:


> I have a problm with deallocating stuff. I call a function with this
> command:

> PyObject *rvalue = PyObject_CallMethod(obj, "execute","",NULL);

> if(rvalue==NULL)
>     PyErr_Print();
> else
>     Py_DECREF(rvalue);


Yes, you are right. I got this issue not in syntax error, I get it on
a runtime error like this:

t = MyImage()
self.test("hallo")   #test takes an integer so this would throw an
exception.

Here you see, an object of the type MyImage() was created but its not
deleted. 't' is a local reference so how can I do that?


It will be deleted after the last reference to it is released, as every  
object in Python. If the above two lines are the whole function code, this  
will happen when exiting the function. If some other object holds a  
reference to this MyImage instance (by example, by doing xxx.attribute =  
t, or inserting it in a list) then that won't happen until all those  
references are released.
Note that, in Python 2.x, the traceback object holds a reference to all  
local variables -- if you store the traceback somewhere, those objects  
won't be deleted.


--
Gabriel Genellina

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API (PyObject_CallMethod clear object)

[googler . 1 . webmaster]

the hook is, how to delete the locals of this function, maybe thats a
workaround?

thanks and bye.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API (PyObject_CallMethod clear object)

[googler . 1 . webmaster]

On 26 Jan., 03:25, "Gabriel Genellina"  wrote:
> En Sun, 25 Jan 2009 23:46:01 -0200,   
> escribió:
>
>
>
> > I have a problm with deallocating stuff. I call a function with this
> > command:
>
> > PyObject *rvalue = PyObject_CallMethod(obj, "execute","",NULL);
>
> > if(rvalue==NULL)
> >     PyErr_Print();
> > else
> >     Py_DECREF(rvalue);
>
> > Can it be, that something is missing here? Imagine I allocate an
> > object of a type:
>
> > t = MyImage()
> > ,;- syntax_error 129=)/%  #is a syntax error
>
> > How you see I would get a syntaxerror, but an object of the type
> > MyImage() is created. But its deallocated when Py_Finalize() is
> > called. What can I do to deallocate this object after
> > PyObject_CallMethod returns NULL, too? I debug it, so if rvalue!=NULL
> > the object is deallocated, but what can I do if rvalue==NULL=
>
> A syntax error is detected *before* the code runs -- so the object isn't  
> created at all.
>
> --
> Gabriel Genellina

Hi!

Yes, you are right. I got this issue not in syntax error, I get it on
a runtime error like this:

t = MyImage()
self.test("hallo")   #test takes an integer so this would throw an
exception.


Here you see, an object of the type MyImage() was created but its not
deleted. 't' is a local reference so how can I do that?


Thanks!

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python C API (PyObject_CallMethod clear object)

[Gabriel Genellina]

En Sun, 25 Jan 2009 23:46:01 -0200,   
escribió:



I have a problm with deallocating stuff. I call a function with this
command:

PyObject *rvalue = PyObject_CallMethod(obj, "execute","",NULL);

if(rvalue==NULL)
PyErr_Print();
else
Py_DECREF(rvalue);

Can it be, that something is missing here? Imagine I allocate an
object of a type:

t = MyImage()
,;- syntax_error 129=)/%  #is a syntax error

How you see I would get a syntaxerror, but an object of the type
MyImage() is created. But its deallocated when Py_Finalize() is
called. What can I do to deallocate this object after
PyObject_CallMethod returns NULL, too? I debug it, so if rvalue!=NULL
the object is deallocated, but what can I do if rvalue==NULL=


A syntax error is detected *before* the code runs -- so the object isn't  
created at all.


--
Gabriel Genellina

--
http://mail.python.org/mailman/listinfo/python-list

Python C API (PyObject_CallMethod clear object)

[googler . 1 . webmaster]

Hi!

I have a problm with deallocating stuff. I call a function with this
command:

PyObject *rvalue = PyObject_CallMethod(obj, "execute","",NULL);

if(rvalue==NULL)
PyErr_Print();
else
Py_DECREF(rvalue);

Can it be, that something is missing here? Imagine I allocate an
object of a type:

t = MyImage()
,;- syntax_error 129=)/%  #is a syntax error

How you see I would get a syntaxerror, but an object of the type
MyImage() is created. But its deallocated when Py_Finalize() is
called. What can I do to deallocate this object after
PyObject_CallMethod returns NULL, too? I debug it, so if rvalue!=NULL
the object is deallocated, but what can I do if rvalue==NULL=


Thanks a lot
--
http://mail.python.org/mailman/listinfo/python-list

Re: Problem using python C API

[marcglec]

Right, thx for your reply, I completely overlooked "examplemodule.so"
in the building step. Maybe I should sleep more these days :)
Btw, I've now an other problem with PyArg_ParseTuple but I guess
it is better to post in the capi-sig mailing list.
Thx again,
Marc.


>
> On Jan 16, 2009, at 5:31 AM, shi dingan wrote:
>
> > void initexample() {
> >  PyObject *m;
> >  m = Py_InitModule("example", exampleMethods);
> > }
> >
> > When I try to import examplemodule, I obtain the following message:
>  import examplemodule
> > Traceback (most recent call last):
> >  File "", line 1, in ?
> > ImportError: dynamic module does not define init function
> > (initexamplemodule)
>
> Hi Marc,
> The tutorial says, "The initialization function must be named
> initname(), where name is the name of the module". The error message
> you got is very accurate: "module does not define init function
> (initexamplemodule)". Since your module is called "examplemodule",
> Python is looking for a function called "initexamplemodule".
>
> Change your build step to build a module called "example" and not
> "examplemodule" and I think you'll be OK.
>
> bye
> Philip
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>


--
http://mail.python.org/mailman/listinfo/python-list