[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-31 Thread Simon King
Hi Dima, On 2014-12-31, Dima Pasechnik wrote: > perhaps it's faster, development time-wise, to implement paths in a > digraph as arrays of integers, Or did I misunderstand? Are you saying we should implement bounded integer sequences (#15820, part of the latest beta) as arrays rather than as bi

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-31 Thread Simon King
Hi Dima, On 2014-12-31, Dima Pasechnik wrote: > perhaps it's faster, development time-wise, to implement paths in a > digraph as arrays of integers, > and pack them into numpy arrays, than to be bothered with std::vector. By #16435, paths in digraphs are implemented by sequences of bounded inte

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-31 Thread Dima Pasechnik
On Tuesday, 30 December 2014 22:36:41 UTC, Simon King wrote: > > Hi Dima, > > On 2014-12-30, Dima Pasechnik > wrote: > > Isn't it true that numpy arrays integrate well with cython? > > At least if all you need is builtin Python object types (e.g. int or > > float), or arrays of pointers to Py

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-30 Thread Volker Braun
On Wednesday, December 31, 2014 12:15:08 AM UTC+1, Simon King wrote: > > So, why is the name not sage.data_structures.bounded_integer_sequences.h? > Don't know, might be a bug. > I wonder how to include the header. It did not end up in local/include/. Yes, I don't think we have that yet. All

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-30 Thread Simon King
Hi Volker, On 2014-12-30, Volker Braun wrote: > Can you give me a pointer how I can use in C++ the "ctypedef struct >> biseq_s" >> that is defined in sage/data_structures/bounded_integer_sequences.pxd? >> > > You just have to make it "ctypedef public struct biseq_s": > > http://docs.cython.org

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-30 Thread Simon King
Hi Dima, On 2014-12-30, Dima Pasechnik wrote: > Isn't it true that numpy arrays integrate well with cython? > At least if all you need is builtin Python object types (e.g. int or > float), or arrays of pointers to Python objects, they may well be used. > http://docs.scipy.org/doc/numpy/reference/

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-30 Thread Dima Pasechnik
On 2014-12-30, Volker Braun wrote: > --=_Part_8311_2092865628.1419974826414 > Content-Type: multipart/alternative; > boundary="=_Part_8312_1157955583.1419974826414" > > --=_Part_8312_1157955583.1419974826414 > Content-Type: text/plain; charset=UTF-8 > > IMHO you need C/C++ if me

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-30 Thread Volker Braun
IMHO you need C/C++ if memory locality is important for you because Python/Cython does not give you control over memory location; PyObject memory is always managed by Python and will be randomly on the heap. Whereas in C++ you can allocate objects in a contiguous array, or even by "placement ne

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-30 Thread Simon King
Hi Volker, On 2014-12-30, Volker Braun wrote: > In C++ just wrap the PyObject* in a class that does it automatically (RAII) "grep" has told me that src/sage/modular/arithgroup/farey.cpp uses PyObject*, so, I'll try to learn from it. > The Python objects of course are not local in memory, they'

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-30 Thread Volker Braun
On Tuesday, December 30, 2014 3:23:11 PM UTC+1, Simon King wrote: > > [...] But my impression is that addition (basically > merging of two sorted lists into one) is rather complicated when sorted > lists are implemented as arrays. > Its just merge sort, right? > [...] get std::vector work wi

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-30 Thread Simon King
Hi Dima, On 2014-12-30, Dima Pasechnik wrote: >> As long as you only insert at the end (append) I don't see why you wouldn't >> use an array. You can append to std::vector in amortized constant time. > > No, the whole point of geobuckets is to make addition fast, and > only moderately more memo

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-30 Thread Dima Pasechnik
On 2014-12-29, Volker Braun wrote: > --=_Part_7027_1620461608.1419885835815 > Content-Type: multipart/alternative; > boundary="=_Part_7028_710634402.1419885835815" > > --=_Part_7028_710634402.1419885835815 > Content-Type: text/plain; charset=UTF-8 > > On Monday, December 29, 201

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-29 Thread Volker Braun
On Monday, December 29, 2014 7:47:18 PM UTC+1, Simon King wrote: > > [...] I suppose I should first read more on typed memoryviews, > right? To me, it looks like the right Cython tool to create arrays of > objects (assuming that I will also turn my C struct terms into objects). > I haven't used

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-29 Thread Simon King
Hi Volker, On 2014-12-29, Volker Braun wrote: >> My reference is "The Geobucket Data Structure for Polynomials" of Thomas >> Yan. His buckets are decreasingly sorted lists of terms. He doesn't >> mention arrays. Do you have a different reference in mind? Or do you >> think I misunderstood what

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-29 Thread Volker Braun
On Monday, December 29, 2014 5:08:23 PM UTC+1, Simon King wrote: > > What do you mean by doubling the number of Python objects? Do you mean: > "The coefficients are Python objects anyway, so, what's the problem if > the terms become Python objects, too?" > Yes, that is what I meant. My refere

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-29 Thread Simon King
Hi Volker, first of all, to the OP, sorry that the original topic of the thread has changed. On 2014-12-29, Volker Braun wrote: > On Monday, December 29, 2014 2:27:49 AM UTC+1, Simon King wrote: >> >> In fact, a year or two ago, I had experimental code where I did follow >> the aproach to use

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-29 Thread Volker Braun
On Monday, December 29, 2014 2:27:49 AM UTC+1, Simon King wrote: > > In fact, a year or two ago, I had experimental code where I did follow > the aproach to use Python objects (of course cythoned). It tought me much > (about theory), but it wasn't really competitive. > Competitive relative to wh

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-28 Thread Simon King
Hi Volker, On 2014-12-28, Volker Braun wrote: > When starting from scratch I would go with the first option. Cleaner code, > easier to write and debug, and if the coefficient is already a Python > object then the overhead is already bounded by a factor of 2. In fact, a year or two ago, I had e

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-28 Thread Volker Braun
When starting from scratch I would go with the first option. Cleaner code, easier to write and debug, and if the coefficient is already a Python object then the overhead is already bounded by a factor of 2. If that is not fast enough then I would implement the polynomials as C++ templates and s

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-28 Thread Simon King
Hi Nils, On 2014-12-28, Nils Bruin wrote: > That's not immediately clear. An array of pointers is a little more compact > than a linked list, and allocating an array at once should be a little more > efficient than allocating multiple nodes of a linked list. An array of fixed size is no option

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-28 Thread Nils Bruin
On Sunday, December 28, 2014 10:44:51 AM UTC-8, Simon King wrote: > > Hi! > > On 2014-12-28, Jeroen Demeyer > > wrote: > > The idea of Cython is really that you shouldn't need to know the C API > > to write good Cython code. If Cython knows the types of the objects, it > > will generate effici

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-28 Thread Simon King
Hi! On 2014-12-28, Jeroen Demeyer wrote: > The idea of Cython is really that you shouldn't need to know the C API > to write good Cython code. If Cython knows the types of the objects, it > will generate efficient code. Then I wonder what to do at #17435. Each term of an element of a path alg

Re: [sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-28 Thread Jeroen Demeyer
On 2014-12-27 13:35, Volker Braun wrote: IMHO you should strive to never use direct CPython API in Cython programs. Almost all uses in Sage are just vestiges of older times where Cython wasn't able to do something. But Cython is much more capable nowadays, use it to write clear and expressive pro

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-27 Thread Volker Braun
On Saturday, December 27, 2014 10:41:22 AM UTC+1, Simon King wrote: > > [...] But I am not sure > if (my_list_pointer)[0] will work, it could be that the cython > compiler complains about accessing an item from a temporary variable. There is no temporary involved, my_list_pointer is just reint

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-27 Thread Simon King
Hi Nathann, On 2014-12-27, Nathann Cohen wrote: > Oh, good to know. But then, would "print ( my_list_pointer)[0]" > do a incref followed by a decref ? If I understand correctly, as soon as you do "O=your_pointer", Python will take care of the object that has just popped up. Hence, when you do "d

Re: [sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-27 Thread Nathann Cohen
Hello ! > Another hint: If P is a pointer to PyObject, then >P > does already increase the reference count. So, no need to call Py(X)INCREF > in that case. Oh, good to know. But then, would "print ( my_list_pointer)[0]" do a incref followed by a decref ? Thanks, Nathann -- You received th

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-27 Thread Simon King
Hi! On 2014-12-26, Nathann Cohen wrote: > Don't forget the decref/incref stuff ! That's for Python's garbage > collector. Otherwise it will not know that the object is being used > and you will be in trouble ! Another hint: If P is a pointer to PyObject, then P does already increase the refer

Re: [sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-26 Thread Volker Braun
You shouldn't tinker with the Python reference counting system if it can be avoided. Just make sure that the Python object that you want to pass to your function is kept alive for as long as you are going to use the C pointer to it.E.g. store it in an attribute of the Python object that is call

Re: [sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-26 Thread Vincent Delecroix
Hello, Note that there is the googlegroups cython-users which might be a better place for cython related questions. Best Vincent 2014-12-26 15:09 UTC+01:00, Nathann Cohen : >> yes yes, that's precisely what I was looking for: >> >> cpdef void *user_param = data >> .. >> data = user_param >

Re: [sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-26 Thread Nathann Cohen
> yes yes, that's precisely what I was looking for: > > cpdef void *user_param = data > .. > data = user_param > > Thanks for the help Don't forget the decref/incref stuff ! That's for Python's garbage collector. Otherwise it will not know that the object is being used and you will be in tro

Re: [sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-26 Thread Jernej Azarija
Hey Nathann, yes yes, that's precisely what I was looking for: cpdef void *user_param = data .. data = user_param Thanks for the help Jernej On Fri, Dec 26, 2014 at 2:19 PM, Nathann Cohen wrote: > Yoo Jernej ! > > Not sure I understood what you wanted, but in > graphs/base/stat

[sage-devel] Re: Cython: passing a Python data structure as a void pointer

2014-12-26 Thread Nathann Cohen
Yoo Jernej ! Not sure I understood what you wanted, but in graphs/base/static_sparse_graph.pyx there is a Cython attribute which points toward a Python list. That's because we have to store the labels of each edge, and because that label can be any python object. If you look for insta