Re: bigint - python long
On 09/10/2012 10:50 PM, Russel Winder wrote: Python 2 and Python 3 are totally different in this regard. I don't have a obvious proposal to make to avoid having PyD for Python 2 and a different PyD for Python 3, but the six package might have some hints as it is intended to support creating Python codebases guaranteed to run under Python 2 and Python 3. Pyd doesn't really have a python codebase, I was talking mostly about PyString_AsString - PyUnicode_Whatever, since even for Python 2, unicode is much more appropriate for anything interfacing with D. For getting pyd to support python 3, its mostly a matter of choosing the right C API functions, and anyways I have version identifiers I can rely on if there is divergence. Wait, CeleriD is python. I might need that six package after all. Thanks for the tip.
Re: bigint - python long
On 09/05/2012 07:10 PM, bearophile wrote: Ellery Newcomer: Yep. Oh, good. Have any suggestions for supported conversion out of the box? There are several important cases, like: Some D lazy ranges == Python lazy iterators/generators array.array == D arrays NumPy arrays == D arrays Welp. I started on NumPy arrays == D arrays, and it turned out to be pretty easy. It's in its own function; maybe I'll put it in pyd.extras or something. But now I have just about all of the above cases working. Bearophile: would you be interested in contributing some code showcasing what we can do with numpy? Just, say, a D function operating on D arrays that does something that maybe numpy doesn't have built in.
Re: D and SCons [ was Re: bigint - python long ]
Am Sun, 09 Sep 2012 12:55:19 -0700 schrieb Brad Roberts bra...@puremagic.com: On 9/9/2012 1:15 AM, Johannes Pfau wrote: Am Sat, 08 Sep 2012 16:25:49 +0100 schrieb Russel Winder rus...@winder.org.uk: On Sat, 2012-09-08 at 07:20 -0700, Ellery Newcomer wrote: […] Okay, here: https://bitbucket.org/ariovistus/deimos-elfutils/overview I have some code with a working makefile and a nonworking SConstruct file. I believe the issue is the header files have pragma(lib, X) in them, and a single call to dmd links the appropriate lib in, but scons' link step loses that information. Do you have any intention of supporting pragma(lib) in scons? If that is valid Dv2 and SCons doesn't deal with it then the SCons D tools are broken and need fixing. Is there a tiny project replicating this that I can turn into a unit/system test. The red so caused will necessitate action :-) Please note that pragma(lib) is an evil feature. For example it will never work in gdc. It's not impossible and never is rather defeatist. Using the frontend as is and grabing the json output, part of which includes the pragmas, would be easy. Then invoking gdc with the appropriate options to get the library linked in. rdmd is a good example of this sort of process. Is there a special flag to enable pragmas for the json output? It does not work with gdc right now, but it should be possible to make it work. Sorry, I should have said 'It'll _probably_ never be supported in gdc'. There are some possible solutions but: * It must be good enough to get approved when gdc is merged into gcc. (remember it must be portable and gpl and you can't use stdout/stdin...) * Someone would have to implement the solution. I guess Iain had his reasons not to implement it so somebody else would have to do that. Of course you can always try to make it work with external build tools. But a solution _in_ gdc seems not very likely. I don't want to badmouth the pragma(lib) feature, in some cases it's nice to have (mainly building simple script-like programs with few source files). But for bigger projects, pragma(lib) makes things difficult (incremental compilation; build tools usually check if a library is available before trying to link against it so they can put out a nice warning. pragma(lib) in dmd subverts this feature; can't specify linker path with pragma lib, can't specify static vs dynamic linking, ...). The C/C++ architecture splits compilation and linking. Trying to conflate those concepts as pragma(lib) does, might even be a good idea(other languages have done it for some time now). But as we have to deal with tools that were designed for C/C++ (linkers, gcc) we'll always hit some issues with pragma(lib).
Re: D and SCons [ was Re: bigint - python long ]
Am Mon, 10 Sep 2012 14:48:30 +0200 schrieb Johannes Pfau nos...@example.com: Sorry, I should have said 'It'll _probably_ never be supported in gdc'. There are some possible solutions but: * It must be good enough to get approved when gdc is merged into gcc. (remember it must be portable and gpl and you can't use stdout/stdin...) * Someone would have to implement the solution. I guess Iain had his reasons not to implement it so somebody else would have to do that. Of course you can always try to make it work with external build tools. But a solution _in_ gdc seems not very likely. For reference: Here's the gdc bug report for pragma(lib): http://d.puremagic.com/issues/show_bug.cgi?id=1690 Filed 2007, closed 2012 as RESOLVED/WONTFIX.
Re: bigint - python long
On 09/05/2012 07:10 PM, bearophile wrote: NumPy arrays == D arrays I've been thinking about this one a bit more, and I am not sure it belongs in pyd. First, the conversion is not symmetric. One can convert a numpy.ndarray to a d array like so: PyObject* ndarray; double[][] matrix = d_type!(double[][])(ndarray); however, going back PyObject* res = _py(matrix); It is not at all clear that the user wants res to be a numpy.ndarray. The problem is partially that d arrays would be overloaded to a few too many things (list, str, array, any iterable, any buffer). That last one is a doozy. d_type never actually touches ndarray's type, so _py can hardly know what to use to convert matrix. (what if ndarray is actually a foo.BizBar matrix?) I could just specialize _py for numpy.ndarrays, defaulting to lists of lists (which is what we do already), but I kinda want a specialized type for numpy.ndarrays. Also, all these conversions imply data copying; is this reasonable for numpy arrays? It is easy enough to get a void* and shape information out of the ndarray, but building a decent matrix type out of them is not trivial. Is there a good matrix library for D that would be suitable for this? Oh yeah, also: rectangular matrices. For static arrays, the conversion is 1 memcpy. For dynamic arrays: lots of memcpys. I suppose I could abuse slicing much.
Re: bigint - python long
Ellery Newcomer: I've been thinking about this one a bit more, and I am not sure it belongs in pyd. I understand. The point of Pyd is to interface D and Python, while NumPy is something external. So if you find difficulties just keep it out. Adding it later is possible. Bye, bearophile
Re: bigint - python long
On 09/10/2012 12:11 PM, bearophile wrote: I understand. The point of Pyd is to interface D and Python, while NumPy is something external. So if you find difficulties just keep it out. Adding it later is possible. Thing is, pyd will convert a ndarray to d array already, it just won't do it as quickly as it could if it made use of the underlying c array, and _py(d_type!(double[][])(ndarray)) will result in a list of lists. So it's really a question of should I add more oddness to an already odd situation. OT Bugger, I'm going to have to go through pyd and replace all usages of str with unicode. /OT
Re: bigint - python long
On Mon, 2012-09-10 at 15:54 -0700, Ellery Newcomer wrote: […] OT Bugger, I'm going to have to go through pyd and replace all usages of str with unicode. /OT Python 2 and Python 3 are totally different in this regard. I don't have a obvious proposal to make to avoid having PyD for Python 2 and a different PyD for Python 3, but the six package might have some hints as it is intended to support creating Python codebases guaranteed to run under Python 2 and Python 3. It is a pity the world doesn't just spontaneously switch to Python 3 so that Python 2 is just a we used to use that technology. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: D and SCons [ was Re: bigint - python long ]
Am Sat, 08 Sep 2012 16:25:49 +0100 schrieb Russel Winder rus...@winder.org.uk: On Sat, 2012-09-08 at 07:20 -0700, Ellery Newcomer wrote: […] Okay, here: https://bitbucket.org/ariovistus/deimos-elfutils/overview I have some code with a working makefile and a nonworking SConstruct file. I believe the issue is the header files have pragma(lib, X) in them, and a single call to dmd links the appropriate lib in, but scons' link step loses that information. Do you have any intention of supporting pragma(lib) in scons? If that is valid Dv2 and SCons doesn't deal with it then the SCons D tools are broken and need fixing. Is there a tiny project replicating this that I can turn into a unit/system test. The red so caused will necessitate action :-) Please note that pragma(lib) is an evil feature. For example it will never work in gdc.
Re: D and SCons [ was Re: bigint - python long ]
On Sun, 2012-09-09 at 10:15 +0200, Johannes Pfau wrote: […] Please note that pragma(lib) is an evil feature. For example it will never work in gdc. So this is a DMD-only (*) feature and not a feature of the D programming language per se? (*) and hence LDC. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: D and SCons [ was Re: bigint - python long ]
On 9/9/2012 1:15 AM, Johannes Pfau wrote: Am Sat, 08 Sep 2012 16:25:49 +0100 schrieb Russel Winder rus...@winder.org.uk: On Sat, 2012-09-08 at 07:20 -0700, Ellery Newcomer wrote: […] Okay, here: https://bitbucket.org/ariovistus/deimos-elfutils/overview I have some code with a working makefile and a nonworking SConstruct file. I believe the issue is the header files have pragma(lib, X) in them, and a single call to dmd links the appropriate lib in, but scons' link step loses that information. Do you have any intention of supporting pragma(lib) in scons? If that is valid Dv2 and SCons doesn't deal with it then the SCons D tools are broken and need fixing. Is there a tiny project replicating this that I can turn into a unit/system test. The red so caused will necessitate action :-) Please note that pragma(lib) is an evil feature. For example it will never work in gdc. It's not impossible and never is rather defeatist. Using the frontend as is and grabing the json output, part of which includes the pragmas, would be easy. Then invoking gdc with the appropriate options to get the library linked in. rdmd is a good example of this sort of process.
Re: bigint - python long
On Sat, 2012-09-08 at 00:58 +0200, bearophile wrote: […] If you support NumPy well and efficiently through Pyd, I think some people will start using the D language just for this :-) NumPy is good but it usually forces you to program vectorially, like Matlab. Sometimes this is not nice, so people write some routines in Fortran, C, etc. With Pyd some scientific programmers maybe will want to use D instead. The idea of a Python/NumPy/D combination, especially if connected to OpenCL does have great appeal. NumPy and D for computation and Python for data visualization and coordination would be an improvement over the currently canonical Python/NumPy/C/C++/Fortran. But there is a large inertia to the system that has built up in the bioinformatics and HEP communities. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: bigint - python long
On Fri, 2012-09-07 at 15:21 -0700, Ellery Newcomer wrote: On 09/06/2012 12:07 AM, Russel Winder wrote: […] just used your scons fork to build the pyd embedded unittests. works pretty nice Splendid :-) If you see any problems or glitches, let me know and/or post a bug report. Although I said use the BitBucket issue system, that was a hang over from the days of a totally separate package for the D system. Now that D support is being evolved as a fork, it has been pointed out to me I should have pointed people to the SCons issue tracker http://scons.tigris.org/project_issues.html for posting reports about D tooling issues as they are in the SCons core. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: bigint - python long
On 09/08/2012 03:09 AM, Russel Winder wrote: On Fri, 2012-09-07 at 15:21 -0700, Ellery Newcomer wrote: On 09/06/2012 12:07 AM, Russel Winder wrote: […] just used your scons fork to build the pyd embedded unittests. works pretty nice Splendid :-) Okay, here: https://bitbucket.org/ariovistus/deimos-elfutils/overview I have some code with a working makefile and a nonworking SConstruct file. I believe the issue is the header files have pragma(lib, X) in them, and a single call to dmd links the appropriate lib in, but scons' link step loses that information. Do you have any intention of supporting pragma(lib) in scons?
Re: D and SCons [ was Re: bigint - python long ]
On Sat, 2012-09-08 at 07:20 -0700, Ellery Newcomer wrote: […] Okay, here: https://bitbucket.org/ariovistus/deimos-elfutils/overview I have some code with a working makefile and a nonworking SConstruct file. I believe the issue is the header files have pragma(lib, X) in them, and a single call to dmd links the appropriate lib in, but scons' link step loses that information. Do you have any intention of supporting pragma(lib) in scons? If that is valid Dv2 and SCons doesn't deal with it then the SCons D tools are broken and need fixing. Is there a tiny project replicating this that I can turn into a unit/system test. The red so caused will necessitate action :-) -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: bigint - python long
On 09/06/2012 09:48 AM, Ellery Newcomer wrote: On 09/05/2012 11:19 PM, Jacob Carlborg wr Associative arrays? check. eh, that was check as in yes, not check as in look it up yourself. didn't seem ambiguous at the time.
Re: bigint - python long
On 09/06/2012 12:07 AM, Russel Winder wrote: I am guessing this is interfacing to CPython, remember there is also PyPy and ActiveState, they have different ways of doing things. Well the ActiveState C API will be very close to the CPython C API, but PyPy (which is the best Python 2.7 just now) doesn't have a C API since it is written in RPython. Yep, CPython for now. If activestate supports the CPython C API, then supporting it will probably just be a matter of linkage. Googling, it looks like PyPy is/was trying to grow some CPython C API compatibility, so hopefully we can squeeze it in at some point. Oh yes. NumPy is basically a subsystem that Python applications make calls into. Although the data structure can be accessed and amended, algorithms on the data structures should never be written in Python, they should always be function calls into the NumPy framework. I had a look at numpy, and at least ndarray supports the new style buffer interface (which is pretty freaking complicated), so convertion is totally doable. I'll fold in support as soon as I can get dmd to stop stabbing me in the face. just used your scons fork to build the pyd embedded unittests. works pretty nice
Re: bigint - python long
Ellery Newcomer: I had a look at numpy, and at least ndarray supports the new style buffer interface (which is pretty freaking complicated), so convertion is totally doable. I'll fold in support as soon as I can get dmd to stop stabbing me in the face. If you support NumPy well and efficiently through Pyd, I think some people will start using the D language just for this :-) NumPy is good but it usually forces you to program vectorially, like Matlab. Sometimes this is not nice, so people write some routines in Fortran, C, etc. With Pyd some scientific programmers maybe will want to use D instead. Bye, bearophile
Re: bigint - python long
On 2012-09-06 04:10, bearophile wrote: There are several important cases, like: Some D lazy ranges == Python lazy iterators/generators array.array == D arrays NumPy arrays == D arrays Associative arrays? -- /Jacob Carlborg
Re: bigint - python long
Sorry, I missed earlier bits of this thread…
On Wed, 2012-09-05 at 19:37 -0700, Ellery Newcomer wrote:
On 09/05/2012 07:10 PM, bearophile wrote:
Some D lazy ranges == Python lazy iterators/generators
I'll look into this one.
array.array == D arrays
just checked, looks like we have it:
PyStmts(q{from array import array; a = array('i', [44,33,22,11]);},
testing);
assert(PyEval!(int[])(a, testing) == [44,33,22,11]);
I think if the python object is iterable, it can be converted to array.
I am guessing this is interfacing to CPython, remember there is also
PyPy and ActiveState, they have different ways of doing things. Well
the ActiveState C API will be very close to the CPython C API, but PyPy
(which is the best Python 2.7 just now) doesn't have a C API since it is
written in RPython.
Matrices might pose a problem. But user can define custom conversions if
need be.
NumPy arrays == D arrays
I have never used NumPy. Are its arrays special?
Oh yes.
NumPy is basically a subsystem that Python applications make calls into.
Although the data structure can be accessed and amended, algorithms on
the data structures should never be written in Python, they should
always be function calls into the NumPy framework.
Also, which of the following looks more appealing to you?
I don't know.
ok, I'll be lazy then.
--
Russel.
=
Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net
41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk
London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
signature.asc
Description: This is a digitally signed message part
Re: bigint - python long
On 05/09/12 21:23, Paul D. Anderson wrote: On Wednesday, 5 September 2012 at 18:13:40 UTC, Ellery Newcomer wrote: Hey. Investigating the possibility of providing this conversion in pyd. Python provides an api for accessing the underlying bytes. std.bigint seemingly doesn't. Am I missing anything? No, I don't believe so. AFAIK there is no public access to the underlying array, but I think it is a good idea. I suspect the reason for not disclosing the details is to disallow anyone putting the data into an invalid state. But read-only access would be safe. No, it's just not disclosed because I didn't know the best way to do it. I didn't want to put something in unless I was sure it was correct. (And a key part of that, is what is required to implement BigFloat).
Re: bigint - python long
Ellery Newcomer:
array.array == D arrays
just checked, looks like we have it:
PyStmts(q{from array import array; a = array('i',
[44,33,22,11]);}, testing);
assert(PyEval!(int[])(a, testing) == [44,33,22,11]);
I think if the python object is iterable, it can be converted
to array.
array.array are special, they aren't Python lists. array.array
contains uniform data, so conversion to D arrays is a memcpy (or
it's nearly nothing if you don't copy the data).
Bye,
bearophile
Re: bigint - python long
On 09/05/2012 11:19 PM, Jacob Carlborg wrote: On 2012-09-06 04:10, bearophile wrote: There are several important cases, like: Some D lazy ranges == Python lazy iterators/generators array.array == D arrays NumPy arrays == D arrays Associative arrays? check. https://bitbucket.org/ariovistus/pyd/wiki/TypeConversion
Re: bigint - python long
On 09/06/2012 04:11 AM, bearophile wrote:
Ellery Newcomer:
array.array == D arrays
just checked, looks like we have it:
PyStmts(q{from array import array; a = array('i', [44,33,22,11]);},
testing);
assert(PyEval!(int[])(a, testing) == [44,33,22,11]);
I think if the python object is iterable, it can be converted to array.
array.array are special, they aren't Python lists. array.array contains
uniform data, so conversion to D arrays is a memcpy (or it's nearly
nothing if you don't copy the data).
Bye,
bearophile
I see. The docs for array.array suggest that it implements the buffer
interface, but it doesn't seem to implement new or old style buffers, at
least according to PyObject_CheckBuffer and PyBuffer_Check.
I think I'll add support for new style buffers anyways. a memoryview
would be good, too.
Guess I'll hack together a special case for array using buffer_info.
Re: bigint - python long
On Wednesday, 5 September 2012 at 18:13:40 UTC, Ellery Newcomer wrote: Hey. Investigating the possibility of providing this conversion in pyd. Python provides an api for accessing the underlying bytes. std.bigint seemingly doesn't. Am I missing anything? No, I don't believe so. AFAIK there is no public access to the underlying array, but I think it is a good idea. I suspect the reason for not disclosing the details is to disallow anyone putting the data into an invalid state. But read-only access would be safe.
Re: bigint - python long
On Wednesday, 5 September 2012 at 19:23:11 UTC, Paul D. Anderson wrote: No, I don't believe so. AFAIK there is no public access to the underlying array, but I think it is a good idea. I meant to say I think that access to the array is a good idea, not the lack of access. Words are hard!
Re: bigint - python long
Ellery Newcomer: Investigating the possibility of providing this conversion in pyd. Are you updating Pyd? :-) Bye, bearophile
Re: bigint - python long
On 09/05/2012 05:02 PM, bearophile wrote: Ellery Newcomer: Investigating the possibility of providing this conversion in pyd. Are you updating Pyd? :-) Bye, bearophile Yep. Have any suggestions for supported conversion out of the box? From the standard library, I already have Complex, Tuple, and now BigInt. Was thinking maybe the datetime stuff.. Also, which of the following looks more appealing to you? wrap_class!(Foo, Def!(Foo.bar, PyName!__len__))(); wrap_class!(Foo, Len!(Foo.bar))(); they wrap Foo and give it an overriden __len__; the latter is how it is done at the moment (with similar templates for the other operators), but boost::python does something akin to the former. PyName is in the former because Def takes too many optional string arguments.
Re: bigint - python long
Ellery Newcomer: Yep. Oh, good. Have any suggestions for supported conversion out of the box? There are several important cases, like: Some D lazy ranges == Python lazy iterators/generators array.array == D arrays NumPy arrays == D arrays Also, which of the following looks more appealing to you? I don't know. Bye, bearophile
Re: bigint - python long
On 09/05/2012 07:10 PM, bearophile wrote:
Some D lazy ranges == Python lazy iterators/generators
I'll look into this one.
array.array == D arrays
just checked, looks like we have it:
PyStmts(q{from array import array; a = array('i', [44,33,22,11]);},
testing);
assert(PyEval!(int[])(a, testing) == [44,33,22,11]);
I think if the python object is iterable, it can be converted to array.
Matrices might pose a problem. But user can define custom conversions if
need be.
NumPy arrays == D arrays
I have never used NumPy. Are its arrays special?
Also, which of the following looks more appealing to you?
I don't know.
ok, I'll be lazy then.
