Re: [Python-Dev] Floating-point implementations

[Mark Dickinson]

On Tue, Dec 9, 2008 at 5:24 PM, Mark Dickinson <[EMAIL PROTECTED]> wrote:
> I don't know of any.  There are certainly places in the codebase that
> assume 56 bits are enough.  (I seem to recall it's something like
> 56 bits for IBM, 53 bits for IEEE 754, 48 for Cray, and 52 or 56 for VAX.)

Quick correction, after actually bothering to look things up rather
than relying on my poor memory:  VAX doubles have either *53*
(not 52) or 56 bit mantissas.  More precisely, the VAX G_floating
format has a 53-bit mantissa (52 bits stored directly, one implicit
'hidden' bit), while the (now rare) D_floating format has a 56-bit
mantissa (again, including the implicit 'hidden' bit).

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime.date.today() raises "AttributeError: time"

[Lennart Regebro]

A funny thing just happened to me. I tried out causing this error,
just to see how the error message was somehow different, by creating a
time.py in /tmp, and running python from there. Then I removed the
time.py, and went on working.

Two days later, my usage of zc.buildout are broken with a "module time
has no attribute time". Huh?

Turns out, I created an empty time.py in /tmp, just to see the error
message. By buildout will when creating eggs from checked out modules,
copy them to a directory under /tmp, and evidently run python from
/tmp to create the eggs. So that process finds the time.pyc, created
from the empty time.py, which I hadn't deleted, and breaks!

Heh. That was funny. Moral of the story: Don't create python modules
with names that clash with build in modules in /tmp, even for testing.
Or at least, of you do, remember to remove the pyc. :-P Or, reboot
your Linux every night.  Or well. I guess this could have been avoided
in many ways. ;-)

On Sun, Nov 16, 2008 at 17:43, Guilherme Polo <[EMAIL PROTECTED]> wrote:
> On Sun, Nov 16, 2008 at 11:55 AM, Tal Einat <[EMAIL PROTECTED]> wrote:
>> Steve Holden wrote:
>>> Tal Einat wrote:
 It this desired behavior?

 At the very least the exception should be more detailed, perhaps to
 the point of suggesting the probable cause of the error (i.e.
 overriding the time module).

>>> How is this different from any other case where you import a module with
>>> a standard library name conflict, thereby confusing modules loaded later
>>> standard library. Should we do the same for any error induced in such a way?
>>
>> The difference is that here the exception is generated directly in the
>> C code so you don't get an intelligible traceback. The C code for
>> datetime imports the time module via the Python C API.
>>
>> In other words, here a function from a module in the stdlib, datetime,
>> barfs unexpectedly because I happen to have a file name time.py
>> hanging around in some directory. There is no traceback and no
>> intelligible exception message, just "AttributeError: time". I had to
>> dig through datetime's C code to figure out which module was being
>> imported via the Python C API, which turned out to be time.
>
> Just like Steve told you, this isn't different from other cases. But,
> at least you get a message a bit more verbose in most cases, like:
>
> Traceback (most recent call last):
>  File "", line 1, in 
> AttributeError: 'module' object has no attribute 'time'
>
> Then I went to look why this wasn't happening with datetime too, and I
> found out that PyObject_CallMethod in abstract.c re sets the exception
> message that would have been set by PyObject_GetAttr by now. Maybe
> someone can tell me why it is doing that, for now a patch is attached
> here (I didn't resist to not remove two trailing whitespaces).
>
>>
>>  This is rare enough that I've never had something like this happen to
>> me in seven years of heavy Python programming.
>>
>> - Tal
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> http://mail.python.org/mailman/options/python-dev/ggpolo%40gmail.com
>>
>
>
>
> --
> -- Guilherme H. Polo Goncalves
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/regebro%40gmail.com
>
>



-- 
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Trap SIGSEGV and SIGFPE

[Victor Stinner]

Hi,

I published a new version of my fault handler: it installs an handler for 
signals SIGFPE and SIGSEGV. Using it, it's possible to catch them and 
continue the execution of your Python program. Example:

   try:
  call_evil_code()
   except MemoryError:
  print "A segfault? Haha, I don't care!"
   print "continue the execution"

(yes, it's possible to continue the execution after a segmentation fault!)

Handled errors:

 - Segmentation fault:
   * invalid memory read
   * invalid memory write
   * stack overflow (stack pointer outside the stack memory)

 - SIGFPE
   * division by zero
   * floating point error?

Such errors may occurs from external libraries (written in C)... or Python 
builtin libraries (eg. imageop). The handler is now only used in 
Py_EvalFrameEx(), but it could be used anywhere.

The patch uses sigsetjmp() in Py_EvalFrameEx() to set a "check point", and 
siglongjmp() in the signal handler to go back to the check point. It also 
uses a separated stack for the signal handler, because on stack overflow you 
can not use the stack (ex: unable to call any function!). With MAXDEPTH=100, 
the memory footprint is ~20 KB. If you call Py_EvalFrameEx() more than 
MAXDEPTH times, the handler will go back to the frame #MAXDEPTH on error (you 
loose the last entries in the Python traceback).

sigsetjmp()/siglongjmp() should be available on many OS. I just know that it 
works perfectly on Linux. sigaltstack() is needed to recover after a stack 
overflow, but other errors can be catched without it.

I didn't run any benchmark yet, but it would be interresting ;-) Changing 
MAXDEPTH constant may changes the speed with many recursive calls (eg. 
MAXDEPTH=1 only set a check for the first call to Py_EvalFrameEx()).

I would appreciate a review, especially for the patch in Python/ceval.c.

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python-3.0, unicode, and os.environ

[Ulrich Eckhardt]

On Tuesday 09 December 2008, Adam Olsen wrote:
> On Tue, Dec 9, 2008 at 11:31 AM, Ulrich Eckhardt
>
> <[EMAIL PROTECTED]> wrote:
> > On Monday 08 December 2008, Adam Olsen wrote:
> >> At this point someone suggests we have a type that can store an
> >> arbitrary mix of unicode and bytes, so the undecodable portions stay
> >> in their original form. :P
> >
> > Well, not an arbitrary mix, but a type that just stores whatever comes
> > from the system without further specifying it as either bytes or Unicode:
> >
> > * If you want a string for displaying it, you first have to extract a
> > string from that thing and there you optionally specify the encoding and
> > error behaviour.
> > * If you want to append a string to it, it is automatically encoded in
> > the default encoding, which obviously can fail.
>
> So the 2.x str, but with a more interesting default encoding than
> ASCII.  It'll work fine on the developer's system, but one day a user
> will present it with strange input, and boom.

If the system's representation of filenames can not represent a Unicode 
codepoint that the user entered, trying to open such a file must fail. If it 
can be represented, for convenience I would allow an implicit conversion.

  for i in readdir():
  copy( i, i+".backup")
  ...

> You have to be pessimistic here.  The default operations should either
> always work or never work.  Using unicode internally and skipping
> garbage input means the operations always work.  Using a bytes API
> means mixing with unicode never works, unless the programmer
> explicitly converts, in which case the onus is on them to use proper
> error handling.

So, if I understand you correctly, you would prefer an explicit conversion to 
the system's representation:

  for i in readdir():
  copy( i, i+path(".backup"))
  ...

> The only thing separating this from a bikeshed discussion is that a
> bikeshed has many equally good solutions, while we have no good
> solutions.  Instead we're trying to find the least-bad one.  The
> unicode/bytes separation is pretty close to that.  Adding a warning
> gets even closer.  Adding magic makes it worse.

Well, I see two cases:
1. Converting from an uncertain representation to a known one.
2. Converting from a known representation to a known one.

The uncertain one is the one used by the filesystem or environment. The known 
representations are the expected(!) encoding for filesystem and environment 
and the internal text in Unicode. For case 1, I would require an explicit 
conversion to make the programmer really aware of the fact that it can fail. 
For the second case, I would allow an implicit conversion even though it can 
fail. Anyhow, that is a matter of taste, and I can actually live with your 
point of view.

However, one question still remains: What about the approach in general, i.e. 
that these texts with an uncertain representation are handled as a separate 
type? I find this much more appealing that duplicating APIs like readdir() 
using either overloading on the arguments or a separate readdirb().

Uli

-- 
Sator Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932

**
   Visit our website at 
**
Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten 
bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen 
Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein 
sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, 
weitergeleitet, veröffentlicht oder anderweitig benutzt werden.
E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte 
Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht 
verantwortlich.

**

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Nick Coghlan]

Antoine Pitrou wrote:
> In all honesty, I admit I am annoyed by all the problems with the buffer API /
> memoryview object, many of which are caused by its utterly bizarre design (and
> the fact that the design team went missing in action after imposing such a
> bizarre and complex design on us), and I'm reluctant to add yet another level 
> of
> byzantine complexity in order to solve those problems. It explains I may 
> sound a
> bit angry at times :-)
> 
> If we really need to change things a lot to make them work, we should re-work
> the buffer API from the ground up, make the Py_buffer struct a true PyObject
> (that is, a true variable-length object so as to solve the shape and strides
> allocation issue) and merge it with the current memoryview implementation. It
> would make things both more simpler and more flexible.

I don't see anything wrong with the PEP 3118 protocol. It does exactly
what it is designed to do: allow the number crunching crowd to share
large datasets between different libraries without copying things around
in memory. Yes, the protocol is complicated, but that is because it is
trying to handle a complicated problem.

The memoryview implementation on the other hand is pretty broken. I do
have a theory on how it ended up in such an unusable state, but I'm not
particularly inclined to share it - this kind of thing can happen
sometimes, and the important question now is how we fix it.

As I see it, memoryview is actually trying to do two things, but the
design for supporting the second of them doesn't appear to have been
adequately thought through in the current implementation.

The first use of a memoryview object is merely to allow access to the
Py_buffer of a data store. This is pretty simple, and aside from
currently getting len() wrong when itemsize > 1, memoryview isn't
terrible at it.

If we left memoryview at that it *would* just be a simple wrapper around
a Py_buffer struct, and it's implementation wouldn't be difficult at all.

Where it gets a bit more complicated is if we want to support slices
(rather than just indexing) on memoryview objects. When you do that, the
memoryview is no longer a simple wrapper around the Py_buffer of the
underlying data store, because it isn't exposing the whole data store
any more - it is only exposing part of it.

Requesting access to only part of a data buffer is NOT part of the PEP
3118 API, and it doesn't need to be: it can be part of a separate object
that adapts from the underlying data store to the desired subview.

The object that is meant to be performing at least simple 1-dimensional
cases of that adaptation is memoryview (or more to the point, memoryview
slices), but it currently *sucks* at this because it relies too heavily
on the info in the Py_buffer that it got from the underlying object.
That Py_buffer describes the *whole* data store, but a memoryview slice
may only be exposing part of it - so while the info in the Py_buffer is
accurate for the underlying object, it is *not* accurate for the
memoryview itself.

Fixing that for the 1 dimensional case shouldn't actually be all that
difficult - the memoryview just needs to maintain its own shape[0] entry
that reflects the number of items in the view rather than the number in
the underlying object.

The multi-dimensional cases get pretty tricky though, since they will
almost always end up dealing with non-contiguous data. The PEP 3118
protocol is up to handling the task, but the implementation of the index
mapping to handle these multi-dimensional cases is highly non-trivial,
and probably best left to third party libraries like numpy.

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Antoine Pitrou]

Nick Coghlan  gmail.com> writes:
> 
> I don't see anything wrong with the PEP 3118 protocol.

Apart from the fact that:
- it uses something (Py_buffer) which is not a PyObject and has totally
different allocation/lifetime semantics (which makes it non-trivial to adapt to
for anyone used to the rest of the C API)
- it has unsolved issues like allocation of the underlying shape and strides 
members
- it doesn't specify how to obtain e.g. a sub-buffer, or even duplicate an
existing one (which seem to be rather fundamental actions to me)

... I agree there's nothing wrong with it!

> That Py_buffer describes the *whole* data store, but a memoryview slice
> may only be exposing part of it - so while the info in the Py_buffer is
> accurate for the underlying object, it is *not* accurate for the
> memoryview itself.

And the problem here is that Py_buffer is/was (*) not flexible enough to allow
easy modification in order to take a sub-buffer without some annoying problems.

(*) my patch solves the one-dimensional case. People interested in the
multi-dimensional case will have to do their homework themselves!

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Greg Ewing]

Antoine Pitrou wrote:


If the memoryview wasn't holding onto a Py_buffer, one couldn't rely on its
length or anything else because the underlying object could be mutated at any
moment


Hmm, it seems there are two different approaches that could
be taken here to the design of a memoryview object.

You seem to be thinking of an "eager" approach where the
memoryview keeps the underlying object's memory locked for
as long as it exists, thus preventing it from being
resized.

Whereas I've been thinking of it as being "lazy", in
the sense that the memoryview simply remembers the slice
parameters it was given, and waits until you access it
before making any GetBuffer calls.

The lazy version would have the characteristic that
creating a slice could succeed even though accessing it
later fails due to a range error. I'm not sure that's
necessarily a fatally bad thing.

I'm also not sure that the eager version would be totally
immune to such things. The PEP seems to permit the shape
to change while the buffer is locked as long as the overall
size and location of the memory doesn't change, so a
subsequent access to a formerly-valid slice could still
fail.

In any case, I think it should be possible to implement
either version without the memoryview having to own
more than one Py_buffer and one set of shape/strides
at a time. Slicing the memoryview creates another
memoryview with its own Py_buffer and shape/strides.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Nick Coghlan]

Greg Ewing wrote:
> In any case, I think it should be possible to implement
> either version without the memoryview having to own
> more than one Py_buffer and one set of shape/strides
> at a time. Slicing the memoryview creates another
> memoryview with its own Py_buffer and shape/strides.

The important point is that the shape information in the Py_buffer
filled in by the underlying object is the shape of *that* object.

Except in the trivial case where the memoryview is exposing the entire
underlying data buffer, the shape information in the Py_buffer has
nothing to do with the shape of the memoryview object itself.

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Holding a Python Language Summit at PyCon

[Frank Wierzbicki]

On Mon, Dec 8, 2008 at 10:31 PM, Brett Cannon <[EMAIL PROTECTED]> wrote:
> On Mon, Dec 8, 2008 at 18:53, A.M. Kuchling <[EMAIL PROTECTED]> wrote:
>> On Sat, Dec 06, 2008 at 02:42:38PM -0800, Brett Cannon wrote:
>>> No, I am saying I had told AMK I was interested in championing the
>>> session. He chose you, and that's that. One less thing for me to worry
>>> about. =)
>>
>> Brett, I actually think you'd be a good champion for the 11AM
>> transition-planning session.
>
> OK, so I guess I do have one more thing to worry about. =) I'd be
> happy to do that session.
Sounds good, and I'm still happy to do the other session even with all
of the heckling :)

-Frank
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Floating-point implementations

[Lie Ryan]

On Tue, 09 Dec 2008 12:15:53 -0500, Steve Holden wrote:

> Is anyone aware of any implementations that use other than 64-bit
> floating-point? I'd be particularly interested in any that use greater
> precision than the usual 56-bit mantissa. Do modern 64-bit systems
> implement anything wider than the normal double?
> 
> regards
>  Steve

Why don't we create a DecimalFloat datatype which is a variable-width 
floating point number. Decimal is variable precision fixed-point number, 
while the plain ol' float would be system dependent floating point.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Travis Oliphant]

Antoine Pitrou wrote:

Hello,

The Py_buffer struct has two pointers named `shape` and `strides`. Each points
to an array of Py_ssize_t values whose length is equal to the number of
dimensions of the buffer object. Unfortunately, the buffer protocol spec doesn't
explain how allocation of these arrays should be handled.



I'm coming in late to this discussion, so I apologize for being out of 
order.   But, as Nick later clarifies, the PEP *does* specify how 
allocation of these arrays is handled.


Specifically, it is the responsibility of the exporter to do it and keep 
them correct as long as the buffer is shared.


I have not been able to keep up with the python-dev mailing lists since 
I have been working full time outside of academia.   I apologize for the 
difficulty this may have caused.  But, I have been available via email 
and am happy to respond to specific questions regarding the buffer 
protocol and its implementation.


I will make some time during December to help clean up confusing issues. 
 There are still pieces to implement as well (the enhancements to the 
struct module, for example), but I will not have time for this in the 
next 6 months because I would like to spend any time I can find on 
porting NumPy to use the new buffer protocol as part of getting NumPy 
ready for 3.0.


-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Floating-point implementations

[Steve Holden]

Lie Ryan wrote:
> On Tue, 09 Dec 2008 12:15:53 -0500, Steve Holden wrote:
> 
>> Is anyone aware of any implementations that use other than 64-bit
>> floating-point? I'd be particularly interested in any that use greater
>> precision than the usual 56-bit mantissa. Do modern 64-bit systems
>> implement anything wider than the normal double?
>>
>> regards
>>  Steve
> 
> Why don't we create a DecimalFloat datatype which is a variable-width 
> floating point number. Decimal is variable precision fixed-point number, 
> while the plain ol' float would be system dependent floating point.
> 
Because it's a large amount of work? For a limited return ... the
implementation is bound to be hugely slow compared with hardware
floating point, and as Martin already pointed out gmpy provides
higher-precision arithmetic where required, and the Decimal module
provides arbitrary-range fixed-point arithmetic.

regards
 Steve
-- 
Steve Holden+1 571 484 6266   +1 800 494 3119
Holden Web LLC  http://www.holdenweb.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Travis Oliphant]

Alexander Belopolsky wrote:

On Mon, Dec 8, 2008 at 6:25 PM, Antoine Pitrou <[EMAIL PROTECTED]> wrote:
..

Alexander's suggestion of going and looking at what the numpy folks have
done in this area is probably a good idea too.

Well, I'm open to others doing this, but I won't do it myself. My interest is in
fixing the most glaring bugs of the buffer API and memoryview object. The numpy
folks are welcome to voice their opinions and give advice on python-dev.



I did not follow numpy development for the last year or more, so I
won't qualify as "the numpy folks," but my understanding is that numpy
does exactly what Nick recommended: the viewed object owns shape and
strides just as it owns the data.  The viewing object increases the
reference count of the viewed object and thus assures that data, shape
and strides don't go away prematurely.

I am copying Travis, the author of the PEP 3118, hoping that he would
step in on behalf of "the numpy folks."


I appreciate the copy, as I mentioned I have not had time to follow 
python-dev in detail this year, but I'm glad to help maintain the buffer 
protocol and share any information I can.


I think Nick understands the situation:  the exporter is responsible for 
 allocating and freeing shape, strides, and suboffsets memory (as well 
as  formats, and buf memory).   How it does this is not specified and 
open for interpretation by the objects.  In the standard library there 
is nothing that needs anything complicated and I'm comfortable with what 
I wrote previously to support the objects in the standard library.


There is a length bug in the memoryview implementation, but that is a 
separate issue and being handled.


NumPy will have to handle sharing shape and strides information and will 
serve as a reference implementation when that support is added.


-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Floating-point implementations

[Mark Dickinson]

On Tue, Dec 9, 2008 at 9:48 PM, Lie Ryan <[EMAIL PROTECTED]> wrote:
> Why don't we create a DecimalFloat datatype which is a variable-width
> floating point number. Decimal is variable precision fixed-point number,
> while the plain ol' float would be system dependent floating point.

Decimal is *already* floating-point.  Its handling of exponents
and significant zeros mean that it can do a pretty good job of
imitating fixed-point as well, but it's still at root a floating-point
type.

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Travis Oliphant]

Antoine Pitrou wrote:

Alexander Belopolsky  gmail.com> writes:

I did not follow numpy development for the last year or more, so I
won't qualify as "the numpy folks," but my understanding is that numpy
does exactly what Nick recommended: the viewed object owns shape and
strides just as it owns the data.  The viewing object increases the
reference count of the viewed object and thus assures that data, shape
and strides don't go away prematurely.


That doesn't work if e.g. you take a slice of a memoryview object, since the
shape changes in the process.
See http://bugs.python.org/issue4580




I think there was some confusion about how to support slicing with 
memory view objects.  I remember thinking about it but not getting to 
the code to write it.   The memory object is both an exporter and 
consumer of the buffer protocol.  It can have it's own semantics about 
storing shape and strides information separate from the buffer protocol.


The memory view object needs some way to translate the information it 
gets from the underlying object to the consumer of the information.


My thinking is that the memory view object itself will allocate shape 
and strides information as it needs it.


-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Travis Oliphant]

Antoine Pitrou wrote:

Nick Coghlan  gmail.com> writes:

For the slicing problem in particular, memoryview is currently trying to
get away with only one Py_buffer object when it needs TWO.


Why should it need two? Why couldn't the embedded Py_buffer fullfill all the
needs of the memoryview object? If the memoryview can't be a relatively thin
object-oriented wrapper around a Py_buffer, then this all screams failure to me.



The advice to look at NumPy is good because memoryview is modeled after 
NumPy -- and never completed.


When a slice view is made, a new memoryview object is created with a 
Py_buffer  structure that needs to allocate it's own shape and strides 
(or something that will allow correct shape and strides to be reported 
to any consumer).  In this way, there are two Py_buffer structures.


I do not remember implementing slicing for memoryview objects and it 
looks like the problem is there.






In all honesty, I admit I am annoyed by all the problems with the buffer API /
memoryview object, many of which are caused by its utterly bizarre design (and
the fact that the design team went missing in action after imposing such a
bizarre and complex design on us), and I'm reluctant to add yet another level of
byzantine complexity in order to solve those problems. It explains I may sound a
bit angry at times :-)


I understand your frustration, but I've been here (just not able to 
follow python-dev), and I've tried to respond to issues that came to my 
attention.   I did not have time to complete the memoryview 
implementation, but that does not meen the buffer API is "bizarre".


Yes, the cobbled together memoryview object itself may be "bizarre", but 
that is sometimes the reality of volunteer work.  Just ignore the 
memoryview object if it does not meet your needs.


Please let me know what other problems exist.



If we really need to change things a lot to make them work, we should re-work
the buffer API from the ground up, make the Py_buffer struct a true PyObject
(that is, a true variable-length object so as to solve the shape and strides
allocation issue) and merge it with the current memoryview implementation. It
would make things both more simpler and more flexible.



The only place there is a shape/strides allocation issue is with the 
memoryview object itself.   There is not an issue as far as I can see 
with the buffer protocol itself.


I'm glad you are trying to help clean up the memoryview implementation. 
I welcome the eyes and the keystrokes.  Are you familiar at all 
with NumPy?  That may help you understand what you currently consider to 
be "utterly bizarre"


Best regards,

-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Travis Oliphant]

Greg Ewing wrote:

Antoine Pitrou wrote:

Why should it need two? Why couldn't the embedded Py_buffer fullfill 
all the
needs of the memoryview object? 


Two things here:

  1) The memoryview should *not* be holding onto a Py_buffer
 in between calls to its getitem and setitem methods. It
 should request one from the underlying object when needed
 and release it again as soon as possible.



This is actually a different design than the PEP calls for.  From the PEP:

   This is functionally similar to the current buffer object except a
reference to base is kept and the memory view is not re-grabbed.
Thus, this memory view object holds on to the memory of base until it
is deleted.

I'm open to this changing, but it is the current PEP.



  2) The "second" Py_buffer referred to above only needs to
 be materialized when someone makes a GetBuffer request on
 the memoryview itself. It's not needed for Python getitem
 and setitem calls. (The implementation might choose to
 implement these by creating a temporary Py_buffer, but
 again, it would only last as long as the call.)


The memoryview object will need to store some information for 
re-calculating strides, shape, and sub-offsets for consumers.





If the memoryview can't be a relatively thin
object-oriented wrapper around a Py_buffer, then this all screams 
failure to me.


It shouldn't be a wrapper around a Py_buffer, it should be a
wrapper around the buffer *interface* of the underlying object.



This is a different object than what was proposed, but I'm not opposed 
to it.



It sounds to me like whoever wrote the memoryview implementation
didn't understand how the buffer interface is meant to be used.
That doesn't mean there's anything wrong with the buffer interface.

I have some doubts myself about whether it needs to be as
complicated as it is, but I think the basic idea is sound:
that Py_buffer objects are ephemeral, to be obtained when
needed and not kept for any longer than necessary.



I'm all for simplifying as much as possible.  There are some things I 
understand very well (like how strides and shape information can be 
shared with views), but others that I'm trying to understand better 
(like whether holding on to a view or re-grabbing the view is better).


I think I'm leaning toward the re-grabbing concept.   I'm all for 
improving the memoryview object, but let's not confuse that effort with 
the buffer API implementation.


I do not think we need to worry about changes to the memoryview object, 
because I doubt anything outside of the standard library is using it yet.



-Travis


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Travis Oliphant]

Nick Coghlan wrote:

Antoine Pitrou wrote:

In all honesty, I admit I am annoyed by all the problems with the buffer API /
memoryview object, many of which are caused by its utterly bizarre design (and
the fact that the design team went missing in action after imposing such a
bizarre and complex design on us), and I'm reluctant to add yet another level of
byzantine complexity in order to solve those problems. It explains I may sound a
bit angry at times :-)

If we really need to change things a lot to make them work, we should re-work
the buffer API from the ground up, make the Py_buffer struct a true PyObject
(that is, a true variable-length object so as to solve the shape and strides
allocation issue) and merge it with the current memoryview implementation. It
would make things both more simpler and more flexible.


I don't see anything wrong with the PEP 3118 protocol. It does exactly
what it is designed to do: allow the number crunching crowd to share
large datasets between different libraries without copying things around
in memory. Yes, the protocol is complicated, but that is because it is
trying to handle a complicated problem.

The memoryview implementation on the other hand is pretty broken. I do
have a theory on how it ended up in such an unusable state, but I'm not
particularly inclined to share it - this kind of thing can happen
sometimes, and the important question now is how we fix it.



Thank you Nick.   This is a correct assessment of the situation.  I'd 
like to help improve memoryview as I can.  It does need thought about 
what you want memoryview to be.


I wanted memoryview to be able to be sliced and diced (much like NumPy 
arrays).  But, I only was able to get around to implementing the (simple 
view of Py_buffer struct).



-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Travis Oliphant]

Antoine Pitrou wrote:

Nick Coghlan  gmail.com> writes:

I don't see anything wrong with the PEP 3118 protocol.


Apart from the fact that:
- it uses something (Py_buffer) which is not a PyObject and has totally
different allocation/lifetime semantics (which makes it non-trivial to adapt to
for anyone used to the rest of the C API)


 * this is a non-issue.   The Py_buffer struct is just a place-holder 
for a bunch of variables.  It could be a Python-object but that was seen 
as unnecessary.


- it has unsolved issues like allocation of the underlying shape and strides 
members


 * this is false.  It does specify how this is handled.


- it doesn't specify how to obtain e.g. a sub-buffer, or even duplicate an
existing one (which seem to be rather fundamental actions to me)


 * this is not part of the PEP.  Whether it's a deficiency or not is 
open to interpretation.




... I agree there's nothing wrong with it!


I'm glad you agree.




That Py_buffer describes the *whole* data store, but a memoryview slice
may only be exposing part of it - so while the info in the Py_buffer is
accurate for the underlying object, it is *not* accurate for the
memoryview itself.


And the problem here is that Py_buffer is/was (*) not flexible enough to allow
easy modification in order to take a sub-buffer without some annoying problems.



You are confusing the intent of the memoryview with the Py_buffer struct.

-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Travis Oliphant]

Greg Ewing wrote:

Nick Coghlan wrote:

Maintaining a PyDict instance to map from view pointers to shapes
and strides info doesn't strike me as a "complex scheme" though.


I don't see why a given buffer provider should ever need
more than one set of shape/strides arrays at a time. It
can allocate them on creation, reallocate them as needed
if the shape of its internal data changes, and deallocate
them when it goes away.



I agree.  NumPy has a single shape/strides array.  The intent was to 
share this through the buffer interface.




If you are creating view objects that present slices or
some other alternative perspective, then the view object
itself is a buffer provider and should maintain shape/stride
arrays for its particular view of the underlying object.


Yes, that is correct.

-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime.date.today() raises "AttributeError: time"

[James Y Knight]

On Dec 10, 2008, at 5:55 AM, Lennart Regebro wrote:


Turns out, I created an empty time.py in /tmp, just to see the error
message. By buildout will when creating eggs from checked out modules,
copy them to a directory under /tmp, and evidently run python from
/tmp to create the eggs. So that process finds the time.pyc, created
from the empty time.py, which I hadn't deleted, and breaks!


Sounds like a security hole in zc.buildout. Imagine someone *else*  
made a time.py in /tmp...


James
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime.date.today() raises "AttributeError: time"

[Lennart Regebro]

On Wed, Dec 10, 2008 at 18:49, James Y Knight <[EMAIL PROTECTED]> wrote:
>
> On Dec 10, 2008, at 5:55 AM, Lennart Regebro wrote:
>
>> Turns out, I created an empty time.py in /tmp, just to see the error
>> message. By buildout will when creating eggs from checked out modules,
>> copy them to a directory under /tmp, and evidently run python from
>> /tmp to create the eggs. So that process finds the time.pyc, created
>> from the empty time.py, which I hadn't deleted, and breaks!
>
> Sounds like a security hole in zc.buildout. Imagine someone *else* made a
> time.py in /tmp...

Yup. Adam Olsen also reminded me of this, and I have filed a bug report.

-- 
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime.date.today() raises "AttributeError: time"

[Ralf Schmitt]

On Wed, Dec 10, 2008 at 6:49 PM, James Y Knight <[EMAIL PROTECTED]> wrote:
>
> On Dec 10, 2008, at 5:55 AM, Lennart Regebro wrote:
>
>> Turns out, I created an empty time.py in /tmp, just to see the error
>> message. By buildout will when creating eggs from checked out modules,
>> copy them to a directory under /tmp, and evidently run python from
>> /tmp to create the eggs. So that process finds the time.pyc, created
>> from the empty time.py, which I hadn't deleted, and breaks!
>
> Sounds like a security hole in zc.buildout. Imagine someone *else* made a
> time.py in /tmp...
>

the current working directory is also added to sys.path if PYTHONPATH
contains an empty element. might be the case here...

- Ralf
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Adam Olsen]

On Wed, Dec 10, 2008 at 4:06 AM, Victor Stinner
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I published a new version of my fault handler: it installs an handler for
> signals SIGFPE and SIGSEGV. Using it, it's possible to catch them and
> continue the execution of your Python program. Example:

This will of course leave the program in an undefined state.  It is
very likely to crash again, emit garbage, hang, or otherwise be
useless.

sigsetjmp() is only safe for code explicitly designed for it.  That
will never be the case for CPython, let alone all the arbitrary
libraries that may be used with it.


-- 
Adam Olsen, aka Rhamphoryncus
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python-3.0, unicode, and os.environ

[Adam Olsen]

On Wed, Dec 10, 2008 at 3:39 AM, Ulrich Eckhardt
<[EMAIL PROTECTED]> wrote:
> On Tuesday 09 December 2008, Adam Olsen wrote:
>> The only thing separating this from a bikeshed discussion is that a
>> bikeshed has many equally good solutions, while we have no good
>> solutions.  Instead we're trying to find the least-bad one.  The
>> unicode/bytes separation is pretty close to that.  Adding a warning
>> gets even closer.  Adding magic makes it worse.
>
> Well, I see two cases:
> 1. Converting from an uncertain representation to a known one.
> 2. Converting from a known representation to a known one.

Not quite:
1. Using a garbage file name locally (within a single process, not
talking to any libs)
2. Using a unicode filename everywhere (libs, saved to config files,
displayed to the user, etc.)

Note that if you have a GUI doing the former, all you technically need
is a placeholder like "".  You might try to
extract some ASCII out of it, but that's just a minor bonus.

On linux the bytes/unicode separation is perfect for this.  You decide
which approach you're using and use it consistently.  If you mess up
(mixing bytes and unicode) you'll consistently get an error.

We currently don't follow this model on windows, so a garbage file
name gets passed around as if it was unicode, but fails when passed to
a lib, saved to a config file, is displayed to a user, etc.
(Depending on the API, as many won't validate either.)


-- 
Adam Olsen, aka Rhamphoryncus
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Victor Stinner]

Oh, I forgot the issue URL:
   http://bugs.python.org/issue3999

I also attached an example of catching segfaults.

> > I published a new version of my fault handler: it installs an handler for
> > signals SIGFPE and SIGSEGV. Using it, it's possible to catch them and
> > continue the execution of your Python program. Example:
>
> This will of course leave the program in an undefined state.  It is
> very likely to crash again, emit garbage, hang, or otherwise be
> useless.

Recover after a segfault is dangerous, but my first goal was to get the Python 
backtrace instead just one line: "Segmentation fault". It helps a lot for 
debug!

I didn't try on real world application, but with a small script the program 
continues its execution without any problem.

But yes, there is a big risk of:
 - leak memory 
 - deadlock
 - context problem, eg. for the GIL, I call PyGILState_Ensure()
 - etc.

I choosed the exceptions MemoryError and ArithmeticError, but we could use 
specific exceptions based on BaseException instead of Exception to avoid 
catching them with "except Exception: ...".

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Benjamin Peterson]

On Wed, Dec 10, 2008 at 12:37 PM, Victor Stinner
<[EMAIL PROTECTED]> wrote:
> Oh, I forgot the issue URL:
>   http://bugs.python.org/issue3999
>
> I also attached an example of catching segfaults.
>
>> > I published a new version of my fault handler: it installs an handler for
>> > signals SIGFPE and SIGSEGV. Using it, it's possible to catch them and
>> > continue the execution of your Python program. Example:
>>
>> This will of course leave the program in an undefined state.  It is
>> very likely to crash again, emit garbage, hang, or otherwise be
>> useless.
>
> Recover after a segfault is dangerous, but my first goal was to get the Python
> backtrace instead just one line: "Segmentation fault". It helps a lot for
> debug!

Exactly! That's why it doesn't belong in the Python core. We can't
guarantee anything about its affects or encourage it.

>
> I didn't try on real world application, but with a small script the program
> continues its execution without any problem.

But as you say, it would be used on real world programs!



-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Adam Olsen]

On Wed, Dec 10, 2008 at 11:37 AM, Victor Stinner
<[EMAIL PROTECTED]> wrote:
> Oh, I forgot the issue URL:
>   http://bugs.python.org/issue3999
>
> I also attached an example of catching segfaults.
>
>> > I published a new version of my fault handler: it installs an handler for
>> > signals SIGFPE and SIGSEGV. Using it, it's possible to catch them and
>> > continue the execution of your Python program. Example:
>>
>> This will of course leave the program in an undefined state.  It is
>> very likely to crash again, emit garbage, hang, or otherwise be
>> useless.
>
> Recover after a segfault is dangerous, but my first goal was to get the Python
> backtrace instead just one line: "Segmentation fault". It helps a lot for
> debug!

It's possible to print the Python stack purely from C, without
invoking any Python code.  Even better, you could print the C stack
while you're at it!  Doing that in a signal handler, and then killing
the process, could be seriously considered.

Take a look at http://www.linuxjournal.com/article/6391 .  You'll
probably need #ifdef's to only use it on certain supported platforms,
and probably disable it by default anyway (configure option?  Not
sure).  Still, it'd be useful to have it there.


-- 
Adam Olsen, aka Rhamphoryncus
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Terry Reedy]

Benjamin Peterson wrote:

On Wed, Dec 10, 2008 at 12:37 PM, Victor Stinner



This will of course leave the program in an undefined state.  It is
very likely to crash again, emit garbage, hang, or otherwise be
useless.

Recover after a segfault is dangerous, but my first goal was to get the Python
backtrace instead just one line: "Segmentation fault". It helps a lot for
debug!


Exactly! That's why it doesn't belong in the Python core. We can't
guarantee anything about its affects or encourage it.


Would it be safe to catch SIGSEGV, output a trace, and then exit?
IE, make the 'first goal' the only goal?

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[BJörn Lindqvist]

One thing i think it would be useful for in the real world is for
unittesting extension modules. You cant profitably write unit tests
for segfaults because that breaks the test harness. In situations like
those, recovering would be likely (caveat emptor of course).


2008/12/10, Adam Olsen <[EMAIL PROTECTED]>:
> On Wed, Dec 10, 2008 at 4:06 AM, Victor Stinner
> <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> I published a new version of my fault handler: it installs an handler for
>> signals SIGFPE and SIGSEGV. Using it, it's possible to catch them and
>> continue the execution of your Python program. Example:
>
> This will of course leave the program in an undefined state.  It is
> very likely to crash again, emit garbage, hang, or otherwise be
> useless.
>
> sigsetjmp() is only safe for code explicitly designed for it.  That
> will never be the case for CPython, let alone all the arbitrary
> libraries that may be used with it.
>
>
> --
> Adam Olsen, aka Rhamphoryncus
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/bjourne%40gmail.com
>


-- 
mvh Björn
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Adam Olsen]

On Wed, Dec 10, 2008 at 12:22 PM, BJörn Lindqvist <[EMAIL PROTECTED]> wrote:
> One thing i think it would be useful for in the real world is for
> unittesting extension modules. You cant profitably write unit tests
> for segfaults because that breaks the test harness. In situations like
> those, recovering would be likely (caveat emptor of course).

The only safe option there is a subprocess.


-- 
Adam Olsen, aka Rhamphoryncus
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[M.-A. Lemburg]

On 2008-12-10 21:05, Adam Olsen wrote:
> On Wed, Dec 10, 2008 at 12:22 PM, BJörn Lindqvist <[EMAIL PROTECTED]> wrote:
>> One thing i think it would be useful for in the real world is for
>> unittesting extension modules. You cant profitably write unit tests
>> for segfaults because that breaks the test harness. In situations like
>> those, recovering would be likely (caveat emptor of course).
> 
> The only safe option there is a subprocess.

True, but that still makes it a little difficult to report the errors
found in the module.

mxTools has an optional safecall() function that allows calling
functions which potentially segfault and still returns control
back to the calling application:

http://www.egenix.com/products/python/mxBase/mxTools/

It's not (yet) documented, but fairly straight forward to use
once you've enabled it in egenix_mx_base.py:

result = mx.Tools.safecall(callable, args, kws)

Using such a function is handy in situations where you have a
multi-process application setup that sometimes needs to call
out to external libraries of varying quality - a situation that's
not uncommon in real-life situations.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 10 2008)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2008-12-02: Released mxODBC.Connect 1.0.0  http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Martin v. Löwis]

> I would appreciate a review, especially for the patch in Python/ceval.c.

In this specific case, it is not clear for what objective you want such
review. For inclusion into Python?

Several people already said (essentially) that: -1. I don't think such
code should be added to the Python core, no matter how smart or correct
it is.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Greg Ewing]

Travis Oliphant wrote:

When a slice view is made, a new memoryview object is created with a 
Py_buffer  structure that needs to allocate it's own shape and strides 
(or something that will allow correct shape and strides to be reported 
to any consumer).  In this way, there are two Py_buffer structures.


To be precise, the important thing is for the memoryview to allocate
its own shape and strides. It's not strictly necessary to keep them
internally in a Py_buffer struct, although that may be a convenient
way to do it.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Merging flow

[Jeffrey Yasskin]

Was there ever a conclusion to this? I need to merge the patches
associated with issue 4597 from trunk to all the maintenance branches,
and I'd like to avoid messing anyone up if possible. If I don't hear
back, I'll plan to svnmerge directly from trunk to each of the
branches, and then block my merge to py3k from being merged again to
release30-maint.

Thanks,
Jeffrey

On Thu, Dec 4, 2008 at 7:12 AM, Christian Heimes <[EMAIL PROTECTED]> wrote:
> Several people have asked about the patch and merge flow. Now that Python
> 3.0 is out it's a bit more complicated.
>
> Flow diagram
> 
>
> trunk ---> release26-maint
>   \->  py3k   ---> release30-maint
>
>
> Patches for all versions of Python should land in the trunk. They are then
> merged into release26-maint and py3k branches. Changes for Python 3.0 are
> merged via the py3k branch.
>
> Christian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Alexander Belopolsky]

On Wed, Dec 10, 2008 at 6:12 PM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
>> I would appreciate a review, especially for the patch in Python/ceval.c.
>
> In this specific case, it is not clear for what objective you want such
> review. For inclusion into Python?
>

Even if it does not result in an inclusion into Python, I personally
would be quite interested in following this thread if discussion of
Victor's patch continues.  It may quite possibly yield some
improvements to python development tools (core and libraries'
development).  Graceful handling of hard errors is an unsolved problem
in Python and it has become more important since ctypes made it to the
standard library and therefore it has become possible to easily
trigger a hard error from pure python code.

> Several people already said (essentially) that: -1. I don't think such
> code should be added to the Python core, no matter how smart or correct
> it is.
>

Looking up the thread, I don't see anyone taking such an extreme
position: never recover from SEGV even if it can be done 100%
correctly.  The sentiment that I see and the one that I share is that
it is extremely difficult (and maybe impossible) to do correctly.
However, if someone comes up with a smart solution, I would be very
much interested to see it.

While by the time you get a SIGSEGV, you process is likely to be
beyond recovery, I don't think the same applies to SIGFPE.   It may
also be possible to get rid of the arbitrary recursion limit on Linux
(I've heard this problem is solved on Windows) by being smart about
handling SIGSEGV.

Finally, providing some diagnostic before exiting on hard errors is
not without precedent: I believe R has such a feature.  It may be
worthwhile to compare Victor's approach to what is done in R.

It may, however, be better to move further discussion to the tracker
(I understand that the patch is at
).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Greg Ewing]

Nick Coghlan wrote:


The multi-dimensional cases get pretty tricky though, since they will
almost always end up dealing with non-contiguous data. The PEP 3118
protocol is up to handling the task, but the implementation of the index
mapping to handle these multi-dimensional cases is highly non-trivial,
and probably best left to third party libraries like numpy.


I'm wondering whether there should be some kind of utility
function provided with the buffer API for doing this. It
would take the shape/strides info from a Py_buffer together
with a set of slicing parameters, and create you another
set of shape/strides info describing the slice.

It seems sensible to put the effort into doing this
correctly once, rather than leave everyone implementing
a memoryview-like object to come up with their own
half-working and/or broken implementation.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Greg Ewing]

Antoine Pitrou wrote:


- it uses something (Py_buffer) which is not a PyObject and has totally
different allocation/lifetime semantics


This was a deliberate decision -- in fact I argued for it myself.
The buffer interface is meant to be a minimal-overhead way for
C code to get at the underlying data. Requiring allocation of
a PyObject would be too expensive.

The way to think about the Py_buffer struct is not as an
object in its own right, but just a place to put some output
parameters from the GetBuffer call.

The lifetime of the information pointed to by the Py_buffer
is the same as the lifetime of the underlying object, and that
object is responsible for managing it.


- it doesn't specify how to obtain e.g. a sub-buffer, or even duplicate an
existing one (which seem to be rather fundamental actions to me)


I don't think they're as fundamental as all that. But some
utilities for doing things like this could be useful, as I
mentioned in another post.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allocation of shape and strides fields in Py_buffer

[Antoine Pitrou]

Greg Ewing  canterbury.ac.nz> writes:
> 
> This was a deliberate decision -- in fact I argued for it myself.
> The buffer interface is meant to be a minimal-overhead way for
> C code to get at the underlying data. Requiring allocation of
> a PyObject would be too expensive.

Tuples are used everywhere throughout the interpreter and yet they are proper
PyObjects. Even simple integers are often wrapped into PyLong objects (see the
getitem/setitem protocol in Py3k). I doubt Py_buffers are more critical for
performance than tuples and integers are.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Merging flow

[Martin v. Löwis]

Jeffrey Yasskin wrote:
> Was there ever a conclusion to this? I need to merge the patches
> associated with issue 4597 from trunk to all the maintenance branches,
> and I'd like to avoid messing anyone up if possible. If I don't hear
> back, I'll plan to svnmerge directly from trunk to each of the
> branches, and then block my merge to py3k from being merged again to
> release30-maint.

No - you should merge from the py3k branch to the release30-maint branch.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Adam Olsen]

On Wed, Dec 10, 2008 at 5:21 PM, Alexander Belopolsky
<[EMAIL PROTECTED]> wrote:
> On Wed, Dec 10, 2008 at 6:12 PM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
>> Several people already said (essentially) that: -1. I don't think such
>> code should be added to the Python core, no matter how smart or correct
>> it is.
>>
>
> Looking up the thread, I don't see anyone taking such an extreme
> position: never recover from SEGV even if it can be done 100%
> correctly.  The sentiment that I see and the one that I share is that
> it is extremely difficult (and maybe impossible) to do correctly.
> However, if someone comes up with a smart solution, I would be very
> much interested to see it.

It is impossible to do in general, and I am -1 on any misguided
attempts to do so.


> While by the time you get a SIGSEGV, you process is likely to be
> beyond recovery, I don't think the same applies to SIGFPE.

No, it's as much about the context as it is the error.  We could write
our own floating point code that can recover from SIGFPE (which isn't
portable, but still mostly doable), but enabling it for arbitrary
third-party libraries is completely unsafe.

Printing a stack trace and then aborting would be possible and useful though.


> It may
> also be possible to get rid of the arbitrary recursion limit on Linux
> (I've heard this problem is solved on Windows) by being smart about
> handling SIGSEGV.

If we could calculate how much stack is left we'd have a much more
robust way of doing recursion limits.  I suppose this could be done by
reading a byte from each page with a temporary SIGSEGV handler
installed, but I'm not convinced you can't ask the platform directly
somehow.  I'd also be considered about thread-safety.


> Finally, providing some diagnostic before exiting on hard errors is
> not without precedent: I believe R has such a feature.  It may be
> worthwhile to compare Victor's approach to what is done in R.
>
> It may, however, be better to move further discussion to the tracker
> (I understand that the patch is at
> ).


-- 
Adam Olsen, aka Rhamphoryncus
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Alexander Belopolsky]

On Wed, Dec 10, 2008 at 8:01 PM, Adam Olsen <[EMAIL PROTECTED]> wrote:
..
> It is impossible to do in general, and I am -1 on any misguided
> attempts to do so.
>

I agree, recovering from segfaults caused by buggy third party C
modules is a losing proposition, but for a limited number of
conditions that can be triggered from python code running on a
non-buggy interpreter (hopefully ctypes included, but that would be
hard), converting signals into exceptions may be possible.
..
> Printing a stack trace and then aborting would be possible and useful though.
>
Even a simple dialog: Python have encountered a segfault, would you
like to dump core? y/n in the interactive session will be quite
useful.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Simon Cross]

On Wed, Dec 10, 2008 at 8:37 PM, Victor Stinner
<[EMAIL PROTECTED]> wrote:
> Recover after a segfault is dangerous, but my first goal was to get the Python
> backtrace instead just one line: "Segmentation fault". It helps a lot for
> debug!

This would be extremely useful. I've had PyGTK segfault on me a number
of times in an app I'm writing and I keep meaning to try get to the
bottom of the issue but it happens infrequently and somehow I never
get around to it. Some indictation of what Python was executing when
the segfault occurred would help narrow now the possibilities rapidly.

Schiavo
Simon
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com