Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Gregory P. Smith 
fwiw i'm also supportive of adding these apis.  Lets PEP away to iron out
any details or document disagreements but overall I'd also like to see
something a lot like these go in.

-gps


On Fri, Jun 14, 2013 at 10:50 PM, Nick Coghlan  wrote:

> On 15 June 2013 11:54, Victor Stinner  wrote:
> > 2013/6/15 Antoine Pitrou :
> >>> http://hg.python.org/cpython/rev/6661a8154eb3
> >>> ...
> >>>   Issue #3329: Add new APIs to customize memory allocators
> >>>
> >>> * Add a new PyMemAllocators structure
> >>> * New functions:
> >>>
> >>>   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free
> memory
> >>> allocator functions
> >>>   - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
> >>>   - PyMem_GetAllocators(), PyMem_SetAllocators()
> >>>   - PyMem_SetupDebugHooks()
> >>>   - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
> >>
> >> My two cents, but I would prefer if this whole changeset was reverted.
> >> I think it adds too much complexity in the memory allocation APIs,
> >> for a pretty specialized benefit. IMHO, we should be able to get by with
> >> less allocation APIs (why the new _Raw APIs) and less hook-setting
> >> functions.
> >
> > Ok, I reverted my commit.
> >
> > I posted my initial patch 3 months ago on the bug tracker. I got some
> > reviews and discussed with Kristján Valur Jónsson who heavily modified
> > Python for his game at CCP. I started two threads on python-dev this
> > week (ok, only two days ago). I thaugh that the last known issues were
> > fixed with the addition of PyMem_SetupDebugHooks() (to avoid an
> > environment variable, as asked by Nick) and PyMem_RawMalloc() (have a
> > GIL-free allocator).
> >
> > I will work on a PEP to explain all these new functions and their use
> cases.
>
> I think the new APIs are mostly valid and well-justified, but agree a
> PEP is a good idea.
>
> Yes, it's a complex solution, but it's solving a complex problem that
> arises when embedding CPython inside executables that need to run on
> non-traditional platforms where the simple C defined
> malloc/realloc/free trio is inadequate.
>
> This is a complementary effort to PEP 432 - that aims to simplify
> embedding CPython in general, while Victor's efforts here specifically
> focus on situations where it is necessary to better map CPython to an
> underlying memory model that differs from the traditional C one. While
> the "single heap" model of memory enshrined in the C standard is
> certainly the most common model, it's far from being the only one, and
> these days CPython also gets used in those other environments.
>
> About the only simplification I can see is that PyMem_RawMalloc(),
> PyMem_RawRealloc(), PyMem_RawFree() could perhaps be handled with
> preprocessor macros instead of permitting runtime reconfiguration.
> Allowing the memory allocations for the CPython runtime to be handled
> separately from those for arbitrary C libraries loaded into the
> process seems reasonable, though.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   [email protected]   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 442 accepted

2013-06-15 Thread Antoine Pitrou 
On Sat, 15 Jun 2013 14:57:49 +1000
Nick Coghlan  wrote:

> On 15 June 2013 03:34, Antoine Pitrou  wrote:
> > On Wed, 5 Jun 2013 09:10:54 -0700
> > Benjamin Peterson  wrote:
> >> I (and Guido) are accepting PEP 442 (Safe object finalization) on the
> >> condition that finalizers are only ever called once globally.
> >
> > Ok, so there's an issue with that condition: it can't be upholded on
> > non-GC objects. Creating a non-GC object is quite obscure and rare,
> > though, since it requires basically a class with no __dict__ and an
> > empty __slots__:
> >
> > class C:
> > __slots__ = ()
> > survivors = []
> >
> > def __del__(self):
> > self.survivors.append(self)
> >
> >
> > In this case, a C instance's __del__ will be called every time
> > destruction is attempted, not only once. Is that a realistic problem?
> 
> So, to trigger that __del__() method a second time, such an object
> would have to be:
> 
> 1. Defined in the first place (the use cases for stateless objects
> with destructors seem rare...)
> 2. Hanging off a reference cycle
> 3. Which then gets resurrected

They don't need to hang off a reference cycle. You can resurrect a
non-cyclic object from __del__ too.

But, yeah, stateless objects must be pretty rare, since by definition
they cannot represent anything (except perhaps "nothing").

> i.e. force them to be GC-aware when they
> define a __del__ method, since they may still be hanging off the edge
> of a reference cycle, even if they can't form one themselves

Objects which hang off the edge of a reference cycle don't need to be
GC-aware, since traversing them isn't needed to be break the cycle.
I think "chalking it up as a CPython wart" is a reasonable solution, I
just wanted to ask.

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Antoine Pitrou 
On Sat, 15 Jun 2013 03:54:50 +0200
Victor Stinner  wrote:
> The addition of PyMem_RawMalloc() is motivated by the issue #18203
> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be
> able to setup a custom allocator for *all* allocation made by Python,
> so malloc() should not be called directly. PyMem_RawMalloc() is
> required in places where the GIL is not held (ex: in os.getcwd() on
> Windows).

We already had this discussion on IRC and this argument isn't very
convincing to me. If os.getcwd() doesn't hold the GIL while allocating
memory, then you should fix it to hold the GIL while allocating memory.

I don't like the idea of adding of third layer of allocation APIs. The
dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit
gratuitous (i.e. not motivated by any actual real-world concern, as
far as I can tell).

As for the debug functions you added: PyMem_GetRawAllocators(),
PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(),
PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(),
_PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't
you try to make that 2 or 3?

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Nick Coghlan 
On 15 June 2013 21:01, Antoine Pitrou  wrote:
> On Sat, 15 Jun 2013 03:54:50 +0200
> Victor Stinner  wrote:
>> The addition of PyMem_RawMalloc() is motivated by the issue #18203
>> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be
>> able to setup a custom allocator for *all* allocation made by Python,
>> so malloc() should not be called directly. PyMem_RawMalloc() is
>> required in places where the GIL is not held (ex: in os.getcwd() on
>> Windows).
>
> We already had this discussion on IRC and this argument isn't very
> convincing to me. If os.getcwd() doesn't hold the GIL while allocating
> memory, then you should fix it to hold the GIL while allocating memory.
>
> I don't like the idea of adding of third layer of allocation APIs. The
> dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit
> gratuitous (i.e. not motivated by any actual real-world concern, as
> far as I can tell).

The only reason for the small object allocator to exist is because
operating system allocators generally aren't optimised for frequent
allocation and deallocation of small objects. You can gain a *lot* of
speed from handling those inside the application. As the allocations
grow in size, though, the application level allocator just becomes
useless overhead, so it's better to delegate those operations directly
to the OS.

However, it's still desirable to be able to monitor those direct
allocations in debug mode, thus it makes sense to have a GIL protected
direct allocation API as well. You could try to hide the existence of
the latter behaviour and treat it as a private API, but why? For
custom allocators, it's useful to be able to *ensure* you can bypass
CPython's small object allocator, rather than having to rely on it
being bypassed for allocations above a certain size.

> As for the debug functions you added: PyMem_GetRawAllocators(),
> PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(),
> PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(),
> _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't
> you try to make that 2 or 3?

Faux simplicity that is achieved only by failing to model a complex
problem domain correctly is a bad idea (if we were satisfied with
that, we could stick with the status quo).

The only question mark in my mind is over the GIL-free raw allocation
APIs. I think it makes sense to at least conditionally define those as
macros so an embedding application can redirect *just* the allocations
made by the CPython runtime (rather than having to redefine
malloc/realloc/free when building Python), but I don't believe the
case has been adequately made for making the raw APIs configurable at
runtime. Dropping that aspect would at least eliminate the
PyMem_(Get|Set)RawAllocators() APIs.

Cheers,
Nick.

--
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Antoine Pitrou 
On Sat, 15 Jun 2013 22:22:33 +1000
Nick Coghlan  wrote:
> On 15 June 2013 21:01, Antoine Pitrou  wrote:
> > On Sat, 15 Jun 2013 03:54:50 +0200
> > Victor Stinner  wrote:
> >> The addition of PyMem_RawMalloc() is motivated by the issue #18203
> >> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be
> >> able to setup a custom allocator for *all* allocation made by Python,
> >> so malloc() should not be called directly. PyMem_RawMalloc() is
> >> required in places where the GIL is not held (ex: in os.getcwd() on
> >> Windows).
> >
> > We already had this discussion on IRC and this argument isn't very
> > convincing to me. If os.getcwd() doesn't hold the GIL while allocating
> > memory, then you should fix it to hold the GIL while allocating memory.
> >
> > I don't like the idea of adding of third layer of allocation APIs. The
> > dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit
> > gratuitous (i.e. not motivated by any actual real-world concern, as
> > far as I can tell).
> 
> The only reason for the small object allocator to exist is because
> operating system allocators generally aren't optimised for frequent
> allocation and deallocation of small objects. You can gain a *lot* of
> speed from handling those inside the application. As the allocations
> grow in size, though, the application level allocator just becomes
> useless overhead, so it's better to delegate those operations directly
> to the OS.

The small object allocator *already* delegates those operations
directly to the OS. You don't need a separate API to do it by hand.

> For
> custom allocators, it's useful to be able to *ensure* you can bypass
> CPython's small object allocator, rather than having to rely on it
> being bypassed for allocations above a certain size.

Which custom allocators?

> > As for the debug functions you added: PyMem_GetRawAllocators(),
> > PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(),
> > PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(),
> > _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't
> > you try to make that 2 or 3?
> 
> Faux simplicity that is achieved only by failing to model a complex
> problem domain correctly is a bad idea (if we were satisfied with
> that, we could stick with the status quo).

Actually, I'm sure almost everyone *is* satisfied with the status quo
here (witness the total absence of bug reports on the matter). Victor's
patch addresses a rare concern compared to the common use cases of
CPython.

And I'm not even sure what "faux simplicity" you are talking about.
What is the supposed complexity that this API is supposed to address?
Why do we need two different pairs of hook-setting functions, rather
than letting each function set / get all hooks at once? And why the
private API functions for setting arena allocators?

Memory allocation APIs are a fundamental part of the C API that many
extension writers have to understand and deal with. I'm opposed to
gratuitous complication when the use cases are not compelling.

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-15 Thread Tres Seaver 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 06/14/2013 04:55 PM, R. David Murray wrote:
> This I sort of agree with.  I've often enough wanted to know if
> something is a non-string iterable.  But you'd have to decide if
> bytes/bytearray is a sequence of integers or a scaler...

In fifteen years of Python programming, I have literally *never* wanted
to iterate over 'str' (or now 'bytes').  I've always considered the fact
that Python made them iterable by default (rather than e.g. defining a
method / property to get to an iterable "view" on the underlying string)
a wart and a serious bug magnet.


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  [email protected]
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlG8Y7MACgkQ+gerLs4ltQ6EbwCfYlC3JKL22HK7WgxJLAh9Gk2H
R4cAn2+ijAkebHuF92txeddBq99L/zqn
=bLkO
-END PGP SIGNATURE-

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner 
Le 15 juin 2013 03:54, "Victor Stinner"
>
a écrit :

> Ok, I reverted my commit.
>
> I will work on a PEP to explain all these new functions and their use
cases.

I created the PEP 445 to reserve the number. It is ready for a review but
already contains some explanation of the new API.

http://www.python.org/dev/peps/pep-0445/

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-15 Thread Donald Stufft 
I never want to iterate, but I love slice syntax and indexing. Don't think you 
can have that w/o being able to loop over it can you? Maybe I'm just thinking 
slow since I just woke up. 

On Jun 15, 2013, at 8:53 AM, Tres Seaver  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 06/14/2013 04:55 PM, R. David Murray wrote:
>> This I sort of agree with.  I've often enough wanted to know if
>> something is a non-string iterable.  But you'd have to decide if
>> bytes/bytearray is a sequence of integers or a scaler...
> 
> In fifteen years of Python programming, I have literally *never* wanted
> to iterate over 'str' (or now 'bytes').  I've always considered the fact
> that Python made them iterable by default (rather than e.g. defining a
> method / property to get to an iterable "view" on the underlying string)
> a wart and a serious bug magnet.
> 
> 
> Tres.
> - -- 
> ===
> Tres Seaver  +1 540-429-0999  [email protected]
> Palladion Software   "Excellence by Design"http://palladion.com
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with undefined - http://www.enigmail.net/
> 
> iEYEARECAAYFAlG8Y7MACgkQ+gerLs4ltQ6EbwCfYlC3JKL22HK7WgxJLAh9Gk2H
> R4cAn2+ijAkebHuF92txeddBq99L/zqn
> =bLkO
> -END PGP SIGNATURE-
> 
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/donald%40stufft.io
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-15 Thread Tres Seaver 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 06/15/2013 09:15 AM, Donald Stufft wrote:
> I never want to iterate, but I love slice syntax and indexing. Don't 
> think you can have that w/o being able to loop over it can you? Maybe 
> I'm just thinking slow since I just woke up.

You could if '__iter__' raised an error ('NotIterable', maybe) which
defeated the '__getitem__'/'__len__'-based fallback.


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  [email protected]
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlG8cFoACgkQ+gerLs4ltQ7KcgCfdFCEkp2gBeuUgn/KooY7F5HX
Jm8An2bwq8QoplwJ1MqIbS76m+xdl/Mk
=6hA9
-END PGP SIGNATURE-

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Nick Coghlan 
On 15 June 2013 22:41, Antoine Pitrou  wrote:
> On Sat, 15 Jun 2013 22:22:33 +1000
> Nick Coghlan  wrote:
>> For
>> custom allocators, it's useful to be able to *ensure* you can bypass
>> CPython's small object allocator, rather than having to rely on it
>> being bypassed for allocations above a certain size.
>
> Which custom allocators?

Those used by companies like Dropbox to speed up frequent allocations
(look up their PyCon 2011 keynote). If we don't provide suitable APIs
that we can still hook in debug mode, they'll bypass our
infrastructure completely and we'll miss significant memory accesses.

>> > As for the debug functions you added: PyMem_GetRawAllocators(),
>> > PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(),
>> > PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(),
>> > _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't
>> > you try to make that 2 or 3?
>>
>> Faux simplicity that is achieved only by failing to model a complex
>> problem domain correctly is a bad idea (if we were satisfied with
>> that, we could stick with the status quo).
>
> Actually, I'm sure almost everyone *is* satisfied with the status quo
> here (witness the total absence of bug reports on the matter). Victor's
> patch addresses a rare concern compared to the common use cases of
> CPython.

Indeed, but they're use cases I care about, Victor cares about,
Kristjan cares about, Greg cares about. It's OK that you don't care
about them, just as 99% of the Python programmers on the planet won't
care about PEP 432 or the various arcane metaclass changes we've made
over the years.

issue 3329 (the one where Victor implemented this) was actually filed
by the folks working on the Symbian port. The last comment on that
issue before Victor restarted was from you, in reply to someone asking
if we had implemented it yet.

> And I'm not even sure what "faux simplicity" you are talking about.
> What is the supposed complexity that this API is supposed to address?

The fact that there is more to the world than x86/x86_64 and the very
simplistic C memory model. Python is growing more popular in
non-traditional execution environments, and we finally have someone
(Victor) interested in doing the work to support them properly. That
should be celebrated, not blocked because it isn't meaningful for the
more common systems where the C memory model is fine.

> Why do we need two different pairs of hook-setting functions, rather
> than letting each function set / get all hooks at once?

I've already said I don't think the raw allocators should be
configurable at runtime. The other is because it's likely people will
only want to replace the lower level allocators and leave the small
object allocator alone. However, they should be able to completely
replace the small object allocator if they want. Making the more
common case more complicated to avoid adding a more appropriate two
level API is the kind of thing I mean by "faux simplicity" - it's
almost certainly going to be harder to use in practice, so trading
multiple functions for fewer functions each taking more parameters
isn't actually a win.

> And why the
> private API functions for setting arena allocators?

Because they're in a different compilation unit...

> Memory allocation APIs are a fundamental part of the C API that many
> extension writers have to understand and deal with. I'm opposed to
> gratuitous complication when the use cases are not compelling.

That's a documentation problem. C extension authors shouldn't be
touching these, and most people embedding CPython shouldn't be
touching them either. They're the C level equivalent of metaclasses:
if you're not sure you need them, you don't need them.

Cheers,
Nick.

--
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Antoine Pitrou 
On Sun, 16 Jun 2013 00:12:02 +1000
Nick Coghlan  wrote:
> On 15 June 2013 22:41, Antoine Pitrou  wrote:
> > On Sat, 15 Jun 2013 22:22:33 +1000
> > Nick Coghlan  wrote:
> >> For
> >> custom allocators, it's useful to be able to *ensure* you can bypass
> >> CPython's small object allocator, rather than having to rely on it
> >> being bypassed for allocations above a certain size.
> >
> > Which custom allocators?
> 
> Those used by companies like Dropbox to speed up frequent allocations
> (look up their PyCon 2011 keynote). If we don't provide suitable APIs
> that we can still hook in debug mode, they'll bypass our
> infrastructure completely and we'll miss significant memory accesses.

I don't understand the concern. People can ignore the Python
allocators, and then use their own debugging infrastructure. This is
what happens everytime you want to use your own infrastructure instead
of a 3rd party-provided one.

> > And I'm not even sure what "faux simplicity" you are talking about.
> > What is the supposed complexity that this API is supposed to address?
> 
> The fact that there is more to the world than x86/x86_64 and the very
> simplistic C memory model.

Then I think it needs a PEP to properly address it and explain it to
everyone.

Moreover, I think you are conflating two issues: the ability to add
memory allocation hooks (for tracing/debugging purposes), and the
adaptation to "non-traditional" memory models (whatever that means).
Those concerns don't necessarily come together.

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-15 Thread Oscar Benjamin 
Your questions/suggestions are off-topic for this list. This belongs
on python-ideas.

On 14 June 2013 20:12, Martin Schultz  wrote:
>
> 2. Testing for empty lists or empty ndarrays:
> 4. Finding the number of elements in an object:
> 6. Detecting None values in a list:

Each of the problems above relates to wanting to use generic Python
containers interchangeably with numpy arrays. The solution is simply
not to do this; there are too many semantic inconsistencies for this
interchangeability to be safe. Here are just a few of them:

>>> import numpy as np
>>> [1, 2, 3] + [4, 5, 6]
[1, 2, 3, 4, 5, 6]
>>> np.array([1, 2, 3]) + np.array([4, 5, 6])
array([5, 7, 9])
>>> [1, 2, 3] * 3
[1, 2, 3, 1, 2, 3, 1, 2, 3]
>>> np.array([1, 2, 3]) * 3
array([3, 6, 9])
>>> [[1, 2, 3], [4, 5, 6]] + [7, 8, 9]
[[1, 2, 3], [4, 5, 6], 7, 8, 9]
>>> np.array([[1, 2, 3], [4, 5, 6]]) + np.array([7, 8, 9])
array([[ 8, 10, 12],
   [11, 13, 15]])
>>> bool([1, 2, 3])
True
>>> bool(np.array([1, 2, 3]))
Traceback (most recent call last):
  File "", line 1, in 
ValueError: The truth value of an array with more than one element is
ambiguous. Use a.any() or a.all()

Numpy provides a very convenient function that will convert objects of
all the mentioned types to ndarray objects: the numpy.array function.
If you have a function and want to accept lists (of lists ...) and
ndarrays interchangeably, simply convert your input with numpy.array
at the start of the function i.e.:

def square(numbers):
numbers = numpy.array(numbers)
print(numbers.size, numbers.shape)  # etc.
return numbers ** 2  # Would be an error for a list of lists

> 3. Testing for scalar:
> 5. Forcing a scalar to become a 1-element list:

numbers = np.array(possibly_scalar_input)
if not numbers.shape:
print('Numbers was scalar: coercing...')
numbers = np.array([numbers])

Usually it is bad design to have an API that interchangeably accepts
scalars and sequences. There are some situations where it does make
sense such as numpy's ufuncs. These are scalar mathematical functions
that can operate distributively on lists of lists, ndarrays etc. and
numpy provides a convenient way for you to define these yourself:

>>> def square(x):
... return x ** 2
...
>>> square = np.frompyfunc(square, 1, 1)
>>> square(2)
4
>>> square([2])
array([4], dtype=object)
>>> square([2, 3, 4])
array([4, 9, 16], dtype=object)
>>> square([[2, 3, 4]])
array([[4, 9, 16]], dtype=object)
>>> square([[]])
array([], shape=(1, 0), dtype=object)

The frompyfunc function unfortunately doesn't quite work like a
decorator but you can easily wrap it do so:

def ufunc(nin, nout):
def wrapper_factory(func):
return np.frompyfunc(func, nin, nout)
return wrapper_factory

@ufunc(1, 1)
def square(x):
return x ** 2


Oscar
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Christian Heimes 
Am 15.06.2013 14:22, schrieb Nick Coghlan:
> However, it's still desirable to be able to monitor those direct
> allocations in debug mode, thus it makes sense to have a GIL protected
> direct allocation API as well. You could try to hide the existence of
> the latter behaviour and treat it as a private API, but why? For
> custom allocators, it's useful to be able to *ensure* you can bypass
> CPython's small object allocator, rather than having to rely on it
> being bypassed for allocations above a certain size.


There is even more to it. We like to keep track of memory allocations in
libraries that are wrapped by Python's extension modules, e.g. expat,
openssl etc. Almost every library has a hook to set a custom memory
allocator, either globally (CRYPTO_set_mem_functions) or for each object
(XML_ParserCreate_MM's XML_Memory_Handling_Suite).

Python releases the GIL around IO or CPU critical sections of the
library. But these sections may call the memory management functions. If
these memory functions use the GIL, then some speed ups of releasing the
GIL in the first place are lost. It might even be possible to walk into
dead lock situations.

For that reason it makes sense to have a set of low level memory
management functions, that don't rely on the GIL for locking. These
functions can still impose their own locking if they need to modify a
global state (e.g. allocation statistics). In normal release mode and on
most platforms the raw memory allocators should be thin wrappers around
malloc(), realloc() and free() -- or perhaps just macros.

Eventually I would like to ban direct usage of malloc() from Python's
core and patch all memory management through our API.

Christian
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Christian Heimes 
Am 15.06.2013 14:57, schrieb Victor Stinner:
> Le 15 juin 2013 03:54, "Victor Stinner"  > a écrit :
> 
>> Ok, I reverted my commit.
>>
>> I will work on a PEP to explain all these new functions and their use
> cases.
> 
> I created the PEP 445 to reserve the number. It is ready for a review
> but already containssome explanation of the new API.

+1

How about you compare your approach with some other libraries, too?
Other projects have dealt with similar issues in the past, too. libxml2
has an extensive API for memory management and debugging.

I have assembled a list of APIs for you:

OpenSSL has CRYPTO_set_mem_functions() to set memory management
functions globally
http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/mem.c;h=f7984fa958eb1edd6c61f6667f3f2b29753be662;hb=HEAD#l124

expat has a per-instance memory handler:
http://hg.python.org/cpython/file/cc27d50bd91a/Modules/expat/xmlparse.c#l717

libtiff has three global hooks _TIFFmalloc(), _TIFFrealloc() and
_TIFFfree() that are used instead of malloc() in its core.
http://trac.imagemagick.org/browser/tiff/trunk/libtiff/tif_unix.c#L258

libxml2 has http://xmlsoft.org/html/libxml-xmlmemory.html

Christian
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-15 Thread Ron Adam 



On 06/14/2013 04:03 PM, PJ Eby wrote:

>Should this be the same?
>
>
>python3 -c 'print(bytes("""\r\n""", "utf8"))'
>b'\r\n'
>
>

eval('print(bytes("""\r\n""", "utf8"))')

>b'\n'

No, but:

eval(r'print(bytes("""\r\n""", "utf8"))')

should be.  (And is.)

What I believe you and Walter are missing is that the \r\n in the eval
strings are converted early if you don't make the enclosing string
raw.  So what you're eval-ing is not what you think you are eval-ing,
hence the confusion.


Yes thanks, seems like an easy mistake to make.

To be clear...

The string to eval is parsed when the eval line is tokenized in the scope 
containing the eval() function.  The eval function then parses the 
resulting string object it receives as it's input.


There is no mention of using raw strings in the docs on evel and exec.  I 
think there should be, because the intention (in most cases) is for eval to 
parse the string, and not for it to be parsed or changed before it's 
evaluated by eval or exec.


An example using a string with escape characters might make it clearer.

Cheers,
   Ron
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-15 Thread Terry Reedy 

On 6/15/2013 8:53 AM, Tres Seaver wrote:


In fifteen years of Python programming, I have literally *never* wanted
to iterate over 'str' (or now 'bytes').


If so, it is because you have always been able to use pre-written 
methods and functions that internally do the iteration for you.


> I've always considered the fact

that Python made them iterable by default (rather than e.g. defining a
method / property to get to an iterable "view" on the underlying string)


.__iter__ is that method.

But this is off-topic for pydev.

--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-15 Thread Guido van Rossum 
The semantics of raw strings are clear. I don't see that they should be
called out especially in any context. (Except for regexps.) Usually exec()
is not used with a literal anyway (what would be the point).

--Guido van Rossum (sent from Android phone)
On Jun 15, 2013 1:03 PM, "Ron Adam"  wrote:

>
>
> On 06/14/2013 04:03 PM, PJ Eby wrote:
>
>> >Should this be the same?
>>> >
>>> >
>>> >python3 -c 'print(bytes("""\r\n""", "utf8"))'
>>> >b'\r\n'
>>> >
>>> >
>>>
 eval('print(bytes("""\r\n"**"", "utf8"))')
>>
> >b'\n'
>>>
>> No, but:
>>
>> eval(r'print(bytes("""\r\n""", "utf8"))')
>>
>> should be.  (And is.)
>>
>> What I believe you and Walter are missing is that the \r\n in the eval
>> strings are converted early if you don't make the enclosing string
>> raw.  So what you're eval-ing is not what you think you are eval-ing,
>> hence the confusion.
>>
>
> Yes thanks, seems like an easy mistake to make.
>
> To be clear...
>
> The string to eval is parsed when the eval line is tokenized in the scope
> containing the eval() function.  The eval function then parses the
> resulting string object it receives as it's input.
>
> There is no mention of using raw strings in the docs on evel and exec.  I
> think there should be, because the intention (in most cases) is for eval to
> parse the string, and not for it to be parsed or changed before it's
> evaluated by eval or exec.
>
> An example using a string with escape characters might make it clearer.
>
> Cheers,
>Ron
> __**_
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/**mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] backported Enum

2013-06-15 Thread Ethan Furman 

So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 in 
the 2.x series.

I would like to put this on PyPI, but the `enum` name is already taken.

Would it be inappropriate to call it `stdlib.enum`?

--
~Ethan~
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backported Enum

2013-06-15 Thread Antoine Pitrou 
On Sat, 15 Jun 2013 12:46:32 -0700
Ethan Furman  wrote:
> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 
> in the 2.x series.
> 
> I would like to put this on PyPI, but the `enum` name is already taken.
> 
> Would it be inappropriate to call it `stdlib.enum`?

`backport.enum`?


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backported Enum

2013-06-15 Thread Barry Warsaw 
On Jun 15, 2013, at 12:46 PM, Ethan Furman wrote:

>So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4
>in the 2.x series.
>
>I would like to put this on PyPI, but the `enum` name is already taken.
>
>Would it be inappropriate to call it `stdlib.enum`?

The last upload was on 2009-08-25.  Maybe Ben Finney's abandoned it and
wouldn't mind giving up the enum PyPI name?

-Barry
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-15 Thread Tres Seaver 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 06/15/2013 04:11 PM, Terry Reedy wrote:

> On 6/15/2013 8:53 AM, Tres Seaver wrote:
> 
>> In fifteen years of Python programming, I have literally *never*
>> wanted to iterate over 'str' (or now 'bytes').
> 
> If so, it is because you have always been able to use pre-written 
> methods and functions that internally do the iteration for you.

Given that strings are implemented in C, there is no real "iteration"
happing (in the Python sense) in their methods.  What stdlib code can you
point to that does iterate over them in Python?  I.e.:

  for c in stringval:
  ...

Even if there were, aren't you proving my point?  The fact that strings
are implicitly iterable injects all kinds of fur into methods which take
either a single value or an iterable as an argument, e.g.:

  def api_function(value_or_values):
  if isinstance(value_or_values, STRING_TYPES):
 value_or_values = [value_or_values]
  elif isinstance(value_or_values, collections.abc.Iterable):
 value_or_values = list(value_or_values)
  else:
 value_or_values = [value_or_values]

The bugs that leaving the first test out yields pop up all over the place.


>> I've always considered the fact
>>> that Python made them iterable by default (rather than e.g.
>>> defining a method / property to get to an iterable "view" on the
>>> underlying string)
> 
> .__iter__ is that method.

But it is *on be default*:  I was describing something which has to be
requested in ways that don't get triggered by syntax, and doesn't make
strings look "non-scalar" for APIs like the one above.  


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  [email protected]
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlG84H0ACgkQ+gerLs4ltQ56RQCgks8R52f41RwJ+v9oteOBC3qY
kIIAoIHmg+zcmJpV3v/gAhkKJfbNKufj
=ZeRB
-END PGP SIGNATURE-

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backported Enum

2013-06-15 Thread Tres Seaver 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 06/15/2013 05:14 PM, Barry Warsaw wrote:
> On Jun 15, 2013, at 12:46 PM, Ethan Furman wrote:
> 
>> So I have the stdlb 3.4 Enum backported for both earlier 3.x and
>> back to 2.4 in the 2.x series.
>> 
>> I would like to put this on PyPI, but the `enum` name is already
>> taken.
>> 
>> Would it be inappropriate to call it `stdlib.enum`?
> 
> The last upload was on 2009-08-25.  Maybe Ben Finney's abandoned it
> and wouldn't mind giving up the enum PyPI name?

That would screw anybody already using the existing distributions pretty
badly.


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  [email protected]
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlG84O0ACgkQ+gerLs4ltQ71qwCgo4uubYXVw/qvARESfzPLzFYZ
Fb8AnjB5ZcwupMoQ6SP6r7Xl26Hg3wpQ
=u3L7
-END PGP SIGNATURE-

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backported Enum

2013-06-15 Thread Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC] 
What if there's some obscure PyPi module that requires that `enum` package, or 
some other project (not hosted on PyPI) that requires Ben Finney's original 
`enum` package? Is there anyway to get usage data to make sure nobody's been 
using it recently?

ML

On Jun 15, 2013, at 5:14 PM, Barry Warsaw wrote:

On Jun 15, 2013, at 12:46 PM, Ethan Furman wrote:

So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4
in the 2.x series.

I would like to put this on PyPI, but the `enum` name is already taken.

Would it be inappropriate to call it `stdlib.enum`?

The last upload was on 2009-08-25.  Maybe Ben Finney's abandoned it and
wouldn't mind giving up the enum PyPI name?

-Barry
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/matthew.lefavor%40nasa.gov

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backported Enum

2013-06-15 Thread Ethan Furman 

On 06/15/2013 01:53 PM, Antoine Pitrou wrote:

On Sat, 15 Jun 2013 12:46:32 -0700
Ethan Furman  wrote:

So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 in 
the 2.x series.

I would like to put this on PyPI, but the `enum` name is already taken.

Would it be inappropriate to call it `stdlib.enum`?


`backport.enum`?


Some other well-meaning soul has already claimed that name, but that one only 
covers 2.6 and 2.7.

--
~Ethan~
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backported Enum

2013-06-15 Thread Gregory P. Smith 
I tend to just pick a name and stick with it.  subprocess32 is subprocess
backported from 3.2 (with the 3.3 timeout feature also in it).  unittest2
is unittest from 2.7.

It tends to work.  and it also emphasizes that i'm unlikely to backport
future non-bugfix updates beyond the release mentioned and merely focus on
keeping it stable and available for use on older pythons.

A "backport" namespace is a neat idea but unless someone's going to create
a system for backports to register which versions they are backports from
and automagically have sub-module imports from backport pick the backported
code or the standard library version when the Python VM is recent enough to
not need a backport I wouldn't bother claiming that name.  (and even that
feels like overengineering)

-gps



On Sat, Jun 15, 2013 at 12:46 PM, Ethan Furman  wrote:

> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to
> 2.4 in the 2.x series.
>
> I would like to put this on PyPI, but the `enum` name is already taken.
>
> Would it be inappropriate to call it `stdlib.enum`?
>
> --
> ~Ethan~
> __**_
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/**mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> greg%40krypto.org
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner 
2013/6/15 Christian Heimes :
> Am 15.06.2013 14:22, schrieb Nick Coghlan:
>> However, it's still desirable to be able to monitor those direct
>> allocations in debug mode, thus it makes sense to have a GIL protected
>> direct allocation API as well. You could try to hide the existence of
>> the latter behaviour and treat it as a private API, but why? For
>> custom allocators, it's useful to be able to *ensure* you can bypass
>> CPython's small object allocator, rather than having to rely on it
>> being bypassed for allocations above a certain size.
>
> There is even more to it. We like to keep track of memory allocations in
> libraries that are wrapped by Python's extension modules, e.g. expat,
> openssl etc. Almost every library has a hook to set a custom memory
> allocator, either globally (CRYPTO_set_mem_functions) or for each object
> (XML_ParserCreate_MM's XML_Memory_Handling_Suite).

I just create the issue http://bugs.python.org/issue18227: "Use Python
memory allocators in external libraries like zlib or OpenSSL".

Is it possible to detect if Python is used as a standalone application
(the classic "python" program) or if Python is embedded? If it is
possible, we can modify the "global" memory allocators of a library.
Otherwise, it is more tricky. Should Python sets its "own" memory
allocators? Maybe only if PyMem_SetRawAllocators() was called?

> Eventually I would like to ban direct usage of malloc() from Python's
> core and patch all memory management through our API.

I already create issue http://bugs.python.org/issue18203 for this part.

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner 
2013/6/15 Antoine Pitrou :
> On Sat, 15 Jun 2013 03:54:50 +0200
> Victor Stinner  wrote:
>> The addition of PyMem_RawMalloc() is motivated by the issue #18203
>> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be
>> able to setup a custom allocator for *all* allocation made by Python,
>> so malloc() should not be called directly. PyMem_RawMalloc() is
>> required in places where the GIL is not held (ex: in os.getcwd() on
>> Windows).
>
> We already had this discussion on IRC and this argument isn't very
> convincing to me. If os.getcwd() doesn't hold the GIL while allocating
> memory, then you should fix it to hold the GIL while allocating memory.

The GIL is released for best performances, holding the GIL would have
an impact on performances.

PyMem_RawMalloc() is needed when PyMem_Malloc() cannot be used because
the GIL was released. For example, for the issue #18227 (reuse the
custom allocator in external libraries), PyMem_Malloc() is usually not
appropriate. PyMem_RawMalloc() should also be used instead of
PyMem_Malloc() in the Python startup sequence, because PyMem_Malloc()
requires the GIL whereas the GIL does not exist yet.

PyMem_RawMalloc() also provides more accurate memory usage if it can
be replaced or hooked (with PyMem_SetRawAllocators).

The issue #18203 explains why I would like to replace direct call to
malloc() with PyMem_Malloc() or PyMem_RawMalloc().

> I don't like the idea of adding of third layer of allocation APIs. The
> dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit
> gratuitous (i.e. not motivated by any actual real-world concern, as
> far as I can tell).

In Python 3.3, PyMem_Malloc() cannot be used instead of malloc() where
the GIL is not held. Instead of adding PyMem_RawMalloc(), an
alternative is to remove the "the GIL must be held" restriction from
PyMem_Malloc() by changing PyMem_Malloc() to make it always call
malloc() (instead of PyObject_Malloc() in debug mode).

With such change, a debug hook cannot rely on the GIL anymore: it
cannot inspect Python objects, get a frame or traceback, etc. To still
get accurate debug report, PyMem_Malloc() should be replaced with
PyObject_Malloc().

I don't understand yet the effect of such change on backport
compatibility. May it break applications?

> As for the debug functions you added: PyMem_GetRawAllocators(),
> PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(),
> PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(),
> _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't
> you try to make that 2 or 3?

Get/SetAllocators of PyMem, PyMem_Raw and PyObject can be grouped into
2 functions (get and set) with an argument to select the API.

It is what I proposed initially. I changed this when I had to choose a
name for the name of the argument ("api", "domain",  something else?)
because there were only two choices. With 3 family of functions
(PyMem, PyMem_Raw and PyObject), it becomes again interesting to have
generic functions.

The arena case is different: pymalloc only uses two functions to
allocate areneas: void* alloc(size_t) and void release(void*, size_t).
The release function has a size argument, which is unusual, but
require to implement it using munmap(). VirtualFree() on Windows
requires also the size.

An application can choose to replace PyObject_Malloc() with its own
allocator, but in my experience, it has an important impact on
performance (Python is slower). To benefit of pymalloc with a custom
memory allocator, _PyObject_SetArenaAllocators() can be used.

I kept _PyObject_SetArenaAllocators() private because I don't like its
API, it is not homogenous with the other SetAllocators functions. I'm
not sure that it would be used, so I prefer to keep it private until
it is tested by some projects.

"Private" functions can be used by applications, it's just that Python
doesn't give any backward compatibility warranty. Am I right?

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner 
2013/6/15 Nick Coghlan :
> The only reason for the small object allocator to exist is because
> operating system allocators generally aren't optimised for frequent
> allocation and deallocation of small objects. You can gain a *lot* of
> speed from handling those inside the application. As the allocations
> grow in size, though, the application level allocator just becomes
> useless overhead, so it's better to delegate those operations directly
> to the OS.

Why not using PyObject_Malloc() for all allocations? PyObject_Malloc()
fallbacks to malloc() if the size is larger than a threshold (512
bytes in Python 3.4).

Are PyObject_Realloc() and PyObject_Free() more expensive than
realloc() and free() (when the memory was allocated by malloc)?

> The only question mark in my mind is over the GIL-free raw allocation
> APIs. I think it makes sense to at least conditionally define those as
> macros so an embedding application can redirect *just* the allocations
> made by the CPython runtime (rather than having to redefine
> malloc/realloc/free when building Python), but I don't believe the
> case has been adequately made for making the raw APIs configurable at
> runtime. Dropping that aspect would at least eliminate the
> PyMem_(Get|Set)RawAllocators() APIs.

PyMem_SetRawAllocators() is required for the two use cases: use a
custom memory allocator (embedded device and Python embedded in an
application) and setup an hook for debug purpose.

Without PyMem_SetRawAllocators(), allocations made by
PyMem_RawMalloc() would go to the same place than the rest of the
"Python memory", nor seen by debug tools. It becomes worse with large
allocations kept for a long time.

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner 
2013/6/15 Antoine Pitrou :
> Moreover, I think you are conflating two issues: the ability to add
> memory allocation hooks (for tracing/debugging purposes), and the
> adaptation to "non-traditional" memory models (whatever that means).
> Those concerns don't necessarily come together.

In my implementation, both uses case use the same API:
PyMem_SetAllocators(), except that hooks need also
PyMem_GetAllocators().

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backported Enum

2013-06-15 Thread Donald Stufft 

On Jun 15, 2013, at 5:43 PM, Ethan Furman  wrote:

> On 06/15/2013 01:53 PM, Antoine Pitrou wrote:
>> On Sat, 15 Jun 2013 12:46:32 -0700
>> Ethan Furman  wrote:
>>> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 
>>> 2.4 in the 2.x series.
>>> 
>>> I would like to put this on PyPI, but the `enum` name is already taken.
>>> 
>>> Would it be inappropriate to call it `stdlib.enum`?
>> 
>> `backport.enum`?
> 
> Some other well-meaning soul has already claimed that name, but that one only 
> covers 2.6 and 2.7.
> 
> --
> ~Ethan~
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/donald%40stufft.io

I claimed backport.enum, but you're welcome to the name. I was going to try and 
backport this PEP under that name anyways.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] eval and triple quoted strings

2013-06-15 Thread Ron Adam 



On 06/15/2013 03:23 PM, Guido van Rossum wrote:

The semantics of raw strings are clear. I don't see that they should be
called out especially in any context. (Except for regexps.) Usually exec()
is not used with a literal anyway (what would be the point).



There are about a hundred instances of eval/exec(some_string_literal) in 
pythons library.  Most of them in the tests, and maybe about half of those 
testing the compiler, eval, and exec.


egrep -owr --include="*.py" "(eval|exec)\(('.*'|\".*\")\)" * | wc -l
114


I have no idea in how many places a string literal is assigned to a name 
first and then used later in eval or exec.  It's harder to grep for but 
would be less than...


egrep -owr --include="*.py" "(eval|exec)\(.*\)" * | wc -l
438

That's overstated because some of those are comments, and some may be 
functions with the name ending with eval or exec.



I do think that eval and exec is a similar case to regexps.  And possibly 
often enough, the string may contain a raw string, regular expression, or a 
file/path name.


Only a short note needed in the docs for eval, nothing more.  And not even 
that if no thinks it's an issue.


cheers,
   Ron









___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Nick Coghlan 
On 16 Jun 2013 10:54, "Victor Stinner"  wrote:
>
> 2013/6/15 Antoine Pitrou :
> > Moreover, I think you are conflating two issues: the ability to add
> > memory allocation hooks (for tracing/debugging purposes), and the
> > adaptation to "non-traditional" memory models (whatever that means).
> > Those concerns don't necessarily come together.
>
> In my implementation, both uses case use the same API:
> PyMem_SetAllocators(), except that hooks need also
> PyMem_GetAllocators().

Right - they're different use cases that share a technical solution, so it
makes sense to consider them together.

Cheers,
Nick.

>
> Victor
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backported Enum

2013-06-15 Thread Ben Finney 
Ethan Furman  writes:

> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back
> to 2.4 in the 2.x series.
>
> I would like to put this on PyPI, but the `enum` name is already
> taken.

I have for a long time approved of ‘flufl.enum’ becoming the One Obvious
Way to do enumerations, and am glad to see PEP 435 bring it into the
standard library.

http://mail.python.org/pipermail/python-ideas/2011-July/010811.html>

There are some people still contacting me about their use of the ‘enum’
package on PyPI, so I think that to just replace it with a different
code base under the same name would not be helpful.

> Would it be inappropriate to call it `stdlib.enum`?

That sounds good to me.

Is there anything I can do to keep the ‘enum’ package online for
continuity but make it clear, to automated tools, that this is
end-of-life and obsoleted by another package?

-- 
 \   “Free thought is a necessary, but not a sufficient, condition |
  `\   for democracy.” —Carl Sagan |
_o__)  |
Ben Finney

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backported Enum

2013-06-15 Thread Donald Stufft 


On Jun 15, 2013, at 10:45 PM, Ben Finney  wrote:

> Ethan Furman  writes:
> 
>> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back
>> to 2.4 in the 2.x series.
>> 
>> I would like to put this on PyPI, but the `enum` name is already
>> taken.
> 
> I have for a long time approved of ‘flufl.enum’ becoming the One Obvious
> Way to do enumerations, and am glad to see PEP 435 bring it into the
> standard library.
> 
> http://mail.python.org/pipermail/python-ideas/2011-July/010811.html>
> 
> There are some people still contacting me about their use of the ‘enum’
> package on PyPI, so I think that to just replace it with a different
> code base under the same name would not be helpful.
> 
>> Would it be inappropriate to call it `stdlib.enum`?
> 
> That sounds good to me.
> 
> Is there anything I can do to keep the ‘enum’ package online for
> continuity but make it clear, to automated tools, that this is
> end-of-life and obsoleted by another package?

Right now the best you can do is make a note of it. Pep 426 let's you do what 
you want. 

> 
> -- 
> \   “Free thought is a necessary, but not a sufficient, condition |
>  `\   for democracy.” —Carl Sagan |
> _o__)  |
> Ben Finney
> 
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/donald%40stufft.io
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-15 Thread Terry Reedy 

On 6/15/2013 5:45 PM, Tres Seaver wrote:


Given that strings are implemented in C,


That is a current implementation detail. String functions were 
originally written in python in string.py. Some used 'for c in s:'. The 
functions only because methods after 2.2. I presume Pypy starts from 
Python code again.


> there is no real "iteration"

happing (in the Python sense) in their methods.


I disagree, but not relevant.


 What stdlib code can you
point to that does iterate over them in Python?  I.e.:

   for c in stringval:


Using Idle's grep (Find in files) for ...Python33/Lib/*.py:
'for c in' 193 hits
'for ch in' 30 hits
'for chr in' 0 hits
'for char in' 14 hits.

Some, especially in the 193, are false positives, but most are not. 
There are at least a few other 'for name in string' uses: for instance, 
'for a in _hexdig for b in _hexdig' where _hexdig is 
'0123456789ABCDEFabcdef'.


--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backported Enum

2013-06-15 Thread Ben Finney 
Donald Stufft  writes:

> On Jun 15, 2013, at 10:45 PM, Ben Finney  wrote:
> > Is there anything I can do to keep the ‘enum’ package online for
> > continuity but make it clear, to automated tools, that this is
> > end-of-life and obsoleted by another package?
>
> Right now the best you can do is make a note of it. Pep 426 let's you
> do what you want.

Thanks. What specifically in PEP 426 do you mean; how would I make a
note of “this package is end-of-life as is, please migrate to
‘flufl.enum’ instead” using PEP 426 metadata?

-- 
 \“There are no significant bugs in our released software that |
  `\ any significant number of users want fixed.” —Bill Gates, |
_o__)   1995-10-23 |
Ben Finney

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backported Enum

2013-06-15 Thread Donald Stufft 

On Jun 16, 2013, at 1:52 AM, Ben Finney  wrote:

> Donald Stufft  writes:
> 
>> On Jun 15, 2013, at 10:45 PM, Ben Finney  wrote:
>>> Is there anything I can do to keep the ‘enum’ package online for
>>> continuity but make it clear, to automated tools, that this is
>>> end-of-life and obsoleted by another package?
>> 
>> Right now the best you can do is make a note of it. Pep 426 let's you
>> do what you want.
> 
> Thanks. What specifically in PEP 426 do you mean; how would I make a
> note of “this package is end-of-life as is, please migrate to
> ‘flufl.enum’ instead” using PEP 426 metadata?

http://www.python.org/dev/peps/pep-0426/#obsoleted-by

Note PEP426 isn't completed and isn't implemented :)

> 
> -- 
> \“There are no significant bugs in our released software that |
>  `\ any significant number of users want fixed.” —Bill Gates, |
> _o__)   1995-10-23 |
> Ben Finney
> 
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com