Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod

2013-06-14 Thread Antoine Pitrou
On Fri, 14 Jun 2013 07:06:49 +0200 (CEST)
raymond.hettinger  wrote:
> http://hg.python.org/cpython/rev/5accb0ac8bfb
> changeset:   84116:5accb0ac8bfb
> branch:  2.7
> parent:  84095:ca8e86711403
> user:Raymond Hettinger 
> date:Fri Jun 14 01:06:33 2013 -0400
> summary:
>   Fix comment blocks.  Adjust blocksize to a power-of-two for better divmod 
> computations.

Is there any rationale for changing the heuristic from "fits in a whole
number of cachelines" to "allows fast division by the blocksize"?

I personally would prefer if such changes were justified a bit more
than by a one-liner changeset message without any corresponding open
issue.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Allow calling PyMem_Malloc() without the GIL held in Python 3.4

2013-06-14 Thread Kristján Valur Jónsson

> -Original Message-
> I would like to remove the "GIL must be held" restriction from
> PyMem_Malloc(). In my opinion, the restriction was motived by a bug in
> Python, bug fixed by the issue #3329. Let me explain why.
> 
...

> 
> Removing the GIL restriction would help to replace direct calls to
> malloc() with PyMem_Malloc(). Using PyMem_SetAllocators(), an application
> would be able to replace memory allocators, and these allocators would be
> used "everywhere".
> => see http://bugs.python.org/issue18203
> 

To keep this interesting, I have a somewhat different opinion to Victor :)
  have put comments in the original defect, but would like to repeat them here.
IMHO, keeping the GIL restriction on PyMem_MALLOC is useful.
1) It allows it to be replaced with PyObject_MALLOC(). Or anything else.  In 
particular, an implementer is free to add memory profiling support and other 
things without worrying about implementation details.  Requiring it to be GIL 
free severely limits what it can do.  For example, it would be forbidden to 
delegate to PyObject_MALLOC when debugging. 

The approach CCP has taken (we have replaced all raw malloc calls with api 
calls) is this:
a) Add a "raw" api, PyMem_MALLOC_RAW.  This is guaranteed to be thread safe and 
call directly to the external memory api of python, as set by Py_SetAllocator()
b) Replace calls to malloc() in the source code with 
PyMem_MALLOC/PyMem_MALLOC_RAW as appropriate (in our case, using  an include 
file with #defines to mimimize changes)

There are only two or three places in the source code that require non-GIL 
protected malloc.  IMHO, requiring PyMem_MALLOC to be threadsafe just to cater 
to those three places is an overkill, and does more harm than good by limiting 
our options.

Cheers!
Kristján

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod

2013-06-14 Thread Serhiy Storchaka

14.06.13 11:46, Antoine Pitrou написав(ла):

On Fri, 14 Jun 2013 07:06:49 +0200 (CEST)
raymond.hettinger  wrote:

http://hg.python.org/cpython/rev/5accb0ac8bfb
changeset:   84116:5accb0ac8bfb
   Fix comment blocks.  Adjust blocksize to a power-of-two for better divmod 
computations.


Is there any rationale for changing the heuristic from "fits in a whole
number of cachelines" to "allows fast division by the blocksize"?

I personally would prefer if such changes were justified a bit more
than by a one-liner changeset message without any corresponding open
issue.


I share the doubts of Antoine and I was going to write the same comment. 
I thought there were good reasons for previous code. What has changed?



___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] cpython: Move test_pep352 over to unittest.main()

2013-06-14 Thread Serhiy Storchaka

14.06.13 04:18, brett.cannon написав(ла):

http://hg.python.org/cpython/rev/af27c661d4fb
changeset:   84115:af27c661d4fb
user:Brett Cannon 
date:Thu Jun 13 21:18:43 2013 -0400
summary:
   Move test_pep352 over to unittest.main()


You forgot about:

from test.support import run_unittest


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] python symbolizition

2013-06-14 Thread Pynix Wang
1.lambda expression

c#
a.(x, y) => x == y
b.() => SomeMethod()

ruby:
 -> {|msg| puts msg}

python can use c# like and remove "lambda" keyword.

2.global variable

ruby
$glo_var

python can use $ or @ or another and remove "global".
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] python symbolizition

2013-06-14 Thread Markus Unterwaditzer
But why?

-- Markus (from phone)

Pynix Wang  wrote:
>1.lambda expression
>
>c#
>a.(x, y) => x == y
>b.() => SomeMethod()
>
>ruby:
> -> {|msg| puts msg}
>
>python can use c# like and remove "lambda" keyword.
>
>2.global variable
>
>ruby
>$glo_var
>
>python can use $ or @ or another and remove "global".
>
>
>
>
>___
>Python-Dev mailing list
>[email protected]
>http://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>http://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] python symbolizition

2013-06-14 Thread R. David Murray
This discussion is better suited for python-ideas than it
is for python-dev.

On Fri, 14 Jun 2013 16:50:53 +0200, Markus Unterwaditzer 
 wrote:
> But why?
> 
> -- Markus (from phone)
> 
> Pynix Wang  wrote:
> >1.lambda expression
> >
> >c#
> >a.(x, y) => x == y
> >b.() => SomeMethod()
> >
> >ruby:
> > -> {|msg| puts msg}
> >
> >python can use c# like and remove "lambda" keyword.
> >
> >2.global variable
> >
> >ruby
> >$glo_var
> >
> >python can use $ or @ or another and remove "global".
> >
> >
> >
> >
> >___
> >Python-Dev mailing list
> >[email protected]
> >http://mail.python.org/mailman/listinfo/python-dev
> >Unsubscribe:
> >http://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net
> 
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] eval and triple quoted strings

2013-06-14 Thread Walter Dörwald

Hello all!

This surprised me:

   >>> eval("'''\r\n'''")
   '\n'

Where did the \r go? ast.literal_eval() has the same problem:

   >>> ast.literal_eval("'''\r\n'''")
   '\n'

Is this a bug/worth fixing?

Servus,
   Walter
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] eval and triple quoted strings

2013-06-14 Thread Guido van Rossum
Not a bug. The same is done for file input -- CRLF is changed to LF before
tokenizing.
On Jun 14, 2013 8:27 AM, "Walter Dörwald"  wrote:

> Hello all!
>
> This surprised me:
>
>>>> eval("'''\r\n'''")
>'\n'
>
> Where did the \r go? ast.literal_eval() has the same problem:
>
>>>> ast.literal_eval("'''\r\n'''")
>'\n'
>
> Is this a bug/worth fixing?
>
> Servus,
>Walter
> __**_
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/**mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] Summary of Python tracker Issues

2013-06-14 Thread Python tracker

ACTIVITY SUMMARY (2013-06-07 - 2013-06-14)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open4042 (+18)
  closed 25946 (+41)
  total  29988 (+59)

Open issues with patches: 1795 


Issues opened (41)
==

#17947: Code, test, and doc review for PEP-0435 Enum
http://bugs.python.org/issue17947  reopened by ncoghlan

#18159: ConfigParser getters not available on SectionProxy
http://bugs.python.org/issue18159  opened by JBernardo

#18161: call fchdir if subprocess.Popen(cwd=integer|fileobject)
http://bugs.python.org/issue18161  opened by mmarkk

#18162: Add index attribute to IndexError
http://bugs.python.org/issue18162  opened by brett.cannon

#18163: Add a 'key' attribute to KeyError
http://bugs.python.org/issue18163  opened by brett.cannon

#18164: Embedding Python doc incorrectly refers to LINKFORSHARED
http://bugs.python.org/issue18164  opened by ned.deily

#18165: Add 'unexpected_type' to TypeError
http://bugs.python.org/issue18165  opened by brett.cannon

#18166: 'value' attribute for ValueError
http://bugs.python.org/issue18166  opened by brett.cannon

#18167: cgi.FieldStorage fails to handle multipart/form-data when \r\n
http://bugs.python.org/issue18167  opened by catlee

#18168: plistlib output self-sorted dictionary
http://bugs.python.org/issue18168  opened by halfjuice

#18170: define built-in exceptions in Python code
http://bugs.python.org/issue18170  opened by brett.cannon

#18173: Add MixedTypeKey to reprlib
http://bugs.python.org/issue18173  opened by ncoghlan

#18174: Make regrtest with --huntrleaks check for fd leaks
http://bugs.python.org/issue18174  opened by sbt

#18176: Builtins documentation refers to old version of UCD.
http://bugs.python.org/issue18176  opened by belopolsky

#18177: Incorect quote marks in code section-headers in PDF version of
http://bugs.python.org/issue18177  opened by icedream91

#18178: Redefinition of malloc(3) family of functions at build time
http://bugs.python.org/issue18178  opened by cvs-src

#18179: SMTP.local_hostname is undocumented
http://bugs.python.org/issue18179  opened by jonash

#18181: super vs. someclass.__getattribute__
http://bugs.python.org/issue18181  opened by ronaldoussoren

#18182: xml.dom.createElement() does not take implicit namespaces into
http://bugs.python.org/issue18182  opened by Alexander.Tobias.Heinrich

#18184: Add range check  for %c in PyUnicode_FromFormat
http://bugs.python.org/issue18184  opened by serhiy.storchaka

#18185: Error in test_set.TestVariousIteratorArgs.test_inline_methods
http://bugs.python.org/issue18185  opened by syeberman

#18188: ERROR: test_no_optimize_flag on Mageia Linux Cauldron x86-64 w
http://bugs.python.org/issue18188  opened by shlomif

#18189: IDLE Improvements: Unit test for Delegator.py
http://bugs.python.org/issue18189  opened by JayKrish

#18191: urllib2/urllib.parse.splitport does not handle IPv6 correctly
http://bugs.python.org/issue18191  opened by jd

#18192: Move imp.get_magic() to importlib
http://bugs.python.org/issue18192  opened by brett.cannon

#18193: Move imp.reload() to importlib
http://bugs.python.org/issue18193  opened by brett.cannon

#18194: Move imp.source_from_cache/cache_from_source to importlib
http://bugs.python.org/issue18194  opened by brett.cannon

#18195: error when deep copying module is confusing
http://bugs.python.org/issue18195  opened by mrjbq7

#18197: insufficient error checking causes crash on windows
http://bugs.python.org/issue18197  opened by maxdeliso

#18199: Windows: support path longer than 260 bytes using "\\?\" prefi
http://bugs.python.org/issue18199  opened by Voo

#18201: distutils write into symlinks instead of replacing them
http://bugs.python.org/issue18201  opened by mgorny

#18202: Minor fixes for test_coding
http://bugs.python.org/issue18202  opened by serhiy.storchaka

#18203: Replace calls to malloc() with PyMem_Malloc()
http://bugs.python.org/issue18203  opened by haypo

#18206: license url in site.py should always use X.Y.Z form of version
http://bugs.python.org/issue18206  opened by py.user

#18207: OpenSSL may ignore seconds in notAfter
http://bugs.python.org/issue18207  opened by christian.heimes

#18209: Bytearray type not supported as a mutable object in the fcntl.
http://bugs.python.org/issue18209  opened by vxgmichel

#18211: -Werror=statement-after-declaration problem
http://bugs.python.org/issue18211  opened by ronaldoussoren

#18212: No way to check whether Future is finished?
http://bugs.python.org/issue18212  opened by cool-RR

#18213: py-bt errors on backtrace with PyRun_SimpleString and friends
http://bugs.python.org/issue18213  opened by pmuldoon

#18214: Stop purging modules which are garbage collected before shutdo
http://bugs.python.org/issue18214  opened by sbt

#18215: Script to test multiple versions of OpenSSL
http://bugs.python.org/issue18215  opened by christian.heime

Re: [Python-Dev] python symbolizition

2013-06-14 Thread Isaac Morland

On Fri, 14 Jun 2013, R. David Murray wrote:


This discussion is better suited for python-ideas than it
is for python-dev.


Better yet, perl-ideas.


On Fri, 14 Jun 2013 16:50:53 +0200, Markus Unterwaditzer 
 wrote:

But why?

-- Markus (from phone)

Pynix Wang  wrote:

1.lambda expression

c#
a.(x, y) => x == y
b.() => SomeMethod()

ruby:
-> {|msg| puts msg}

python can use c# like and remove "lambda" keyword.

2.global variable

ruby
$glo_var

python can use $ or @ or another and remove "global".


Isaac Morland   CSCF Web Guru
DC 2554C, x36650WWW Software Specialist
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 442 accepted

2013-06-14 Thread Antoine Pitrou
On Wed, 5 Jun 2013 09:10:54 -0700
Benjamin Peterson  wrote:
> I (and Guido) are accepting PEP 442 (Safe object finalization) on the
> condition that finalizers are only ever called once globally.

Ok, so there's an issue with that condition: it can't be upholded on
non-GC objects. Creating a non-GC object is quite obscure and rare,
though, since it requires basically a class with no __dict__ and an
empty __slots__:

class C:
__slots__ = ()
survivors = []

def __del__(self):
self.survivors.append(self)


In this case, a C instance's __del__ will be called every time
destruction is attempted, not only once. Is that a realistic problem?

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 442 accepted

2013-06-14 Thread Antoine Pitrou
On Fri, 14 Jun 2013 19:34:41 +0200
Antoine Pitrou  wrote:
> On Wed, 5 Jun 2013 09:10:54 -0700
> Benjamin Peterson  wrote:
> > I (and Guido) are accepting PEP 442 (Safe object finalization) on the
> > condition that finalizers are only ever called once globally.
> 
> Ok, so there's an issue with that condition: it can't be upholded on
> non-GC objects. Creating a non-GC object is quite obscure and rare,
> though, since it requires basically a class with no __dict__ and an
> empty __slots__: [...]

One more problem, albeit minor: there's a test_descr failure in some
part marked as "CPython implementation detail":

==
FAIL: test_subtype_resurrection (test.test_descr.ClassPropertiesAndMethods)
--
Traceback (most recent call last):
  File "/home/antoine/cpython/finalize/Lib/test/test_descr.py", line 3749, in 
test_subtype_resurrection
self.assertEqual(len(C.container), 1)
AssertionError: 0 != 1


However, the third-party test suites (Tornado, Tulip, SQLAlchemy,
Django, zope.interface) show no regressions.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] eval and triple quoted strings

2013-06-14 Thread Ron Adam



On 06/14/2013 10:36 AM, Guido van Rossum wrote:

Not a bug. The same is done for file input -- CRLF is changed to LF before
tokenizing.



Should this be the same?


python3 -c 'print(bytes("""\r\n""", "utf8"))'
b'\r\n'


>>> eval('print(bytes("""\r\n""", "utf8"))')
b'\n'



Ron




On Jun 14, 2013 8:27 AM, "Walter Dörwald" mailto:[email protected]>> wrote:

Hello all!

This surprised me:

>>> eval("'''\r\n'''")
'\n'

Where did the \r go? ast.literal_eval() has the same problem:

>>> ast.literal_eval("'''\r\n'''")
'\n'

Is this a bug/worth fixing?

Servus,
Walter
_
Python-Dev mailing list
[email protected] 
http://mail.python.org/__mailman/listinfo/python-dev

Unsubscribe:
http://mail.python.org/__mailman/options/python-dev/__guido%40python.org 




___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-14 Thread Martin Schultz
As much as I love python, the following drives me crazy, and I would wish
that some future version would come up with a more consistent approach for
this. And please don't reply with "Too bad if you don't know what type your
data are..." - if I want to implement some generic functionality, I want to
avoid constrains on the user where not absolutely necessary, and I believe
this approach is truely pythonic.

OK - here comes the problem set. These are in fact several related issues.
Clearly, a solution exists for each of them, but you will have to admit
that they are all very different in style and therefore unnecessarily
complicate matters. If you want to write code which works under many
circumstances, it will become quite convoluted.

1. Testing for None:

 From what I read elsewhere, the preferred solution is `if x is None` (or
`if not x is None` in the opposite case). This is fine and it works for
scalars, lists, sets, numpy ndarrays,...

2. Testing for empty lists or empty ndarrays:

 In principle, `len(x) == 0` will do the trick. **BUT** there are several
caveats here:
   - `len(scalar)` raises a TypeError, so you will have to use try and
except or find some other way of testing for a scalar value
   - `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises
a TypeError ("unsized object")
   - `len([[]])` returns a length of 1, which is somehow understandable,
but - I would argue - perhaps not what one might expect initially

 Alternatively, numpy arrays have a size attribute, and
`numpy.array([]).size`, `numpy.array(8.).size`, and
`numpy.array([8.]).size` all return what you would expect. And even
`numpy.array([[]]).size` gives you 0. Now, if I could convert everything to
a numpy array, this might work. But have you ever tried to assign a list of
mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])`
will fail, even though the list inside is perfectly fine as a list.

3. Testing for scalar:

 Let's suppose we knew the number of non-empty elements, and this is 1.
Then there are occasions when you want to know if you have in fact `6` or
`[6]` as an answer (or maybe even `[[6]]`). Obviously, this question is
also relevant for numpy arrays. For the latter, a combination of size and
ndim can help. For other objects, I would be tempted to use something like
`isiterable()`, however, this function doesn't exist, and there are
numerous discussions how one could or should find out if an object is
iterable - none of them truly intuitive. (and is it true that **every**
iterable object is a descendant of collections.Iterable?)

4. Finding the number of elements in an object:

 From the discussion above, it is already clear that `len(x)` is not very
robust for doing this. Just to mention another complication: `len("abcd")`
returns 4, even though this is only one string. Of course this is correct,
but it's a nuisance if you need to to find the number of elements of a list
of strings and if it can happen that you have a scalar string instead of a
1-element list. And, believe me, such situations do occur!

5. Forcing a scalar to become a 1-element list:

 Unfortunately, `list(77)` throws an error, because 77 is not iterable.
`numpy.array(77)` works, but - as we saw above - there will be no len
defined for it. Simply writing `[x]` is dangerous, because if x is a list
already, it will create `[[77]]`, which you generally don't want. Also,
`numpy.array([x])` would create a 2D array if x is already a 1D array or a
list. Often, it would be quite useful to know for sure that a function
result is provided as a list, regardless of how many elements it contains
(because then you can write `res[0]` without risking the danger to throw an
exception). Does anyone have a good suggestion for this one?

6. Detecting None values in a list:

 This is just for completeness. I have seen solutions using `all` which
solve this problem (see [question #1270920][1]). I haven't digged into them
extensively, but I fear that these will also suffer from the
above-mentioned issues if you don't know for sure if you are starting from
a list, a numpy array, or a scalar.

Enough complaining. Here comes my prayer to the python gods: **Please**

 - add a good `isiterable` function
 - add a `size` attribute to all objects (I wouldn't mind if this is None
in case you don't really know how to define the size of something, but it
would be good to have it, so that `anything.size` would never throw an error
 - add an `isscalar` function which would at least try to test if something
is a scalar (meaning a single entity). Note that this might give different
results compared to `isiterable`, because one would consider a scalar
string as a scalar even though it is iterable. And if `isscalar` would
throw exceptions in cases where it doesn't know what to do: fine - this can
be easily captured.
 - enable the `len()` function for scalar variables such as integers or
floats. I would tend to think that 1 is a natural answer to what the l

Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-14 Thread Brett Cannon
On Fri, Jun 14, 2013 at 3:12 PM, Martin Schultz  wrote:

> As much as I love python, the following drives me crazy, and I would wish
> that some future version would come up with a more consistent approach for
> this. And please don't reply with "Too bad if you don't know what type your
> data are..." - if I want to implement some generic functionality, I want to
> avoid constrains on the user where not absolutely necessary, and I believe
> this approach is truely pythonic.
>
> OK - here comes the problem set. These are in fact several related issues.
> Clearly, a solution exists for each of them, but you will have to admit
> that they are all very different in style and therefore unnecessarily
> complicate matters. If you want to write code which works under many
> circumstances, it will become quite convoluted.
>
> 1. Testing for None:
>
>  From what I read elsewhere, the preferred solution is `if x is None` (or
> `if not x is None` in the opposite case). This is fine and it works for
> scalars, lists, sets, numpy ndarrays,...
>

Should actually be ``if x is not None``.

[SNIP]

>
>  - add a good `isiterable` function
>

Done: isinstance(x, collections.abc.Iterable)


>  - add a `size` attribute to all objects (I wouldn't mind if this is None
> in case you don't really know how to define the size of something, but it
> would be good to have it, so that `anything.size` would never throw an error
>

This is what len() is for. I don't know why numpy doesn't define the
__len__ method on their array types for that. We can't force every object
to have it as it doesn't make sense in all cases.


>  - add an `isscalar` function which would at least try to test if
> something is a scalar (meaning a single entity). Note that this might give
> different results compared to `isiterable`, because one would consider a
> scalar string as a scalar even though it is iterable. And if `isscalar`
> would throw exceptions in cases where it doesn't know what to do: fine -
> this can be easily captured.
>

The numbers module has a bunch of ABCs you can use to test for integrals,
etc. just as I suggested for iterables.


>   - enable the `len()` function for scalar variables such as integers or
> floats. I would tend to think that 1 is a natural answer to what the length
> of a number is.
>

The len() function is specifically for containers so it would not make
sense to ask for the length of something you can't put something into or
iterate over.

This is actually a perfect case of using the new single-dispatch generic
function work that has landed in Python 3.4 (
http://docs.python.org/3.4/library/functools.html#functools.singledispatch).
With that you could write your own custom_len() function that dispatches on
the type to return exactly what you are after. E.g.::

  @functools.singledispatch
  def custom_length(ob): return len(ob)

  @custom_length.register(int)
  _(ob): return 1

And on you go for all the types you want to special-case.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-14 Thread Robert Kern

On 2013-06-14 21:03, Brett Cannon wrote:


On Fri, Jun 14, 2013 at 3:12 PM, Martin Schultz mailto:[email protected]>> wrote:



  - add a `size` attribute to all objects (I wouldn't mind if this is None
in case you don't really know how to define the size of something, but it
would be good to have it, so that `anything.size` would never throw an error

This is what len() is for. I don't know why numpy doesn't define the __len__
method on their array types for that.


It does. It gives the size of the first axis, i.e. the one accessed by simple 
indexing with an integer: some_array[i]. The `size` attribute givens the total 
number of items in the possibly-multidimensional array. However, one of the 
other axes can be 0-length, so the array will have no elements but the length 
will be nonzero.


[~]
|4> np.empty([3,4,0])
array([], shape=(3, 4, 0), dtype=float64)

[~]
|5> np.empty([3,4,0])[1]
array([], shape=(4, 0), dtype=float64)

[~]
|6> len(np.empty([3,4,0]))
3

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-14 Thread R. David Murray
On Fri, 14 Jun 2013 21:12:00 +0200, Martin Schultz  wrote:
> 2. Testing for empty lists or empty ndarrays:
> 
>  In principle, `len(x) == 0` will do the trick. **BUT** there are several
> caveats here:
>- `len(scalar)` raises a TypeError, so you will have to use try and
> except or find some other way of testing for a scalar value
>- `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises
> a TypeError ("unsized object")
>- `len([[]])` returns a length of 1, which is somehow understandable,
> but - I would argue - perhaps not what one might expect initially
> 
>  Alternatively, numpy arrays have a size attribute, and
> `numpy.array([]).size`, `numpy.array(8.).size`, and
> `numpy.array([8.]).size` all return what you would expect. And even
> `numpy.array([[]]).size` gives you 0. Now, if I could convert everything to
> a numpy array, this might work. But have you ever tried to assign a list of
> mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])`
> will fail, even though the list inside is perfectly fine as a list.

In general you test whether nor not something is empty in Python by
testing its truth value.  Empty things are False.  Numpy seems to
follow this using size, from the limited examples you have given

   >>> bool(numpy.array([[]])
   False
   >>> bool(numpy.array([[1]])
   True

I have no idea what the definition of numpy.array.size is that it would
return 0 for [[]], so its return value obviously defies my intuition
as much as len([[]]) seems to have initially defied yours :)

> 3. Testing for scalar:
> 
>  Let's suppose we knew the number of non-empty elements, and this is 1.
> Then there are occasions when you want to know if you have in fact `6` or
> `[6]` as an answer (or maybe even `[[6]]`). Obviously, this question is
> also relevant for numpy arrays. For the latter, a combination of size and
> ndim can help. For other objects, I would be tempted to use something like
> `isiterable()`, however, this function doesn't exist, and there are
> numerous discussions how one could or should find out if an object is
> iterable - none of them truly intuitive. (and is it true that **every**
> iterable object is a descendant of collections.Iterable?)

No, but...I'm not 100% sure about this as I tend to stay away from ABCs
myself, but my understanding is that collections.Iterable checks if an
object is iterable when you use it in an isinstance check.  There are
probably ways to fool it, but I think you could argue that any such data
types are broken.

> 4. Finding the number of elements in an object:
> 
>  From the discussion above, it is already clear that `len(x)` is not very
> robust for doing this. Just to mention another complication: `len("abcd")`
> returns 4, even though this is only one string. Of course this is correct,
> but it's a nuisance if you need to to find the number of elements of a list
> of strings and if it can happen that you have a scalar string instead of a
> 1-element list. And, believe me, such situations do occur!

len is robust when you consider that it only applies to sequences (see
below).  (I don't know what it means to "code a scaler as a numpy
array", but if it is still a scaler, it makes sense that it raises
a TypeError on len...it should.)

> 5. Forcing a scalar to become a 1-element list:
> 
>  Unfortunately, `list(77)` throws an error, because 77 is not iterable.
> `numpy.array(77)` works, but - as we saw above - there will be no len
> defined for it. Simply writing `[x]` is dangerous, because if x is a list
> already, it will create `[[77]]`, which you generally don't want. Also,
> `numpy.array([x])` would create a 2D array if x is already a 1D array or a
> list. Often, it would be quite useful to know for sure that a function
> result is provided as a list, regardless of how many elements it contains
> (because then you can write `res[0]` without risking the danger to throw an
> exception). Does anyone have a good suggestion for this one?

Well, no.  If the list is empty res[0] will throw an error.  You need
to know both that it is indexable (note: not iterable...an object can be
iterable without being indexable) and that it is non-empty.  Well behaved
objects should I think pass an isinstance check against collections.Sequence.
(I can't think of a good way to check for indexability without the abc.)

> Enough complaining. Here comes my prayer to the python gods: **Please**
> 
>  - add a good `isiterable` function

That would be spelled isinstance(x, collections.Iterable), it seems.

>  - add a `size` attribute to all objects (I wouldn't mind if this is None
> in case you don't really know how to define the size of something, but it
> would be good to have it, so that `anything.size` would never throw an error

Why?  What is the definition of 'size' that makes it useful outside
of numpy?

>  - add an `isscalar` function which would at least try to test if something
> is a scalar (meaning a single entity). Note that thi

Re: [Python-Dev] eval and triple quoted strings

2013-06-14 Thread PJ Eby
On Fri, Jun 14, 2013 at 2:11 PM, Ron Adam  wrote:
>
>
> On 06/14/2013 10:36 AM, Guido van Rossum wrote:
>>
>> Not a bug. The same is done for file input -- CRLF is changed to LF before
>> tokenizing.
>
>
>
> Should this be the same?
>
>
> python3 -c 'print(bytes("""\r\n""", "utf8"))'
> b'\r\n'
>
>
 eval('print(bytes("""\r\n""", "utf8"))')
> b'\n'

No, but:

eval(r'print(bytes("""\r\n""", "utf8"))')

should be.  (And is.)

What I believe you and Walter are missing is that the \r\n in the eval
strings are converted early if you don't make the enclosing string
raw.  So what you're eval-ing is not what you think you are eval-ing,
hence the confusion.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-14 Thread Robert Kern

On 2013-06-14 21:55, R. David Murray wrote:

On Fri, 14 Jun 2013 21:12:00 +0200, Martin Schultz  wrote:

2. Testing for empty lists or empty ndarrays:

  In principle, `len(x) == 0` will do the trick. **BUT** there are several
caveats here:
- `len(scalar)` raises a TypeError, so you will have to use try and
except or find some other way of testing for a scalar value
- `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises
a TypeError ("unsized object")
- `len([[]])` returns a length of 1, which is somehow understandable,
but - I would argue - perhaps not what one might expect initially

  Alternatively, numpy arrays have a size attribute, and
`numpy.array([]).size`, `numpy.array(8.).size`, and
`numpy.array([8.]).size` all return what you would expect. And even
`numpy.array([[]]).size` gives you 0. Now, if I could convert everything to
a numpy array, this might work. But have you ever tried to assign a list of
mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])`
will fail, even though the list inside is perfectly fine as a list.


In general you test whether nor not something is empty in Python by
testing its truth value.  Empty things are False.  Numpy seems to
follow this using size, from the limited examples you have given

>>> bool(numpy.array([[]])
False
>>> bool(numpy.array([[1]])
True


numpy does not do so. Empty arrays are extremely rare and testing for them rarer 
(rarer still is testing for emptiness not knowing if it is an array or some 
other sequence). What people usually want from bool(some_array) is either 
some_array.all() or some_array.any(). In the face of this ambiguity, numpy 
refuses the temptation to guess and raises an exception explaining matters.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

2013-06-14 Thread Robert Kern

On 2013-06-14 23:31, Robert Kern wrote:

On 2013-06-14 21:55, R. David Murray wrote:

On Fri, 14 Jun 2013 21:12:00 +0200, Martin Schultz  wrote:

2. Testing for empty lists or empty ndarrays:

  In principle, `len(x) == 0` will do the trick. **BUT** there are several
caveats here:
- `len(scalar)` raises a TypeError, so you will have to use try and
except or find some other way of testing for a scalar value
- `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises
a TypeError ("unsized object")
- `len([[]])` returns a length of 1, which is somehow understandable,
but - I would argue - perhaps not what one might expect initially

  Alternatively, numpy arrays have a size attribute, and
`numpy.array([]).size`, `numpy.array(8.).size`, and
`numpy.array([8.]).size` all return what you would expect. And even
`numpy.array([[]]).size` gives you 0. Now, if I could convert everything to
a numpy array, this might work. But have you ever tried to assign a list of
mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])`
will fail, even though the list inside is perfectly fine as a list.


In general you test whether nor not something is empty in Python by
testing its truth value.  Empty things are False.  Numpy seems to
follow this using size, from the limited examples you have given

>>> bool(numpy.array([[]])
False
>>> bool(numpy.array([[1]])
True


numpy does not do so. Empty arrays are extremely rare and testing for them rarer
(rarer still is testing for emptiness not knowing if it is an array or some
other sequence). What people usually want from bool(some_array) is either
some_array.all() or some_array.any(). In the face of this ambiguity, numpy
refuses the temptation to guess and raises an exception explaining matters.


Actually, that's a bit of a lie. In the empty case and the one-element case, we 
do return a bool, False for empty and bool(element) for whatever that one 
element is. Anything else raises the exception since we don't know whether it is 
all() or any() that was desired.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] [Python-checkins] cpython: Closes issue 17947. Adds PEP-0435 (Enum, IntEnum) to the stdlib.

2013-06-14 Thread Brett Cannon
Ethan, did you forget to run ``hg add`` before committing? If not then why
the heck did we argue over enums for so long if this was all it took to
make everyone happy? =)


On Fri, Jun 14, 2013 at 3:31 AM, ethan.furman wrote:

> http://hg.python.org/cpython/rev/fae92309c3be
> changeset:   84117:fae92309c3be
> parent:  84115:af27c661d4fb
> user:Ethan Furman 
> date:Fri Jun 14 00:30:27 2013 -0700
> summary:
>   Closes issue 17947.  Adds PEP-0435 (Enum, IntEnum) to the stdlib.
>
> files:
>   Doc/library/datatypes.rst |  1 +
>   1 files changed, 1 insertions(+), 0 deletions(-)
>
>
> diff --git a/Doc/library/datatypes.rst b/Doc/library/datatypes.rst
> --- a/Doc/library/datatypes.rst
> +++ b/Doc/library/datatypes.rst
> @@ -30,3 +30,4 @@
> copy.rst
> pprint.rst
> reprlib.rst
> +   enum.rst
>
> --
> Repository URL: http://hg.python.org/cpython
>
> ___
> Python-checkins mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Allow calling PyMem_Malloc() without the GIL held in Python 3.4

2013-06-14 Thread Victor Stinner
I commited the new API (little bit different than my last patch on issue #3329):
http://hg.python.org/cpython/rev/6661a8154eb3

The documentation will be available in a few minutes at:
http://docs.python.org/3/c-api/memory.html

2013/6/14 Kristján Valur Jónsson :
>> Removing the GIL restriction would help to replace direct calls to
>> malloc() with PyMem_Malloc(). Using PyMem_SetAllocators(), an application
>> would be able to replace memory allocators, and these allocators would be
>> used "everywhere".
>> => see http://bugs.python.org/issue18203
>
> To keep this interesting, I have a somewhat different opinion to Victor :)
>   have put comments in the original defect, but would like to repeat them 
> here.
> IMHO, keeping the GIL restriction on PyMem_MALLOC is useful.
> 1) It allows it to be replaced with PyObject_MALLOC(). Or anything else.  In 
> particular, an implementer is free to add memory profiling support and other 
> things without worrying about implementation details.  Requiring it to be GIL 
> free severely limits what it can do.  For example, it would be forbidden to 
> delegate to PyObject_MALLOC when debugging.

For my own pytracemalloc tool, holding the GIL while calling
PyMem_Malloc() is required to be able to retrieve the Python filename
and line number of the caller. So you convinced me :-) I am also
worried by the backward compatibility, even if I expect that only a
very few developers replaced Python memory allocators. A custom memory
allocator may not be thread-safe, so the GIL can also be convinient.

I added new functions in the "final" API: PyMem_RawMalloc(),
PyMem_RawRealloc(), PyMem_RawFree(). These functions are just wrapper
for malloc(), realloc() and free(). The GIL does not need to be hold.
No process is done before/after at all. Behaviour of
PyMem_RawMalloc(0) is platform depend for example. "size >
PY_SSIZE_T_MAX" check is not done neither, but it may be interesting
to add this check for security reasons (it is already in place for
PyMem_Malloc and PyObject_Malloc).

Using these new functions instead of malloc/realloc/free is
interesting because the internal functions can be replaced with
PyMem_SetRawAllocators() and many checks are added in debug mode (ex:
check for buffer under- and overflow).

PyObject_Malloc() was not documented, so I did not document
PyObject_SetAllocators().

In the final API, I added a new PyMemAllocators structure to simplify
the API. I also made _PyObject_SetArenaAllocators() private because I
don't like its API (it is not homogenous with PyMem_SetAllocators) and
it is concerned by less use cases. I prefer to wait a little before
making this API public.

I didn't use "#ifndef Py_LIMITED_API", so all new functions are part
of the stable API. Is it correct?

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-14 Thread Antoine Pitrou
On Sat, 15 Jun 2013 00:44:11 +0200 (CEST)
victor.stinner  wrote:
> http://hg.python.org/cpython/rev/6661a8154eb3
> changeset:   84127:6661a8154eb3
> user:Victor Stinner 
> date:Sat Jun 15 00:37:46 2013 +0200
> summary:
>   Issue #3329: Add new APIs to customize memory allocators
> 
> * Add a new PyMemAllocators structure
> * New functions:
> 
>   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
> allocator functions
>   - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
>   - PyMem_GetAllocators(), PyMem_SetAllocators()
>   - PyMem_SetupDebugHooks()
>   - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()

My two cents, but I would prefer if this whole changeset was reverted.
I think it adds too much complexity in the memory allocation APIs,
for a pretty specialized benefit. IMHO, we should be able to get by with
less allocation APIs (why the new _Raw APIs) and less hook-setting
functions.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-14 Thread Victor Stinner
2013/6/15 Antoine Pitrou :
>> http://hg.python.org/cpython/rev/6661a8154eb3
>> ...
>>   Issue #3329: Add new APIs to customize memory allocators
>>
>> * Add a new PyMemAllocators structure
>> * New functions:
>>
>>   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
>> allocator functions
>>   - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
>>   - PyMem_GetAllocators(), PyMem_SetAllocators()
>>   - PyMem_SetupDebugHooks()
>>   - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
>
> My two cents, but I would prefer if this whole changeset was reverted.
> I think it adds too much complexity in the memory allocation APIs,
> for a pretty specialized benefit. IMHO, we should be able to get by with
> less allocation APIs (why the new _Raw APIs) and less hook-setting
> functions.

Ok, I reverted my commit.

I posted my initial patch 3 months ago on the bug tracker. I got some
reviews and discussed with Kristján Valur Jónsson who heavily modified
Python for his game at CCP. I started two threads on python-dev this
week (ok, only two days ago). I thaugh that the last known issues were
fixed with the addition of PyMem_SetupDebugHooks() (to avoid an
environment variable, as asked by Nick) and PyMem_RawMalloc() (have a
GIL-free allocator).

I will work on a PEP to explain all these new functions and their use cases.

**

The addition of PyMem_RawMalloc() is motivated by the issue #18203
(Replace calls to malloc() with PyMem_Malloc()). The goal is to be
able to setup a custom allocator for *all* allocation made by Python,
so malloc() should not be called directly. PyMem_RawMalloc() is
required in places where the GIL is not held (ex: in os.getcwd() on
Windows).

PyMem_Malloc() is misused (called without the GIL held) in different
places. Examples: the readline modules and functions called at Python
startup, including main(). Replacing PyMem_Malloc() with malloc()
would not allow to use the custom allocator everywhere, so
PyMem_RawMalloc() is also required here.

The last point is an extension of the issue #18203: some external
libraries like zlib or OpenSSL are also calling malloc() directly. But
Python can configure these libraries to use a custom memory allocator.
I plan to configure external libraries to use PyMem_GetRawAllocators()
if PyMem_SetRawAllocators() was called (if PyMem_RawMalloc is not
simply malloc) and if setting a custom allocator only affect a
function and not the whole library.

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] eval and triple quoted strings

2013-06-14 Thread Greg Ewing

Guido van Rossum wrote:
Not a bug. The same is done for file input -- CRLF is changed to LF 
before tokenizing.


I'm not convinced it's reasonable behaviour to re-scan the
string as though it's being read from a file. It's a Python
string, so it's already been through whatever line-ending
transformation is appropriate to get it into memory.

--
Greg
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 442 accepted

2013-06-14 Thread Nick Coghlan
On 15 June 2013 03:34, Antoine Pitrou  wrote:
> On Wed, 5 Jun 2013 09:10:54 -0700
> Benjamin Peterson  wrote:
>> I (and Guido) are accepting PEP 442 (Safe object finalization) on the
>> condition that finalizers are only ever called once globally.
>
> Ok, so there's an issue with that condition: it can't be upholded on
> non-GC objects. Creating a non-GC object is quite obscure and rare,
> though, since it requires basically a class with no __dict__ and an
> empty __slots__:
>
> class C:
> __slots__ = ()
> survivors = []
>
> def __del__(self):
> self.survivors.append(self)
>
>
> In this case, a C instance's __del__ will be called every time
> destruction is attempted, not only once. Is that a realistic problem?

So, to trigger that __del__() method a second time, such an object
would have to be:

1. Defined in the first place (the use cases for stateless objects
with destructors seem rare...)
2. Hanging off a reference cycle
3. Which then gets resurrected

If it isn't easy to fix by changing the way such classes are
constructed by typeobject.c (i.e. force them to be GC-aware when they
define a __del__ method, since they may still be hanging off the edge
of a reference cycle, even if they can't form one themselves), I'd say
chalk it up as a CPython wart and not worry about it for now.

As far as the test_subtype_resurrection failure goes, I think the
associated comment says it all: "(On non-CPython implementations,
however, __del__ is typically not called again.)"

The test should just be rewritten to check that __del__ is only called
once in the face of resurrection.

Cheers,
Nick.

--
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] eval and triple quoted strings

2013-06-14 Thread Nick Coghlan
On 15 June 2013 14:08, Greg Ewing  wrote:
> Guido van Rossum wrote:
>>
>> Not a bug. The same is done for file input -- CRLF is changed to LF before
>> tokenizing.
>
>
> I'm not convinced it's reasonable behaviour to re-scan the
> string as though it's being read from a file. It's a Python
> string, so it's already been through whatever line-ending
> transformation is appropriate to get it into memory.

No, that's not the way the Python compiler works. The transformation
Guido is talking about is the way the tokenizer identifiers "NEWLINE"
tokens:

>>> list(tokenize.tokenize((l for l in (b"""'\r\n'""", b"")).__next__))[2]
TokenInfo(type=4 (NEWLINE), string='\r\n', start=(1, 1), end=(1, 3),
line="'\r\n'")

This long predates universal newlines mode - it's part of the
compilation process, not part of the IO system. The compiler then sees
the NEWLINE token in the tokenizer output, and inserts a "\n" into the
triple-quoted string.

Cheers,
Nick.

--
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-14 Thread Nick Coghlan
On 15 June 2013 11:54, Victor Stinner  wrote:
> 2013/6/15 Antoine Pitrou :
>>> http://hg.python.org/cpython/rev/6661a8154eb3
>>> ...
>>>   Issue #3329: Add new APIs to customize memory allocators
>>>
>>> * Add a new PyMemAllocators structure
>>> * New functions:
>>>
>>>   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
>>> allocator functions
>>>   - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
>>>   - PyMem_GetAllocators(), PyMem_SetAllocators()
>>>   - PyMem_SetupDebugHooks()
>>>   - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
>>
>> My two cents, but I would prefer if this whole changeset was reverted.
>> I think it adds too much complexity in the memory allocation APIs,
>> for a pretty specialized benefit. IMHO, we should be able to get by with
>> less allocation APIs (why the new _Raw APIs) and less hook-setting
>> functions.
>
> Ok, I reverted my commit.
>
> I posted my initial patch 3 months ago on the bug tracker. I got some
> reviews and discussed with Kristján Valur Jónsson who heavily modified
> Python for his game at CCP. I started two threads on python-dev this
> week (ok, only two days ago). I thaugh that the last known issues were
> fixed with the addition of PyMem_SetupDebugHooks() (to avoid an
> environment variable, as asked by Nick) and PyMem_RawMalloc() (have a
> GIL-free allocator).
>
> I will work on a PEP to explain all these new functions and their use cases.

I think the new APIs are mostly valid and well-justified, but agree a
PEP is a good idea.

Yes, it's a complex solution, but it's solving a complex problem that
arises when embedding CPython inside executables that need to run on
non-traditional platforms where the simple C defined
malloc/realloc/free trio is inadequate.

This is a complementary effort to PEP 432 - that aims to simplify
embedding CPython in general, while Victor's efforts here specifically
focus on situations where it is necessary to better map CPython to an
underlying memory model that differs from the traditional C one. While
the "single heap" model of memory enshrined in the C standard is
certainly the most common model, it's far from being the only one, and
these days CPython also gets used in those other environments.

About the only simplification I can see is that PyMem_RawMalloc(),
PyMem_RawRealloc(), PyMem_RawFree() could perhaps be handled with
preprocessor macros instead of permitting runtime reconfiguration.
Allowing the memory allocations for the CPython runtime to be handled
separately from those for arbitrary C libraries loaded into the
process seems reasonable, though.

Cheers,
Nick.

--
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com