Re: [Python-Dev] PEP 461 updates

[Ethan Furman]

On 01/15/2014 06:12 PM, Glenn Linderman wrote:

On 1/15/2014 4:13 PM, Ethan Furman wrote:


  - no value generated errors

...


%c will insert a single byte, either from an int in range(256), or from
a bytes argument of length 1.


what does

x = 354
b"%c" % x

produce?  Seems that construct produces a value dependent error in both python 2 
& 3 (although it takes a much bigger
value to produce the error in python 3, with str %... with bytes %, the problem 
with be reached at 256, just like python 2).

Is this an intended exception to the overriding principle?


Hmm, thanks for spotting that.  Yes, that would be a value error if anything over 255 is used, both currently in Py2, 
and for bytes in Py3.  As Carl suggested, a little more explanation is needed in the PEP.


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Ethan Furman]

On 01/15/2014 06:45 AM, Brett Cannon wrote:


This is why I have argued that if you specify it as "if there is a format spec 
specified, then the return value from
calling __format__() will have str.decode('ascii', 'strict') called on it" you 
get the support for the various
number-specific format specs for free.


It may work like this under the hood, but it's an implementation detail.  Since the numeric format codes will call int, 
index, or float on the object (to handle subclasses), we could then call __format__ on the resulting int or float to do 
the heavy lifting; but since __format__ on anything else would never be called I don't want to give that impression.



It also means if you pass in a string that you just want the strict ASCII bytes
of then you can get it with {:s}.


This isn't going to happen.  If the user wants a string to be in the byte stream, it has to either be a bytes literal or 
explicitly encoded [1].


--
~Ethan~

[1] Apologies if this has already been answered.  I wanted to make sure I responded to all the ideas/objects, and I may 
have responded more than once to some.  It's been a long few threads.  ;)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460 reboot

[Ethan Furman]

On 01/14/2014 10:58 PM, Stephen J. Turnbull wrote:


At the very least, the "iterated interpolation is a bad idea"
misfeature needs to be documented.


I'm not sure it needs any extra attention.  Even with str, iterated interpolation is tricky -- I've been bitten by it 
more than once, and that even when I controlled the source!  :/


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 updates

[Ethan Furman]

On 01/15/2014 05:17 PM, Carl Meyer wrote:


I think the PEP could really use a rationale section


It will have one before it's done.



Also I think it would be useful to have a section summarizing the
primary objections that have been raised, and why those objections have
been overruled


Excellent point.  That section will also be present.



In order to avoid the problems of auto-conversion and value-generated
exceptions,
all object checking will be done via isinstance, not by values contained
in a
Unicode representation.  In other words::

   - duck-typing to allow/reject entry into a byte-stream
   - no value generated errors


This seems self-contradictory; "isinstance" is type-checking, which is
the opposite of duck-typing.


Good point, I'll reword that.  It will be duck-typing.



I think it might also be good to expand (very) slightly on what "the
problems of auto-conversion and value-generated exceptions" are


Will do.



.. [2] TypeError, ValueError, or UnicodeEncodeError?


TypeError seems right to me. Definitely not UnicodeEncodeError - refusal
to implicitly encode is not at all the same thing as an encoding error.


That's the direction I'm leaning, too.

Thanks for your comments!

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AC Derby and accepting None for optional positional arguments

[Larry Hastings]

On 01/15/2014 10:21 PM, Georg Brandl wrote:

Am 16.01.2014 05:35, schrieb Ryan Smith-Roberts:

On Wed, Jan 15, 2014 at 7:57 PM, Ryan Smith-Roberts mailto:r...@lab.net>> wrote:

 socket.getservbyname(servicename[, protocolname])
 ->
 socket.getservbyname(servicename, protocolname=None)


Here is a more complicated example, since the above does technically have an
alternative fix:

sockobj.sendmsg(buffers[, ancdata[, flags[, address]]])
->
sockobj.sendmsg(buffers, ancdata=None, flags=0, address=None)

As far as I understand you should convert these with the "optional group" syntax
(i.e. brackets).


That's correct.  Functions that use PyArg_ParseTuple should continue to 
use PyArg_ParseTuple.  Ryan: add a "/" on a line by itself after the 
last parameter of each of these functions, and that will let you use the 
[ ] syntax too.  Please see the Argument Clinic howto for more.



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AC Derby and accepting None for optional positional arguments

[Georg Brandl]

Am 16.01.2014 05:35, schrieb Ryan Smith-Roberts:
> On Wed, Jan 15, 2014 at 7:57 PM, Ryan Smith-Roberts  > wrote:
> 
> socket.getservbyname(servicename[, protocolname])
> ->
> socket.getservbyname(servicename, protocolname=None)
> 
> 
> Here is a more complicated example, since the above does technically have an
> alternative fix:
> 
> sockobj.sendmsg(buffers[, ancdata[, flags[, address]]])
> ->
> sockobj.sendmsg(buffers, ancdata=None, flags=0, address=None)

As far as I understand you should convert these with the "optional group" syntax
(i.e. brackets).

Georg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AC Derby and accepting None for optional positional arguments

[Guido van Rossum]

On Wed, Jan 15, 2014 at 9:55 PM, Larry Hastings  wrote:
>
> On 01/15/2014 09:37 PM, Guido van Rossum wrote:
>
> Well, I think these are mostly artifacts from old times, and usually passing
> None *should* be the same as omitting the argument. But check each case!
>
>
> Vajrasky Kok's recent posting to python-dev discusses the same problem.  His
> example is itertools.repeat's second parameter, which is slightly nastier.
> Consider the implementation:
>
> static PyObject *
> repeat_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
> {
> repeatobject *ro;
> PyObject *element;
> Py_ssize_t cnt = -1;
> static char *kwargs[] = {"object", "times", NULL};
>
> if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|n:repeat", kwargs,
>  &element, &cnt))
> return NULL;
>
> if (PyTuple_Size(args) == 2 && cnt < 0)
> cnt = 0;
>
>
> I draw your attention to the last two lines.  And remember, Argument Clinic
> doesn't provide the "args" and "kwargs" parameters to the "impl" function.
> That means two things:
>
> itertools.repeat deliberately makes it impossible to provide an argument for
> "times" that behaves the same as not providing the "times" argument, and
> there is currently no way to implement this behavior using Argument Clinic.
> (I'd have to add a hack where impl functions also get args and kwargs.)
>
>
> Passing in "None" here is inconvenient as it's an integer argument.  -1
> actually seems like a pretty sane default to mean "repeat forever", but the
> author has gone to some effort to prevent this.  I therefore assume they had
> a very good reason.  So again we seem stuck.
>
> Are you suggesting that, when converting builtins to Argument Clinic with
> unrepresentable default values, we're permitted to tweak the defaults to
> something representable?

In this specific case it's clear to me that the special-casing of
negative count is intentional -- presumably it emulates sequence
repetition, where e.g. 'a'*-1 == ''.

But not accepting None is laziness -- accepting either a number or
None requires much more effort, if you need to have the number as a C
integer. I'm not sure how AC could make this any easier, unless you
want to special-case maxint or -maxint-1.

In the sha1 example, however, accepting None and converting it to NULL
(without a reference leak, please :-) seems fine though.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AC Derby and accepting None for optional positional arguments

[Larry Hastings]

On 01/15/2014 09:37 PM, Guido van Rossum wrote:
Well, I think these are mostly artifacts from old times, and usually 
passing None *should* be the same as omitting the argument. But check 
each case!


Vajrasky Kok's recent posting to python-dev discusses the same problem.  
His example is itertools.repeat's second parameter, which is slightly 
nastier.  Consider the implementation:


   static PyObject *
   repeat_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
   {
repeatobject *ro;
PyObject *element;
Py_ssize_t cnt = -1;
static char *kwargs[] = {"object", "times", NULL};

if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|n:repeat", kwargs,
 &element, &cnt))
return NULL;

if (PyTuple_Size(args) == 2 && cnt < 0)
cnt = 0;


I draw your attention to the last two lines.  And remember, Argument 
Clinic doesn't provide the "args" and "kwargs" parameters to the "impl" 
function.  That means two things:


 * itertools.repeat deliberately makes it impossible to provide an
   argument for "times" that behaves the same as not providing the
   "times" argument, and
 * there is currently no way to implement this behavior using Argument
   Clinic.  (I'd have to add a hack where impl functions also get args
   and kwargs.)


Passing in "None" here is inconvenient as it's an integer argument. -1 
actually seems like a pretty sane default to mean "repeat forever", but 
the author has gone to some effort to prevent this.  I therefore assume 
they had a very good reason.  So again we seem stuck.


Are you suggesting that, when converting builtins to Argument Clinic 
with unrepresentable default values, we're permitted to tweak the 
defaults to something representable?



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AC Derby and accepting None for optional positional arguments

[Guido van Rossum]

Well, I think these are mostly artifacts from old times, and usually
passing None *should* be the same as omitting the argument. But check each
case!

On Wednesday, January 15, 2014, Larry Hastings  wrote:

>
> On 01/15/2014 08:35 PM, Ryan Smith-Roberts wrote:
>
>  On Wed, Jan 15, 2014 at 7:57 PM, Ryan Smith-Roberts 
> 
> > wrote:
>
>>  socket.getservbyname(servicename[, protocolname])
>> ->
>> socket.getservbyname(servicename, protocolname=None)
>>
>
>  Here is a more complicated example, since the above does technically
> have an alternative fix:
>
>  sockobj.sendmsg(buffers[, ancdata[, flags[, address]]])
> ->
> sockobj.sendmsg(buffers, ancdata=None, flags=0, address=None)
>
>
> I feel like Ryan didn't sufficiently explain the problem.  I'll elaborate.
>
>
> For functions implemented in Python, it's always true that:
>
>- a parameter that is optional always has a default value, and
>- this default value is visible to Python and is a Python value.
>
> The default value of every parameter is part of the function's
> signature--you can see them with inspect.signature() or
> inspect.getfullargspec().
>
> A corollary of this: for every function implemented in Python, you can
> construct a call to it where you fill in every optional value with its
> published default value, and this is exactly equivalent to calling it
> without specifying those arguments:
>
> def foo(a=any_python_value): ...
>
> sig = inspect.signature(foo)
> defaults = [p.default for p in sig.parameters.values()]
> foo(*defaults) == foo()
>
> Assuming neither foo nor "any_python_value" have side effects, those two
> calls to foo() will be exactly the same in every way.
>
>
> But!  Builtin functions implemented in C commonly have optional parameters
> whose default value is not only opaque to Python, it's not renderable as a
> Python value.  That default value is NULL.  Consider the first two
> paragraphs of SHA1_new() in Modules/sha1module.c:
>
> static PyObject *
> SHA1_new(PyObject *self, PyObject *args, PyObject *kwdict)
> {
> static char *kwlist[] = {"string", NULL};
> SHA1object *new;
> PyObject *data_obj = NULL;
> Py_buffer buf;
>
> if (!PyArg_ParseTupleAndKeywords(args, kwdict, "|O:new", kwlist,
>  &data_obj)) {
> return NULL;
> }
> ...
>
> The "string" parameter is optional.  Its value, if specified, is written
> to "data_obj".  "data_obj" has a default value of NULL.  There is no Python
> value you could pass in to this function that would result in "data_obj"
> being (redundantly) set to NULL.  In Python SHA1_new is known as
> "_sha1.sha1".  And:
>
> sig = inspect.signature(_sha1.sha1)
> defaults = [p.default for p in sig.parameters.values()]
> _sha1.sha1(*defaults) == _sha1.sha1()
>
> There's no value we could put in the signature for _sha1.sha1 that would
> behave exactly the same as that NULL.
>
> The ultimate goal of Argument Clinic is to provide introspection
> information for builtins.  But we can't provide a default value to Python
> for the "string" parameter to _sha1.sha1() without changing the semantics
> of the function.  We're stuck.
>
> Ryan's question, then, is "can we change a function like this so it
> accepts None?"  My initial reaction is "no".  That might be okay for
> _sha1.sha1(), but it'd be best to avoid.
>
> In the specific case of SHA1_new's "string" parameter, we could lie and
> claim that the default value is b''.  Internally we could still use NULL as
> a default and get away with it.  But this is only a happy coincidence.
> Many (most?) functions like this won't have a clever Python value we can
> trick you with.
>
> What else could we do?  We could add a special value, let's call it
> sys.NULL, whose specific semantics are "turns into NULL when passed into
> builtins".  This would solve the problem but it's really, really awful.
>
> The only other option I can see: don't convert SHA1_new() to use Argument
> Clinic, and don't provide introspection information for it either.
>
> Can you, gentle reader, suggest a better option?
>
>
> */arry*
>
> p.s. Ryan's function signatures above suggest that he's converting code
> from using PyArg_ParseTuple into using PyArg_ParseTupleAndKeywords.  I
> don't think he's *actually* doing that, and if I saw that in patches
> submitted to me I would ask that it be fixed.
>


-- 
--Guido van Rossum (on iPad)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AC Derby and accepting None for optional positional arguments

[Larry Hastings]

On 01/15/2014 08:35 PM, Ryan Smith-Roberts wrote:
On Wed, Jan 15, 2014 at 7:57 PM, Ryan Smith-Roberts > wrote:


socket.getservbyname(servicename[, protocolname])
->
socket.getservbyname(servicename, protocolname=None)


Here is a more complicated example, since the above does technically 
have an alternative fix:


sockobj.sendmsg(buffers[, ancdata[, flags[, address]]])
->
sockobj.sendmsg(buffers, ancdata=None, flags=0, address=None)


I feel like Ryan didn't sufficiently explain the problem.  I'll elaborate.


For functions implemented in Python, it's always true that:

 * a parameter that is optional always has a default value, and
 * this default value is visible to Python and is a Python value.

The default value of every parameter is part of the function's 
signature--you can see them with inspect.signature() or 
inspect.getfullargspec().


A corollary of this: for every function implemented in Python, you can 
construct a call to it where you fill in every optional value with its 
published default value, and this is exactly equivalent to calling it 
without specifying those arguments:


   def foo(a=any_python_value): ...

   sig = inspect.signature(foo)
   defaults = [p.default for p in sig.parameters.values()]
   foo(*defaults) == foo()

Assuming neither foo nor "any_python_value" have side effects, those two 
calls to foo() will be exactly the same in every way.



But!  Builtin functions implemented in C commonly have optional 
parameters whose default value is not only opaque to Python, it's not 
renderable as a Python value.  That default value is NULL. Consider the 
first two paragraphs of SHA1_new() in Modules/sha1module.c:


   static PyObject *
   SHA1_new(PyObject *self, PyObject *args, PyObject *kwdict)
   {
static char *kwlist[] = {"string", NULL};
SHA1object *new;
PyObject *data_obj = NULL;
Py_buffer buf;

if (!PyArg_ParseTupleAndKeywords(args, kwdict, "|O:new", kwlist,
 &data_obj)) {
return NULL;
}
   ...

The "string" parameter is optional.  Its value, if specified, is written 
to "data_obj".  "data_obj" has a default value of NULL. There is no 
Python value you could pass in to this function that would result in 
"data_obj" being (redundantly) set to NULL.  In Python SHA1_new is known 
as "_sha1.sha1".  And:


   sig = inspect.signature(_sha1.sha1)
   defaults = [p.default for p in sig.parameters.values()]
   _sha1.sha1(*defaults) == _sha1.sha1()

There's no value we could put in the signature for _sha1.sha1 that would 
behave exactly the same as that NULL.


The ultimate goal of Argument Clinic is to provide introspection 
information for builtins.  But we can't provide a default value to 
Python for the "string" parameter to _sha1.sha1() without changing the 
semantics of the function.  We're stuck.


Ryan's question, then, is "can we change a function like this so it 
accepts None?"  My initial reaction is "no".  That might be okay for 
_sha1.sha1(), but it'd be best to avoid.


In the specific case of SHA1_new's "string" parameter, we could lie and 
claim that the default value is b''.  Internally we could still use NULL 
as a default and get away with it.  But this is only a happy 
coincidence.  Many (most?) functions like this won't have a clever 
Python value we can trick you with.


What else could we do?  We could add a special value, let's call it 
sys.NULL, whose specific semantics are "turns into NULL when passed into 
builtins".  This would solve the problem but it's really, really awful.


The only other option I can see: don't convert SHA1_new() to use 
Argument Clinic, and don't provide introspection information for it either.


Can you, gentle reader, suggest a better option?


//arry/

p.s. Ryan's function signatures above suggest that he's converting code 
from using PyArg_ParseTuple into using PyArg_ParseTupleAndKeywords.  I 
don't think he's *actually* doing that, and if I saw that in patches 
submitted to me I would ask that it be fixed.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AC Derby and accepting None for optional positional arguments

[Ryan Smith-Roberts]

On Wed, Jan 15, 2014 at 7:57 PM, Ryan Smith-Roberts  wrote:

> socket.getservbyname(servicename[, protocolname])
> ->
> socket.getservbyname(servicename, protocolname=None)
>

Here is a more complicated example, since the above does technically have
an alternative fix:

sockobj.sendmsg(buffers[, ancdata[, flags[, address]]])
->
sockobj.sendmsg(buffers, ancdata=None, flags=0, address=None)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460 reboot

[Stephen J. Turnbull]

Nick Coghlan writes:

 > Yes, I'm currently thinking the appropriate approach to the docs
 > will be to remove the current "these have most of the str methods
 > too" paragraph for binary sequences and instead create three
 > completely explicit lists of methods:

 >   - provided, works with arbitrary data

 >   - provided, assumes the use of an ASCII compatible data format

I'm not sure what that means.  If you mean that in the format string
for .format() and %-formatting, bytes 0-127 must always have ASCII
coded character semantics with bytes 128-255 unrestricted, indeed,
that is the pragmatic restriction.  Is there anything else?

The implications of this should be made clear, though: funky Asian
encodings cannot be safely used in format strings for format(),
GB18030 isn't safe in %-formatting either, and the value returned by
these operations should be assumed to be non-ASCII-compatible unless
proven otherwise (no iterated formatting).

I think you also need

  - provided, assumes pure ASCII-encoded text

since as far as I know the only strictly ASCII-compatible binary
formats are ISO 2022-compatible encodings and UTF-8, ie, text, and the
characters represented with bytes in the range 128-255 are not handled
by bytes versions of the case-checking and case-converting operations,
and so have extremely dubious semantics unless the data is pure ASCII.
This is also true of most of the is_* operations.

Note that .center and .strip have pretty dubious semantics for
arbitrary "ASCII-compatible" data:

>>> b"abc\r\n".center(15)
b' abc\r\n '

>>> " \xA0abc\xA0 ".strip()
'abc'
>>> b" \xA0abc\xA0 ".strip()
b'\xa0abc\xa0'

Of course the case of .center() is purely a programmer error, and I
don't have a use case where it's problematic in practice.  But it's
sort of unpleasant.

Although I have internalized Guido's point that what's important is
that there be no implicit conversions between bytes and str, I still
worry that this slew of subtle semantic differences when moving str
methods wholesale to bytes is a bug magnet.

I have an especially bad feeling about str-into-bytes interpolation.
If people want that, they should use a type like asciistr that
provides more or less firm guarantees that the content is pure ASCII.

 >   - not provided

 > PEP 461 would add a fourth category, of being provided, but with
 > more restricted semantics.

I haven't looked closely at PEP 461 yet, and I'm not sure I'm going to
have time this week.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] AC Derby and accepting None for optional positional arguments

[Ryan Smith-Roberts]

One of the downsides of converting positional-only functions to Argument
Clinic is that it can result in misleading docstring signatures. Example:

socket.getservbyname(servicename[, protocolname])
->
socket.getservbyname(servicename, protocolname=None)

The problem with the new signature is that it indicates passing None for
protocolname is the same as omitting it (the other, much larger problem is
that it falsely indicates keyword compatibility, but that's a separate
indoor elephant).

My question:

Is it OK to change a longstanding function to treat None like an absent
parameter, where previously it was an error? (This also entails a docs
update and maybe a changelog entry)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 updates

[Greg Ewing]

Glenn Linderman wrote:


x = 354
b"%c" % x

Is this an intended exception to the overriding principle?


I think it's an unavoidable one, unless we want to
introduce an "integer in the range 0-255" type. But
that would just push the problem into another place,
since

   b"%c" % byte(x)

would then blow up on byte(x) if x were out of
range.

If you really want to make sure it won't crash, you
can always do

  b"%c" % (x & 0xff)

or whatever your favourite method of mangling out-
of-range ints is.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Greg Ewing]

Ethan Furman wrote:

Well, I'm not sure what "booted into touch" means,


It's a rugby term, referring to kicking the ball
over the touch line.

As a metaphor, it seems to mean making a problem
go away.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Signature of function with default value uncapturable in Python and C

[Terry Reedy]

On 1/15/2014 9:25 PM, Vajrasky Kok wrote:

Dear friends,


from itertools import repeat
list(repeat('a', 3))

['a', 'a', 'a']

list(repeat('a', 0))

[]

repeat.__doc__

'repeat(object [,times]) -> create an iterator which returns the
object\nfor the specified number of times.  If not specified, returns
the object\nendlessly.'


I think that the doc should say that a negative value is treated as 0 
and that this is enough for a tracker issue after you get more feedback 
or gather more info. There is at least one other builtin/stdlib function 
that does this.



If you omit the times argument:


list(repeat('a'))

... unlimited of a  sometimes it can hang your machine 

In the C code it self, the default value of variable handling times
argument is -1.


Is is necessary to give times a pseudo-default? What is done in other 
places (which are many) where a parameter is optional, with no default?



It checks how many arguments you give to the function.
So if you give -1 directly:


list(repeat('a', -1))

[]

Negative value of times argument means 0.

So what is the correct signature of this function? The value is not
really capturable in Python and C.


The signature in the doc is correct: times is optional, with no default 
value. Instead, the function has a default behavior that does not need 
the value. There are other examples. The (nearly) 'equivalent' Python 
code in the doc fakes this with times=None, but passing None fails. I 
think the same issue occurs in the random module.



repeat(object [,times = unlimited]) 

Can we do this in Clinic? If not, should we?


I should hope that Clinic (and signature objects) can handle no-default 
optional args, as there are several.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 updates

[Guido van Rossum]

Surprisingly, in this case the exception is just what the doctor ordered. :-)

On Wed, Jan 15, 2014 at 6:12 PM, Glenn Linderman  wrote:
> On 1/15/2014 4:13 PM, Ethan Furman wrote:
>
>   - no value generated errors
>
> ...
>
>
> %c will insert a single byte, either from an int in range(256), or from
> a bytes argument of length 1.
>
>
> what does
>
> x = 354
> b"%c" % x
>
> produce?  Seems that construct produces a value dependent error in both
> python 2 & 3 (although it takes a much bigger value to produce the error in
> python 3, with str %... with bytes %, the problem with be reached at 256,
> just like python 2).
>
> Is this an intended exception to the overriding principle?
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Signature of function with default value uncapturable in Python and C

[Vajrasky Kok]

Dear friends,

>>> from itertools import repeat
>>> list(repeat('a', 3))
['a', 'a', 'a']
>>> list(repeat('a', 0))
[]
>>> repeat.__doc__
'repeat(object [,times]) -> create an iterator which returns the
object\nfor the specified number of times.  If not specified, returns
the object\nendlessly.'

If you omit the times argument:

>>> list(repeat('a'))
... unlimited of a  sometimes it can hang your machine 

In the C code it self, the default value of variable handling times
argument is -1. It checks how many arguments you give to the function.
So if you give -1 directly:

>>> list(repeat('a', -1))
[]

Negative value of times argument means 0.

So what is the correct signature of this function? The value is not
really capturable in Python and C.

repeat(object [,times = unlimited]) 

Can we do this in Clinic? If not, should we?

Vajrasky
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 updates

[Glenn Linderman]

On 1/15/2014 4:13 PM, Ethan Furman wrote:
  - no value generated errors 

...


%c will insert a single byte, either from an int in range(256), or from
a bytes argument of length 1. 


what does

x = 354
b"%c" % x

produce?  Seems that construct produces a value dependent error in both 
python 2 & 3 (although it takes a much bigger value to produce the error 
in python 3, with str %... with bytes %, the problem with be reached at 
256, just like python 2).


Is this an intended exception to the overriding principle?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Steven D'Aprano]

On Wed, Jan 15, 2014 at 05:46:07PM -0800, Glenn Linderman wrote:
> On 1/15/2014 4:03 PM, Steven D'Aprano wrote:
> >What precisely does it do? If it's so obvious, why is this thread so
> >long?
> 
> It produces a formatted representation of the object in bytes.  For 
> numbers, that would probably be expected to be ASCII digits and punctuation.
> 
> But other items are not as obvious.

My point exactly.


-- 
Steven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Glenn Linderman]

On 1/15/2014 4:03 PM, Steven D'Aprano wrote:

What precisely does it do? If it's so obvious, why is this thread so
long?


It produces a formatted representation of the object in bytes.  For 
numbers, that would probably be expected to be ASCII digits and punctuation.


But other items are not as obvious.

bytes would probably be expected not to have a __bytes_format__, but if 
a subclass defined one, it might be HEX or Base64 of the base bytes. Or 
if the subclass is ASCII text oriented, it might be the ASCII text 
version of the base bytes (which would be identical to the base bytes, 
except for the type transformation).


str  would probably be expected not to have a __bytes_format__,  but if 
a subclass defined one, it might be HEX or Base64, or it might be a 
specific encoding of the base str.


Other objects might generate an ASCII __repr__, if they define the method.


It took a lot of talk to reach the conclusion, if it has been reached, 
that none of the solution are general enough without defining something 
like __bytes_format__. And before that, a lot of talk to decide that % 
interpolation already had an ASCII bias.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 updates

[Carl Meyer]

Hi Ethan,

I haven't chimed into this discussion, but the direction it's headed
recently seems right to me. Thanks for putting together a PEP. Some
comments on it:

On 01/15/2014 05:13 PM, Ethan Furman wrote:
> 
> Abstract
> 
> 
> This PEP proposes adding the % and {} formatting operations from str to
> bytes [1].

I think the PEP could really use a rationale section summarizing _why_
these formatting operations are being added to bytes; namely that they
are useful when working with various ASCIIish-but-not-properly-text
network protocols and file formats, and in particular when porting code
dealing with such formats/protocols from Python 2.

Also I think it would be useful to have a section summarizing the
primary objections that have been raised, and why those objections have
been overruled (presuming the PEP is accepted). For instance: the main
objection, AIUI, has been that the bytes type is for pure bytes-handling
with no assumptions about encoding, and thus we should not add features
to it that assume ASCIIness, and that may be attractive nuisances for
people writing bytes-handling code that should not assume ASCIIness but
will once they use the feature. And the refutation: that the bytes type
already provides some operations that assume ASCIIness, and these new
formatting features are no more of an attractive nuisance than those;
since the syntax of the formatting mini-languages themselves itself
assumes ASCIIness, there is not likely to be any temptation to use it
with binary data that cannot.

Although it can be hard to arrive at accurate and agreed-on summaries of
the discussion, recording such summaries in the PEP is important; it may
help save our future selves and colleagues from having to revisit all
these same discussions and megathreads.

> Overriding Principles
> =
> 
> In order to avoid the problems of auto-conversion and value-generated
> exceptions,
> all object checking will be done via isinstance, not by values contained
> in a
> Unicode representation.  In other words::
> 
>   - duck-typing to allow/reject entry into a byte-stream
>   - no value generated errors

This seems self-contradictory; "isinstance" is type-checking, which is
the opposite of duck-typing. A duck-typing implementation would not use
isinstance, it would call / check for the existence of a certain magic
method instead.

I think it might also be good to expand (very) slightly on what "the
problems of auto-conversion and value-generated exceptions" are; that
is, that the benefit of Python 3's model is that encoding is explicit,
not implicit, making it harder to unwittingly write code that works as
long as all data is ASCII, but fails as soon as someone feeds in
non-ASCII text data.

Not everyone who reads this PEP will be steeped in years of discussion
about the relative merits of the Python 2 vs 3 models; it doesn't hurt
to spell out a few assumptions.


> Proposed semantics for bytes formatting
> ===
> 
> %-interpolation
> ---
> 
> All the numeric formatting codes (such as %x, %o, %e, %f, %g, etc.)
> will be supported, and will work as they do for str, including the
> padding, justification and other related modifiers, except locale.
> 
> Example::
> 
>>>> b'%4x' % 10
>b'   a'
> 
> %c will insert a single byte, either from an int in range(256), or from
> a bytes argument of length 1.
> 
> Example:
> 
> >>> b'%c' % 48
> b'0'
> 
> >>> b'%c' % b'a'
> b'a'
> 
> %s is restricted in what it will accept::
> 
>   - input type supports Py_buffer?
> use it to collect the necessary bytes
> 
>   - input type is something else?
> use its __bytes__ method; if there isn't one, raise an exception [2]
> 
> Examples:
> 
> >>> b'%s' % b'abc'
> b'abc'
> 
> >>> b'%s' % 3.14
> Traceback (most recent call last):
> ...
> TypeError: 3.14 has no __bytes__ method
> 
> >>> b'%s' % 'hello world!'
> Traceback (most recent call last):
> ...
> TypeError: 'hello world' has no __bytes__ method, perhaps you need
> to encode it?
> 
> .. note::
> 
>Because the str type does not have a __bytes__ method, attempts to
>directly use 'a string' as a bytes interpolation value will raise an
>exception.  To use 'string' values, they must be encoded or otherwise
>transformed into a bytes sequence::
> 
>   'a string'.encode('latin-1')
> 
> format
> --
> 
> The format mini language codes, where they correspond with the
> %-interpolation codes,
> will be used as-is, with three exceptions::
> 
>   - !s is not supported, as {} can mean the default for both str and
> bytes, in both
> Py2 and Py3.
>   - !b is supported, and new Py3k code can use it to be explicit.
>   - no other __format__ method will be called.
> 
> Numeric Format Codes
> 
> 
> To properly handle int and float subclasses, int(), index(), and float()
> will be called on the
> obje

Re: [Python-Dev] PEP 460 reboot

[Nick Coghlan]

On 15 Jan 2014 20:58, "Stephen J. Turnbull"  wrote:
>
> Aside: OK, Guido, ya got me.
>
> I have a separate screed recounting the reasons for my apostasy, but
> that's probably not interesting any more.  I'll send it to individuals
> on request.
>
>  > But in terms of explaining the text model, that
>  > separation is important enough that
>  >
>  > (1)  We should be reluctant to strengthen the
>  >  "its really just ASCII" messages.
>
> True.  I think the right message is is "Unless you know why you
> *desperately* want this, not only don't you need it, but using it is
> the Python equivalent of skydiving without a parachute."
>
> N.B. Don't take the metaphor as an insult.  I think it's become clear
> that those who "desperately want this" not only use parachutes, they
> pack their own.  No need to worry about them.
>
>  > (2)  It *may* be worth creating a virtual
>  >  split in the documentation.
>
> Please don't.  All we need to tell naive users is:
>
> Look at the structure of the bytes.  If that structure is "text",
> convert to str using .decode().  Please don't use bytes.
>
> If that structure isn't text, you're in a specialist domain, and
> it's your problem.  Many structured uses of bytes use ASCII-
> encoded keywords: we provide bytes methods for handling them, but
> you *must* be aware that these methods *cannot* distinguish "bytes
> representing text encoded as ASCII" from "any old bytes".  Be
> warned: They will happily -- and silently -- corrupt the latter.
> Make sure you respect the higher-level structure of your data when
> using them.

Yes, I'm currently thinking the appropriate approach to the docs will be to
remove the current "these have most of the str methods too" paragraph for
binary sequences and instead create three completely explicit lists of
methods:

- provided, works with arbitrary data
- provided, assumes the use of an ASCII compatible data format
- not provided

PEP 461 would add a fourth category, of being provided, but with more
restricted semantics.

Cheers,
Nick.

>
>  > Virtual subclass ASCIIStructuredBytes
>  > 
>  >
>  > One particularly common use of bytes is to represent
>  > the contents of a file, or of a network message.  In
>  > these cases, the bytes will often represent Text
>  > *in a specific encoding* and that encoding will usually
>  > be a superset of ASCII.  Rather than create and support
>  > an ASCIIStructuredBytes subclass, Python simply added
>  > support for these use cases straight to Bytes objects,
>  > and assumes that this support simply won't be used when
>  > when it does not make sense. For example, bytes literals
>
> This is going quite the wrong direction, I think.  The only people who
> should care about "Text *in a specific encoding* and that encoding
> will usually be a superset of ASCII" are codec writers, and by now
> writing those is a very rare task.  Everybody else uses ASCII keywords
> in a simple formal language.
>
>  > *could* be used to construct a sound sample, but the
>  > literals will be far easier to read when they are used
>  > to represent (encoded) ASCII text, such as "OPEN".
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 461 updates

[Ethan Furman]

Current copy of PEP, many modifications from all the feedback.  Thank you to 
everyone.

I know it's been a long week (feels a lot longer!) while all this was hammered 
out, but I think we're getting close!


Abstract


This PEP proposes adding the % and {} formatting operations from str to bytes 
[1].


Overriding Principles
=

In order to avoid the problems of auto-conversion and value-generated 
exceptions,
all object checking will be done via isinstance, not by values contained in a
Unicode representation.  In other words::

  - duck-typing to allow/reject entry into a byte-stream
  - no value generated errors


Proposed semantics for bytes formatting
===

%-interpolation
---

All the numeric formatting codes (such as %x, %o, %e, %f, %g, etc.)
will be supported, and will work as they do for str, including the
padding, justification and other related modifiers, except locale.

Example::

   >>> b'%4x' % 10
   b'   a'

%c will insert a single byte, either from an int in range(256), or from
a bytes argument of length 1.

Example:

>>> b'%c' % 48
b'0'

>>> b'%c' % b'a'
b'a'

%s is restricted in what it will accept::

  - input type supports Py_buffer?
use it to collect the necessary bytes

  - input type is something else?
use its __bytes__ method; if there isn't one, raise an exception [2]

Examples:

>>> b'%s' % b'abc'
b'abc'

>>> b'%s' % 3.14
Traceback (most recent call last):
...
TypeError: 3.14 has no __bytes__ method

>>> b'%s' % 'hello world!'
Traceback (most recent call last):
...
TypeError: 'hello world' has no __bytes__ method, perhaps you need to 
encode it?

.. note::

   Because the str type does not have a __bytes__ method, attempts to
   directly use 'a string' as a bytes interpolation value will raise an
   exception.  To use 'string' values, they must be encoded or otherwise
   transformed into a bytes sequence::

  'a string'.encode('latin-1')

format
--

The format mini language codes, where they correspond with the %-interpolation 
codes,
will be used as-is, with three exceptions::

  - !s is not supported, as {} can mean the default for both str and bytes, in 
both
Py2 and Py3.
  - !b is supported, and new Py3k code can use it to be explicit.
  - no other __format__ method will be called.

Numeric Format Codes


To properly handle int and float subclasses, int(), index(), and float() will 
be called on the
objects intended for (d, i, u), (b, o, x, X), and (e, E, f, F, g, G).

Unsupported codes
-

%r (which calls __repr__), and %a (which calls ascii() on __repr__) are not 
supported.

!r and !a are not supported.

The n integer and float format code is not supported.


Open Questions
==

Currently non-numeric objects go through::

  - Py_buffer
  - __bytes__
  - failure

Do we want to add a __format_bytes__ method in there?

  - Guaranteed to produce only ascii (as in b'10', not b'\x0a')
  - Makes more sense than using __bytes__ to produce ascii output
  - What if an object has both __bytes__ and __format_bytes__?

Do we need to support all the numeric format codes?  The floating point
exponential formats seem less appropriate, for example.


Proposed variations
===

It was suggested to let %s accept numbers, but since numbers have their own
format codes this idea was discarded.

It has been suggested to use %b for bytes instead of %s.

  - Rejected as %b does not exist in Python 2.x %-interpolation, which is
why we are using %s.

It has been proposed to automatically use .encode('ascii','strict') for str
arguments to %s.

  - Rejected as this would lead to intermittent failures.  Better to have the
operation always fail so the trouble-spot can be correctly fixed.

It has been proposed to have %s return the ascii-encoded repr when the value
is a str  (b'%s' % 'abc'  --> b"'abc'").

  - Rejected as this would lead to hard to debug failures far from the problem
site.  Better to have the operation always fail so the trouble-spot can be
easily fixed.


Footnotes
=

.. [1] string.Template is not under consideration.
.. [2] TypeError, ValueError, or UnicodeEncodeError?
==

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Steven D'Aprano]

On Wed, Jan 15, 2014 at 10:34:48PM +, Mark Lawrence wrote:
> On 15/01/2014 22:22, Brett Cannon wrote:
> >
> >
> >
> >On Wed, Jan 15, 2014 at 5:00 PM, Steven D'Aprano  >> wrote:
> >
> >On Thu, Jan 16, 2014 at 10:55:31AM +1300, Greg Ewing wrote:
> > > Neil Schemenauer wrote:
> > > >Objects that implement __str__ can also implement __bytes__ if they
> > > >can guarantee that ASCII characters are always returned,
> > >
> > > I think __ascii_ would be a better name. I'd expect
> > > a method called __bytes__ on an int to return some
> > > version of its binary value.
> >
> >+1
> >
> >
> >If we are going the route of a new magic method then __ascii__ or
> >__bytes_format__ get my vote as long as they only return bytes (I see no
> >need to abbreviate to __bformat__ or __formatb__ when we have method
> >names as long as __text_signature__ now).
> >
> 
> __bytes_format__ gets my vote as it's blatantly obvious what it does. 

What precisely does it do? If it's so obvious, why is this thread so 
long?


> I'm against __ascii__ as I'd automatically associate that with ascii in 
> the same way that I associate str with __str__ and repr with __repr__.

That's a good point. I forgot about ascii().


-- 
Steven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Common subset of python 2 and python 3

[Steven D'Aprano]

On Wed, Jan 15, 2014 at 09:25:04AM -0500, Eric V. Smith wrote:
> On 1/15/2014 8:21 AM, Chris Angelico wrote:
> > On Wed, Jan 15, 2014 at 11:22 PM, "Martin v. Löwis"  
> > wrote:
> >> I don't think that it is possible to write an interpreter that is fully
> >> compatible for all it accepts. Would you think that the program
> >>
> >> print(repr(2**80).endswith("L"))
> >>
> >> is in the subset that should be supported by both Python 2 and Python 3?
> > 
> > Easiest fix for that would be to have long.__repr__ omit the L tag.
> > Then it'll do the same as it would in Py3.
> 
> I think Martin's point is not this specific thing, but that such a
> subset would be useless. Would you drop dict.items() because it returns
> different things in both languages? Drop range() because it's different?
> There are many, many such differences. The common subset is not useful.

To expand on this, the common subset is not useful, not well-defined, 
and it is not needed.

Not well-defined because neither "Python 2" nor "Python 3" are 
well-defined. Most of the code I write supports Python 2.4 onwards, and 
there are a lot of features (including syntax!) that exist in 2.7 but 
not 2.4. Likewise there are features in 3.3 that aren't in 3.2.

But most importantly, limiting yourself to just the common subset isn't 
needed to write polyglot code that works over 2.x and 3.x. For the most 
part, a few conditional tests will let you write code that works across 
multiple versions. I prefer to check for features than test the version 
number:

try:
next
except NameError:
# Python 2.4 or 2.5
def next(obj):
return type(obj).__next__()


sort of thing. 

Syntax changes are more difficult to deal with. I deal with the lack of 
ternary if operator in 2.4 by just avoiding it. The other day I really, 
really wanted to use a with statement, but still be compatible with 2.4. 
I started off messing about with exec, but eventually rejected that in 
favour of a conditional import: I lifted that one function using a with 
statement into a module of its own, then tried importing it. If it 
failed, I fell back to a second module which implemented the same thing 
using nested try...except blocks. There's a tiny bit of duplicated 
code, but less than a dozens lines including a docstring.

Given how easy it usually is to write 2/3 compatible code, I don't think 
that limiting myself to a subset that works unchanged in both would be 
useful to me. That would be a step backward, like going back to Python 
1.5 or 2.0, where the language is still recognisably the same, but it's 
missing so many features we take for granted that it's painful to work 
with.


-- 
Steven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Mark Lawrence]

On 15/01/2014 22:22, Brett Cannon wrote:




On Wed, Jan 15, 2014 at 5:00 PM, Steven D'Aprano mailto:st...@pearwood.info>> wrote:

On Thu, Jan 16, 2014 at 10:55:31AM +1300, Greg Ewing wrote:
 > Neil Schemenauer wrote:
 > >Objects that implement __str__ can also implement __bytes__ if they
 > >can guarantee that ASCII characters are always returned,
 >
 > I think __ascii_ would be a better name. I'd expect
 > a method called __bytes__ on an int to return some
 > version of its binary value.

+1


If we are going the route of a new magic method then __ascii__ or
__bytes_format__ get my vote as long as they only return bytes (I see no
need to abbreviate to __bformat__ or __formatb__ when we have method
names as long as __text_signature__ now).



__bytes_format__ gets my vote as it's blatantly obvious what it does. 
I'm against __ascii__ as I'd automatically associate that with ascii in 
the same way that I associate str with __str__ and repr with __repr__.


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Ethan Furman]

On 01/15/2014 08:33 AM, Mark Lawrence wrote:


For completeness I believe %r and %a should be included here as well.


Good point.  Done.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Brett Cannon]

On Wed, Jan 15, 2014 at 5:00 PM, Steven D'Aprano wrote:

> On Thu, Jan 16, 2014 at 10:55:31AM +1300, Greg Ewing wrote:
> > Neil Schemenauer wrote:
> > >Objects that implement __str__ can also implement __bytes__ if they
> > >can guarantee that ASCII characters are always returned,
> >
> > I think __ascii_ would be a better name. I'd expect
> > a method called __bytes__ on an int to return some
> > version of its binary value.
>
> +1
>

If we are going the route of a new magic method then __ascii__ or
__bytes_format__ get my vote as long as they only return bytes (I see no
need to abbreviate to __bformat__ or __formatb__ when we have method names
as long as __text_signature__ now).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Brett Cannon]

On Wed, Jan 15, 2014 at 4:24 PM, Ethan Furman  wrote:

> On 01/15/2014 06:45 AM, Brett Cannon wrote:
>
>>
>> I also think that a 'b' conversion be added to bytes.format(). This
>> doesn't have the same issue as %b if you make {}
>> implicitly mean {!b} in Python 3.5 as {} will mean what is the most
>> accurate for bytes.format() in either version. It
>> also allows for explicit support where you know you only want a byte and
>> allows {!s} to mean you only want a string (and
>> thus throw an error otherwise).
>>
>
> Given that !b does not exist in Py2, !s (like %s) has to mean bytes when
> working with a byte stream.  Given that, !s and !b would mean the same
> thing, so it worth adding !b?
>

I disagree with the assertion. %s has to mean bytes for Python 2
compatibility because there is no equivalent to '{}' (no conversion or
format spec specified); basically %s represents "no conversion" for the %
operator. But since format() has the concept of a default conversion as
well as explicit conversions you can lean on that fact and let the default
conversion do what makes sense for that version of Python.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Eric V. Smith]

On 1/15/2014 4:32 PM, Ethan Furman wrote:
> A question for anyone that has extensive experience in both %-formatting
> and .format-formatting:  Would it be possible, at least for int and
> float, to take whatever is in the specifier and convert to %?  Example:
> 
>   "Weight: {wgt:-07f}".format(wgt=137.23)
> 
> would take the "-07f" and basically do a "%-07f" % 137.23 to get the
> ASCII to use?

I think the int.__format__ version might be a superset. Specifically,
the "n" and "%" types. There may well be others.

But I think we could say we're not going to support these in b"".format().

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Steven D'Aprano]

On Thu, Jan 16, 2014 at 10:55:31AM +1300, Greg Ewing wrote:
> Neil Schemenauer wrote:
> >Objects that implement __str__ can also implement __bytes__ if they
> >can guarantee that ASCII characters are always returned,
> 
> I think __ascii_ would be a better name. I'd expect
> a method called __bytes__ on an int to return some
> version of its binary value.

+1


-- 
Steven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Greg Ewing]

Neil Schemenauer wrote:

Objects that implement __str__ can also implement __bytes__ if they
can guarantee that ASCII characters are always returned,


I think __ascii_ would be a better name. I'd expect
a method called __bytes__ on an int to return some
version of its binary value.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Ethan Furman]

On 01/15/2014 06:45 AM, Brett Cannon wrote:
The PEP currently says::


format
--

The format mini language will be used as-is, with the behaviors as listed
for %-interpolation.


That's too vague; % interpolation does not support other format operators in 
the same way as str.format() does. %
interpolation has specific code to support %d, etc. But str.format() gets 
supported for {:d} not from special code but
because e.g. float.__format__('d') works. So you can't say "bytes.format() 
supports {:d} just like %d works with string
interpolation" since the mechanisms are fundamentally different.


A question for anyone that has extensive experience in both %-formatting and .format-formatting:  Would it be possible, 
at least for int and float, to take whatever is in the specifier and convert to %?  Example:


  "Weight: {wgt:-07f}".format(wgt=137.23)

would take the "-07f" and basically do a "%-07f" % 137.23 to get the ASCII to 
use?

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Ethan Furman]

On 01/15/2014 06:45 AM, Brett Cannon wrote:


I also think that a 'b' conversion be added to bytes.format(). This doesn't 
have the same issue as %b if you make {}
implicitly mean {!b} in Python 3.5 as {} will mean what is the most accurate 
for bytes.format() in either version. It
also allows for explicit support where you know you only want a byte and allows 
{!s} to mean you only want a string (and
thus throw an error otherwise).


Given that !b does not exist in Py2, !s (like %s) has to mean bytes when working with a byte stream.  Given that, !s and 
!b would mean the same thing, so it worth adding !b?


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Neil Schemenauer]

Glenn Linderman  wrote:
> On 1/15/2014 7:52 AM, Eric V. Smith wrote:
>> Either that, or we're back to encoding the result of __format__ and
>> accepting that sometimes it might throw errors, depending on the values
>> being passed into format().

That would take us back to Python 2 hell.  Please no.  I don't like
checking for types either, we should have a special method.

> Looks like you need to invent  __formatb__ to produce only ASCII. 
> Objects that have __formatb__ can be formatted by bytes.format.  To 
> avoid coding, it could be possible that __formatb__ might be a callable
> in which case it is called to get the result, or not a callable, in 
> which case one calls __format__ and converts the result to ASCII, 
> __formatb__ just indicating a guarantee that only ASCII will result.

Just do:

def __formatb__(self, spec):
return MyClass.__format__(self, spec).encode('ascii')

Note that I think it is better to explicitly use the __format__
method rather than using self.__format__.  My reasoning is that a
subclass might implement a __format__ that returns non-ASCII
characters.

We don't need a special bytes version of __str__ since the
%-operator can call __formatb__ with the correct format spec.

   Neil

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Glenn Linderman]

On 1/15/2014 7:52 AM, Eric V. Smith wrote:

So basically I think we'll have to hard-code the types that .format()
will support, and never call __format__, or only call __format__ if we
know that it's a exact type where we know that __format__ will return
(strict ASCII).

Either that, or we're back to encoding the result of __format__ and
accepting that sometimes it might throw errors, depending on the values
being passed into format().


Looks like you need to invent  __formatb__ to produce only ASCII. 
Objects that have __formatb__ can be formatted by bytes.format.  To 
avoid coding, it could be possible that __formatb__ might be a callable, 
in which case it is called to get the result, or not a callable, in 
which case one calls __format__ and converts the result to ASCII, 
__formatb__ just indicating a guarantee that only ASCII will result.


Or it could be that __formatb__ replaces __format__ and str.__format__, 
if it finds no __format__ looks for __formatb__, calls that, and 
converts the result to Unicode.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Guido van Rossum]

All sounds good.

A fleeting thought about constructors: you can always add alternative
constructors as class methods (like datetime does).

On Wed, Jan 15, 2014 at 10:09 AM, Neil Schemenauer  wrote:
> Antoine Pitrou  wrote:
>> On Wed, 15 Jan 2014 15:47:43 + (UTC) Neil S wrote:
>>>
>>> Objects that implement __str__ can also implement __bytes__ if they
>>> can guarantee that ASCII characters are always returned, no matter
>>> what the *value*
>>
>> I think that's a slippery slope. __bytes__ should mean that the object
>> has a well-known bytes equivalent or encoding, not that its __str__
>> happens to be pure ASCII.
>
> After poking around some more into the Python 3 source, I agree.  It
> seems too late to change bytes() and bytearray().
> We should have used a keyword only argument but too late now (tp_new
> is a mess).
>
> I can also agree that pushing the ASCII-centric behavior into the
> bytes() constructor goes too far.  If we limit the ASCII-centric
> behavior solely to % and format(), that seems a reasonable trade-off
> for usability.  As others have argued, once you are using format
> codes, you are pretty clearly dealing with ASCII encoding.
>
> I feel strongly that % and format on bytes needs to use duck-typing
> and not type checking.  Also, formatting falures must be due to
> types and not due to values.  If we can get agreement on these two
> principles, that will help guide the design.
>
> Those principles absolutely rule out call calling encode('ascii')
> automatically.  I'm not deeply intimate with format() but I think it
> also rules out calling __format__.
>
> Could we introduce only __bformat__ and have the % operator call it?
> That would only require implementing one new special method instead
> of two.
>
>   Neil
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Neil Schemenauer]

Antoine Pitrou  wrote:
> On Wed, 15 Jan 2014 15:47:43 + (UTC) Neil S wrote:
>> 
>> Objects that implement __str__ can also implement __bytes__ if they
>> can guarantee that ASCII characters are always returned, no matter
>> what the *value*
>
> I think that's a slippery slope. __bytes__ should mean that the object
> has a well-known bytes equivalent or encoding, not that its __str__
> happens to be pure ASCII.

After poking around some more into the Python 3 source, I agree.  It
seems too late to change bytes() and bytearray().
We should have used a keyword only argument but too late now (tp_new
is a mess).

I can also agree that pushing the ASCII-centric behavior into the
bytes() constructor goes too far.  If we limit the ASCII-centric
behavior solely to % and format(), that seems a reasonable trade-off
for usability.  As others have argued, once you are using format
codes, you are pretty clearly dealing with ASCII encoding.

I feel strongly that % and format on bytes needs to use duck-typing
and not type checking.  Also, formatting falures must be due to
types and not due to values.  If we can get agreement on these two
principles, that will help guide the design.

Those principles absolutely rule out call calling encode('ascii')
automatically.  I'm not deeply intimate with format() but I think it
also rules out calling __format__.

Could we introduce only __bformat__ and have the % operator call it?
That would only require implementing one new special method instead
of two.

  Neil

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Isaac Morland]

On Wed, 15 Jan 2014, Antoine Pitrou wrote:


On Wed, 15 Jan 2014 15:47:43 + (UTC)
Neil Schemenauer  wrote:


Objects that implement __str__ can also implement __bytes__ if they
can guarantee that ASCII characters are always returned, no matter
what the *value*


I think that's a slippery slope. __bytes__ should mean that the object
has a well-known bytes equivalent or encoding, not that its __str__
happens to be pure ASCII.


+1


(for example, it would be fine for a HTTP message class to define a
__bytes__ method)

Also, consider that if e.g. float had a __bytes__ method, then
bytes(2.0) would start returning b'2.0', while bytes(2) would still
need to return b'\x00\x00'.


Not actually suggesting the following for a number of reasons including
but not limited to the consistency of floating point formats across
different implementations, but it would make more sense for bytes (2.0) to
return the 8-byte IEEE representation than for it to return the ASCII
encoding of the decimal representation of the number.

Isaac Morland   CSCF Web Guru
DC 2619, x36650 WWW Software Specialist
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Ethan Furman]

On 01/15/2014 08:51 AM, Brett Cannon wrote:

On Wed, Jan 15, 2014 at 10:57 AM, Ethan Furman wrote:


Thanks for your comments.  I've only barely touched format, so it's not an area 
of strength for me.  :)


Time to strengthen it if you are proposing a PEP that is going to affect it. =)


I am.  You're helping.  :)

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Brett Cannon]

On Wed, Jan 15, 2014 at 10:57 AM, Ethan Furman  wrote:

> On 01/15/2014 06:45 AM, Brett Cannon wrote:
>
>> bytes.format() below. I'll leave it to you to decide if they warrant
>> using, leaving as an open question, or rejecting.
>>
>
> Thanks for your comments.  I've only barely touched format, so it's not an
> area of strength for me.  :)
>

Time to strengthen it if you are proposing a PEP that is going to affect
it. =)


>
> --
> ~Ethan~
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Brett Cannon]

On Wed, Jan 15, 2014 at 10:52 AM, Eric V. Smith  wrote:

> On 1/15/2014 9:45 AM, Brett Cannon wrote:
>
> > That's too vague; % interpolation does not support other format
> > operators in the same way as str.format() does. % interpolation has
> > specific code to support %d, etc. But str.format() gets supported for
> > {:d} not from special code but because e.g. float.__format__('d') works.
> > So you can't say "bytes.format() supports {:d} just like %d works with
> > string interpolation" since the mechanisms are fundamentally different.
> >
> > This is why I have argued that if you specify it as "if there is a
> > format spec specified, then the return value from calling __format__()
> > will have str.decode('ascii', 'strict') called on it" you get the
> > support for the various number-specific format specs for free. It also
> > means if you pass in a string that you just want the strict ASCII bytes
> > of then you can get it with {:s}.
> >
> > I also think that a 'b' conversion be added to bytes.format(). This
> > doesn't have the same issue as %b if you make {} implicitly mean {!b} in
> > Python 3.5 as {} will mean what is the most accurate for bytes.format()
> > in either version. It also allows for explicit support where you know
> > you only want a byte and allows {!s} to mean you only want a string (and
> > thus throw an error otherwise).
> >
> > And all of this means that much like %s only taking bytes, the only way
> > for bytes.format() to accept a non-byte argument is for some format spec
> > to be specified to trigger the .encode('ascii', 'strict') call.
>
> Agreed. With %-formatting, you can start with the format strings and
> then decide what you want to do with the passed in objects. But with
> .format, it's the other way around: you have to look at the passed in
> objects being formatted, and then decide what the format specifier means
> to that type.
>
> So, for .format, you could say "hey, that object's an int, and I happen
> to know how to format ints, outside of calling it's .__format__". Or you
> could even call its __format__ because you know that it will only be
> ASCII. But to take this approach, you're going to have to hard-code the
> types. And subclasses are probably out, since there you don't know what
> the subclass's __format__ will return. It could be non-ASCII.
>
> >>> class Int(int):
> ...   def __format__(self, fmt):
> ... return u'foo'
> ...
> >>> '{}'.format(Int(3))
> 'foo'
>
> So basically I think we'll have to hard-code the types that .format()
> will support, and never call __format__, or only call __format__ if we
> know that it's a exact type where we know that __format__ will return
> (strict ASCII).
>
> Either that, or we're back to encoding the result of __format__ and
> accepting that sometimes it might throw errors, depending on the values
> being passed into format().
>

I say accept that an error might get thrown as there is precedent of
specifying a format spec that an object's __format__() method doesn't
recognize::

  >>> '{:s}'.format(1)
  Traceback (most recent call last):
File "", line 1, in 
  ValueError: Unknown format code 's' for object of type 'int'

IOW I'm actively trying to avoid type-restricting the semantics for
bytes.format() for a consistent, clear mental model. Remembering that "any
format spec leads to calling .encode('ascii', 'strict') on the result" is
simple compared to "ASCII bytes will be returned for ints and floats when
passed in, otherwise all other types follow these rules".

As the zen says:

  Errors should never pass silently.
  Special cases aren't special enough to break the rules.


-Brett
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Mark Lawrence]

On 14/01/2014 19:56, Ethan Furman wrote:

Duh.  Here's the text, as well.  ;)

%s, because it is the most general, has the most convoluted resolution:

   - input type is bytes?
 pass it straight through

   - input type is numeric?
 use its __xxx__ [1] [2] method and ascii-encode it (strictly)

   - input type is something else?
 use its __bytes__ method; if there isn't one, raise an exception [3]

Examples:

 >>> b'%s' % b'abc'
 b'abc'

 >>> b'%s' % 3.14
 b'3.14'

 >>> b'%s' % 'hello world!'
 Traceback (most recent call last):
 ...
 TypeError: 'hello world' has no __bytes__ method, perhaps you need
to encode it?



For completeness I believe %r and %a should be included here as well. 
FTR %a appears to have been introduced in 3.2, but I couldn't find 
anything in the What's New and there's no note in the docs 
http://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting 
to indicate when it first came into play.


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Ethan Furman]

On 01/15/2014 07:47 AM, Neil Schemenauer wrote:


Thanks for writing the PEP.


Thank you for your comments!

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Ethan Furman]

On 01/15/2014 08:04 AM, Antoine Pitrou wrote:

On Wed, 15 Jan 2014 15:47:43 + (UTC)
Neil Schemenauer  wrote:


Objects that implement __str__ can also implement __bytes__ if they
can guarantee that ASCII characters are always returned, no matter
what the *value*


I think that's a slippery slope. __bytes__ should mean that the object
has a well-known bytes equivalent or encoding, not that its __str__
happens to be pure ASCII.


Agreed.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Ethan Furman]

On 01/14/2014 02:41 PM, Mark Lawrence wrote:

On 14/01/2014 19:55, Ethan Furman wrote:

This PEP goes a but further than PEP 460 does, and hopefully spells
things out in enough detail so there is no confusion as to what is meant.

--
~Ethan~


Out of plain old curiosity do we have to consider PEP 292 string templates in 
any way, shape or form, or regarding this
debate have they been safely booted into touch?


Well, I'm not sure what "booted into touch" means, but yes, we can ignore 
string templates.  :)

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Neil Schemenauer]

Neil Schemenauer  wrote:
> We should use duck-typing and that means a special method, I
> think.  We could introduce a new one but __bytes__ looks like it
> can work.  Otherwise, maybe __ascii__ is a good name.

I poked around the Python 3 source.  Using __bytes__ has some
downsides, e.g. the following would happen:

>>> bytes(12)
b'12'

Perhaps that's a little too ASCII-centric.  OTOH, UTF-8 seems to be
winning the encoding war and so the above could be argued as
reasonable behavior.  I think forcing people to explicitly choose an
encoding for str objects will be sufficient to avoid the bytes/str
mess we have in Python 2.

Unfortunately, that change conflicts with the current behavior:

>>> bytes(12)
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

Would it be too disruptive to change that?  It doesn't appear to be
too useful and we could do it using a keyword argument, e.g.:

bytes(size=12)

I notice something else surprising to me:

>>> class Test(object):
... def __bytes__(self):
... return b'test'
...
>>> with open('test', 'wb') as fp:
... fp.write(Test())
...
Traceback (most recent call last):
  File "", line 2, in 
TypeError: 'Test' does not support the buffer interface

I'd expect that to write b'test' to the file, not raise an error.

Regards,

  Neil

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Changing Clinic's output

[Serhiy Storchaka]

15.01.14 17:37, Larry Hastings написав(ла):

On 01/15/2014 01:59 AM, Serhiy Storchaka wrote:

15.01.14 00:47, Larry Hastings написав(ла):

I also can not change the text, but twice I was a witness as others
did. And I see that make this mistake very easily.


I also didn't modify the generated text, but twice I was a witness as
others did. And I see that make this mistake very easily.


That isn't what I wrote...?


Gotcha! How I got this?

I wanted to quote:


I no longer ever mistake generated code for handwritten code, and I don't ever 
modify the generated text.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Antoine Pitrou]

On Wed, 15 Jan 2014 15:47:43 + (UTC)
Neil Schemenauer  wrote:
> 
> Objects that implement __str__ can also implement __bytes__ if they
> can guarantee that ASCII characters are always returned, no matter
> what the *value*

I think that's a slippery slope. __bytes__ should mean that the object
has a well-known bytes equivalent or encoding, not that its __str__
happens to be pure ASCII.

(for example, it would be fine for a HTTP message class to define a
__bytes__ method)

Also, consider that if e.g. float had a __bytes__ method, then
bytes(2.0) would start returning b'2.0', while bytes(2) would still
need to return b'\x00\x00'.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Ethan Furman]

On 01/15/2014 06:45 AM, Brett Cannon wrote:

bytes.format() below. I'll leave it to you to decide if they warrant using, 
leaving as an open question, or rejecting.


Thanks for your comments.  I've only barely touched format, so it's not an area 
of strength for me.  :)

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Eric V. Smith]

On 1/15/2014 9:45 AM, Brett Cannon wrote:

> That's too vague; % interpolation does not support other format
> operators in the same way as str.format() does. % interpolation has
> specific code to support %d, etc. But str.format() gets supported for
> {:d} not from special code but because e.g. float.__format__('d') works.
> So you can't say "bytes.format() supports {:d} just like %d works with
> string interpolation" since the mechanisms are fundamentally different.
> 
> This is why I have argued that if you specify it as "if there is a
> format spec specified, then the return value from calling __format__()
> will have str.decode('ascii', 'strict') called on it" you get the
> support for the various number-specific format specs for free. It also
> means if you pass in a string that you just want the strict ASCII bytes
> of then you can get it with {:s}.
> 
> I also think that a 'b' conversion be added to bytes.format(). This
> doesn't have the same issue as %b if you make {} implicitly mean {!b} in
> Python 3.5 as {} will mean what is the most accurate for bytes.format()
> in either version. It also allows for explicit support where you know
> you only want a byte and allows {!s} to mean you only want a string (and
> thus throw an error otherwise).
> 
> And all of this means that much like %s only taking bytes, the only way
> for bytes.format() to accept a non-byte argument is for some format spec
> to be specified to trigger the .encode('ascii', 'strict') call.

Agreed. With %-formatting, you can start with the format strings and
then decide what you want to do with the passed in objects. But with
.format, it's the other way around: you have to look at the passed in
objects being formatted, and then decide what the format specifier means
to that type.

So, for .format, you could say "hey, that object's an int, and I happen
to know how to format ints, outside of calling it's .__format__". Or you
could even call its __format__ because you know that it will only be
ASCII. But to take this approach, you're going to have to hard-code the
types. And subclasses are probably out, since there you don't know what
the subclass's __format__ will return. It could be non-ASCII.

>>> class Int(int):
...   def __format__(self, fmt):
... return u'foo'
...
>>> '{}'.format(Int(3))
'foo'

So basically I think we'll have to hard-code the types that .format()
will support, and never call __format__, or only call __format__ if we
know that it's a exact type where we know that __format__ will return
(strict ASCII).

Either that, or we're back to encoding the result of __format__ and
accepting that sometimes it might throw errors, depending on the values
being passed into format().

Eric.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Neil Schemenauer]

This looks pretty good to me.  I don't think we should limit
operands based on type, that's anti-Pythonic IMHO.  We should use
duck-typing and that means a special method, I think.  We could
introduce a new one but __bytes__ looks like it can work.
Otherwise, maybe __ascii__ is a good name.

Objects that implement __str__ can also implement __bytes__ if they
can guarantee that ASCII characters are always returned, no matter
what the *value* (we don't want to repeat the hell of Python 2's
unicode to str coercion which depends on the value of the unicode
object).  Objects that already contain encoded bytes or arbitrary
bytes can also implement __bytes__.

Ethan Furman  wrote:
> %s, because it is the most general, has the most convoluted resolution:

This becomes much simpler:

 - does the object implement __bytes__?
   call it and use the value otherwise raise TypeError

> It has been suggested to use %b for bytes instead of %s.
>
>- Rejected as %b does not exist in Python 2.x %-interpolation, which is
>  why we are using %s.

+1.  %b might be conceptually neater but ease of migration trumps
that, IMHO.

> It has been proposed to automatically use .encode('ascii','strict') for str
> arguments to %s.
>
>- Rejected as this would lead to intermittent failures.  Better to have the
>  operation always fail so the trouble-spot can be correctly fixed.

Right.  That would put us back in Python 2 unicode -> str coercion
hell.

Thanks for writing the PEP.

  Neil

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Changing Clinic's output

[Larry Hastings]

On 01/15/2014 01:59 AM, Serhiy Storchaka wrote:

15.01.14 00:47, Larry Hastings написав(ла):
I also can not change the text, but twice I was a witness as others 
did. And I see that make this mistake very easily.


I also didn't modify the generated text, but twice I was a witness as 
others did. And I see that make this mistake very easily.


That isn't what I wrote...?


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Common subset of python 2 and python 3

[Serhiy Storchaka]

15.01.14 16:34, Antoine Pitrou написав(ла):

If you explicitly create a long the L will always be printed:


long(0)

0L


Hey! long is not in common subset of Python 2 and Python 3.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Binding problem

[R. David Murray]

On Wed, 15 Jan 2014 11:53:05 +1100, "Rob Ward"  wrote:
> I apologise if I have come to the wrong place here, but 12hrs
> searching, plus experimenting,  on the WWW for advice on it has not
> yielded any successful advice to resolve the issue.

This is indeed the wrong place.  Your best bet for getting an
answer would probably be the python-list mailing list/newsgroup.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

[Brett Cannon]

bytes.format() below. I'll leave it to you to decide if they warrant using,
leaving as an open question, or rejecting.


On Tue, Jan 14, 2014 at 2:56 PM, Ethan Furman  wrote:

> Duh.  Here's the text, as well.  ;)
>
>
> PEP: 461
> Title: Adding % and {} formatting to bytes
> Version: $Revision$
> Last-Modified: $Date$
> Author: Ethan Furman 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 2014-01-13
> Python-Version: 3.5
> Post-History: 2014-01-13
> Resolution:
>
>
> Abstract
> 
>
> This PEP proposes adding the % and {} formatting operations from str to
> bytes.
>
>
> Proposed semantics for bytes formatting
> ===
>
> %-interpolation
> ---
>
> All the numeric formatting codes (such as %x, %o, %e, %f, %g, etc.)
> will be supported, and will work as they do for str, including the
> padding, justification and other related modifiers.
>
> Example::
>
>>>> b'%4x' % 10
>b'   a'
>
> %c will insert a single byte, either from an int in range(256), or from
> a bytes argument of length 1.
>
> Example:
>
> >>> b'%c' % 48
> b'0'
>
> >>> b'%c' % b'a'
> b'a'
>
> %s, because it is the most general, has the most convoluted resolution:
>
>   - input type is bytes?
> pass it straight through
>
>   - input type is numeric?
> use its __xxx__ [1] [2] method and ascii-encode it (strictly)
>
>   - input type is something else?
> use its __bytes__ method; if there isn't one, raise an exception [3]
>
> Examples:
>
> >>> b'%s' % b'abc'
> b'abc'
>
> >>> b'%s' % 3.14
> b'3.14'
>
> >>> b'%s' % 'hello world!'
> Traceback (most recent call last):
> ...
> TypeError: 'hello world' has no __bytes__ method, perhaps you need to
> encode it?
>
> .. note::
>
>Because the str type does not have a __bytes__ method, attempts to
>directly use 'a string' as a bytes interpolation value will raise an
>exception.  To use 'string' values, they must be encoded or otherwise
>transformed into a bytes sequence::
>
>   'a string'.encode('latin-1')
>
>
> format
> --
>
> The format mini language will be used as-is, with the behaviors as listed
> for %-interpolation.
>

That's too vague; % interpolation does not support other format operators
in the same way as str.format() does. % interpolation has specific code to
support %d, etc. But str.format() gets supported for {:d} not from special
code but because e.g. float.__format__('d') works. So you can't say
"bytes.format() supports {:d} just like %d works with string interpolation"
since the mechanisms are fundamentally different.

This is why I have argued that if you specify it as "if there is a format
spec specified, then the return value from calling __format__() will have
str.decode('ascii', 'strict') called on it" you get the support for the
various number-specific format specs for free. It also means if you pass in
a string that you just want the strict ASCII bytes of then you can get it
with {:s}.

I also think that a 'b' conversion be added to bytes.format(). This doesn't
have the same issue as %b if you make {} implicitly mean {!b} in Python 3.5
as {} will mean what is the most accurate for bytes.format() in either
version. It also allows for explicit support where you know you only want a
byte and allows {!s} to mean you only want a string (and thus throw an
error otherwise).

And all of this means that much like %s only taking bytes, the only way for
bytes.format() to accept a non-byte argument is for some format spec to be
specified to trigger the .encode('ascii', 'strict') call.

-Brett


>
>
> Open Questions
> ==
>
> For %s there has been some discussion of trying to use the buffer protocol
> (Py_buffer) before trying __bytes__.  This question should be answered
> before
> the PEP is implemented.
>
>
> Proposed variations
> ===
>
> It has been suggested to use %b for bytes instead of %s.
>
>   - Rejected as %b does not exist in Python 2.x %-interpolation, which is
> why we are using %s.
>
> It has been proposed to automatically use .encode('ascii','strict') for str
> arguments to %s.
>
>   - Rejected as this would lead to intermittent failures.  Better to have
> the
> operation always fail so the trouble-spot can be correctly fixed.
>
> It has been proposed to have %s return the ascii-encoded repr when the
> value
> is a str  (b'%s' % 'abc'  --> b"'abc'").
>
>   - Rejected as this would lead to hard to debug failures far from the
> problem
> site.  Better to have the operation always fail so the trouble-spot
> can be
> easily fixed.
>
>
> Foot notes
> ==
>
> .. [1] Not sure if this should be the numeric __str__ or the numeric
> __repr__,
>or if there's any difference
> .. [2] Any proper numeric class would then have to provide an ascii
>representation of its value, either via __repr__ or __str__
> (whichever
>we choose in [1]).
> .. [3] TypeError, V

Re: [Python-Dev] Common subset of python 2 and python 3

[Chris Angelico]

On Thu, Jan 16, 2014 at 1:25 AM, Eric V. Smith  wrote:
>> Easiest fix for that would be to have long.__repr__ omit the L tag.
>> Then it'll do the same as it would in Py3.
>
> I think Martin's point is not this specific thing, but that such a
> subset would be useless. Would you drop dict.items() because it returns
> different things in both languages? Drop range() because it's different?
> There are many, many such differences. The common subset is not useful.

Fair enough.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Common subset of python 2 and python 3

[Antoine Pitrou]

On Wed, 15 Jan 2014 16:31:17 +0200
Serhiy Storchaka  wrote:
> 15.01.14 14:22, "Martin v. Löwis" написав(ла):
> > I don't think that it is possible to write an interpreter that is fully
> > compatible for all it accepts. Would you think that the program
> >
> > print(repr(2**80).endswith("L"))
> >
> > is in the subset that should be supported by both Python 2 and Python 3?
> >
> > Notice that it prints "True" in Python 2 and "False" in Python 3.
> 
> This is implementation details. On 128-bit platform special build of 
> Python 2 can print False.

If you explicitly create a long the L will always be printed:

>>> long(0)
0L

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Common subset of python 2 and python 3

[Serhiy Storchaka]

15.01.14 14:22, "Martin v. Löwis" написав(ла):

I don't think that it is possible to write an interpreter that is fully
compatible for all it accepts. Would you think that the program

print(repr(2**80).endswith("L"))

is in the subset that should be supported by both Python 2 and Python 3?

Notice that it prints "True" in Python 2 and "False" in Python 3.


This is implementation details. On 128-bit platform special build of 
Python 2 can print False.


Of course there are many other differences between Python 2 and Python 3 
besides unicode model and unified integers.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Common subset of python 2 and python 3

[Eric V. Smith]

On 1/15/2014 8:21 AM, Chris Angelico wrote:
> On Wed, Jan 15, 2014 at 11:22 PM, "Martin v. Löwis"  
> wrote:
>> I don't think that it is possible to write an interpreter that is fully
>> compatible for all it accepts. Would you think that the program
>>
>> print(repr(2**80).endswith("L"))
>>
>> is in the subset that should be supported by both Python 2 and Python 3?
> 
> Easiest fix for that would be to have long.__repr__ omit the L tag.
> Then it'll do the same as it would in Py3.

I think Martin's point is not this specific thing, but that such a
subset would be useless. Would you drop dict.items() because it returns
different things in both languages? Drop range() because it's different?
There are many, many such differences. The common subset is not useful.

Eric.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Changing Clinic's output

[Stefan Krah]

Larry Hastings  wrote:
> https://bitbucket.org/larry/clinic-buffer-samples/src

Thanks for doing this!

+1 for the sidefile, -1 for the current approach, +-0 for the rest.


Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Common subset of python 2 and python 3

[Chris Angelico]

On Wed, Jan 15, 2014 at 11:22 PM, "Martin v. Löwis"  wrote:
> I don't think that it is possible to write an interpreter that is fully
> compatible for all it accepts. Would you think that the program
>
> print(repr(2**80).endswith("L"))
>
> is in the subset that should be supported by both Python 2 and Python 3?

Easiest fix for that would be to have long.__repr__ omit the L tag.
Then it'll do the same as it would in Py3.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Common subset of python 2 and python 3

[Antoine Pitrou]

On Wed, 15 Jan 2014 13:22:44 +0100
"Martin v. Löwis"  wrote:
> Am 12.01.14 18:39, schrieb Nachshon David Armon:
> >>> I propose that this new version of python use the python 3 unicode model.
> >>> As the version of python will be fully compatible with both python 2 and
> >>> with python 3 but NOT necsesarily with all existing code in either. It is
> >>> designed as a porting tool only.
> 
> I don't think that it is possible to write an interpreter that is fully
> compatible for all it accepts. Would you think that the program
> 
> print(repr(2**80).endswith("L"))
> 
> is in the subset that should be supported by both Python 2 and Python 3?
> 
> Notice that it prints "True" in Python 2 and "False" in Python 3.

We probably need an "asciibool" that is both true and false.

(for some value of "we" ;-))

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Common subset of python 2 and python 3

[Martin v. Löwis]

Am 12.01.14 18:39, schrieb Nachshon David Armon:
>>> I propose that this new version of python use the python 3 unicode model.
>>> As the version of python will be fully compatible with both python 2 and
>>> with python 3 but NOT necsesarily with all existing code in either. It is
>>> designed as a porting tool only.

I don't think that it is possible to write an interpreter that is fully
compatible for all it accepts. Would you think that the program

print(repr(2**80).endswith("L"))

is in the subset that should be supported by both Python 2 and Python 3?

Notice that it prints "True" in Python 2 and "False" in Python 3. So if
this common-version interpreter *rejects* the above program, which
operation (**, repr, endswith) would you want to ban from subset?

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460 reboot

[Stephen J. Turnbull]

Aside: OK, Guido, ya got me.

I have a separate screed recounting the reasons for my apostasy, but
that's probably not interesting any more.  I'll send it to individuals
on request.

 > But in terms of explaining the text model, that
 > separation is important enough that
 > 
 > (1)  We should be reluctant to strengthen the
 >  "its really just ASCII" messages.

True.  I think the right message is is "Unless you know why you
*desperately* want this, not only don't you need it, but using it is
the Python equivalent of skydiving without a parachute."

N.B. Don't take the metaphor as an insult.  I think it's become clear
that those who "desperately want this" not only use parachutes, they
pack their own.  No need to worry about them.

 > (2)  It *may* be worth creating a virtual
 >  split in the documentation.

Please don't.  All we need to tell naive users is:

Look at the structure of the bytes.  If that structure is "text",
convert to str using .decode().  Please don't use bytes.

If that structure isn't text, you're in a specialist domain, and
it's your problem.  Many structured uses of bytes use ASCII-
encoded keywords: we provide bytes methods for handling them, but
you *must* be aware that these methods *cannot* distinguish "bytes
representing text encoded as ASCII" from "any old bytes".  Be
warned: They will happily -- and silently -- corrupt the latter.
Make sure you respect the higher-level structure of your data when
using them.

 > Virtual subclass ASCIIStructuredBytes
 > 
 > 
 > One particularly common use of bytes is to represent
 > the contents of a file, or of a network message.  In
 > these cases, the bytes will often represent Text
 > *in a specific encoding* and that encoding will usually
 > be a superset of ASCII.  Rather than create and support
 > an ASCIIStructuredBytes subclass, Python simply added
 > support for these use cases straight to Bytes objects,
 > and assumes that this support simply won't be used when
 > when it does not make sense. For example, bytes literals

This is going quite the wrong direction, I think.  The only people who
should care about "Text *in a specific encoding* and that encoding
will usually be a superset of ASCII" are codec writers, and by now
writing those is a very rare task.  Everybody else uses ASCII keywords
in a simple formal language.

 > *could* be used to construct a sound sample, but the
 > literals will be far easier to read when they are used
 > to represent (encoded) ASCII text, such as "OPEN". 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460: allowing %d and %f and mojibake

[Stephen J. Turnbull]

Tres Seaver writes:
 > -BEGIN PGP SIGNED MESSAGE-
 > Hash: SHA1
 > 
 > On 01/15/2014 12:57 AM, Stephen J. Turnbull wrote:
 > 
 > > asciistr *canonizes* something as an ASCII string, and therefore 
 > > compatible with both bytes and str.  It can't *create* such a thing
 > > ex nihilo.
 > 
 > How many miracles must be attested?

You'll have to ask Pope Benno I.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Changing Clinic's output

[Serhiy Storchaka]

15.01.14 00:24, Larry Hastings написав(ла):

But there's one important caveat to the above.  As I recall, Guido has
stated that he hates storing generated code in separate files. He has
yet to rescind or weaken that pronouncement.  Until such time as he
does, the "side file" approach is off the table.  I implemented it in
the prototype purely for the purpose of fostering debate, so the "side
file" proponents can try to convince him that it's necessary or that
it's not so bad.  But it's not going in without Guido's approval.  As
you yourself say--"Python is Guido's language, he just lets us use it."


Yes, we know. The conviction of Guido is the purpose of this topic. I 
hope that his silence is a good sign. Perhaps he doubts and weighs 
arguments.


Personally, I believe that if we don't change Argument Clinic output 
now, we will have to do it in 3.5 or 4.0, when many people work with the 
code, but at the cost of history cluttering, what I want to avoid.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Changing Clinic's output

[Serhiy Storchaka]

15.01.14 00:47, Larry Hastings написав(ла):

I also can not change the text, but twice I was a witness as others did. And I 
see that make this mistake very easily.


I also didn't modify the generated text, but twice I was a witness as 
others did. And I see that make this mistake very easily.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com