date:20180306

On Wed, Mar 7, 2018 at 9:09 AM, Ooomzay  wrote:
>> I'm not trying to dissuade you from using RAII in your own applications,
>> if it works for you, great.
>
> Unfortunately, despite having conquered it, without a _guarantee_ of this
> behaviour from the language, or at least one mainstream implementation,
> I will not invest in python again. Nor recommend any one else with a serious
> real world resource management application to do so. This was the original
> motive for my PEP.

What a pity Python will have to lose someone whose sole goal was to
write C++ code. Had you but chosen to write Python code instead, you
could have saved us all a hundred emails or so and just used the
'with' statement, same as the rest of us do. Considering that "real
world resource management" is *exactly* the purpose of the 'with'
statement, I honestly don't see what the problem is.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bitstream -- Binary Data for Humans

2018-03-06 Thread Roel Schroeven

Sébastien Boisgérault schreef op 6/03/2018 10:23:
I had a look at the AIS message format from your link(https://en.wikipedia.org/wiki/Automatic_identification_system#Message_format) and this seems to be a nice use case; all the data components seem to be nicely aligned on the byte boundary ... until you see that the payload uses ASCII6(*), which I didn't know about.

Thanks again for the info!

ASCII6 is only the first layer where your Bitstream could be useful.
It's the easy part :).

The main work comes after that: decoding ASCII6 results in binary data
which we need to decode into AIS messages. You can see the most
important ones at https://navcen.uscg.gov/?pageName=AISMessagesA and
https://navcen.uscg.gov/?pageName=AISMessagesAStatic but there are many
more.

So for each message we first extract the first 6 bits, which tells us
the message identifier. What we need to do afterwards depends on that
message identifier. The links above describe how to deal with messages
1, 2 and 3 (they are identical except for the message identifier; I
don't know why there are 3 instead of just one) and 5. Basically it's
things like 2 bits for this, next 30 bits for that, etc. That's where
I'm hoping your Bitstream can really shine.

Also note, but now I'm getting off topic, that we don't have to deal
with the low-level data as is described in your link: that is handled by
the AIS transponder or AIS receiver, from which we receive it using
simple NMEA 0183 serial sentences: see
https://en.wikipedia.org/w/index.php?title=NMEA_0183&oldid=828034316#Message_structure

--
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
-- Isaac Asimov

Roel Schroeven

--
https://mail.python.org/mailman/listinfo/python-list

Re: RFC: Proposal: Deterministic Object Destruction

2018-03-06 Thread Ooomzay

On Monday, 5 March 2018 23:06:53 UTC, Steven D'Aprano  wrote:
> On Mon, 05 Mar 2018 09:22:33 -0800, Ooomzay wrote:
> [...]
> > If you would like to have a shot at coding this without RAII, but
> > preserving the OO design, you will find that it is considerably
> > simpler than the with/context manager approach.
> 
> Preserving the OO design, you say? Okay, since my application apparently 
> isn't allowed to know that it is processing two files, I'll simply 
> delegate that to the object:
> 
> class C(A, B):
> def run(self):
> with open(self.a) as a:
> with open(self.b) as b:
> process(a, b)
> 
> # Later.
> if __name__ = '__main__':
> # Enter the Kingdom of Nouns.
> c = C()
> c.run()
> 
> There you go. Encapsulation and OO design.

1. This does not execute. It is only when you actually flesh out 
these deliberately minimal classes A, B & C with PEP232 paraphernalia
that we all be able to see clearly how much cleaner or messier 
things are with PEP232. 

2. Your Class C breaks A & B's encapsulation by inheriting rather than 
composing them. Apart from increasing coupling and preventing 
substitution, C contained A & B in this example to illustrate that there is 
no RAII related burden whatever_on this intermediate class as 
it manages no external resources directly. It does not even 
need to implement __del__.

3. You have assumed a single threaded application. Please imagine 
that A and B are classes managing some remote valves and maintain 
their own threads as well as a serial port for comms. And C is there to 
coordinate them towards some greater purpose.

If you would like to try and add PEP232 support to
classes A,B & C to the point that you can create
and destroy c = C() in an exception-safe way we may 
all learn something.

> I'm not trying to dissuade you from using RAII in your own applications, 
> if it works for you, great.

Unfortunately, despite having conquered it, without a _guarantee_ of this 
behaviour from the language, or at least one mainstream implementation, 
I will not invest in python again. Nor recommend any one else with a serious 
real world resource management application to do so. This was the original 
motive for my PEP.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

2018-03-06 Thread Peter Otten

Chris Angelico wrote:

> On Wed, Mar 7, 2018 at 2:12 AM, Kirill Balunov 
> wrote:
>>
>>
>> 2018-03-06 17:55 GMT+03:00 Chris Angelico :
>>>
>>> On Wed, Mar 7, 2018 at 1:48 AM, Kirill Balunov 
>>> wrote:
>>> > Note: For some historical reasons as the first argument you can use
>>> > None instead of function, in this case the identity function is
>>> > assumed. That is, all elements of iterable that are false are removed
>>> > which is equivalent
>>> > to (item for item in iterable if item). Currently, for the same
>>> > purpose the
>>> > preferred form is `filter(bool, iterable)`.
>>> >
>>>
>>> I'd prefer to word it something like:
>>>
>>> If the first argument is None, the identity function is assumed. That
>>> is, all elements of the iterable that are false are removed; it is
>>> equivalent to (item for item in iterable if item). It is approximately
>>> equivalent to (but faster than) filter(bool, iterable).
>>>
>>> ChrisA
>>> --
>>> https://mail.python.org/mailman/listinfo/python-list
>>
>>
>> I do not want to seem rude and stubborn, but how much faster is it to
>> highlight or emphasize it:
>>
> 
> Timings mean little. Why do we write:
> 
> if lst:
> 
> instead of:
> 
> if bool(lst):
> 
> ? Because it's unnecessary and pointless to call bool() on something
> before using it in a boolean context. If that concept causes you
> problems, it's not the fault of the filter function; filter simply
> uses something in a boolean context.
> 
> So "the identity function" is more correct than "the bool() function".

Fun fact: CPython handles filter(bool) and filter(None) the same way, it 
sets the checktrue flag and chooses the fast path:

static PyObject *
filter_next(filterobject *lz)
{
PyObject *item;
PyObject *it = lz->it;
long ok;
PyObject *(*iternext)(PyObject *);
int checktrue = lz->func == Py_None || lz->func == (PyObject 
*)&PyBool_Type;

iternext = *Py_TYPE(it)->tp_iternext;
for (;;) {
item = iternext(it);
if (item == NULL)
return NULL;

if (checktrue) {
ok = PyObject_IsTrue(item);
} else {
PyObject *good;
good = PyObject_CallFunctionObjArgs(lz->func, item, NULL);
if (good == NULL) {
Py_DECREF(item);
return NULL;
}
ok = PyObject_IsTrue(good);
Py_DECREF(good);
}
if (ok > 0)
return item;
Py_DECREF(item);
if (ok < 0)
return NULL;
}
}

If there were a built-in identity() function it could be treated the same 
way.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

On Wed, Mar 7, 2018 at 2:33 AM, Kirill Balunov  wrote:
>
>
> 2018-03-06 17:55 GMT+03:00 Chris Angelico :
>>
>> If the first argument is None, the identity function is assumed. That
>> is, all elements of the iterable that are false are removed; it is
>> equivalent to (item for item in iterable if item). It is approximately
>> equivalent to (but faster than) filter(bool, iterable).
>>
>> ChrisA
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>
>
> If you look in C source for `filter_next`
> https://github.com/python/cpython/blob/5d92647102fac9e116b98ab8bbc632eeed501c34/Python/bltinmodule.c#L593,
> there is a line:
>
> int checktrue = lz->func == Py_None || lz->func == (PyObject *)&PyBool_Type;
>
> So the only difference between `filter(None, ls`) and `filter(bool, ls)` is
> LOAD_NAME vs LOAD_CONST and that `None` is checked before than `bool`.
>

Assuming that nobody's shadowed the name 'bool' anywhere, which has to
be checked for at run time. (Which is the job of LOAD_NAME.)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

2018-03-06 17:55 GMT+03:00 Chris Angelico :

> If the first argument is None, the identity function is assumed. That
> is, all elements of the iterable that are false are removed; it is
> equivalent to (item for item in iterable if item). It is approximately
> equivalent to (but faster than) filter(bool, iterable).
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>

If you look in C source for `filter_next`
https://github.com/python/cpython/blob/5d92647102fac9e116b98ab8bbc632eeed501c34/Python/bltinmodule.c#L593,
there is a line:

int checktrue = lz->func == Py_None || lz->func == (PyObject
*)&PyBool_Type;

So the only difference between `filter(None, ls`) and `filter(bool, ls)` is
LOAD_NAME vs LOAD_CONST and that `None` is checked before than `bool`.


With kind regards,
-gdg
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

On Wed, Mar 7, 2018 at 2:12 AM, Kirill Balunov  wrote:
>
>
> 2018-03-06 17:55 GMT+03:00 Chris Angelico :
>>
>> On Wed, Mar 7, 2018 at 1:48 AM, Kirill Balunov 
>> wrote:
>> > Note: For some historical reasons as the first argument you can use None
>> > instead of function, in this case the identity function is assumed. That
>> > is, all elements of iterable that are false are removed which is
>> > equivalent
>> > to (item for item in iterable if item). Currently, for the same purpose
>> > the
>> > preferred form is `filter(bool, iterable)`.
>> >
>>
>> I'd prefer to word it something like:
>>
>> If the first argument is None, the identity function is assumed. That
>> is, all elements of the iterable that are false are removed; it is
>> equivalent to (item for item in iterable if item). It is approximately
>> equivalent to (but faster than) filter(bool, iterable).
>>
>> ChrisA
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>
>
> I do not want to seem rude and stubborn, but how much faster is it to
> highlight or emphasize it:
>

Timings mean little. Why do we write:

if lst:

instead of:

if bool(lst):

? Because it's unnecessary and pointless to call bool() on something
before using it in a boolean context. If that concept causes you
problems, it's not the fault of the filter function; filter simply
uses something in a boolean context.

So "the identity function" is more correct than "the bool() function".

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

2018-03-06 17:55 GMT+03:00 Chris Angelico :

> On Wed, Mar 7, 2018 at 1:48 AM, Kirill Balunov 
> wrote:
> > Note: For some historical reasons as the first argument you can use None
> > instead of function, in this case the identity function is assumed. That
> > is, all elements of iterable that are false are removed which is
> equivalent
> > to (item for item in iterable if item). Currently, for the same purpose
> the
> > preferred form is `filter(bool, iterable)`.
> >
>
> I'd prefer to word it something like:
>
> If the first argument is None, the identity function is assumed. That
> is, all elements of the iterable that are false are removed; it is
> equivalent to (item for item in iterable if item). It is approximately
> equivalent to (but faster than) filter(bool, iterable).
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>

I do not want to seem rude and stubborn, but how much faster is it to
highlight or emphasize it:

from random import randint
for i in [1, 10, 100, 1000, 1, 10]:
ls = [randint(0,1) for _ in range(i)]
%timeit [*filter(None, ls)]
%timeit [*filter(bool, ls)]
print()

272 ns ± 0.0346 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)
282 ns ± 0.0714 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)

283 ns ± 0.0645 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)
296 ns ± 0.116 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)

1.4 µs ± 1.32 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)
1.41 µs ± 4.05 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)

14.7 µs ± 40.1 ns per loop (mean ± std. dev. of 7 runs, 10 loops each)
14.7 µs ± 23.2 ns per loop (mean ± std. dev. of 7 runs, 10 loops each)

137 µs ± 186 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)
137 µs ± 24.7 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)

1.32 ms ± 285 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1.32 ms ± 908 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

With kind regards,
-gdg
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

On Wed, Mar 7, 2018 at 1:48 AM, Kirill Balunov  wrote:
> Note: For some historical reasons as the first argument you can use None
> instead of function, in this case the identity function is assumed. That
> is, all elements of iterable that are false are removed which is equivalent
> to (item for item in iterable if item). Currently, for the same purpose the
> preferred form is `filter(bool, iterable)`.
>

I'd prefer to word it something like:

If the first argument is None, the identity function is assumed. That
is, all elements of the iterable that are false are removed; it is
equivalent to (item for item in iterable if item). It is approximately
equivalent to (but faster than) filter(bool, iterable).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

2018-03-06 16:58 GMT+03:00 Jason Friedman :

>
> as a ordinary Python user I'd be interested in improvements to the
> documentation, including suggestions on real-world usage.
>

I'm just an ordinary user, just like you :)


> Kirill, taking deprecation/removal off the table, what changes would you
> recommend to the documentation?
>

My English is about basic level, so you may need to fix the spelling. But I
would write as follows:


filter(function, iterable)
Construct an iterator from those elements of iterable for which function
returns truthy values. iterable may be either a sequence, a container which
supports iteration, or an iterator.

Note that filter(function, iterable) is equivalent to the generator
expression (item for item in iterable if function(item)). In cases when
function corresponds to a simple lambda function a generator expression
should be preferred. For example, when it is necessary to eliminate all
multiples of three (x for x in range(100) if x % 3) should be used, instead
of filter(lambda x: x % 3, range(100))

See itertools.filterfalse() for the complementary function that returns
elements of iterable for which function returns false.

Note: For some historical reasons as the first argument you can use None
instead of function, in this case the identity function is assumed. That
is, all elements of iterable that are false are removed which is equivalent
to (item for item in iterable if item). Currently, for the same purpose the
preferred form is `filter(bool, iterable)`.

p.s.:
maybe _function_ should be changed to _callable_.

With kind regards,
-gdg
-- 
https://mail.python.org/mailman/listinfo/python-list

"except" and "subclasscheck" changed between CPython2 and 3

2018-03-06 Thread Youta TAKAOKA

Thank you, Steven.

> There is a feature-request to support that (as Python 2.7 does):
>
> https://bugs.python.org/issue12029
>
> but it is stalled.

I passed over the ticket.

Now, I know that this is a bug, but has not fixed yet.
There are (or ware ?) problems about performance and integrity for handling
exceptions.

It seems that a narrow range of use-cases cause the stalling.
Someday we have a type-focused VERY NICE library, (I hope that) it will be
resolved.

Thanks.

--
yout...@gmail.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Layers of abstraction, was Re: RFC: Proposal: Deterministic Object Destruction

2018-03-06 Thread Peter Otten

Chris Angelico wrote:

> On Tue, Mar 6, 2018 at 10:04 AM, Steven D'Aprano
>  wrote:
>> # Later.
>> if __name__ = '__main__':
>> # Enter the Kingdom of Nouns.
> 
> Don't you need a NounKingdomEnterer to do that for you?

No, for some extra flexibility there should be a NounKingdomEntererFactory 
-- which of course has to implement the AbstractNounKingdomEntererFactory 
interface. Instantiating the right NounKingdomEntererFactory can easily be 
delegated to a NounKingdomEntererFactoryProducer. It's turtles^Wfactories 
all the way down...

I don't know how this will ever do anything -- my theory is that some good 
soul managed to sneak a few lines of javascript into the giant xml 
configuration file ;)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

2018-03-06 16:35 GMT+03:00 Chris Green :

> It's 'deprecation', depreciation is something quite different.  People
> replying have spelt it correctly so you might possibly have noticed I
> thought/hoped.
>
> ... and it does matter a bit because it's not just a mis-spelling, the
> word you are using has its own meaning and could thus cause confusion.
>
> ... and, yes, I know it's a very common and easily made mistake. :-)
>
>
I did not ;) Thank you!

With kind regards,
-gdg
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

2018-03-06 16:51 GMT+03:00 Chris Angelico :

> On Wed, Mar 7, 2018 at 12:23 AM, Kirill Balunov 
> wrote:
> > Filter is generally faster than list comprehension or generators.
> >
> > %timeit [*filter(lambda x: x % 3, range(1000))]
> > 100 µs ± 16.4 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)
> >
> > f = lambda x: x % 3
> >
> > %timeit [*(f(i) for i in range(1000))]
> > 132 µs ± 73.5 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)
> >
> > %timeit [f(i) for i in range(1000)]
> > 107 µs ± 179 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)
> >
>
> These don't do the same thing, though. A more comparable comprehension is:
>
> [i for i in range(1000) if i % 3]
>
> rosuav@sikorsky:~$ python3 -m timeit '[i for i in range(1000) if i % 3]'
> 1 loops, best of 5: 34.5 usec per loop
> rosuav@sikorsky:~$ python3 -m timeit '[*filter(lambda x: x % 3,
> range(1000))]'
> 5000 loops, best of 5: 81.1 usec per loop
>
> And my point about comprehensions was that you do NOT use a pointless
> function for them - you just have inline code. If there is a
> pre-existing function, sure! Use it. But when you use filter or map
> with a lambda function, you should probably use a comprehension
> instead.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Thank you, I did not understand you at first, now everything is clear. In
this sense of `x % 3`, I fully agree with you.

With kind regards,
-gdg
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

2018-03-06 Thread Jason Friedman

On Tue, Mar 6, 2018 at 1:52 AM, Kirill Balunov 
wrote:

>
> I propose to delete all references in the `filter` documentation that the
> first argument can be `None`, with possible depreciation of `None` as the
> the first argument - FutureWarning in Python 3.8+ and deleting this option
> in Python 4. Personally, regarding the last point - depreciation, I do not
> see this as a great necessity, but I do not find that the option with
> `None`
> should be offered and suggested through the documentation. Instead, it is
> better to show an example with using `filter(bool, iterable)` which is
> absolutely
> equivalent, more readable, but a little bit slower.
>
> Currently documentation for `None` case uses `identity function is
> assumed`, what is this `identity` and how it is consistent with
> truthfulness?
>
> In addition, this change makes the perception of `map` and `filter` more
> consistent,with the rule that first argument must be `callable`.
>
> I see only one moment with `None`, since `None` is a keyword, the behavior
> of `filter(None, iterable)` is alsways guaranteed, but with `bool` it is
> not. Nevertheless, we are all adults here.
>

I won't pretend I am qualified to debate the technical aspects here, but
regarding this snippet:

... Instead, it is better to show an example with using `filter(bool,
iterable)` which is absolutely equivalent, more readable ...

as a ordinary Python user I'd be interested in improvements to the
documentation, including suggestions on real-world usage.  For example,
Chris Angelico below says in part:

... that said, though, any use of filter() that involves a lambda
function should
probably become list comps or genexps, so filter itself should only be used
when ...

Kirill, taking deprecation/removal off the table, what changes would you
recommend to the documentation?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

On Wed, Mar 7, 2018 at 12:23 AM, Kirill Balunov  wrote:
> Filter is generally faster than list comprehension or generators.
>
> %timeit [*filter(lambda x: x % 3, range(1000))]
> 100 µs ± 16.4 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)
>
> f = lambda x: x % 3
>
> %timeit [*(f(i) for i in range(1000))]
> 132 µs ± 73.5 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)
>
> %timeit [f(i) for i in range(1000)]
> 107 µs ± 179 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)
>

These don't do the same thing, though. A more comparable comprehension is:

[i for i in range(1000) if i % 3]

rosuav@sikorsky:~$ python3 -m timeit '[i for i in range(1000) if i % 3]'
1 loops, best of 5: 34.5 usec per loop
rosuav@sikorsky:~$ python3 -m timeit '[*filter(lambda x: x % 3, range(1000))]'
5000 loops, best of 5: 81.1 usec per loop

And my point about comprehensions was that you do NOT use a pointless
function for them - you just have inline code. If there is a
pre-existing function, sure! Use it. But when you use filter or map
with a lambda function, you should probably use a comprehension
instead.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

2018-03-06 Thread Chris Green

Kirill Balunov  wrote:
> 
> As I wrote, __possible depreciation__, I also do not see the point of just

It's 'deprecation', depreciation is something quite different.  People
replying have spelt it correctly so you might possibly have noticed I
thought/hoped.

... and it does matter a bit because it's not just a mis-spelling, the
word you are using has its own meaning and could thus cause confusion.

... and, yes, I know it's a very common and easily made mistake. :-)

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

2018-03-06 13:18 GMT+03:00 Chris Angelico :

> The identity function is:
>
> filter(lambda x: x, range(10))
>
> How is it consistent with truthiness? Exactly the same way the
> underlying object is. There's no requirement for the predicate
> function to return True or False - it's perfectly acceptable, for
> instance, to do this:
>
> filter(lambda x: x % 3, range(10))
>
> to eliminate all multiples of three.
>

Yes there is no reason to return True and False, but in the case of `None`
and `bool` under the hood there will be no difference and the form with
`bool` is much more readable.


>
> That said, though, any use of filter() that involves a lambda function
> should probably become list comps or genexps, so filter itself should
> only be used when there really IS a pre-existing function that does
> the job.


Filter is generally faster than list comprehension or generators.

%timeit [*filter(lambda x: x % 3, range(1000))]
100 µs ± 16.4 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)

f = lambda x: x % 3

%timeit [*(f(i) for i in range(1000))]
132 µs ± 73.5 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)

%timeit [f(i) for i in range(1000)]
107 µs ± 179 ns per loop (mean ± std. dev. of 7 runs, 1 loops each)


> So, for instance, you could strip out every occurrence of the
> string "0" with:
>
> filter(int, list_of_strings)
>
> And that still depends on the normal Python rules for boolification.
> If that's valid, then it should be just as viable to say
> "filter(identity-function, ...)", which is spelled "filter(None,
> ...)".
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

2018-03-06 14:17 GMT+03:00 Steven D'Aprano <
steve+comp.lang.pyt...@pearwood.info>:

> On Tue, 06 Mar 2018 11:52:22 +0300, Kirill Balunov wrote:
>
> > I propose to delete all references in the `filter` documentation that
> > the first argument can be `None`, with possible depreciation of `None`
> > as the the first argument - FutureWarning in Python 3.8+ and deleting
> > this option in Python 4.
>
> Even if we agreed that it is unfortunate that filter accepts None as an
> argument, since it does (and has done since Python 1.0) there is nothing
> to be gained by deprecating and removing it.
>
> Deprecating and removing it will break code that currently works, for no
> good reason; removing the documentation is unacceptable, as that makes it
> too difficult for people to find out what `filter(None, values)` does.
>

As I wrote, __possible depreciation__, I also do not see the point of just
breaking someone's code. But I didn't see any benefit to explicitly promote
`filter(None, iterable)` form in the documentation as a good style.


> > Instead, it is better to show an example with using
> > `filter(bool, iterable)` which is absolutely
> > equivalent, more readable, but a little bit slower.
>
> So long as `filter(None, ...)` is still documented, I don't mind what
> example is given.
>
> But the idiom `filter(None, ...)` is an old, common idiom, very familiar
> to many people who have a background in functional programming.
>

While this form familiar and common idiom for those who are familiar with
Python from versions < 2.3, before `bool` type was introduced. It looks
kinky for newcomers and not obvious at a glance. In functional programming
we use a predicate, and `None` does not match predicate definition, while
`bool` does!


> It is unfortunate that filter takes the arguments in the order it does.
> Perhaps it would have been better to write it like this:
>
> def filter(iterable, predicate=None):
> ...
>
>
> Then `filter(values, None)` would be a standard Python idiom, explicitly
> saying to use the default predicate function. There is no difference to
> `filter(None, values)` except the order is (sadly) reversed.
>

If such a form was in Python, I probably would agree with you. Although in
its present form I like it a lot more and find it more intuitive.

> Currently documentation for `None` case uses `identity function is
> > assumed`, what is this `identity` and how it is consistent with
> > truthfulness?
>
> The identity function is a mathematical term for a function that returns
> its argument unchanged:
>
> def identity(x):
> return x
>
> So `filter(func, values)` filters according to func(x); using None
> instead filters according to x alone, without the expense of calling a do-
> nothing function:
>
> # slow because it has to call the lambda function each time;
> filter(lambda x: x, values)
>
> # fast because filter takes an optimized path
> filter(None, values)
>


> Since filter filters according to the truthy or falsey value of x, it
> isn't actually necessary to call bool(x). In Python, all values are
> automatically considered either truthy or falsey. The reason to call
> bool() is to ensure you have a canonical True/False value, and there's no
> need for that here.


I went over a bit with the question what is identity function :) But I have
a feeling that I perceive all of the above quite the contrary in the
context of a `filter` function. And since filter filters according to the
truthy or falsey value of x. `None` and `bool` should behave totally
equivalent under the hood and I'm 99% sure that it is so.


> So the identity function should be preferred to bool,
> for those who understand two things:
>
> - the identity function (using None as the predicate function)
>   returns x unchanged;
>

Sorry, but how does the above relates to the `filter` discussion?


>
> - and that x, like all values, automatically has a truthy value in a
>   boolean context (which includes filter).
>
>
Yes, and that is why there is no point to `None` since they will do the
same thing in context of `filter` function.


> > In addition, this change makes the perception of `map` and `filter` more
> > consistent,with the rule that first argument must be `callable`.
>
> I consider that a flaw in map. map should also accept None as the
> identity function, so that map(None, iterable) returns the values of
> iterable unchanged.
>
> def map(function=None, *iterables):
> if len(iterables) == 0:
> raise TypeError("map() must have at least two arguments.")
> if function is None:
> if len(iterables) > 1:
> return zip(*iterables)
> else:
> assert len(iterables) == 1
> return iter(iterables[0])
> elif len(iterables) > 1:
> return (function(*args) for args in zip(*iterables))
> else:
> assert len(iterables) == 1
> return (function(arg) for arg in iterables[0])
>

And what will be the practical reason to have this? :)

W

Re: Do not promote `None` as the first argument to `filter` in documentation.

2018-03-06 Thread Steven D'Aprano

On Tue, 06 Mar 2018 11:52:22 +0300, Kirill Balunov wrote:

> I propose to delete all references in the `filter` documentation that
> the first argument can be `None`, with possible depreciation of `None`
> as the the first argument - FutureWarning in Python 3.8+ and deleting
> this option in Python 4.

Even if we agreed that it is unfortunate that filter accepts None as an 
argument, since it does (and has done since Python 1.0) there is nothing 
to be gained by deprecating and removing it.

Deprecating and removing it will break code that currently works, for no 
good reason; removing the documentation is unacceptable, as that makes it 
too difficult for people to find out what `filter(None, values)` does.

> Instead, it is better to show an example with using
> `filter(bool, iterable)` which is absolutely
> equivalent, more readable, but a little bit slower.

So long as `filter(None, ...)` is still documented, I don't mind what 
example is given.

But the idiom `filter(None, ...)` is an old, common idiom, very familiar 
to many people who have a background in functional programming.

It is unfortunate that filter takes the arguments in the order it does. 
Perhaps it would have been better to write it like this:

def filter(iterable, predicate=None):
...

Then `filter(values, None)` would be a standard Python idiom, explicitly 
saying to use the default predicate function. There is no difference to 
`filter(None, values)` except the order is (sadly) reversed.

> Currently documentation for `None` case uses `identity function is
> assumed`, what is this `identity` and how it is consistent with
> truthfulness?

The identity function is a mathematical term for a function that returns 
its argument unchanged:

def identity(x):
return x

So `filter(func, values)` filters according to func(x); using None 
instead filters according to x alone, without the expense of calling a do-
nothing function:

# slow because it has to call the lambda function each time;
filter(lambda x: x, values)

# fast because filter takes an optimized path
filter(None, values)

Since filter filters according to the truthy or falsey value of x, it 
isn't actually necessary to call bool(x). In Python, all values are 
automatically considered either truthy or falsey. The reason to call 
bool() is to ensure you have a canonical True/False value, and there's no 
need for that here. So the identity function should be preferred to bool, 
for those who understand two things:

- the identity function (using None as the predicate function) 
  returns x unchanged;

- and that x, like all values, automatically has a truthy value in a
  boolean context (which includes filter).

> In addition, this change makes the perception of `map` and `filter` more
> consistent,with the rule that first argument must be `callable`.

I consider that a flaw in map. map should also accept None as the 
identity function, so that map(None, iterable) returns the values of 
iterable unchanged.

def map(function=None, *iterables):
if len(iterables) == 0:
raise TypeError("map() must have at least two arguments.")
if function is None:
if len(iterables) > 1:
return zip(*iterables)
else:
assert len(iterables) == 1
return iter(iterables[0])
elif len(iterables) > 1:
return (function(*args) for args in zip(*iterables))
else:
assert len(iterables) == 1
return (function(arg) for arg in iterables[0])

-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bitstream -- Binary Data for Humans (Posting On Python-List Prohibited)

Le mardi 6 mars 2018 11:15:15 UTC+1, Terry Reedy a écrit :
> On 3/6/2018 3:58 AM, Sébastien Boisgérault wrote:
> > Hi Lawrence,
> > 
> > Le mardi 6 mars 2018 01:20:36 UTC+1, Lawrence D’Oliveiro a écrit :
> >> On Tuesday, March 6, 2018 at 8:06:00 AM UTC+13, Sébastien Boisgérault 
> >> wrote:
> >>> I have released bitstream, a Python library to manage binary data
> >>> (at the byte or bit level), hopefully without the pain that this kind
> >>> of thing usually entails :)
> >>
> >>> byte_index = offset / 8
> >>
> >> This will return a float.
> 
> byte_index // 8
> will give you the int index directly in both late 2.x and 3.x.

Indeed! And since this is Cython code, I *guess* that I should combine `//` 
with the 'cdivision' set to True to get the syntax that everyone understands 
*and* the pure C speed (see 
http://cython.readthedocs.io/en/latest/src/reference/compilation.html). I need 
to run some experiments to make sure that this behaves as expected, but this is 
very likely the way to go.

> -- 
> Terry Jan Redey

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do not promote `None` as the first argument to `filter` in documentation.

On Tue, Mar 6, 2018 at 7:52 PM, Kirill Balunov  wrote:
> This thought occurred to me several times, but I could not decide to write.
> And since `filter` is a builtin, I think this change should be discussed
> here, before opening an issue on bug tracker.
>
> I propose to delete all references in the `filter` documentation that the
> first argument can be `None`, with possible depreciation of `None` as the
> the first argument - FutureWarning in Python 3.8+ and deleting this option
> in Python 4. Personally, regarding the last point - depreciation, I do not
> see this as a great necessity, but I do not find that the option with `None`
> should be offered and suggested through the documentation. Instead, it is
> better to show an example with using `filter(bool, iterable)` which is
> absolutely
> equivalent, more readable, but a little bit slower.
>
> %timeit [*filter(None, range(10))]
> 503 ns ± 0.259 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)
>
> %timeit [*filter(bool, range(10))]
> 512 ns ± 1.09 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)
>
> Currently documentation for `None` case uses `identity function is
> assumed`, what is this `identity` and how it is consistent with
> truthfulness?

The identity function is:

filter(lambda x: x, range(10))

How is it consistent with truthiness? Exactly the same way the
underlying object is. There's no requirement for the predicate
function to return True or False - it's perfectly acceptable, for
instance, to do this:

filter(lambda x: x % 3, range(10))

to eliminate all multiples of three.

That said, though, any use of filter() that involves a lambda function
should probably become list comps or genexps, so filter itself should
only be used when there really IS a pre-existing function that does
the job. So, for instance, you could strip out every occurrence of the
string "0" with:

filter(int, list_of_strings)

And that still depends on the normal Python rules for boolification.
If that's valid, then it should be just as viable to say
"filter(identity-function, ...)", which is spelled "filter(None,
...)".

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bitstream -- Binary Data for Humans (Posting On Python-List Prohibited)

2018-03-06 Thread Terry Reedy


On 3/6/2018 3:58 AM, Sébastien Boisgérault wrote:

Hi Lawrence,

Le mardi 6 mars 2018 01:20:36 UTC+1, Lawrence D’Oliveiro a écrit :

On Tuesday, March 6, 2018 at 8:06:00 AM UTC+13, Sébastien Boisgérault wrote:

I have released bitstream, a Python library to manage binary data
(at the byte or bit level), hopefully without the pain that this kind
of thing usually entails :)



byte_index = offset / 8


This will return a float.


byte_index // 8
will give you the int index directly in both late 2.x and 3.x.

--
Terry Jan Redey


--
https://mail.python.org/mailman/listinfo/python-list

Re: Ways to make a free variable local to a function?

On Tue, Mar 6, 2018 at 8:02 PM, Kirill Balunov  wrote:
>
>
> 2018-03-05 17:34 GMT+03:00 Chris Angelico :
>>
>> In theory, the CPython bytecode compiler (don't know about other
>> Python implementations) could just add these as constants. They'd then
>> be bound at either compile time or function definition time (by
>> default the former, I think, but the latter would be more useful), and
>> be looked up as quickly as locals. I'm not sure how useful this would
>> be, though.
>
>
> With some assumptions, It will be useful for every function call:-)
>
>> If PEP 572 [1] were to be accepted, you could do something like this:
>>
>> def func(numb):
>> if ((int as int), (float as float)):
>> res = []
>> for i in range(numb):
>> res.append(int(i) + float(i))
>> return res
>>
>> Syntactically a bit clunky, but keeps everything inside the function,
>> and DOES create local variables. Not sure it's better than your other
>> options, but it is another option.
>
>
> While I'm +0.5 on yours PEP 572 idea, especially in `while` and `if`
> statements, this example is an overuse of the proposed syntax ;-) Also it
> will add an overhead on every function call, and as you said  -
> "Syntactically a bit clunky".
>

The run-time overhead should be insignificant; this kind of
optimization is done when you're running a tight loop, so it's the run
time of the loop body that dominates the function. That's also why I
do NOT want this to happen at compile time, even though that would be
the easiest. The very latest this should happen is function definition
time; it would be extremely surprising otherwise. And if it happens
once when the function's called, that's usually not going to be much
cost compared to the saving of LOAD_FAST instead of LOAD_GLOBAL in
each iteration of the loop.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ways to make a free variable local to a function?

 2018-03-05 21:44 GMT+03:00 Terry Reedy :

> Yes, what we really want for this sort of thing are unrebindable local
> constants.  A simple syntax change could do it.
>
>  def func_local_1(numb; int = int, float = float, range = range):
>
> The binding after ';' belong in the header because they should be done
> once.
>
> They'd then
>> be bound at either compile time or function definition time (by
>> default the former, I think, but the latter would be more useful), and
>> be looked up as quickly as locals. I'm not sure how useful this would
>> be, though.
>>
>
> I believe that the occasional practice of re-binding built-in names to
> locals can be shown to speed up loops run enough times.


Yes "_unrebindable local constants_" it is what I was thinking about. But I
do not agree that they must be passed through arguments, because they
should be somewhat static for a function, and could not be changed by any
means after function is compiled.
Alternative option, more dynamic - to allow injecting local variables into
the function via some interface. Currently, there is no such _feature_ in
Python, at least I do not know. There was _somewhat_ related discussion
about how to change the locals of a frame (https://bugs.python.org/
issue1654367) by making `frame.f_locals` writable, but it seems that it is
dead.

With kind regards,
-gdg
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bitstream -- Binary Data for Humans (Posting On Python-List Prohibited)

Le mardi 6 mars 2018 10:23:02 UTC+1, Lawrence D’Oliveiro a écrit :
> On Tuesday, March 6, 2018 at 9:59:55 PM UTC+13, Sébastien Boisgérault wrote:
> > 
> > Le mardi 6 mars 2018 01:20:36 UTC+1, Lawrence D’Oliveiro a écrit :
> >
> >> On Tuesday, March 6, 2018 at 8:06:00 AM UTC+13, Sébastien Boisgérault 
> >> wrote:
> >>>
> >>>byte_index = offset / 8
> >> 
> >> This will return a float.
> > 
> > The implementation is in Cython, which allows to declare types. 
> > The variable byte_index is defined as a size_t. 
> 
> Ah, I see. Still it seems unPythonic to use ”/” for integer division. Does it 
> not allow “//”?

Yes, '//' works, see:


https://mybinder.org/v2/gh/boisgera/jupyter-cython/master?filepath=Integer%20Division.ipynb

Actually, in this Jupyter notebook setting, this is '/' that doesn't work!
I should have a new look at this; since performance also matters,
I'd also like to have the Cython code with the smallest overhead 
(even if it is less readable/Pythonic).

Cheers,

SB
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ways to make a free variable local to a function?

2018-03-06 Thread Wolfgang Maier


On 03/05/2018 07:44 PM, Terry Reedy wrote:

On 3/5/2018 9:34 AM, Chris Angelico wrote:

On Tue, Mar 6, 2018 at 12:52 AM, Terry Reedy  wrote:

On 3/5/2018 7:12 AM, Kirill Balunov wrote:

# 1. By passing through local variable's default values

   def func_local_1(numb, _int = int, _float = float, _range = range):



You are not required to mangle the names.

def func_local_1(numb, int = int, float = float, range = range):
...



Even so, this does mess up the function's signature,


Which I why I only said that using the original names solves the syntax
highlighting issue (of marking built-ins as built-ins).


leaving your
callers wondering if they can call it with some sort of range
parameter. (Though in this particular instance, range() is only called
once, so it's pretty much useless to try to optimize it.)

In theory, the CPython bytecode compiler (don't know about other
Python implementations) could just add these as constants.


Yes, what we really want for this sort of thing are unrebindable local
constants.  A simple syntax change could do it.

   def func_local_1(numb; int = int, float = float, range = range):

The binding after ';' belong in the header because they should be done once.



Ah, I did not really understand initially what Kirill was trying to 
achieve by putting the name binding into the function signature.
Now I do, but I don't think it is a good idea. Sanctioning this with 
dedicated syntax would only make Python more static because for any 
function defined this way, you would lose the ability to alter the 
behavior of that function through changing the global binding after the 
function has been called (in the example above, you could no longer mock 
replace int, float and range on subsequent func_local_1 calls) and I 
don't think this is something that should be encouraged.


Wolfgang


--
https://mail.python.org/mailman/listinfo/python-list

Re: Bitstream -- Binary Data for Humans

Le mardi 6 mars 2018 09:26:50 UTC+1, Sébastien Boisgérault a écrit :
> Le mardi 6 mars 2018 00:29:25 UTC+1, Roel Schroeven a écrit :
> > Sébastien Boisgérault schreef op 5/03/2018 20:05:
> > > I have released bitstream, a Python library to manage binary data (at the 
> > > byte or bit level),
> >  > hopefully without the pain that this kind of thing usually entails :)
> > > 
> > > If you have struggled with this topic in the past, please take a look at 
> > > the documentation
> >  > (http://boisgera.github.io/bitstream/) and tell me what you think.
> > 
> > Hi Sébastien,
> > 
> > At work I have some Python code to decode AIS[1] messages, for which I 
> > created my own code for handling binary data. It works, but is pretty 
> > slow. Not surprising, since handling data at the bit level is not 
> > exactly Python's strength. If I find the time, I'll try to replace my 
> > code with your bitstream and see if it does what I need it to do, and if 
> > it's any faster.
> > 
> > If/when I actually get around to it, I'll keep you informed.
> > 
> > [1] https://en.wikipedia.org/wiki/Automatic_identification_system
> > 
> > 
> > Best regards,
> > Roel
> > 
> > -- 
> > The saddest aspect of life right now is that science gathers knowledge
> > faster than society gathers wisdom.
> >-- Isaac Asimov
> > 
> > Roel Schroeven
> 
> Great, thanks !

Hi again Roel,

I had a look at the AIS message format from your link 
(https://en.wikipedia.org/wiki/Automatic_identification_system#Message_format) 
and this seems to be a nice use case; all the data components seem to be nicely 
aligned on the byte boundary ... until you see that the payload uses ASCII6(*), 
which I didn't know about. 

Thanks again for the info!

Cheers,

(*) ASCII6 code = ASCII code + 48; 
(http://catb.org/gpsd/AIVDM.html#_aivdm_aivdo_sentence_layer)
SB



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ways to make a free variable local to a function?

2018-03-05 17:34 GMT+03:00 Chris Angelico :

> In theory, the CPython bytecode compiler (don't know about other
> Python implementations) could just add these as constants. They'd then
> be bound at either compile time or function definition time (by
> default the former, I think, but the latter would be more useful), and
> be looked up as quickly as locals. I'm not sure how useful this would
> be, though.
>

With some assumptions, It will be useful for every function call:-)

If PEP 572 [1] were to be accepted, you could do something like this:
>
> def func(numb):
> if ((int as int), (float as float)):
> res = []
> for i in range(numb):
> res.append(int(i) + float(i))
> return res
>
> Syntactically a bit clunky, but keeps everything inside the function,
> and DOES create local variables. Not sure it's better than your other
> options, but it is another option.
>

While I'm +0.5 on yours PEP 572 idea, especially in `while` and `if`
statements, this example is an overuse of the proposed syntax ;-) Also it
will add an overhead on every function call, and as you said  -
"Syntactically a bit clunky".

With kind regards,
-gdg
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bitstream -- Binary Data for Humans (Posting On Python-List Prohibited)

Hi Lawrence,

Le mardi 6 mars 2018 01:20:36 UTC+1, Lawrence D’Oliveiro a écrit :
> On Tuesday, March 6, 2018 at 8:06:00 AM UTC+13, Sébastien Boisgérault wrote:
> > I have released bitstream, a Python library to manage binary data
> > (at the byte or bit level), hopefully without the pain that this kind
> > of thing usually entails :)
> 
> >byte_index = offset / 8
> 
> This will return a float.

The implementation is in Cython, which allows to declare types. 
The variable byte_index is defined as a size_t. 
Did I miss something? Do you mean that an intermediate float 
is used in the generated C code? I guess I should check that.
I realize now that I sometimes use the code above to get the bit
and bytes index and sometimes divmod ...

> 
> Also I notice you count bit positions from the top of each byte, rather than 
> from the bottom. Any reason for this?

I suppose that my mental representation of the bitstream is left-to-right 
(think sequence of bits) with bits being "big-endian"ish in each byte, 
therefore new bits enter from the left with big weights. Which should translate 
to your description. So, no, no specific reason for this I guess. Is one of the 
representation better that the other (wrt performance for example)?.

Cheers,

SB

-- 
https://mail.python.org/mailman/listinfo/python-list

Do not promote `None` as the first argument to `filter` in documentation.

This thought occurred to me several times, but I could not decide to write.
And since `filter` is a builtin, I think this change should be discussed
here, before opening an issue on bug tracker.

I propose to delete all references in the `filter` documentation that the
first argument can be `None`, with possible depreciation of `None` as the
the first argument - FutureWarning in Python 3.8+ and deleting this option
in Python 4. Personally, regarding the last point - depreciation, I do not
see this as a great necessity, but I do not find that the option with `None`
should be offered and suggested through the documentation. Instead, it is
better to show an example with using `filter(bool, iterable)` which is
absolutely
equivalent, more readable, but a little bit slower.

%timeit [*filter(None, range(10))]
503 ns ± 0.259 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit [*filter(bool, range(10))]
512 ns ± 1.09 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)

Currently documentation for `None` case uses `identity function is
assumed`, what is this `identity` and how it is consistent with
truthfulness?

In addition, this change makes the perception of `map` and `filter` more
consistent,with the rule that first argument must be `callable`.

I see only one moment with `None`, since `None` is a keyword, the behavior
of `filter(None, iterable)` is alsways guaranteed, but with `bool` it is
not. Nevertheless, we are all adults here.

With kind regards,
-gdg
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bitstream -- Binary Data for Humans