Python told me a Joke

[Alan Bawden via Python-list]

Python 3.10.5 (v3.10.5:f37715, Jul 10 2022, 00:26:17) [GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x,_,z = [1,2,3]

Works as expected.

Now I didn't expect the following to work (but Python sometimes
surprises me!), so I tried:
 
>>> x,2,z = [1,2,3]
  File "", line 1
x,2,z = [1,2,3]
^^^
SyntaxError: invalid syntax. Maybe you meant '==' or ':=' instead of '='?

Yeah, that makes sense, no surprises today...  Except "maybe you meant
'=='..." caught my attention.  _Could_ that be what someone would want
in this situation I wondered?  So I tried:

>>> x,2,z == [1,2,3]
(1, 2, False)

Now that made me laugh.

- Alan

[ Some people reading this will be tempted to explain what's really
  going on here -- it's not hard to understand.  But please remember that
  a joke is never funny if you have to explain it. ]
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to discover what values produced an exception?

[Alan Bawden via Python-list]

Thomas Passin  writes:

   On 5/3/2024 9:56 AM, Johanne Fairchild via Python-list wrote:
   > How to discover what values produced an exception?  Or perhaps---why
   > doesn't the Python traceback show the values involved in the TypeError?
   > For instance:
   >
   > --8<>8---
    (0,0) < 4
   > Traceback (most recent call last):
   >File "", line 1, in 
   > TypeError: '<' not supported between instances of 'tuple' and 'int'
   > --8<>8---
   >
   > It could have said something like:
   >
   > --8<>8---
   > TypeError: '<' not supported between instances of 'tuple' and 'int'
   >in (0,0) < 4.
   > --8<>8---
   >
   > We would know which were the values that caused the problem, which would
   > be very helpful.

   In this example it would not help at all to know the actual values. Knowing
   that you are trying to compare incomparable types is enough.

In general, it absolutely can help.  The programmer can sometimes
recognize where a value of unexpected type came from just by looking at
it, allowing her to quickly deduce just what went wrong without further
investigation.

A good error message shouldn't withhold any information that can
_easily_ be included.  Debugging is more art than science, so there is
no real way to predict what information might prove useful in solving
the crime.  I emphasized "easily" because of course you have to draw the
line somewhere.

The fact that Python error messages often fail to mention the actual
objects that caused the error has always annoyed me.  I've always
presumed that for some reason it just wasn't easy to do.  And it's never
been more than a minor annoyance to me.

So the OP is not wrong for wishing for this.  Other programming
languages do it.  Other Python programmers miss it.

- Alan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Dialogs (Posting On Python-List Prohibited)

[Alan Bawden via Python-list]

Lawrence D'Oliveiro  writes:

   > Assume you have an expression "s.replace('a','b').replace('c','d').
   > replace('e','f').replace('g','h')". Its value is a string which
   > is the value of s, but with "a" replaced by "b", "c" replaced by
   > "d", "e" replaced by "f" and "g" replaced by "h". How to modify
   > this expression, so that "a", "c", "e", and "g", respectively,
   > are replaced only if they are words (but not parts of words)?

   import re

   replacements = (("a", "b"), ("c", "d"), ("e", "f"), ("g", "h"))

   text = "this be a test g eg"

   "".join \
 (
   repl.get(s, s)
   for repl in (dict(replacements),)
   for s in
   re.split("\\b(" + "|".join(re.escape(s[0]) for s in 
replacements) + ")\\b", text)
 )

How about just:

  repl = {
  "a" : "b",
  "c" : "d",
  "e" : "f",
  "g" : "h",
  }

  "".join(repl.get(s, s) for s in re.split(r"\b", text))

- Alan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to implement the ** operator on a custom object

[Alan Bawden via Python-list]

Chris Angelico  writes:

   > On 08Feb2024 12:21, tony.fl...@btinternet.com  
wrote:
   > >I know that mappings by default support the ** operator, to unpack the
   > >mapping into key word arguments.
   > >
   > >Has it been considered implementing a dunder method for the **
   > >operator so you could unpack an object into a key word argument, and
   > >the developer could choose which keywords would be generated (or could
   > >even generate 'virtual' attributes).

   I presume this is more like:

   obj = SomeObject()
   func(**obj)

   ie making the object behave in a dict-like way. I can't remember how
   this is implemented, but you can create the necessary methods to have
   your object produce whatever it likes.

All you need to do is subclass collections.abc.Mapping, and
implement __len__, __iter__, and __getitem__.  Pretty easy.

- Alan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: on writing a number as 2^s * q, where q is odd

[Alan Bawden via Python-list]

jak  writes:

   Oscar Benjamin ha scritto:
   ...
   If we now use the function being discussed:

   powers_of_2_in(n)
   (63, 1)

   we can see that the bit_count() method had to do 63 iterations to count
   the bits

I certainly hope that the bit_count method doesn't count bits by
iterating over them one-by-one.  You can count bits _much_ faster than
that.

You can count the bits in a 62-bit number as follows:

   def bit_count_62(n):
   n = (n - ((n >> 1) & 0o3)
  - ((n >> 2) & 0o1))
   n = (   (n & 0o307070707070707070707)
+ ((n & 0o070707070707070707070) >> 3))
   return n % 63

Then if you want to count the bits in arbitrarily large integers, you
iterate over them in N-bit chunks, for some N <= 62.  Your choice of N
will depend on how you represent your big integers.  Probably N is 56 or
48 or 32.

And why 62 bits?  Because the bit_count method is certainly written in
C, where every step in bit_count_62 would use 64-bit integers.

If you like this sort of stuff, check out the book "Hacker's Delight" by
Henry Warren.  See .

- Alan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: on writing a number as 2^s * q, where q is odd

[Alan Bawden via Python-list]

jak  writes:

   Alan Bawden ha scritto:
   > Julieta Shem  writes:
   >
   > How would you write this procedure?
   > def powers_of_2_in(n):
   > ...
   >
   > def powers_of_2_in(n):
   >  return (n ^ (n - 1)).bit_count() - 1
   >

   Great solution, unfortunately the return value is not a tuple as in the
   OP version. Maybe in this way?

   def powers_of_2_inB(n):
   bc = (n ^ (n - 1)).bit_count() - 1
   return bc, int(n / (1 << bc))

Good point.  I overlooked that.  I should have written:

def powers_of_2_in(n):
bc = (n ^ (n - 1)).bit_count() - 1
return bc, n >> bc
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: on writing a number as 2^s * q, where q is odd

[Alan Bawden via Python-list]

Julieta Shem  writes:

   How would you write this procedure?
   def powers_of_2_in(n):
   ...

def powers_of_2_in(n):
return (n ^ (n - 1)).bit_count() - 1
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Checking if email is valid

[Alan Bawden via Python-list]

Chris Angelico  writes:

   On Thu, 2 Nov 2023 at 15:20, AVI GROSS via Python-list
wrote:

   > Yes, it would be nice if there was a syntax for sending a test
   > message sort of like an ACK that is not delivered to the recipient
   > but merely results in some status being sent back such as
   > DELIVERABLE or NO SUCH USER or even MAILBOX FULL.

   Yes, it would! Spammers would be able to use this syntax to figure out
   exactly which addresses actually have real people connected to it. It
   would save them so much trouble! Brilliant idea.

That sounds like the SMTP "VRFY" command.  And spammers _did_ abuse it
in exactly this manner.  And so pretty much every mail server in the
world disabled VRFY sometime in the 90s.

- Alan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python list insert iterators

[Alan Bawden]

Guenther Sohler  writes:

   Hi Python community,

   I have a got an example list like

   1, 2, 3, 4, 5, 6, 7, 8, 9, 10
T   T

   and i  eventually want to insert items in the given locations
   (A shall go between 2 and 3,  B shall go between 6 and 7)

   Right now i just use index numbers to define the place:

   A shall insert in position 2
   B shall insert in position 6

   However when i insert A in position 2, the index for successful insertion
   of B gets wrong
   (should now be 7 instead of 6)

   No, it's not an option to sort the indexes and start inserting from the
   back.

If you are definitely inserting from the front, then you can use
negative indexes, since the positions relative to the end of the list
won't be changing.

If you must insert in a random order, that won't help, but you haven't
told us what your real constraints are.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Regular Expression bug?

[Alan Bawden]

jose isaias cabrera  writes:

   On Thu, Mar 2, 2023 at 2:38 PM Mats Wichmann  wrote:

   This re is a bit different than the one I am used. So, I am trying to match
   everything after 'pn=':

   import re
   s = "pm=jose pn=2017"
   m0 = r"pn=(.+)"
   r0 = re.compile(m0)
   s0 = r0.match(s)
   >>> print(s0)
   None

Assuming that you were expecting to match "pn=2017", then you probably
don't want the 'match' method.  Read its documentation.  Then read the
documentation for the _other_ methods that a Pattern supports.  Then you
will be enlightened.

- Alan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: random.SystemRandom().randint() inefficient

[Alan Bawden]

Cecil Westerhof  writes:

   Alan Bawden  writes:

   > Cecil Westerhof  writes:
   >
   >Yes, I try to select a random element, but it has also to be removed,
   >because an element should not be used more as once.
   >
   > Instead of using pop to do that why not something like:
   >
   > def lazy_shuffle(seq):
   > """
   > Generate the elements of the given sequence in a random order.
   > """
   > # Delete the next line if you want to use a different version of
   > # randrange:
   > from random import randrange
   > # Delete the next line if SEQ is already a mutable sequence and you
   > # are willing to have it destroyed by this process:
   > seq = list(seq)
   > n = len(seq)
   > while n:
   > i = randrange(n)
   > yield seq[i]
   > n -= 1
   > if i < n:
   > seq[i] = seq[n]

   That looks interesting.
   But there is on problem. (I think.)
   The list is only partly eaten and I will eat a lot of sequences. So a
   lot of sequences will be left in the runtime.
   Or is there a way to destroy the lazy_shuffle when it is not needed
   anymore?

You don't have to worry about that.  That's what Garbage Collection is
for.  After you drop the last reference to the generator, the GC will
destroy it for you.  Welcome to Python.

BTW, what I wrote might be slightly faster if you replace it with:

while n:
i = randrange(n)
yield seq[i]
n -= 1
seq[i] = seq[n]

That will save you some time spent testing and branching.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: random.SystemRandom().randint() inefficient

[Alan Bawden]

Cecil Westerhof  writes:

   Yes, I try to select a random element, but it has also to be removed,
   because an element should not be used more as once.

Instead of using pop to do that why not something like:

def lazy_shuffle(seq):
"""
Generate the elements of the given sequence in a random order.
"""
# Delete the next line if you want to use a different version of
# randrange:
from random import randrange
# Delete the next line if SEQ is already a mutable sequence and you
# are willing to have it destroyed by this process:
seq = list(seq)
n = len(seq)
while n:
i = randrange(n)
yield seq[i]
n -= 1
if i < n:
    seq[i] = seq[n]

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: on a statement followed by an expression

[Alan Bawden]

Meredith Montgomery  writes:

   How can I execute a statement followed by a value in a single line?

   def roberry_via_reduce(rs):
 return my_reduce(rs, lambda d, r: ``increment and return d'', {})

The grammar or Python is deliberately designed so that the body of a
lambda expression cannot contain any statements.  So your original code,
where you defined a separate `count_in' procedure, is probably what you
want.

Although using `reduce' is kind of silly in this case because you aren't
computing a new dictionary every time, but instead you are mutating the
_same_ dictionary every time.  Your original code that used a `for' loop
is actually much clearer.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: min, max with position

[Alan Bawden]

"Michael F. Stemper"  writes:

   Are there similar functions that return not only the minimum
   or maximum value, but also its position?

>>> specialmin(l)
(0,1.618033)
>>> specialmax(l)
3.141593
>>>

I believe that what you are looking for is usually called "argmax" and
"argmin" (see ).  These don't exist in
the standard Python library as far as I can tell, but numpy does have
"argmax" and "argmin" routines.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tail

[Alan Bawden]

Marco Sulla  writes:

   On Mon, 9 May 2022 at 19:53, Chris Angelico  wrote:
   ...
   Nevertheless, tail is a fundamental tool in *nix. It's fast and
   reliable. Also the tail command can't handle different encodings?

It definitely can't.  It works for UTF-8, and all the ASCII compatible
single byte encodings, but feed it a file encoded in UTF-16, and it will
sometimes screw up.  (And if you don't redirect the output away from
your terminal, and your terminal encoding isn't also set to UTF-16, you
will likely find yourself looking at gibberish -- but that's another
problem...)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to check if there is internet?

[Alan Bawden]

And I missed one that was just published last month:

https://datatracker.ietf.org/doc/html/rfc9171

Unlike RFC 5050, this version of the protocol actually claims to be a
"Proposed Standard".

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to check if there is internet?

[Alan Bawden]

Greg Ewing  writes:

   On 8/02/22 8:51 am, Chris Angelico wrote:
   > Some day, we'll have people on Mars. They won't have TCP connections -
   > at least, not unless servers start supporting connection timeouts
   > measured in minutes or hours - but it wouldn't surprise me if some
   > sort of caching proxy system is deployed.

   Or the internet acquires a new protocol that's designed
   for very-long-latency connections.

Very much a hot topic:

https://www.nasa.gov/directorates/heo/scan/engineering/technology/disruption_tolerant_networking_overview
https://en.wikipedia.org/wiki/Delay-tolerant_networking
https://datatracker.ietf.org/doc/html/rfc4838
https://datatracker.ietf.org/doc/html/rfc5050

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Advantages of Default Factory in Dataclasses

[Alan Bawden]

David Lowry-Duda  writes:

   ...

   For the same reason that the following code doesn't do what some people 
   might expect it to:

   ```python
   def add_to(elem, inlist=[]):
   inlist.append(elem)
   return inlist

   list1 = add_to(1)
   list2 = add_to(2)
   print(list1)  # prints [1]
   print(list2)  # prints [1, 2], potentially confusing
   ```

Not only does it not print what "most people" expect.  It also doesn't
print what _you_ expect!  (But you made your point.)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: New assignmens ...

[Alan Bawden]

It seemed weird to me that only an identifier was allowed to be the
target of an assignment expression.  Then it occurred to me that
function definitions are another place where only identifiers are
allowed, but where I could imagine an attributeref or a subscription
being used.  E.g.

  def table[i](x):
  ...

would mean the same thing as:

  def _temp_(x):
  ...
  table[i] = _temp_

I don't immediately see that this would screw up the grammar in any way,
so why not allow it?  A `def` statement is just another form of
assignment, so just as for `:=` expressions, we should allow the target
to be as general as possible!

I'm guessing that about 70% of you will think that this is a horrible
idea, 10% of you will find it compelling, and the remaining 20% will
find themselves conflicted.  You can count me in that last category...

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Strange disassembly

[Alan Bawden]

Chris Angelico  writes:

   >>> sys.version
   '3.10.0b2+ (heads/3.10:33a7a24288, Jun  9 2021, 20:47:39) [GCC 8.3.0]'
   >>> def chk(x):
   ... if not(0 < x < 10): raise Exception
   ...
   >>> dis.dis(chk)
 2   0 LOAD_CONST   1 (0)
 2 LOAD_FAST0 (x)
 4 DUP_TOP
 6 ROT_THREE
 8 COMPARE_OP   0 (<)
10 POP_JUMP_IF_FALSE   11 (to 22)
12 LOAD_CONST   2 (10)
14 COMPARE_OP   0 (<)
16 POP_JUMP_IF_TRUE14 (to 28)
18 LOAD_GLOBAL  0 (Exception)
20 RAISE_VARARGS1
   >>   22 POP_TOP
24 LOAD_GLOBAL  0 (Exception)
26 RAISE_VARARGS1
   >>   28 LOAD_CONST   0 (None)
30 RETURN_VALUE
   >>>

   Why are there two separate bytecode blocks for the "raise Exception"?
   I'd have thought that the double condition would still be evaluated as
   one thing, or at least that the jump destinations for both the
   early-abort and the main evaluation should be the same.

Looks like in 3.8 it compiles more like the way you expected.

I didn't try 3.9, but it looks like a recent change to te compiler.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Alan Bawden]

Sibylle Koczian  writes:

   Am 20.03.2021 um 09:34 schrieb Alan Bawden:
   >
   > When you write that code to capitalize your book titles, you should be
   > calling .title() rather than .upper() if you are doing it right.
   >
   But that's exactly what he's doing, with a result which is documented, but
   not really satisfactory.

Sorry, what I wrote was ambiguous.  What I _meant_ was that when you
replace x.title() with my_title(x) , then in the definition of my_title
you will be calling both .lower() and .title() on individual characters,
but you will probably _never_ be calling .upper().

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Alan Bawden]

The real reason Python strings support a .title() method is surely
because Unicode supports upper, lower, _and_ title case letters, and
tells you how to map between them.  Consider:

   >>> '\u01f1'.upper()
   '\u01f1'

This is the "DZ" character.

   >>> '\u01f1'.lower()
   '\u01f3'

This is the "dz" character.

   >>> '\u01f1'.title()
   '\u01f2'

This is the "Dz" character.

When you write that code to capitalize your book titles, you should be
calling .title() rather than .upper() if you are doing it right.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about generators

[Alan Bawden]

   >>>
   >>> s = []
   >>> s.append(((b, c) for b, c in a))
   >>> s
   [ at 0x019FC3F863C0>]
   >>>

   TIA for any insights.

Replace "append" above with "extend" and observe the results.  Then
ponder the difference between append and extend.  I suspect that the
heart of your confusion actually has nothing to do with generators.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Control stript which is runing in background.

[Alan Bawden]

jak  writes:

   Il 02/01/2021 01:07, Alan Bawden ha scritto:
   > jak  writes:
   >
   > Il 01/01/2021 06:23, Alan Bawden ha scritto:
   > > jak  writes:
   > >
   > > Running the command:
   > >
   > > $ cat bible.txt > cmdpipe & cat bible.txt > cmdpipe & cat 
bible.txt > cmdpipe
   > >
   > > the three texts do not mix
   > >
   > > The three texts do not mix as long at the contents of bible.txt is 
short
   > > enough (and provided `cat' only calls `write' once).  In the POSIX
   > > specification, the manual page for the `write' system call describes
   > > writing to a pipe or FIFO as follows:
   > >
   > >Write requests of {PIPE_BUF} bytes or less shall not be 
interleaved
   > >with data from other processes doing writes on the same pipe.  
Writes
   > >of greater than {PIPE_BUF} bytes may have data interleaved, on
   > >arbitrary boundaries, with writes by other processes, whether or 
not
   > >the O_NONBLOCK flag of the file status flags is set.
   > >
   > ...  Just to make sure I wasn't missing
   > something, I tested your exact command before I sent my previous reply.
   > They mixed.
   >

   This is really strange. On which system did you test? unix, linux or a
   surrogate (cygwin, msys)? I asked this because bible.txt is 4.25MB size
   (https://github.com/mxw/grmr/blob/master/src/finaltests/bible.txt)...
   and the OP needs to send only commands (I hope smaller than the bible).

I tested it on both Linux and FreeBSD.  My test data was a 289M file I
happend to have handy.  I didn't try to see how much smaller that file
could be to still see mixing, but I can tell you that on FreeBSD
PIPE_BUF is 512 bytes and on Linux PIPE_BUF is 4096 bytes.  Recent
versions of POSIX apparently require PIPE_BUF to be at least 512.  (My
older copy of POSIX doesn't mention any lower bound at all!)

So as long as the OP's commands are no longer than 512 bytes, and as
long as they are careful to send commands in a single call to write(),
and as long as the commands don't require a reply, they can get away
with using a pipe/FIFO.  (Or maybe they can know that there will only
ever be one client.)

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Control stript which is runing in background.

[Alan Bawden]

jak  writes:

   Il 01/01/2021 06:23, Alan Bawden ha scritto:
   > jak  writes:
   >
   > Running the command:
   >
   > $ cat bible.txt > cmdpipe & cat bible.txt > cmdpipe & cat bible.txt > 
cmdpipe
   >
   > the three texts do not mix
   >
   > The three texts do not mix as long at the contents of bible.txt is short
   > enough (and provided `cat' only calls `write' once).  In the POSIX
   > specification, the manual page for the `write' system call describes
   > writing to a pipe or FIFO as follows:
   >
   >Write requests of {PIPE_BUF} bytes or less shall not be interleaved
   >with data from other processes doing writes on the same pipe.  Writes
   >of greater than {PIPE_BUF} bytes may have data interleaved, on
   >arbitrary boundaries, with writes by other processes, whether or not
   >the O_NONBLOCK flag of the file status flags is set.
   >
   Ok. And...
...Running the command:

$ cat bible.txt > cmdpipe & cat bible.txt > cmdpipe & cat bible.txt > 
cmdpipe

the three texts do not mix

Saying it again doesn't make it any more true.  If bible.txt is large
enough, they most definitely DO mix!  Just to make sure I wasn't missing
something, I tested your exact command before I sent my previous reply.
They mixed.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Control stript which is runing in background.

[Alan Bawden]

jak  writes:

   Running the command:

   $ cat bible.txt > cmdpipe & cat bible.txt > cmdpipe & cat bible.txt > cmdpipe

   the three texts do not mix

The three texts do not mix as long at the contents of bible.txt is short
enough (and provided `cat' only calls `write' once).  In the POSIX
specification, the manual page for the `write' system call describes
writing to a pipe or FIFO as follows:

  Write requests of {PIPE_BUF} bytes or less shall not be interleaved
  with data from other processes doing writes on the same pipe.  Writes
  of greater than {PIPE_BUF} bytes may have data interleaved, on
  arbitrary boundaries, with writes by other processes, whether or not
  the O_NONBLOCK flag of the file status flags is set.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Class Definitions

[Alan Bawden]

r...@zedat.fu-berlin.de (Stefan Ram) writes:

 I expected this solution:

   class Main:
   def __init__( self ):
   self.value = 0
   def count( self ):
   self.value += 1

 but a student turned in the following solution:

   class Main:
   value = 0
   def count(self):
   self.value += 1

 I thought that the solution of the student had a shortcoming
 but I was not able to put my finger on it. Can you?

Not exactly a shortcoming, but the fact that it works as a solution to
your problem may cause the student to someday write something like:

  class Main:
  value = []
  def add(self, x):
  self.value += [x]

and be suprised by the resulting behavior.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: IP address to binary conversion

[Alan Bawden]

al.alex...@gmail.com writes:

> Just for the records and to have a fully working bidirectional solution:
> 
> >>> ip
> '10.44.32.0'
> >>> struct.unpack('L', socket.inet_aton(ip))[0]
> 2108426
> >>> socket.inet_ntoa(struct.pack(' '10.44.32.0'
> >>>
> 
> Good luck ;-)

This will not work as expected on a big-endian machine, because 'L' means
_native_ byte order, but 'https://mail.python.org/mailman/listinfo/python-list

Re: How to test the data type of a variable

[Alan Bawden]

Deac-33 Lancaster  writes:

> I'm aware that you can find the type of a variable with 
>type(var)

(Strictly speaking, variables don't have types.  This gives you the type of
the variable's current value.  But we know what you meant.)

> But are there Boolean operators in Python3.8 to test the data type, e.g.
>   is_floate(var)
>   is_string(var)
> etc. ?

You should probably be using isinstance(), as in:
  isinstance(var, float)
  isinstance(var, str)

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Python-ideas] Re: Magnitude and ProtoMagnitude ABCs — primarily for argument validation

[Alan Bawden]

Chris Angelico  writes:
> People keep saying this - that nan exists to avoid exceptions - but
> that doesn't take signalling nans into account. Even in C, they are
> expected to raise an exception or equivalent.

Actually, its not that far from the truth to say that NaNs "exist to avoid
exceptions".  According to Kahan, NaNs were an improvement over previous
NaN-like mechanisms that existed in older computer hardware precisely
because they avoid exceptions.  Kahan says of those older mechanisms: "But
nobody used them because they trap when touched."

I believe that signalling NaNs were added to the IEEE standard late in the
process because people still wanted something you could store in a
uninitialized variable that would trap if you accidentally used it.  Unlike
all the IEEE operations that result in a NaN, using an uninitialized
variable is clearly an error.  Kahan doesn't think much of signalling NaNs,
writing that they "exist mainly for political reasons and are rarely used".

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A small quiz

[Alan Bawden]

r...@zedat.fu-berlin.de (Stefan Ram) writes:
...
>   So I was looking for a predefined object from the standard
>   library that already /is/ an iterator (with no need to use
>   »iter«).
> 
>   I found exactly one such object (which can be used after a
>   »from ... import ...« statement). Can you guess what it is?

Well, there's sys.stdin.  

But I would expect beginning students to find the effect of typing
"next(sys.stdin)" to the commnd prompt to be a bit confusing...

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: WedWonder: Scripts and Modules

[Alan Bawden]

DL Neil  writes:

> ... However, reversing the question in my mind led me to ask (myself):
> how many scripts do I have (in "production use") which are ever used
> (also) as a module by some other script? I think the answer is/was:
> "none"! Accordingly, (spoiler alert: this statement may be heresy) I
> stopped using the "if".

I found that the problem with doing this is that `pydoc' now _runs_ my
scripts, when all I wanted was a quick look at the constants and procedures
defined in them.

It's pretty handy sometimes to fire up `pydoc' in http server mode in a
directory full of random Python code (some scripts and some modules), and
use it to browse around to figure out what everything does.  Scripts that
aren't protected with the usual `if __name__' are more likely to crash
in that situation, often crashing `pydoc' in the process.

In fact, I have been known to add `if __name__' to my colleagues' Python
scripts, just so that I can safely `pydoc' them.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: a,b = 2,3 and [a,b] = [2,3]

[Alan Bawden]

Chris Angelico  writes:

> On Mon, Sep 2, 2019 at 12:36 PM Alan Bawden  wrote:
...
> > > > a,b = 2,3 and [a,b] = [2,3]
...
> > It looks to me like they generate identical code.  The first one calls the
> > construction of a tuple, where the second one calls for the construction of
> > a list.  It would be surprising if the compiler optimized the tuple
> > away, but failed to optimize the list away!
> >
> 
> Well, you can find out with the 'dis' module.

Actually, when I wrote "It looks to me", I was in fact looking at the
results of calling `dis.dis'!  But I had unconciously changed what the OP
wrote to make it more realistic.  I tested:

>>> def f(x, y):
  x, y = y, x
  [x, y] = [y, x]
>>> dis.dis(f)
  2   0 LOAD_FAST1 (y)
  3 LOAD_FAST0 (x)
  6 ROT_TWO
  7 STORE_FAST   0 (x)
 10 STORE_FAST   1 (y)

  3  13 LOAD_FAST1 (y)
 16 LOAD_FAST0 (x)
 19 ROT_TWO
 20 STORE_FAST   0 (x)
 23 STORE_FAST   1 (y)
 26 LOAD_CONST   0 (None)
 29 RETURN_VALUE

And I observed that whatever peephole optimization got rid of the tuple
also got rid of the list.  Clearly "BUILD_LIST 2" or "BUILD_TUPLE 2"
followed by "UNPACK_SEQUENCE 2" can be replaced by "ROT_TWO", and the
compiler knew that!  It never occured to me that the case where all the
elements of the sequence to the right of the "=" were constants would
change that, but yes, you are right about that (but read on...):

> >>> def f():
> ... a, b = 2, 3
> ... a, b = [2, 3]
> ...
> >>> dis.dis(f)
>   2   0 LOAD_CONST   1 ((2, 3))
>   2 UNPACK_SEQUENCE  2
>   4 STORE_FAST   0 (a)
>   6 STORE_FAST   1 (b)

And indeed, if they are all constants, then the tuple itself is a constant,
and the obvious peephole optimization is no longer available!

But wait...

>   3   8 LOAD_CONST   2 (2)
>  10 LOAD_CONST   3 (3)
>  12 BUILD_LIST   2
>  14 UNPACK_SEQUENCE  2
>  16 STORE_FAST   0 (a)
>  18 STORE_FAST   1 (b)

Dang!  There is exactly the instruction sequence I just argued could be
optimized away still sitting right there.  So maybe my belief that this is
being done by peephole optimization is in fact incorrect?  So I went and
tried again:

bash-4.2$ python3 -E
Python 3.6.6 (default, Aug 13 2018, 18:24:23) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def f():
... a, b = 2, 3
... a, b = [2, 3]
... a, b = b, a
... a, b = [b, a]
... 
>>> import dis
>>> dis.dis(f)
  2   0 LOAD_CONST   3 ((2, 3))
  2 UNPACK_SEQUENCE  2
  4 STORE_FAST   0 (a)
  6 STORE_FAST   1 (b)

  3   8 LOAD_CONST   1 (2)
 10 LOAD_CONST   2 (3)
 12 ROT_TWO
 14 STORE_FAST   0 (a)
 16 STORE_FAST   1 (b)

  4  18 LOAD_FAST1 (b)
 20 LOAD_FAST0 (a)
 22 ROT_TWO
 24 STORE_FAST   0 (a)
 26 STORE_FAST   1 (b)

  5  28 LOAD_FAST1 (b)
 30 LOAD_FAST0 (a)
 32 ROT_TWO
 34 STORE_FAST   0 (a)
 36 STORE_FAST   1 (b)
 38 LOAD_CONST   0 (None)
 40 RETURN_VALUE

OK, now I'm confused.  How come I'm not seeing the
BUILD_LIST/UNPACK_SEQUENCE sequence that you're seeing?  

> This is with CPython 3.9. It's entirely possible that other Pythons
> and/or other versions of CPython may give different results, but with
> this particular interpreter, the list is not optimized away.

I actually did try this with several different versions of CPython going
back to 2.4 and up to 3.6, and they all behave this way for me.  Maybe
something changed after 3.6?  Weird...

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: a,b = 2,3 and [a,b] = [2,3]

[Alan Bawden]

Eko palypse  writes:

> Am Montag, 2. September 2019 00:49:05 UTC+2 schrieb Hongyi Zhao:
> > Hi,
> > 
> > What's differences:
> > 
> > a,b = 2,3 and [a,b] = [2,3]
> > 
> > Regards
> 
> In this example the result is the same but the second one
> builds, internally, an additional list, therefore isn't as sufficient
> as the first one.

It looks to me like they generate identical code.  The first one calls the
construction of a tuple, where the second one calls for the construction of
a list.  It would be surprising if the compiler optimized the tuple
away, but failed to optimize the list away!

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: An "Object" class?

[Alan Bawden]

Cristian Cocos  writes:

> Thank you! I can see that the taxonomy of built-in classes (i.e. the
> subclass/superclass relations) is not very developed. At the very least I
> would have loved to see stuff such as int declared as a subClass/subType
> of float and the like--that is, a class taxonomy in tune with standard
> mathematical practice, but I am guessing that mathematical kosher-ness had
> to take a back seat to implementational concerns.

Except that numbers of type `int' are _not_ a subset of numbers of
type `float'!  Some ints are much larger that the largest float.

In fact, both `int' and `float' are subclasses of `numbers.Real'.  While it
is true that `numbers.Real' does not appear in the list returned by
`type.mro(int)', nevertheless `issubclass(int, numbers.Real)' and
`isinstance(2, numbers.Real)' are true.  `type.mro' tells you something
about the _implementation_ of `int' and `float' that you _usually_ shouldn't
concern yourself with.  Stick to `isinstance' and `issubclass' and
everthing looks pretty kosher.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What make function with huge list so slow

[Alan Bawden]

Windson Yang  writes:

> 'I'm just running them in succession and seeing how long they'. The full
> code looks like this, this is only an example.py here. and I run 'time
> python3 example.py' for each function.
> 
> def fib_dp(n):
> dp = [0] * (n+1)
> if n <= 1:
> return n
> dp[0], dp[1] = 0, 1
> for i in range(2, n+1):
> dp[i] = dp[i-1] + dp[i-2]
> return dp[-1]
...
> # run the function only once
> # fib_dp(40) # took more than 60s
> # fib_dp2(40) # took about 2s

Figure out how much memory fib_dp is holding on to right before it returns
the answer.  fib(40) is a _big_ number!  And so is fib(39), and
fib(38), and fib(37), etc.  By the time you're done, you're holding
on to quite a huge pile of storage here.  Depending on how much physical
memory you have, you much actually be swapping before you're done.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: My pseudocode to Python?

[Alan Bawden]

Alan Bawden  writes:

> r...@zedat.fu-berlin.de (Stefan Ram) writes:
> > for i in range( len( list )- 1, 0, -1 ):
> > if list[ i ]is None: del list[ i ]
> 
> list = [x for x in list if x is not None]

Except 'list' is a bad name to use...

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: My pseudocode to Python?

[Alan Bawden]

r...@zedat.fu-berlin.de (Stefan Ram) writes:
> for i in range( len( list )- 1, 0, -1 ):
> if list[ i ]is None: del list[ i ]

list = [x for x in list if x is not None]

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: FDs will be closed after exception automatically in python2.7?

[Alan Bawden]

D'Arcy Cain  writes:
> On 2019-06-10 15:46, Alan Bawden wrote:
> > D'Arcy Cain  writes:
> >> with open("file","w+") as fd:
> > 
> > That makes the window smaller, but it doesn't actually eliminate it.  Look
> > at the generated byte code.  In both cases the call to open() is over and
> > the open file is created _before_ the SETUP_WITH instruction is executed.
> 
> Am I correct in assuming that the window is there for process interrupts
> but not threads?

I believe that _either_ a signal handler invocation _or_ a thread switch
might happen immediately before the SETUP_WITH instruction, so there is a
window in either case.

You probably do have less to worry about in the thread switch case,
because eventually the thread will resume where it left off (there being no
way to kill a thread).

In the interrupt case it is possible that a KeyboardInterrupt will kill
your thread right before the SETUP_WITH, and then the stream's __exit__
method will never be called.  But you don't have much to worry about in
that case either because the ONLY reference to the stream is on the top of
the stack, so after the thread is unwound the stream will become garbage,
and then the garbage collector will close it.

But my point was that replacing:

f = open(...)
with f:

with

with open(...) as f:

doesn't fully close _any_ timing windows, it's just clearer to write it
that way.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: FDs will be closed after exception automatically in python2.7?

[Alan Bawden]

D'Arcy Cain  writes:
> On 2019-06-10 10:48, Michael Torrie wrote:
> > fd = open("file","w+")
> > with fd:
> 
> There is still a small window there if there are asynchronous events
> happening.  Use this instead:
> 
> with open("file","w+") as fd:

That makes the window smaller, but it doesn't actually eliminate it.  Look
at the generated byte code.  In both cases the call to open() is over and
the open file is created _before_ the SETUP_WITH instruction is executed.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Testing the data type of a value

[Alan Bawden]

Luuk  writes:
...
> Maybe i should have asked this:
> 
> What is the difference between 'type(5)==int'  and 'isinstance(5,int)'
> 
> and, if there is no difference why did someone invent 'isinstance()' ...

Look:

  Python 3.6.6 (default, Aug 13 2018, 18:24:23)
  [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> isinstance(True, int)
  True
  >>> type(True) == int
  False
  >>> type(True)   
  

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lifetime of a local reference

[Alan Bawden]

r...@zedat.fu-berlin.de (Stefan Ram) writes:

> Alan Bawden  writes:
> >The Java compiler has no way to know whether a variable references an
> >object with a finalize() method that has side effects
> 
>   java.lang.Object#finalize() is deprecated since Java 9. 

And we are advised to use a "Cleaner" or a "PhantomReference" instead.  So
there are still non-deprecated mechanisms in Java 11 you can use to run
cleanup code when an object becomes unreachable.  And the language I quoted
from the Java 8 spec is still there in the Java 11 spec.  So the situation
is unchanged with respect to the point I was making.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lifetime of a local reference

[Alan Bawden]

Gregory Ewing  writes:

> Alan Bawden wrote:
> > the Java Language
> > Specification contains the following language:
> >Optimizing transformations of a program can be designed that reduce
> >the number of objects that are reachable to be less than those which
> >would naively be considered reachable.  For example, a Java compiler
> >or code generator may choose to set a variable or parameter that
> >will no longer be used to null to cause the storage for such an
> >object to be potentially reclaimable sooner.
> 
> However, it only makes sense to do that if the compiler can be
> sure that reclaiming the object can't possibly have any side
> effects. That's certainly not true of things like file objects
> that reference resources outside of the program. I'd be pretty
> upset if a Java implementation prematurely closed my files on
> the basis of this clause.

The Java compiler has no way to know whether a variable references an
object with a finalize() method that has side effects, so that quote from
the Specification licenses a Java implementation to do exactly the thing
you say will make you upset.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lifetime of a local reference

[Alan Bawden]

Marko Rauhamaa  writes:
> def fun():
> f = open("lock")
> flock.flock(f, fcntl.LOCK_EX)
> do_stuff()
> sys.exit(0)
> 
> Question: can a compliant Python implementation close f (and,
> consequently, release the file lock) before/while do_stuff() is
> executed?

A correct-but-fails-to-answer-your-real-question answer would be: "If you
_care_ about when f gets closed, then just call f.close() yourself."  So
if you want to be _sure_ the lock stays locked while you "do stuff", you
should write:

   def fun():
   f = open("lock")
   flock.flock(f, fcntl.LOCK_EX)
   do_stuff()
   f.close()
   sys.exit(0)

And now you don't have to worry about the details of variable lifetimes.

But I appreciate that that isn't the true question that you wanted to ask!
You are wondering if a Python implementation is _permitted_ to treat the
code you wrote _as if_ you had written:

   def fun():
   f = open("lock")
   flock.flock(f, fcntl.LOCK_EX)
   del f
   do_stuff()
   sys.exit(0)

which deletes the variable f from the local environment at a point where it
will never be used again.  (Which could cause the lock to be released
before do_stuff is even called.)

This is an interesting question, and one that any garbage collected
language should probably address somehow.  For example, the Java Language
Specification contains the following language:

   Optimizing transformations of a program can be designed that reduce the
   number of objects that are reachable to be less than those which would
   naively be considered reachable.  For example, a Java compiler or code
   generator may choose to set a variable or parameter that will no longer be
   used to null to cause the storage for such an object to be potentially
   reclaimable sooner.

(This is from section 12.6.1 of the Java 8 version, which is what I had
handy.)

So you're not crazy to ask such a question.

> I couldn't find an immediate answer in the documentation.

I suspect that given the history of Python, pretty much everybody has
always assumed that a Python implementation will not delete local variables
early.  But I agree with you that the Python Language Reference does not
appear to address this question anywhere!

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Improvement Request

[Alan Bawden]

Gilmeh Serda  writes:

> On Fri, 23 Nov 2018 13:44:35 +0530, Sayan Kundu wrote:
> 
> > *** Please make "Table of Contents" an independent scrolling panel*
> > For a long documentation, users have to scroll up all the way to
> > navigate a different sub-topic.
> 
> And just what in the world are you talking about? And more to the point, 
> what does it have to do with the Python language?

He/She is talking about the python documentation pages.  See
<https://docs.python.org/3.8/library/pickle.html> for example -- scroll
down far enough and the table of contents disappears off the top.

It's a reasonable request, but this probably isn't the most effective place
to make it.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Odd truth result with in and ==

[Alan Bawden]

Chris Angelico  writes:

> Or you could say "not [1,2,3] == not True",
> but that's a bit less clear

It's less clear in more ways than one:

   3.6> not [1,2,3] == not True
 File "", line 1
   not [1,2,3] == not True
^
   SyntaxError: invalid syntax
   3.6> not [1,2,3] == (not True)
   True
   3.6> not [] == (not True)
   True
   3.6> (not []) == (not True)
   False

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python indentation (3 spaces)

[Alan Bawden]

Chris Angelico  writes:
> On Mon, Oct 15, 2018 at 9:56 AM Alan Bawden  wrote:
> > In my experience this is a very common way to assume that tabs will be
> > interpreted.  Virtually every source-code file I have encountered since the
> > mid 1970s (for any programming language or operating system) has assumed
> > either this convention or, slightly less often, its 4-column variant.
> >
> > It's surprising that you've never encountered it.
> 
> I've seen the four-column variant, but never the two. Maybe I just
> move in different circles. In any case, I wouldn't say that "two space
> indents, collapsed to a tab after eight" is somehow a universal
> convention any more than four-space with or without collapsing.

I think we're talking past each other somehow here.  I described the common
8-column interpretation, and the less common 4-column interpretation.  I
did not mention, nor do I ever remember encountering in the wild, the
2-column variation.  (I _have_ seen the 10-column variation used -- it was
the standard on Multics.)

The ANSI standard for ASCII only describes the tab character as follows:

   7.21 HT (HORIZONTAL TABULATION). A format effector that causes the
   active position to advance to the next predetermined character position.

This says nothing about where those "predetermined character positions"
happen to be.  They could be set at columns 5, 17, 23, 99 and 100, or they
could be set every N columns for some N, or wherever you want, it's up to
you.  (Note that this is exactly the way mechanical typewriters used to
work: "tab stops" were originally physical sliders that you would position
manually, and then when you pressed the "tab" key, a spring would pull the
carriage holding the paper to the left until it would be physically
_stopped_ by the tab stop.)

Given that the ANSI standard is silent on where the tab stops should be set
for the HT character, we can only ask where they are set _in practice_,
based on looking at he contents of files and the behavior of various pieces
of software.  For example, in the Linux termios(3) manual page describes
one of the options you can set to control output to a terminal:

   TABDLY Horizontal tab delay mask.  Values are TAB0, TAB1, TAB2, TAB3 (or
  XTABS).  A value of TAB3, that is, XTABS, expands tabs to spaces
  (with tab stops every eight columns).

So if tabs are going to be converted to spaces on output, your only choice
is the every-8-columns interpretation.  (Interestingly, setting TABDLY to
TAB3 to convert tabs to spaces is part of POSIX, but the POSIX standard
does _not_ appear to specify the positions of the tab stops!)

The 8-column interpretation is also the default behavior in every text
editor I have ever used (except, as I noted above, on Multics!).

The 8-column interpretation also seems most common in source code files,
although 4-column files do show up occasionally.

So when you say:

   I've literally NEVER come across this as a convention.  Not a single
   file that I have ever worked with has used it.  Where is this convention
   from?

I feel that there _must_ be some misunderstanding here.  Maybe you
misinterpreted Marko's description of how 8-column tabs are used to achieve
2-columns-per-level indentation.  Or maybe I'm misunderstanding what you
mean by "convention".  Or maybe one of us just beamed in from parallel
universe...

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python indentation (3 spaces)

[Alan Bawden]

Chris Angelico  writes:
> On Mon, Oct 15, 2018 at 8:56 AM Marko Rauhamaa  wrote:
> > Chris Angelico :
...
> > That *could* be the situation. However, it is trumped by an older
> > convention whereby the indentation levels go as follows:
> >
> >0:
> >1: SPC SPC
> >2: SPC SPC SPC SPC
> >3: SPC SPC SPC SPC SPC SPC
> >4: TAB
> >5: TAB SPC SPC
> >6: TAB SPC SPC SPC SPC
> >7: TAB SPC SPC SPC SPC SPC SPC
> >8: TAB TAB
> 
> I've literally NEVER come across this as a convention. Not a single
> file that I have ever worked with has used it. Where is this
> convention from?

I just picked a C source code file AT RANDOM from the FreeBSD source tree
on the machine I'm using to compose this message
(/usr/src/gnu/usr.bin/man/man/glob.c), and I find that, as I expected, it
uses exactly that convention.  And in that file, the comments don't line up
as the author clearly intended unless the tab stops are set every 8
columns.

In my experience this is a very common way to assume that tabs will be
interpreted.  Virtually every source-code file I have encountered since the
mid 1970s (for any programming language or operating system) has assumed
either this convention or, slightly less often, its 4-column variant.

It's surprising that you've never encountered it.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quickest way to concatenate strings

[Alan Bawden]

"Frank Millman"  writes:
> I have often read that the quickest way to concatenate a number of strings
> is to place them in a list and 'join' them -
> 
> C:\Users\User>python -m timeit -s "x='a'*500; y='b'*500; z='c'*500"
> ''.join([x, y, z])
...
> 
> I seem to have found a quicker method, using the new 'f' format operator -
> 
> C:\Users\User>python -m timeit -s "x='a'*500; y='b'*500; z='c'*500"
> f'{x}{y}{z}'

If you look at the compiled byte code for the f-string case you'll see why
it's faster: there's nothing used other than opcodes designed specifically
for building strings.

I note that although 'x + y' is faster than f'{x}{y}', 'x + y + z' is
slower than f'{x}{y}{z}'.  Apparently the overhead of making just _one_
temporary intermediate string is enough to tip the balance.  I'm a bit
surprised that it doesn't take a few more than that.  Although I imagine
that that balance might tip at a different point in future releases of
Python (I tested using 3.6.6).

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: List replication operator

[Alan Bawden]

If the behaviour of * on a sequence is confusing, then isn't the following
behaviour also confusing for exactly the same reason?

   >>> a = [[]]
   >>> b = a + a
   >>> b
   [[], []]
   >>> b[0].append(5)
   >>> b
   [[5], [5]]

And if so, shouldn't there also be a concatenation operator that performs
a copy of some kind on its operands?

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Usenet Gateway

[Alan Bawden]

Gene Heskett  writes:

> You are stating an opinion, but no facts to back it up, so describe your 
> environment that makes you write that, please.

If he describes his environment and why he likes it, will that be a
"fact"?  Or will you dismiss that as just another "opinion"?

You asked:

> can someone explain to me why the mailing list (spam free) is not used by
> everybody?

And the fact is that in many people's opinions, "the Usenet/NNTP interface
(with a good newsreader) is so much better!"  That _is_ the answer to your
question.

I'm using Gnus myself, and I find that for a list/group like this, it works
better than any mail client I have ever used.  The threading support is
excellent.  And the scoring system lets me not only filter out the spam,
but it also lets me tune out the people I think are idiots, who would show
up even on the mailing list.

That's just my opinion, of course.  
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Other difference with Perl: Python scripts in a pipe

[Alan Bawden]

Fillmore  writes:

> On 3/10/2016 7:08 PM, INADA Naoki wrote:
...
> I don't like it. It makes Python not so good for command-line utilities
>

You can easily restore the standard Unix command-line-friendly behavior
by doing:

import signal
signal.signal(signal.SIGINT, signal.SIG_DFL)
signal.signal(signal.SIGPIPE, signal.SIG_DFL)

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Make a unique filesystem path, without creating the file

[Alan Bawden]

Cameron Simpson  writes:
> On 22Feb2016 12:34, Alan Bawden  wrote:

I have deleted the part of discussion where it seems that we must simply
agree to disagree.  You think mktemp() is _way_ more dangerous that I
do.

>>> In fact your use case isn't safe, because _another_ task using mktemp
>>> in conflict as a plain old temporary file may grab your fifo.
>>
>>But here in very last sentence I really must disagree.  If the code I
>>wrote above is "unsafe" because some _other_ process might be using
>>mktemp() badly and stumble over the same path, then the current
>>implementation of tempfile.mkdtemp() is also "unsafe" for exactly the
>>same reason: some other process using mktemp() badly to create its own
>>directory might accidentally grab the same directory.
>
> When the other taks goes mkdir with the generated name it will fail, so no.

Quite right.  I sabotaged my own argument by picking mkdtemp() instead
of mkstemp().  I was trying to shorten my text by taking advantage of
the fact that I had _already_ mentioned that mkdtemp() performs exactly
the same generate-and-open loop than the code I had written.  I
apologize for the confusion.

In fact, mkstemp() also performs that same generate-and-open loop, and of
course it is careful to use os.O_EXCL along with os.O_CREAT when it
opens the file.  So let me re-state my argument using mkstemp() instead:

If the code I wrote in my original message is "unsafe" because some
_other_ process might be using mktemp() badly and stumble over the same
path, then the current implementation of tempfile.mkstemp() is also
"unsafe" for exactly the same reason: some other process badly using
mktemp() to create its own file might accidentally grab the same file.

In other words, if that other process does:

  path = mktemp()
  tmpfp = open(path, "w")

Then yes indeed, it might accidentally grab my fifo when I used my
original code for making a temporary fifo.  But it might _also_ succeed
in grabbing any temporary files I make using tempfile.mkstemp()!  So if
you think what I wrote is "unsafe", it seems that you must conclude that
the standard tempfile.mkstemp() is exactly as "unsafe".  So is that what
you think?

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Make a unique filesystem path, without creating the file

[Alan Bawden]

Cameron Simpson  writes:

> On 16Feb2016 19:24, Alan Bawden  wrote:
>>So in the FIFO case, I might write something like the following:
>>
>>def make_temp_fifo(mode=0o600):
>>while True:
>>path = tempfile.mktemp()
>>try:
>>os.mkfifo(path, mode=mode)
>>except FileExistsError:
>>pass
>>else:
>>return path
>>
>>So is there something wrong with the above code?  Other than the fact
>>that the documentation says something scary about mktemp()?
>
> Well, it has a few shortcomings.
>
> It relies on mkfifo reliably failing if the name exists. It shounds like
> mkfifo is reliable this way, but I can imagine analogous use cases without
> such a convenient core action, and your code only avoids mktemp's security
> issue _because_ mkfifo has that fortuitous aspect.

I don't understand your use of the word "fortuitous" here.  mkfifo is
defined to act that way according to POSIX.  I wrote the code that way
precisely because of that property.  I sometimes write code knowing that
adding two even numbers together results in an even answer.  I suppose
you might describe that as "fortuitous", but it's just things behaving
as they are defined to behave!

> Secondly, why is your example better than::
>
>  os.mkfifo(os.path.join(mkdtemp(), 'myfifo'))

My way is not much better, but I think it is a little better because
your way I have to worry about deleting both the file and the directory
when I am done, and I have to get the permissions right on two
filesystem objects.  (If I can use a TemporaryDirectory() context
manager, the cleaning up part does get easier.)

And it also seems wasteful to me, given that the way mkdtemp() is
implemented is to generate a possible name, try creating it, and loop if
the mkdir() call fails.  (POSIX makes the same guarantee for mkdir() as
it does for mkfifo().)  Why not just let me do an equivalent loop
myself?

> On that basis, this example doesn't present a use case what can't be
> addressed by mkstemp or mkdtemp.

Yes, if mktemp() were taken away from me, I could work around it.  I'm
just saying that in order to justify taking something like this away, it
has to be both below some threshold of utility and above some threshold
of dangerousness.  In the canonical case of gets() in C, not only is
fgets() almost a perfectly exact replacement for gets(), gets() is
insanely dangerous.  But the case of mktemp() doesn't seem to me to come
close to this combination of redundancy and danger.

> You _do_ understand the security issue, yes? I sure looked like you did,
> until here.

Well, it's always dangerous to say that you understand all the security
issues of anything.  In part that is why I wrote the code quoted above.
I am open to the possibility that there is a security problem here that
I haven't thought of.  But so far the only problem anybody has with it
is that you think there is something "fortuitous" about the way that it
works.

>>(As if that would be of any use in the
>>situation above!)  It looks like anxiety that some people might use
>>mktemp() in a stupid way has caused an over-reaction.
>
> No, it is anxiety that mktemp's _normal_ use is inherently unsafe.

So are you saying that the way I used mktemp() above is _abnormal_?

> [ Here I have removed some perfectly reasonable text describing the
>   race condition in question -- yes I really do understand that. ]
>
> This is neither weird nor even unlikely which is why kmtemp is strongly
> discouraged - naive (and standard) use is not safe.
>
> That you have contrived a use case where you can _carefully_ use mktemp in
> safety in no way makes mktemp recommendable.

OK, so you _do_ seem to be saying that I have used mktemp() in a
"contrived" and "non-standard" (and "non-naive"!) way.  I'm genuinely
surprised.  I though I was just writing straightforward correct code and
demonstrating that this was a useful utility that it was not hard to use
safely.  You seem to think what I did is something that ordinary
programmers can not be expected to do.  Your judgement is definitely
different from mine!

And ultimately this does all boil down to making judgements.  It does
make sense to remove things from libraries that are safety hazards (like
gets() in C), I'm just trying to argue that mktemp() isn't nearly
dangerous enough to deserve more than a warning in its documentation.
You don't agree.  Oh well...

Up until this point, you haven't said anything that I actually think is
flat out wrong, we just disagree about what tools it is reasonable to
take away from _all_ programmers just because _some_ programmers might
use them t

Re: Make a unique filesystem path, without creating the file

[Alan Bawden]

Ben Finney  writes:

> Cameron Simpson  writes:
>
>> I've been watching this for a few days, and am struggling to
>> understand your use case.
>
> Yes, you're not alone. This surprises me, which is why I'm persisting.
>
>> Can you elaborate with a concrete example and its purpose which would
>> work with a mktemp-ish official function?
>
> An example::

Let me present another example that might strike some as more
straightforward.

If I want to create a temporary file, I can call mkstemp().  
If I want to create a temporary directory, I can call mkdtemp().

Suppose that instead of a file or a directory, I want a FIFO or a
socket.

A FIFO is created by passing a pathname to os.mkfifo().  A socket is
created by passing a pathname to an AF_UNIX socket's bind() method.  In
both cases, the pathname must not name anything yet (not even a symbolic
link), otherwise the call will fail.

So in the FIFO case, I might write something like the following:

def make_temp_fifo(mode=0o600):
while True:
path = tempfile.mktemp()
try:
os.mkfifo(path, mode=mode)
except FileExistsError:
pass
else:
return path

mktemp() is convenient here, because I don't have to worry about whether
I should be using "/tmp" or "/var/tmp" or "c:\temp", or whether the
TMPDIR environment variable is set, or whether I have permission to
create entries in those directories.  It just gives me a pathname
without making me think about the rest of that stuff.  Yes, I have to
defend against the possibility that somebody else creates something with
the same name first, but as you can see, I did that, and it wasn't
rocket science.

So is there something wrong with the above code?  Other than the fact
that the documentation says something scary about mktemp()?

It looks to me like mktemp() provides some real utility, packaged up in
a way that is orthogonal to the type of file system entry I want to
create, the permissions I want to give to that entry, and the mode I
want use to open it.  It looks like a useful, albeit low-level,
primitive that it is perfectly reasonable for the tempfile module to
supply.

And yet the documentation condemns it as "deprecated", and tells me I
should use mkstemp() instead.  (As if that would be of any use in the
situation above!)  It looks like anxiety that some people might use
mktemp() in a stupid way has caused an over-reaction.  Let the
documentation warn about the problem and point to prepackaged solutions
in the common cases of making files and directories, but I see no good
reason to deprecate this useful utility.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What is a function parameter =[] for?

[Alan Bawden]

Chris Angelico  writes:

> On Thu, Nov 26, 2015 at 1:08 PM, Alan Bawden  wrote:
>> (Note that nothing in the documentation I can find actually _guarantees_
>> that a Python implementation will only have one unique empty tuple, but
>> I wouldn't be suprised if the following is nonetheless true in all
>> current implementations:
>>
>>>>> tuple([]) is tuple([])
>>True
>>
>> )
>
> Jython 2.5.3 (, Oct 8 2014, 03:39:09)
> [OpenJDK 64-Bit Server VM (Oracle Corporation)] on java1.7.0_85
> Type "help", "copyright", "credits" or "license" for more information.
>>>> tuple([]) is tuple([])
> False
>
> Python 2.7.8 (2.4.0+dfsg-3, Dec 20 2014, 13:30:46)
> [PyPy 2.4.0 with GCC 4.9.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>>> tuple([]) is tuple([])
> False
>
> ChrisA

I said I wouldn't be suprised if it was always true, but that doesn't
imply that I need to be suprised if it is sometimes false!

Having said that, I _am_ curious whether that remains False for various
other variant expressions.  "tuple([])" is actually a rather perverse
way to obtain an empty tuple.  How about plain "() is ()"?  While
"tuple([]) is tuple([])" is False is some implementations, it is still
possible that "() is ()" is True in those same implementations,
depending on how that compiles.

An even more perverse way to obtain an empty tuple is "tuple(x for x in ())"
and in fact in CPython:

   >>> tuple(x for x in ()) is tuple(x for x in ())
   False

If I had thought to try that earlier, I wouldn't have bothered making
the parenthetical remark to are responding to -- because none of this
has any relevance to my original point about "BUILD_TUPLE 0".

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What is a function parameter =[] for?

[Alan Bawden]

Steven D'Aprano  writes:
> In fact, it's easy to find cases even now where the compiler is
> insufficiently smart to recognise all the possible optimizations available.
> There's no tuple simpler than the empty tuple, but Python 3.3 at least
> fails to optimize that case:
>
> py> dis(compile("()", '', 'eval'))
>   1   0 BUILD_TUPLE  0
>   3 RETURN_VALUE

Actually, it would be silly to compile that as a LOAD_CONST of an empty
tuple.  "BUILD_TUPLE 0" doesn't allocate storage, it returns the same
empty tuple every time!  In effect "BUILD_TUPLE 0" is just a global
load.  A different disassembler might decide to print "BUILD_TUPLE 0" as
"LOAD_EMPTY_TUPLE".

Using LOAD_CONST would just waste space in the function's vector
of constants.

(Note that nothing in the documentation I can find actually _guarantees_
that a Python implementation will only have one unique empty tuple, but
I wouldn't be suprised if the following is nonetheless true in all
current implementations:

   >>> tuple([]) is tuple([])
   True

)

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to "wow" someone new to Python

[Alan Bawden]

Alan Bawden  writes:
> ...  Score one for untyped languages.

Drat.  I should have writted "dynamically typed languages".

The language has changed.  When I was a novice Lisp hacker, we were
comfortable saying that Lisp was "untyped".  But nowadays we always say
that Lisp is "dynamically typed".  I could write an essay about why...

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to "wow" someone new to Python

[Alan Bawden]

Chris Angelico  writes:
> ..., and I would guess a 64-bit Java would
> also raise the limit.

Even in a 64-bit Java, the _type_ returned by String.length() is 'int',
and is thus at most (2**31 - 1).  This isn't a problem for strings,
which never get that long in practice, but for some other Java datatypes
(e.g., Buffer) it is a real problem.  Score one for untyped languages.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .write() behavior

[Alan Bawden]

Marko Rauhamaa  writes:

> Let me mention a related problem I ran into a couple of weeks ago.
> Linux's standard C library (glibc) implements fread(3) differently in
> RHEL 5 and RHEL 6/7. In RHEL 5, it blocks in a loop until it has read in
> the desired number of records. In RHEL 6 and 7, it appears to block
> until it can return at least one whole record.

By my reading of the POSIX standard, that latter behavior would be
incorrect.  The "RETURN VALUE" section of the POSIX description for
fread() starts:

   Upon successful completion, fread() shall return the number of
   elements successfully read which is less than nitems only if a read
   error or end-of-file is encountered. ...

If in whatever version of glibc comes with RHEL 6/7, it is the case that
a call to fread() can return less than the desired number of records,
and you have not reached EOF, and no read error has occurred, that would
be a serious bug.  The world is full of C programs that call fread() and
fwrite() without wrapping them in defensive
maybe-I-didn't-get-everything loops.

>> Am I missing something? There seem to be some primitive IO facilities
>> in Python 3 that make a distinction between blocking and non-blocking
>> mode, but that distinction doesn't seem to be available when I just
>> call open().

Since I wrote this, I have discovered and read PEP-3116, and using that
as a road-map to understanding the Python 3 documentation it is now
clear to me that in Python 3, if you have not gone out of your way to
enable non-blocking, a call to the .write() method on any object that
might normally be returned by a call to open(), will always write _all_
of your data (or raise an exception).  There is never any need for a
defensive loop.

As further evidence that this is the case, note that PEP-3116 says right
at the end of its "Rationale and Goals" section:

   ...  Programmers who don't want to muck about in the new I/O world
   can expect that the open() factory method will produce an object
   backwards-compatible with old-style file objects.

Since old-style file objects didn't return a count, this goal can only
be achieved if programmers are safe in _ignoring_ that count in Python 3.

I am relieved to discover that when I am finally able to port my Python
code from Python 2 to Python 3, I will not have to worry that .write()
might perform an incomplete write where it previously did not.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

.write() behavior

[Alan Bawden]

Marko Rauhamaa  writes:

> Marko Rauhamaa :
> Actually, that's mistaken as well. The sys.std* handles and pipes
> returned by subprocess are accessed using file.write() and thus may
> return partial writes.

I find this very surprising.  In Python 2, where file.write() seems to
always return None, are you telling me that a call to file.write() might
silently return after only writing _some_ of my output?  Without raising
an exception or returning any other indication of failure?  How am I
supposed to defend myself against that?

You might be right, because nothing in the Python 2 documentation I can
find _explicitly_ says that file.write() is guaranteed to write
everything I told it to, but that seems like a sufficiently surprising
fact that I would expect the documentation to emphasize the danger.

In Python 3 the situation is more puzzling: The documentation for open()
explains that the type of object returned depends on the mode argument,
and several possible candidates for the class of the file object
returned are mentioned.  Some of those classes document a .write()
method that may indeed perform a partial write and return a count of how
far it got.  Other classes don't say that they might do partial writes,
but neither do they say that they don't do partial writes.  It
seems possible that the _intent_ is that text mode opens return a file
object that guarantees to always do a full write.  But maybe not.

OK, so maybe the situation is that in Python 2 file.write() is
guaranteed to never do a partial write and that in Python 3 there are no
such guarantees.  That seems like a big change to watch out for when
migrating my code from 2 to 3.  But I can't seem to find any warnings
about that in the Python 3 documentation.

Just to be clear here, are you saying that in Python 3, I have to start
using a utility like:

  def guaranteed_write(file, data):
  while data:
  n = file.write(data)
  if n < len(data):
  data = data[n:]
  else:
  return

for any file object returned by open()?  Or maybe just for file objects
opened in "binary mode"?

Am I missing something?  There seem to be some primitive IO facilities
in Python 3 that make a distinction between blocking and non-blocking
mode, but that distinction doesn't seem to be available when I just call
open().

Maybe there is some design document for Python 3 IO that I should read
that would explain the rationale for all this?

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposal: === and !=== operators

[Alan Bawden]

Steven D'Aprano  writes:

> But perhaps we only care about changes in value, not type. NAN or no NAN, 
> list equality works fine:
>
> py> data = [1.0, 2.0, float('nan'), 4.0]
> py> old = data[:]
> py> old == data  # No changes made yet, should return True
> True

You lost me right here.  If list equality is determined by comparing
lists element-by-element, and the second element of old is _not_ equal
to the second element of data, then why should old and data be equal?

In fact, I find myself puzzled about exactly how list equality is
actually defined.  Consider:

  >>> a = float('nan')
  >>> x = [1, a, 9]
  >>> y = [1, a, 9.0]
  >>> x == y
  True

So is there some equality predicate where corresponding elements of x
and y are equal?  

  >>> map(operator.eq, x, y)
  [True, False, True]

It's not "==".

  >>> map(operator.is_, x, y)
  [True, True, False]

And it's not "is".

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Explanation of list reference

[Alan Bawden]

Marko Rauhamaa  writes:

> Case in point, if everything is a reference, how come:
>
>>>> "hello".__str__()
>'hello'
>>>> 1.__str__()
>SyntaxError: invalid syntax

   >>> (1).__str__()
   '1'

   >>> 1..__str__()
   '1.0'

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: interactive help on the base object

[Alan Bawden]

Chris Angelico  writes:
> How does that work, exactly? How do you have a class inherit
> (ultimately) from itself, and how does that impact the component class
> list?

How does it work "exactly"?  You're asking me about a feature I never
made use of, in a system I have no source for, and that I haven't used
in over 25 years!  If it wasn't mentioned in a parenthetical comment in
the 32-year-old documentation I still have on my bookshelf (Lisp Machine
Manual, 4th edition, July 1981, blue cover), I wouldn't trust my own
memory that such a thing ever even existed.

So you're not getting anything "exact" here!

I have no idea exactly how it worked, but imagine something that walked
the superclass graph _as_ _if_ it was a tree, collected classes in some
order, and that just skips any class it encounters for the second time.
That results in _some_ linear ordering of all the classes you can reach
from the starting class.  Now use that.

Now the results aren't going to be very good by modern standards.  As
the Common Lisp Object System, Dylan, Python, and others, have all
discovered, you really want something that is at _least_ a topological
sort of the superclass graph -- and there is no topological sort at all
unless your superclass graph is acyclic.

But even if the results aren't up to modern standards, you can write a
hell of a lot of code before the deficiencies have hurt enough people
enough times to motivate you to do better.  After all modern Python
classes didn't start using their current ordering until Python 2.3.

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: interactive help on the base object

[Alan Bawden]

Steven D'Aprano  writes:

> - If all classes are part of a single hierarchy, it must logically end at 
> one (or more, if you support multiple inheritance, which Python does) 
> bases classes. (Unless there are loops, which are generally prohibited in 
> all OOP systems I know of). The simplest way to do this is with a single 
> base class.

The original Lisp Machine "Flavors" system (one of the ancestors of the
Common Lisp Object System) allowed inheritance to be an arbitrary
directed graph -- possibly with cycles.

I don't believe that this was done for any deep principled reason, but
rather it was just permitted because the algorithm for computing method
resolution order didn't actually care whether there were inheritance
cycles -- it still terminated and returned an ordered list of component
classes.

I do not remember seeing any code that made use of this ability, so
don't ask me if it was good for anything, but it was definitely there...

-- 
Alan Bawden
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: 10 sec poll - please reply!

[Alan Bawden]

Since the event being generated is commonly called a "keystroke", and since
my dictionary defines the noun "stroke" as being: "the act of striking", a
good verb to choose for the action itself would seem to be "strike":

strike('a')
-- 
http://mail.python.org/mailman/listinfo/python-list