Re: A technique from a chatbot

[Mark Bourne via Python-list]

Stefan Ram wrote:

Mark Bourne  wrote or quoted:

I don't think there's a tuple being created.  If you mean:
 ( word for word in list_ if word[ 0 ]== 'e' )
...that's not creating a tuple.  It's a generator expression, which
generates the next value each time it's called for.  If you only ever
ask for the first item, it only generates that one.


   Yes, that's also how I understand it!

   In the meantime, I wrote code for a microbenchmark, shown below.

   This code, when executed on my computer, shows that the
   next+generator approach is a bit faster when compared with
   the procedural break approach. But when the order of the two
   approaches is being swapped in the loop, then it is shown to
   be a bit slower. So let's say, it takes about the same time.


There could be some caching going on, meaning whichever is done second 
comes out a bit faster.



   However, I also tested code with an early return (not shown below),
   and this was shown to be faster than both code using break and
   code using next+generator by a factor of about 1.6, even though
   the code with return has the "function call overhead"!


To be honest, that's how I'd probably write it - not because of any 
thought that it might be faster, but just that's it's clearer.  And if 
there's a `do_something_else()` that needs to be called regardless of 
the whether a word was found, split it into two functions:

```
def first_word_beginning_with_e(target, wordlist):
for w in wordlist:
if w.startswith(target):
return w
return ''

def find_word_and_do_something_else(target, wordlist):
result = first_word_beginning_with_e(target, wordlist)
do_something_else()
return result
```


   But please be aware that such results depend on the implementation
   and version of the Python implementation being used for the benchmark
   and also of the details of how exactly the benchmark is written.

import random
import string
import timeit

print( 'The following loop may need a few seconds or minutes, '
'so please bear with me.' )

time_using_break = 0
time_using_next = 0

for repetition in range( 100 ):
 for i in range( 100 ): # Yes, this nesting is redundant!

 list_ = \
 [ ''.join \
   ( random.choices \
 ( string.ascii_lowercase, k=random.randint( 1, 30 )))
   for i in range( random.randint( 0, 50 ))]

 start_time = timeit.default_timer()
 for word in list_:
 if word[ 0 ]== 'e':
 word_using_break = word
 break
 else:
 word_using_break = ''
 time_using_break += timeit.default_timer() - start_time

 start_time = timeit.default_timer()
 word_using_next = \
 next( ( word for word in list_ if word[ 0 ]== 'e' ), '' )
 time_using_next += timeit.default_timer() - start_time

 if word_using_next != word_using_break:
 raise Exception( 'word_using_next != word_using_break' )

print( f'{time_using_break = }' )
print( f'{time_using_next = }' )
print( f'{time_using_next / time_using_break = }' )


--
https://mail.python.org/mailman/listinfo/python-list

Re: A technique from a chatbot

[Mark Bourne via Python-list]

next` (nor anything else) on it, so 
no need for it to be kept hanging around in memory.



Perhaps worse, imagine doing the search in parallel and as sone as it is found 
anywhere, ...



-Original Message-
From: Python-list  On 
Behalf Of Mark Bourne via Python-list
Sent: Thursday, April 4, 2024 3:04 PM
To: python-list@python.org
Subject: Re: A technique from a chatbot

Thomas Passin wrote:

On 4/2/2024 1:47 PM, Piergiorgio Sartor via Python-list wrote:

On 02/04/2024 19.18, Stefan Ram wrote:

Some people can't believe it when I say that chatbots improve
my programming productivity. So, here's a technique I learned
from a chatbot!
It is a structured "break". "Break" still is a kind of jump,
you know?
So, what's a function to return the first word beginning with
an "e" in a given list, like for example
[ 'delta', 'epsilon', 'zeta', 'eta', 'theta' ]

? Well it's
def first_word_beginning_with_e( list_ ):
  for word in list_:
  if word[ 0 ]== 'e': return word

. "return" still can be considered a kind of "goto" statement.
It can lead to errors:

def first_word_beginning_with_e( list_ ):
  for word in list_:
  if word[ 0 ]== 'e': return word
  something_to_be_done_at_the_end_of_this_function()
The call sometimes will not be executed here!
So, "return" is similar to "break" in that regard.
But in Python we can write:
def first_word_beginning_with_e( list_ ):
  return next( ( word for word in list_ if word[ 0 ]== 'e' ), None )


Doesn't look a smart advice.


. No jumps anymore, yet the loop is aborted on the first hit


It's worse than "not a smart advice". This code constructs an
unnecessary tuple, then picks out its first element and returns that.


I don't think there's a tuple being created.  If you mean:
  ( word for word in list_ if word[ 0 ]== 'e' )

...that's not creating a tuple.  It's a generator expression, which
generates the next value each time it's called for.  If you only ever
ask for the first item, it only generates that one.

When I first came across them, I did find it a bit odd that generator
expressions look like the tuple equivalent of list/dictionary
comprehensions.

FWIW, if you actually wanted a tuple from that expression, you'd need to
pass the generator to tuple's constructor:
  tuple(word for word in list_ if word[0] == 'e')
(You don't need to include an extra set of brackets when passing a
generator a the only argument to a function).


--
https://mail.python.org/mailman/listinfo/python-list

Re: A technique from a chatbot

[Mark Bourne via Python-list]

Thomas Passin wrote:

On 4/2/2024 1:47 PM, Piergiorgio Sartor via Python-list wrote:

On 02/04/2024 19.18, Stefan Ram wrote:

   Some people can't believe it when I say that chatbots improve
   my programming productivity. So, here's a technique I learned
   from a chatbot!
   It is a structured "break". "Break" still is a kind of jump,
   you know?
   So, what's a function to return the first word beginning with
   an "e" in a given list, like for example
[ 'delta', 'epsilon', 'zeta', 'eta', 'theta' ]

   ? Well it's
def first_word_beginning_with_e( list_ ):
 for word in list_:
 if word[ 0 ]== 'e': return word

   . "return" still can be considered a kind of "goto" statement.
   It can lead to errors:

def first_word_beginning_with_e( list_ ):
 for word in list_:
 if word[ 0 ]== 'e': return word
 something_to_be_done_at_the_end_of_this_function()
   The call sometimes will not be executed here!
   So, "return" is similar to "break" in that regard.
   But in Python we can write:
def first_word_beginning_with_e( list_ ):
 return next( ( word for word in list_ if word[ 0 ]== 'e' ), None )


Doesn't look a smart advice.


   . No jumps anymore, yet the loop is aborted on the first hit


It's worse than "not a smart advice". This code constructs an 
unnecessary tuple, then picks out its first element and returns that.


I don't think there's a tuple being created.  If you mean:
( word for word in list_ if word[ 0 ]== 'e' )

...that's not creating a tuple.  It's a generator expression, which 
generates the next value each time it's called for.  If you only ever 
ask for the first item, it only generates that one.


When I first came across them, I did find it a bit odd that generator 
expressions look like the tuple equivalent of list/dictionary 
comprehensions.


FWIW, if you actually wanted a tuple from that expression, you'd need to 
pass the generator to tuple's constructor:

tuple(word for word in list_ if word[0] == 'e')
(You don't need to include an extra set of brackets when passing a 
generator a the only argument to a function).


--
Mark.
--
https://mail.python.org/mailman/listinfo/python-list

Re: A missing iterator on itertools module?

[Mark Bourne via Python-list]

Stefan Ram wrote:

ast  wrote or quoted:

Why did you renamed itertools as _itertools ?


   Assume I have a module A.py:

import math
def f(): pass

   . Assume I have an additional module B.py:

import A

   . Now, when I'm editing "B.py" in IDLE and type "A.", IIRC
   IDLE will offer me two possible completions: "A.math" and
   "A.f". The "A.math" makes no sense to me.


`import math` imports the `math` module and binds it to `math` in the 
global namespace of the `A` module.  Since it doesn't have a leading 
underscore, by default it's considered to be a public attribute of the 
`A` module, and IDLE is offering all the public attributes of the `A` 
module for completion.



I want it to go
   away. Therefore, I rewrite A.py as:

import math as _math
def f(): pass

   . Now, Idle will only offer the completion "A.f".

   So, I sometimes use this "import math as _math" style. But then,
   it is simpler for me to /always/ use this style; after all: you
   can't know whether someone eventually will import your module!


You can explicitly declare the public interface of a module by defining 
`__all__`, listing the names which should be considered part of the 
module's public interface; see:

- https://docs.python.org/3/reference/simple_stmts.html#the-import-statement
- https://peps.python.org/pep-0008/#public-and-internal-interfaces

Although `from A import *` is generally discouraged, if `A` defines 
`__all__` then only the names listed in `__all__` are bound in the 
importing module's namespace.  Otherwise, all names from `A` which don't 
have a leading underscore are considered to be public and bound in the 
importing module.


I don't use IDLE, but it may be that it also uses `__all__` to determine 
a module's public API.  In that case, setting `__all__ = ["f"]` in `A` 
should prevent it from offering `math` as a completion (nor any other 
name that's not in the `__all__` list).


--
Mark.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Using a background thread with asyncio/futures with flask

[Mark Bourne via Python-list]

Thomas Nyberg wrote:

Hi,

Yeah so flask does support async (when installed with `pip3 install 
flask[async]), but you are making a good point that flask in this case 
is a distraction. Here's an example using just the standard library that 
exhibits the same issue:


`app.py`
```
import asyncio
import threading
import time
from queue import Queue


in_queue = Queue()
out_queue = Queue()


def worker():
     print("worker started running")
     while True:
     future = in_queue.get()
     print(f"worker got future: {future}")
     time.sleep(5)
     print("worker sleeped")
     out_queue.put(future)


def finalizer():
     print("finalizer started running")
     while True:
     future = out_queue.get()
     print(f"finalizer got future: {future}")
     future.set_result("completed")
     print("finalizer set result")


threading.Thread(target=worker).start()
threading.Thread(target=finalizer).start()


async def main():
     future = asyncio.get_event_loop().create_future()
     in_queue.put(future)
     print(f"main put future: {future}")
     result = await future
     print(result)


if __name__ == "__main__":
     loop = asyncio.get_event_loop()
     loop.run_until_complete(main())
```

If I run that I see the following printed out (after which is just hangs):

```
$ python3 app.py
worker started running
finalizer started running
main put future: 
worker got future: 
worker sleeped
finalizer got future: 
finalizer set result
```

I believe async uses a cooperative multitasking setup under the hood, so 
I presume the way I'm doing this threading just isn't playing well with 
that (and presumably some csp yield isn't happening somewhere). Anyway 
at this point I feel like the easiest approach is to just throw away 
threads entirely and learn how to do all I want fully in the brave new 
async world, but I'm still curious why this is failing and how to make 
this sort of setup work since it points to my not understanding the 
basic implementation/semantics of async in python.


Thanks for any help!

/Thomas

On 3/22/24 08:27, Lars Liedtke via Python-list wrote:

Hey,

As far as I know (might be old news) flask does not support asyncio.

You would have to use a different framework, like e.g. FastAPI or 
similar. Maybe someone has already written "flask with asyncio" but I 
don't know about that.


Cheers

Lars


Lars Liedtke
Lead Developer

[Tel.]  +49 721 98993-
[Fax]   +49 721 98993-
[E-Mail]    l...@solute.de


solute GmbH
Zeppelinstraße 15
76185 Karlsruhe
Germany

[Marken]

Geschäftsführer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten
Webseite | www.solute.de 
Sitz | Registered Office: Karlsruhe
Registergericht | Register Court: Amtsgericht Mannheim
Registernummer | Register No.: HRB 748044
USt-ID | VAT ID: DE234663798



Informationen zum Datenschutz | Information about privacy policy
https://www.solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php 






Am 20.03.24 um 09:22 schrieb Thomas Nyberg via Python-list:

Hello,

I have a simple (and not working) example of what I'm trying to do. 
This is a simplified version of what I'm trying to achieve (obviously 
the background workers and finalizer functions will do more later):


`app.py`

```
import asyncio
import threading
import time
from queue import Queue

from flask import Flask

in_queue = Queue()
out_queue = Queue()


def worker():
    print("worker started running")
    while True:
    future = in_queue.get()
    print(f"worker got future: {future}")
    time.sleep(5)
    print("worker sleeped")
    out_queue.put(future)


def finalizer():
    print("finalizer started running")
    while True:
    future = out_queue.get()
    print(f"finalizer got future: {future}")
    future.set_result("completed")
    print("finalizer set result")


threading.Thread(target=worker, daemon=True).start()
threading.Thread(target=finalizer, daemon=True).start()

app = Flask(__name__)


@app.route("/")
async def root():
    future = asyncio.get_event_loop().create_future()
    in_queue.put(future)
    print(f"root put future: {future}")
    result = await future
    return result


if __name__ == "__main__":
    app.run()
```

If I start up that server, and execute `curl http://localhost:5000`, 
it prints out the following in the server before hanging:


```
$ python3 app.py
worker started running
finalizer started running
* Serving Flask app 'app'
* Debug mode: off
WARNING: This is a development server. Do not use it in a production 
deployment. Use a production WSGI server instead.

* Running on http://127.0.0.1:5000
Press CTRL+C to quit
root put future: 
worker got future: 
worker sleeped
finalizer got future: 
finalizer set result
```

Judging by what's printing out, the `final result = await future` 
doesn't seem to be happy here.


Maybe someone sees something obvious I'm doing wrong here? I presume 
I'm mixing 

Re: Popping key causes dict derived from object to revert to object

[Mark Bourne via Python-list]

Loris Bennett wrote:

Hi,

I am using SQLAlchemy to extract some rows from a table of 'events'.
 From the call to the DB I get a list of objects of the type

   sqlalchemy.orm.state.InstanceState

I would like to print these rows to the terminal using the 'tabulate'
package, the documentation for which says

   The module provides just one function, tabulate, which takes a list of
   lists or another tabular data type as the first argument, and outputs
   a nicely formatted plain-text table

So as I understand it, I need to convert the InstanceState-objects to,
say, dicts, in order to print them.  However I also want to remove one
of the keys from the output and assumed I could just pop it off each
event dict, thus:
  
 event_dicts = [vars(e) for e in events]

 print(type(event_dicts[0]))
 event_dicts = [e.pop('_sa_instance_state', None) for e in event_dicts]
 print(type(event_dicts[0]))


vars() returns the __dict__ attribute of the object.  It may not be a 
good idea to modify that dictionary directly (it will also affect the 
object), although it might be OK if you're not going to do anything else 
with the original objects.  To be safer, you could copy the event objects:

event_dicts = [dict(vars(e)) for e in events]
or:
event_dicts = [vars(e).copy()]


However, this prints

   
   

If I comment out the third line, which pops the unwanted key, I get

   
   

Why does popping one of the keys cause the elements of the list to
revert back to their original class?


As Dieter pointed out, the main problem here is that pop() returns the 
value removed, not the dictionary with the rest of the values.  You 
probably want something more like:

for e in event_dicts:
del e['_sa_instance_state']
(There's not really any point popping the value if you're not going to 
do anything with it - just delete the key from the dictionary)


--
Mark.
--
https://mail.python.org/mailman/listinfo/python-list

Attaching a mock function to another mock breaks reset_mock()

[Mark Bourne via Python-list]

I've came across an issue with attaching a mock function to another mock 
object.  It looks like this might be a bug in unittest.mock, but it's 
possible I'm misunderstanding or doing something wrong.


I'm currently using Python 3.8.10, which is the default installed on 
Ubuntu 20.04.  I don't have time to install and try other versions right 
now but from a quick look at the source at 
, it 
looks like the same issue would still occur with the latest.  I have a 
workaround, but would still like to know if I'm doing something wrong 
which I can correct, or should report this as a bug.


The class I'm testing (let's call it A) is given an instance of another 
class (let's call it B).  In normal operation, some other code adds a 
callback function as an attribute to B before passing it to A, and A 
calls that callback at certain points.  I agree this isn't particularly 
nice; I didn't write the code and don't have time to sort out all the 
structural issues at the moment.


So, in setting up the mock B, I need to attach a mock callback function 
to it.  I'd like the mock function to raise an exception if it's called 
with the wrong arguments.  I can create such a mock function with, for 
example:

```
mock_callback = mock.create_autospec(lambda a, b, c: None)
```
Calls like mock_callback(1,2,3) or mock_callback(a=1,b=2,c=3) are fine, 
and the test can later check the exact values passed in, while 
mock_callback(1,2) or mock_callback(1,2,3,x=9) raise an exception at the 
time of the call - exactly what I want.


However, when I attach this mock callback to the mock instance of B, the 
reset_mock() method breaks.  For example, using object in place of B, 
since the rest of its implementation is irrelevant to the issue:

```
mock_callback = mock.create_autospec(lambda a, b, c: None)
mock_b = mock.create_autospec(object, spec_set=False, instance=True)
mock_b.attach_mock(mock_callback, 'some_callback')
mock_b.reset_mock()
```

The call to reset_mock() results in:
```
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.8/unittest/mock.py", line 603, in reset_mock
child.reset_mock(visited)
TypeError: reset_mock() takes 0 positional arguments but 1 was given
```

This seems to occur because, when creating a mock of a function or 
method, create_autospec calls _setup_func (via _set_signature), which 
sets the mock callback's reset_mock method to a function which doesn't 
accept any arguments.  When mock_b.reset_mock() is called, that 
recursively calls reset_mock() on all the child mocks (including the 
mock callback), passing a number of arguments - which causes the above 
exception.


I thought I might be able to just assign the mock callback to an 
attribute of the mock B, rather than using attach_mock, and avoid the 
recursive call to its reset_mock():

```
mock_b.some_callback = mock_callback
```
But mock_b.reset_mock() still raises that exception.  I think some magic 
in the mocks automatically attaches mock_callback as a child of mock_b 
even in that case.


My current workaround is to just use
```
mock_callback = mock.Mock(spec_set=lambda: None)
```
instead.  While using spec_set there prevents incorrect use of things 
like mock_b.some_callback.random_attribute, it doesn't enforce the 
arguments that the function can be called with, even if I do include 
them in the lambda (as in the first example).


Is there something I'm doing wrong here?  Or does this seem like a bug 
in unittest.mock that I should report?  Perhaps this is something that's 
not often done, so the issue hasn't been noticed before.  Trying to 
search for information generally leads back to the unittest.mock 
documentation, and general tutorials on using create_autospec, 
attach_mock, etc. without anything specific about this case.


--
Mark.

--
https://mail.python.org/mailman/listinfo/python-list

Re: f-string syntax deficiency?

[Mark Bourne via Python-list]

Roel Schroeven wrote:

Op 6/06/2023 om 16:08 schreef Chris Angelico:

On Wed, 7 Jun 2023 at 00:06, Neal Becker  wrote:
>
> The following f-string does not parse and gives syntax error on 3.11.3:
>
> f'thruput/{"user" if opt.return else "cell"} vs. elevation\n'
>
> However this expression, which is similar does parse correctly:
>
> f'thruput/{"user" if True else "cell"} vs. elevation\n'
>
> I don't see any workaround.  Parenthesizing doesn't help:
>  f'thruput/{"user" if (opt.return) else "cell"} vs. elevation\n'
>
> also gives a syntax error

Is this a problem with the f-string, or with the expression
opt.return? That's a keyword.

'return' being a keyowrd is definitely going to be the problem.

Neal, I assume you're using 'opt.return' also outside of that f-string. 
Does that work? How did you manage to do that? I tried to make a simple 
class with an attribute called 'return', but that already fails with a 
syntax error.


Just for fun, here's a class which has any attribute you like, including 
`return`:

```
class AnyAttr:
def __getattr__(self, name):
return f'This is the value of the <{name}> attribute'
```

The usual "dot" notation to access the attributes works for arbitrary 
attributes, but not for `return` because, as mentioned, that's a keyword:

```
>>> aa = AnyAttr()
>>> aa.myattribute
'This is the value of the  attribute'
>>> aa.random
'This is the value of the  attribute'
>>> aa.return
  File "", line 1
aa.return
   ^
SyntaxError: invalid syntax
```

If you really do have an instance with a `return` attribute (perhaps 
because it does fun things with `__getattr__`), you can access it if 
necessary using `getattr`:

```
>>> getattr(aa, 'return')
'This is the value of the  attribute'
```

You can even access attributes with spaces and other usually-invalid 
characters:

```
>>> getattr(aa, 'This really is an attribute name!')
'This is the value of the  attribute'
```

Using `getattr` to access the value, Neal's f-string can be made to work:
```
>>> f'thruput/{"user" if getattr(aa, "return") else "cell"} vs. 
elevation\n'

'thruput/user vs. elevation\n'
```

But if you have control over the implementation of that `opt` object, 
it's probably better to give the attribute a different name which is a 
valid identifier.  PEP-8 suggests a convention of using a single 
trailing underscore (e.g. `return_`) to avoid conflicts with keywords, 
if there's no better name.


Perhaps `opt` is the arguments returned by `argparse.ArgumentParser` and 
you want the command-line option to be `--return`.  In that case, see 
the `dest` argument to `add_argument()` which can specify a different 
name for the attribute used in code (it's almost like they thought about 
this type of problem ;o)).  If it's from `optparse`, that has a similar 
argument, but `optparse` is deprecated so consider updating to `argparse`.




(Recently there has been an effort to provide clearer and more useful 
error messages; this seems to be a case where there is still room for 
improvement: "SyntaxError: invalid syntax" doesn't immediately remind me 
of that fact that 'return' is a keyword and therefor can't be used as an 
attribute.)


--
Mark.
--
https://mail.python.org/mailman/listinfo/python-list