Re: From JoyceUlysses.txt -- words occurring exactly once
On Sat, Jun 8, 2024 at 10:39 AM Mats Wichmann via Python-list < python-list@python.org> wrote: > On 6/5/24 05:10, Thomas Passin via Python-list wrote: > > > Of course, we see this lack of clarity all the time in questions to the > > list. I often wonder how these askers can possibly come up with > > acceptable code if they don't realize they don't truly know what it's > > supposed to do. > > Fortunately, having to explain to someone else why something is giving > you trouble can help shed light on the fact the problem statement isn't > clear, or isn't clearly understood. Sometimes (sadly, many times it > doesn't). The original question struck me as homework or an interview question for a junior position. But having no clear requirements or specifications is good training for the real world where that is often the case. When you question that, you are told to just do something, and then you’re told it’s not what is wanted. That frustrates people but it’s often part of the process. People need to see something to help them know what they really want. > -- https://mail.python.org/mailman/listinfo/python-list
Re: Serializing pydantic enums
On Wed, May 29, 2024 at 12:27 PM Larry Martell wrote: > > On Tue, May 28, 2024 at 11:46 AM Left Right via Python-list > wrote: > > > > Most Python objects aren't serializable into JSON. Pydantic isn't > > special in this sense. > > > > What can you do about this? -- Well, if this is a one-of situation, > > then, maybe just do it by hand? > > > > If this is a recurring problem: json.dumps() takes a cls argument that > > will be used to do the serialization. Extend json.JSONEncoder and > > implement the encode() method for the encoder class you are passing. I > > believe that the official docs have some information about this too. > > Yeah, I know I can do this, but I seem to recall reading that pydantic > handled serialization. Guess not. Actually it's as simple as adding this to any model that uses an enum model: class Config: use_enum_values = True > > On Tue, May 28, 2024 at 2:50 PM Larry Martell via Python-list > > wrote: > > > > > > Just getting started with pydantic. I have this example code: > > > > > > class FinishReason(Enum): > > > stop = 'stop' > > > > > > class Choice(BaseModel): > > > finish_reason: FinishReason = Field(...) > > > > > > > > > But I cannot serialize this: > > > > > > json.dumps(Choice(finish_reason=FinishReason.stop).dict()) > > > *** TypeError: Object of type FinishReason is not JSON serializable > > > > > > > > > I get the object not the value: > > > > > > (Pdb) Choice(finish_reason=FinishReason.stop) > > > Choice(finish_reason=) > > > > > > > > > Also tried it with .value, same result. > > > > > > What am I missing here? -- https://mail.python.org/mailman/listinfo/python-list
Re: Serializing pydantic enums
On Tue, May 28, 2024 at 11:46 AM Left Right via Python-list wrote: > > Most Python objects aren't serializable into JSON. Pydantic isn't > special in this sense. > > What can you do about this? -- Well, if this is a one-of situation, > then, maybe just do it by hand? > > If this is a recurring problem: json.dumps() takes a cls argument that > will be used to do the serialization. Extend json.JSONEncoder and > implement the encode() method for the encoder class you are passing. I > believe that the official docs have some information about this too. Yeah, I know I can do this, but I seem to recall reading that pydantic handled serialization. Guess not. > On Tue, May 28, 2024 at 2:50 PM Larry Martell via Python-list > wrote: > > > > Just getting started with pydantic. I have this example code: > > > > class FinishReason(Enum): > > stop = 'stop' > > > > class Choice(BaseModel): > > finish_reason: FinishReason = Field(...) > > > > > > But I cannot serialize this: > > > > json.dumps(Choice(finish_reason=FinishReason.stop).dict()) > > *** TypeError: Object of type FinishReason is not JSON serializable > > > > > > I get the object not the value: > > > > (Pdb) Choice(finish_reason=FinishReason.stop) > > Choice(finish_reason=) > > > > > > Also tried it with .value, same result. > > > > What am I missing here? -- https://mail.python.org/mailman/listinfo/python-list
Serializing pydantic enums
Just getting started with pydantic. I have this example code: class FinishReason(Enum): stop = 'stop' class Choice(BaseModel): finish_reason: FinishReason = Field(...) But I cannot serialize this: json.dumps(Choice(finish_reason=FinishReason.stop).dict()) *** TypeError: Object of type FinishReason is not JSON serializable I get the object not the value: (Pdb) Choice(finish_reason=FinishReason.stop) Choice(finish_reason=) Also tried it with .value, same result. What am I missing here? -- https://mail.python.org/mailman/listinfo/python-list
Re: PyCon
LOn Fri, May 17, 2024 at 8:57 PM Larry Martell wrote: > I’m at PyCon in Pittsburgh and I’m haven’t an amazing time! s/haven’t/having/ -- https://mail.python.org/mailman/listinfo/python-list
PyCon
I’m at PyCon in Pittsburgh and I’m haven’t an amazing time! -- https://mail.python.org/mailman/listinfo/python-list
Re: Version of NymPy
On Wed, May 15, 2024 at 2:43 PM Popov, Dmitry Yu via Python-list wrote: > > What would be the easiest way to learn which version of NumPy I have with my > Anaconda distribution? >>> import numpy >>> numpy.__version__ '1.24.4' -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract lines from file, add to new files
On Tue, Jan 30, 2024 at 1:13 AM AVI GROSS via Python-list wrote: > > It can be quite frustrating figuring out what someone wants, Grant, > especially when they just change it. > > It is worse when instead of starting a new thread with an appropriate > subject line, it continues and old one that was also frustrating to > understand. Is it worse than top posting? -- https://mail.python.org/mailman/listinfo/python-list
Re: Running a subprocess in a venv
On Sat, Oct 21, 2023 at 12:10 PM Johannes Findeisen wrote: > > On Sat, 21 Oct 2023 11:32:03 -0400 > Larry Martell wrote: > > > On Sat, Oct 21, 2023 at 9:49 AM Johannes Findeisen > > wrote: > > > > > > On Sat, 21 Oct 2023 09:01:18 -0400 > > > Larry Martell via Python-list wrote: > > > > > > > I have a python script, and from that I want to run another > > > > script in a subprocess in a venv. What is the best way to do > > > > that? I could write a file that activates the venv then runs the > > > > script, then run that file, but that seems messy. Is there a > > > > better way? > > > > > > How do you do that? > > > > How? Open a file and write the commands I need then invoke that. > > > > > It sounds messy but not wrong... > > > > > > I would activate the venv and then run my Python script. In the > > > Python script you can call another python script in a subprocess > > > like this: > > > > > > import sys > > > import subprocess > > > > > > # > > > https://docs.python.org/3/library/subprocess.html#popen-constructor > > > proc = subprocess.Popen([sys.executable, > > > "/path/to/an/otherscript.py"]) > > > > > > # https://docs.python.org/3/library/subprocess.html#popen-objects > > > # Do your process communication/handling... proc.communicate(), > > > # proc.wait(), proc.terminate(), proc.kill() etc. > > > > > > Is this the answer you are looking for? > > > > > > Detailed docs: https://docs.python.org/3/library/subprocess.html > > > > I know how to use Popen. What I was missing was running the script > > using sys.executable. Thanks. > > sys.executable is the path to the actual Python binary, e.g. > "/usr/bin/python". You could add "/usr/bin/python" there manually but > this is not portable to Windows for example. > > When you add a shebang line to your other script and the file is > executable, you may not need to add sys.executable as first argument to > Popen but using sys.executable is the most reliable way to do this... ;) I need the path to whichever venv is being used so sys.executable works for me. -- https://mail.python.org/mailman/listinfo/python-list
Re: Running a subprocess in a venv
On Sat, Oct 21, 2023 at 9:49 AM Johannes Findeisen wrote: > > On Sat, 21 Oct 2023 09:01:18 -0400 > Larry Martell via Python-list wrote: > > > I have a python script, and from that I want to run another script in > > a subprocess in a venv. What is the best way to do that? I could write > > a file that activates the venv then runs the script, then run that > > file, but that seems messy. Is there a better way? > > How do you do that? How? Open a file and write the commands I need then invoke that. > It sounds messy but not wrong... > > I would activate the venv and then run my Python script. In the Python > script you can call another python script in a subprocess like this: > > import sys > import subprocess > > # https://docs.python.org/3/library/subprocess.html#popen-constructor > proc = subprocess.Popen([sys.executable, "/path/to/an/otherscript.py"]) > > # https://docs.python.org/3/library/subprocess.html#popen-objects > # Do your process communication/handling... proc.communicate(), > # proc.wait(), proc.terminate(), proc.kill() etc. > > Is this the answer you are looking for? > > Detailed docs: https://docs.python.org/3/library/subprocess.html I know how to use Popen. What I was missing was running the script using sys.executable. Thanks. -- https://mail.python.org/mailman/listinfo/python-list
Running a subprocess in a venv
I have a python script, and from that I want to run another script in a subprocess in a venv. What is the best way to do that? I could write a file that activates the venv then runs the script, then run that file, but that seems messy. Is there a better way? -- https://mail.python.org/mailman/listinfo/python-list
Re: path to python in venv
On Wed, Sep 27, 2023 at 12:53 PM Niktar Lirik wrote: > > Hi Larry, > > You could just create venv with option '—copies' > > > > For example: > > python -m venv -–copies .venv Thanks! That is just what I was looking for. > From: Larry Martell via Python-list > Sent: 27 сентября 2023 г. 22:48 > To: Jon Ribbens > Cc: python-list@python.org > Subject: Re: path to python in venv > > > > On Wed, Sep 27, 2023 at 12:42 PM Jon Ribbens via Python-list > > wrote: > > > > > > On 2023-09-27, Larry Martell wrote: > > > > I was under the impression that in a venv the python used would be in > > > > the venv's bin dir. But in my venvs I see this in the bin dirs: > > > > > > > > lrwxrwxrwx 1 larrymartell larrymartell7 Sep 27 11:21 python -> python3 > > > > lrwxrwxrwx 1 larrymartell larrymartell 16 Sep 27 11:21 python3 -> > > > > /usr/bin/python3 > > > ... > > > > Not sure what this really means, nor how to get python to be in my venv. > > > > > > WHy do you want python to be "in your venv"? > > > > Isn't that the entire point of a venv? To have a completely self > > contained env? So if someone messes with the system python it will not > > break code running in the venv. > > -- > > https://mail.python.org/mailman/listinfo/python-list > > -- https://mail.python.org/mailman/listinfo/python-list
Re: path to python in venv
On Wed, Sep 27, 2023 at 12:42 PM Jon Ribbens via Python-list wrote: > > On 2023-09-27, Larry Martell wrote: > > I was under the impression that in a venv the python used would be in > > the venv's bin dir. But in my venvs I see this in the bin dirs: > > > > lrwxrwxrwx 1 larrymartell larrymartell7 Sep 27 11:21 python -> python3 > > lrwxrwxrwx 1 larrymartell larrymartell 16 Sep 27 11:21 python3 -> > > /usr/bin/python3 > ... > > Not sure what this really means, nor how to get python to be in my venv. > > WHy do you want python to be "in your venv"? Isn't that the entire point of a venv? To have a completely self contained env? So if someone messes with the system python it will not break code running in the venv. -- https://mail.python.org/mailman/listinfo/python-list
path to python in venv
I was under the impression that in a venv the python used would be in the venv's bin dir. But in my venvs I see this in the bin dirs: lrwxrwxrwx 1 larrymartell larrymartell7 Sep 27 11:21 python -> python3 lrwxrwxrwx 1 larrymartell larrymartell 16 Sep 27 11:21 python3 -> /usr/bin/python3 Googling this I read: The presence of symbolic links like python and python3 in the bin directory of your virtual environment pointing to the system Python executable (/usr/bin/python) suggests that the virtual environment was created using the system Python interpreter rather than a standalone Python installation. This can happen if you create a virtual environment using a system-wide Python interpreter, and the virtual environment inherits some of the symbolic links or shortcuts from the system Python installation. In this case, your virtual environment is not fully isolated because it still relies on the system Python interpreter. Not sure what this really means, nor how to get python to be in my venv. -- https://mail.python.org/mailman/listinfo/python-list
Re: Why do I always get an exception raised in this __init__()?
On Thu, Aug 31, 2023 at 3:19 PM Chris Green via Python-list
wrote:
>
> I'm obviously doing something very silly here but at the moment I
> can't see what.
>
> Here's the code:-
>
> #!/usr/bin/python3
> #
> #
> # GPIO
> #
> import gpiod
> #
> #
> # Simple wrapper class for gpiod to make set and clearing outputs
> easier
> #
> class Gpiopin:
>
> def __init__(self, pin):
> #
> #
> # scan through the GPIO chips to find the line/pin we want
> #
> for c in ['gpiochip0', 'gpiochip1', 'gpiochip2', 'gpiochip3']:
>
> chip = gpiod.Chip(c)
> for l in range(32):
> line = chip.get_line(l)
> if pin in line.name():
> print("Found: ", line.name())
> return
> else:
> raise ValueError("Can't find pin '" + pin + "'")
>
> def print_name(self):
> print (self.line.name())
>
> def set(self):
> self.line.set_value(1)
>
> def clear(self):
> self.line.set_value(0)
>
>
> This is by no means the final code, the print() in the __init__() is
> just a diagnostic for example. However I really can't understand why I
> see the following when I try it:-
>
> >>> import ngp
> >>> ngp.Gpiopin("P9_23")
> Found: P9_23
> Traceback (most recent call last):
> File "", line 1, in
> File "/home/chris/.cfg/hosts/bbb/bin/ngp.py", line 24, in __init__
> return
> ValueError: Can't find pin 'P9_23'
> >>>
>
> Does a return in __init__() not do what I think it does?
>
> How else could/should I do this?
Change the return to a break
--
https://mail.python.org/mailman/listinfo/python-list
Re: pip-sync
On Fri, Jul 21, 2023 at 11:08 AM Larry Martell wrote: > > I am trying to set up and maintain a venv with pip-sync. On my bare > metal I have the apparmor python package installed, but it is not > installed in my venv and it's not in my requirements file. When I run > pip-sync I get: > > Found existing installation: apparmor 2.13.3 > ERROR: Cannot uninstall 'apparmor'. It is a distutils installed > project and thus we cannot accurately determine which files belong to > it which would lead to only a partial uninstall. > > Since it's not installed in the venv why does it want to uninstall it? Found out what was going on - I had pip-sync installed on the bare metal and when I ran it from within the venv it was picking that one up, which then looked at packages outside the venv. After I uninstalled it from the bare metal it was then working as expected. -- https://mail.python.org/mailman/listinfo/python-list
pip-sync
I am trying to set up and maintain a venv with pip-sync. On my bare metal I have the apparmor python package installed, but it is not installed in my venv and it's not in my requirements file. When I run pip-sync I get: Found existing installation: apparmor 2.13.3 ERROR: Cannot uninstall 'apparmor'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. Since it's not installed in the venv why does it want to uninstall it? -- https://mail.python.org/mailman/listinfo/python-list
Re: TypeError: can only concatenate str (not "int") to str
On Sun, Feb 26, 2023 at 5:46 PM Chris Angelico wrote: > On Mon, 27 Feb 2023 at 12:44, MRAB wrote: > > Oh dear. An example of Godwin's Law. > > Yeah, is that finally enough to get this user banned ? I hope so > -- https://mail.python.org/mailman/listinfo/python-list
Re: Rob Cliffe should stop sending me rude email messages.
On Sun, Feb 26, 2023 at 3:49 PM Hen Hanna wrote: > > Rob Cliffe should stop sending me rude email messages. You should stop spamming this lists with with meaningless posts. -- https://mail.python.org/mailman/listinfo/python-list
Re: flattening lists
On Tue, Oct 11, 2022 at 12:48 PM SquidBits _ wrote: > > Does anyone else think there should be a flatten () function, which just > turns a multi-dimensional list into a one-dimensional list in the order it's > in. e.g. > > [[1,2,3],[4,5,6,7],[8,9]] becomes [1,2,3,4,5,6,7,8,9]. > > I have had to flatten lists quite a few times and it's quite tedious to type > out. It feels like this should be something built in to python, anyone else > think this way? x = [[1,2,3],[4,5,6,7],[8,9]] [i for j in x for i in j] [1, 2, 3, 4, 5, 6, 7, 8, 9] -- https://mail.python.org/mailman/listinfo/python-list
Re: Function to Print a nicely formatted Dictionary or List?
On Thu, Jun 9, 2022 at 11:44 AM Dave wrote:
>
> Hi,
>
> Before I write my own I wondering if anyone knows of a function that will
> print a nicely formatted dictionary?
>
> By nicely formatted I mean not all on one line!
>>> import json
>>> d = {'John': 'Cleese', 'Eric': "Idle", 'Micheal': 'Palin'}
>>> print(json.dumps(d, indent=4))
{
"John": "Cleese",
"Eric": "Idle",
"Micheal": "Palin"
}
--
https://mail.python.org/mailman/listinfo/python-list
Re: terminate called after throwing an instance of 'boost::python::error_already_set
On Fri, May 27, 2022 at 5:51 PM dn wrote: > On 28/05/2022 08.14, Larry Martell wrote: > > I have a script that has literally been running for 10 years. > > Suddenly, for some runs it crashes with the error: > > > > terminate called after throwing an instance of > 'boost::python::error_already_set > > > > No stack trace. Anyone have any thoughts on what could cause this > > and/or how I can track it down? > > 1 a change to Python interpreter being used > > 2 a change to the boost library being used > > 3 a change to lower levels in 'the s/w stack' or h/w > > 4 a change to the data being passed-across Definitely not 1. 4 is always the case - every run is with different data. 2 and 3 I don’t know. What is boost and how does Python use it? None of my code is importing it. How can get a stack trace when it crashes with just that message? > -- https://mail.python.org/mailman/listinfo/python-list
terminate called after throwing an instance of 'boost::python::error_already_set
I have a script that has literally been running for 10 years. Suddenly, for some runs it crashes with the error: terminate called after throwing an instance of 'boost::python::error_already_set No stack trace. Anyone have any thoughts on what could cause this and/or how I can track it down? -- https://mail.python.org/mailman/listinfo/python-list
Re: Why no list as dict key?
On Wed, Apr 20, 2022 at 2:23 PM Abdur-Rahmaan Janhangeer wrote: > > Greetings list, > > Using Python3.9, i cannot assign a list [1, 2] as key > to a dictionary. Why is that so? Thanks in advanced! Dict keys cannot be mutable. Use a tuple instead. -- https://mail.python.org/mailman/listinfo/python-list
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: One final aside. Let me preface this by saying: I'm not proposing the following for graphlib.TopologicalSort. It's obviously too late to change that object this much. It's just something I'm thinking about--maybe I'll use this in my own library. Where we are now, the graphlib.TopologicalSort object is simultaneously a static representation of a graph--which the object doesn't provide a public API for you to examine!--and a stateful single-use iterator over that graph. My proposed change to the API seems to increase the tension between these two sets of semantics. Perhaps a better set of semantics, that more successfully maps my use case to the graph object, would be as follows: * you can add() nodes and edges at any time. * get_ready() always returns the current list of nodes with no prececessors. There is no internal "we already told you about this one" state. * replace done() with remove(), which removes the node and all edges from/to it from the graph. * static_order() is still fine. I think this would make it easy to reason about the object's behavior, and would be a better match to my use case where you're continually adding (and removing?) nodes, not just during an initial "populate the graph" phase. Again, not appropriate to consider for graphlib.TopologicalSort. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: I agree that the API should have as few surprises as possible. AFAICT you haven't made any terminal pronouncements like "it's impossible to add this feature without too many unacceptable surprises", so I'll proceed assuming we can find an API (and semantics) that we're all happy with. I've come around on the idea that forgetting "done" nodes is too surprising. The least surprising behavior is: once we've added a node to the graph, the graph remembers it. Now I'll propose a second simple rule, which you've already touched on: you can't add a dependency after a node has been returned by get_ready(). Attempting this always throws an exception; which exception specifically TBD. "Tough beans". I think those two rules are easy enough to remember and to reason about. It meaningfully expands the utility of the library with minimal surprise. The library behaves identically for users of the existing API but permits new use cases too. Adding this functionality would therefore mean fewer users would discover too late their use case isn't supported. As far as my "long-lived graph objects will consume more and more memory over time" caveat, there's a better solution than "forget": graph.remove(*nodes). (My version already has a "remove" method, and I forgot that graphlib's doesn't.) Allowing the user to remove a node from the graph gives them explicit control, and the semantics should be obvious and unsurprising; if you--the user--remove a node, and later you--the user--re-add that node to the graph, it behaves identically to any other node the graph has never seen before. Removing a node intuitively removes all edges to that node. Two notes on "remove" if we decide to go that route. First, I'd ensure you can remove a node at any time. Nodes have three externally visible states wrt TopologicalSort: 1) added but not published by get_ready, 2) published by get_ready but not returned using done, and 3) done. You should be able to remove a node in any of those three states. Removing a node in 2) should be equivalent to calling done before calling remove; that is, if you're removing the node anyway, you don't need to call done. Second, the current underlying implementation would make remove really slow. Nodes don't remember their predecessors, only their successors, so removing a node would be O(n). If we expected remove to get a lot of use, we'd probably want to change how the graph is stored to speed up this operation. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: I'm not sure I follow you. What do you suggest graphlib is guessing at? I thought we were discussing what graphlib's (documented, intentional) behavior should be; if there was any guessing going on, it was us doing it, guessing at what behavior the user would prefer. I appreciate you corresponding on the issue, but I'm having difficulty understanding what light you're shining on the problem. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: Having slept on it, I think the footgun is slightly bigger than I gave it credit for. Also the problem of nodes lingering forever is easily solved: give the user control. My next iteration will keep the done nodes around, but I'll also add a forget() method that compels the graph to forget all done nodes. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: calling a function asynchronously
On Wed, Mar 30, 2022 at 2:40 PM Kirill Ratkin via Python-list
wrote:
>
> Hi again,
>
> I changed a bit your example and it works as you expected I hope.
>
> import asyncio
>
>
> async def long():
> for i in range(100):
> await asyncio.sleep(10)
> print("long is done")
>
>
> loop = asyncio.get_event_loop()
>
> task = loop.create_task(long())
> print('after asyncio.run')
> loop.run_until_complete(asyncio.gather(task))
>
>
> But how I wrote before ... if you are in big Django project just look at
> existent django libraries for long task running. One of it I sent you.
> There is 'celery' as well.
>
> It's more pragmatic way ...
Appreciate the reply. I did not know about django-background-tasks -
thanks. I've been trying to make use of that. I do not get the errors
I was getting before but it does not appear that my long running task
is running at all. Still debugging. But concerting asyncio - doesn't
run_until_complete block until long() completes?
>
> 30.03.2022 19:10, Larry Martell пишет:
> > import asyncio
> > import time
> >
> > async def long():
> > for i in range(100):
> > time.sleep(10)
> >
> > asyncio.run(long())
> > print('after asyncio.run')
> --
> https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
calling a function asynchronously
I have a django app, and for a certain request I need to kick off a
long running task. I want to do this asynchronously and immediately
return a response. I tried using subprocess.Process() but the forked
process does not have a django database connection. I then tried
posting a request using ajax but that does not have a django session
so it's not authorized. Tried using asyncio but I am finding the
caller of the async function does not return until the async function
returns - maybe I am doing something wrong, as it does appear to be
actually asynchronous. I tried this test:
import asyncio
import time
async def long():
for i in range(100):
time.sleep(10)
asyncio.run(long())
print('after asyncio.run')
The final print does not come out until after long() completes.
Is there any way to do this?
--
https://mail.python.org/mailman/listinfo/python-list
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: > Assuming we do want to be able to add() after a get_ready(), is there > a reason that "forgetting" already-produced nodes is the correct > behavior, as opposed to remembering all nodes ever added, and > raising iff the addition creates a cycle among all nodes ever > added or depends on an already-yielded node? I'm not sure "correct" applies here, because I don't have a sense that one behavior is conceptually more correct than the other. But in implementing my API change, forgetting about the done nodes seemed more practical. The "benefit" to remembering done nodes: the library can ignore dependencies to them in the future, forever. Adding a dependency to a node that's already been marked as "done" doesn't make much conceptual sense to me, but as a practical thing maybe it's useful? I'm not sure, but it doesn't seem all that useful. I can only come up with a marginal reason why remembering done nodes is useful. Let's say all your tasks fan out from some fundamental task, like "download master list of work" or something. One of your tasks might discover additional tasks that need to run, and conceptually those tasks might depend on your "download master list of work" task. If the graph remembers the done list forever, then adding that dependency is harmless. If the graph forgets about done nodes, then adding that dependency could re-introduce that task to the graph, which could goof things up. So maybe it's a little bit of a footgun? But on the other hand: you already know you're running, and you're a task that was dependent on the master list of work, which means you implicitly know that dependency has been met. So just skip adding the redundant dependency and you're fine. On the other hand, forgetting about the nodes has a definite practical benefit: the graph consumes less memory. If you use a graph object for a long time, the list of done nodes it's holding references to would continue to grow and grow and grow. If we forget about done nodes, we free up all that memory, and done membership testing maybe gets faster. I guess I'm not married to the behavior. If someone had a great conceptual or practical reason why remembering the done nodes forever was better, I'd be willing to listen to reason. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: I'm using my graph library to manage a list of tasks that need doing in some sort of proper order. One task may spawn other tasks at runtime, and we don't necessarily know what the tasks will be until runtime. It's way more convenient to simply add such tasks on demand, rather than trying to preemptively pre-add all such possible tasks before preparing the graph, or creating additional graphs. For example, consider a tool that downloads and digests zip files full of music from an online store. Your initial tasks might represent "download the zip file", then "decompress and examine the contents of the zip file". You could then iterate over the contents of the zip file, adding different tasks based on what you find--one pipeline of tasks for media files (FLAC/MP3/OGG/etc), another for the playlist, a third if you don't *find* a playlist, a fourth for image files, etc. (Not that you'd necessarily write such a tool this way, but it's at least plausible.) The new nodes needn't be connected to the existing nodes for this to still be useful. You could reuse the same graph object for all your tasks. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
New submission from Larry Hastings : I've maintained my own topological sort class for a while now. The API I came up with (or was it Tim and I?) was adopted for graphlib.TopologicalSort(). Some of my method names are slightly different, but the APIs are so similar, I can do a little renaming and run Lib/test/test_graphlib using my implementation. (For clarity I'm going to write the rest of this issue using the names from graphlib.) Recently I made an improvement to my version: it no longer has the modal behavior where there's a "before prepare" phase where you add nodes and an "after prepare" phase where you can call get_ready and done. Instead, in my new and improved version the graph is always maintained in a "ready" state. You can call add, get_ready, and done in any order, as much as you like. prepare doesn't do anything besides the cycle check--but read on! This approach required tweaking some semantics behind the scenes. My graph object now maintains an internal "dirty" flag. It's set to True after any edges are added, and only gets cleared after checking for cycles. prepare runs this cycle check, but get_ready *also* runs this cycle check. (Though in both cases, only when the dirty flag is set, of course.) In theory this means you no longer need the prepare call. However, it's still useful, because you can call it to check for a CycleError. So, on the one hand, this means that get_ready can throw a CycleError, which it never did before. On the other hand, it won't throw for existing users of the library, because they never add edges after their prepare call. So this semantic change shouldn't break existing users, assuming they're not misusing the existing API. Another wrinkle: adding a dependency on a node that's already been handed out by get_ready (but hasn't been returned by done) is ignored. Again, not something existing users of graphlib can do, so this shouldn't break code. There's one final semantic change I made worth knowing about: when you return a node via done, the graph simply forgets about it. This means you could add it a second time (!) and it could go for a complete ride through the graph again, wh! This also means that when all nodes have been returned by done, the graph is essentially returned to its initial pristine state. This too seems completely harmless, because with the existing library it's illegal to add nodes to a graph after calling prepare, so nobody's doing it, so it won't break any code. I'm happy to contribute my version. But I was lazy and didn't worry about speed or memory implementation; it's strewn with sets and things. I figured I could always optimize it later. But "later" could become "now" if we were interested in trying to merge this behavior into Python's graphlib. Interested? -- assignee: larry components: Library (Lib) messages: 416182 nosy: eric.smith, larry, pablogsal, tim.peters priority: normal severity: normal stage: needs patch status: open title: Improve graphlib.TopologicalSort by removing the prepare step type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: Ooh, good one. I don't know anybody in the Django project to contact though. Anybody have any leads? -- ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: I heard back from both Samuel Colvin (Pydantic) and Sebastián Ramírez (FastAPI). They said neither of them use update_wrapper with partial objects. They also took a peek in a bunch of other projects (FastAPI, Typer, SQLModel, Asyncer, SQLAlchemy, Trio, and AnyIO) and nobody was doing it. So honestly it seems like nobody (but me!) calls update_wrapper on partial objects, and we can just fix it. Graham, any final thoughts before we start pulling levers and merging PRs? For now I just want to fix this bug. I'm in favor of re-engineering the relevant objects so they write their own __signature__ objects, so inspect.Signature doesn't have to understand the internals of objects from other modules. But maybe we save that for another day. -- ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: > Performance wise... The SHA series have hardware acceleration on > modern CPUs and SoCs. External libraries such as OpenSSL are in a > position to provide implementations that make use of that. Same with > the Linux Kernel CryptoAPI (https://bugs.python.org/issue47102). > > Hardware accelerated SHAs are likely faster than blake3 single core. > And certainly more efficient in terms of watt-secs/byte. I don't know if OpenSSL currently uses the Intel SHA1 extensions. A quick google suggests they added support in 2017. And: * I'm using a recent CPU that AFAICT supports those extensions. (AMD 5950X) * My Python build with BLAKE3 support is using the OpenSSL implementation of SHA1 (_hashlib.openssl_sha1), which I believe is using the OpenSSL provided by the OS. (I haven't built my own OpenSSL or anything.) * I'm using a recent operating system release (Pop!_OS 21.10), which currently has OpenSSL version 1.1.1l-1ubuntu1.1 installed. * My Python build with BLAKE3 doesn't support multithreaded hashing. * In that Python build, BLAKE3 is roughly twice as fast as SHA1 for non-trivial workloads. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: I can't answer why the Rust one is so much larger--that's a question for Jack. But the blake3-py you built might (should?) have support for SIMD extensions. See the setup.py for how that works; it appears to at least try to use the SIMD extensions on x86 POSIX (32- and 64-bit), x86_64 Windows, and 64-bit ARM POSIX. If you were really curious, you could run some quick benchmarks, then hack your local setup.py to not attempt adding support for those (see "portable code only" in setup.py) and do a build, and run your benchmarks again. If BLAKE3 got a lot slower, yup, you (initially) built it with SIMD extension support. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: The Rust version is already quite "lean". And it can be much faster than the C version, because it supports internal multithreading. Even without multithreading I bet it's at least a hair faster. Also, Jack has independently written a Python package based around the C version: https://github.com/oconnor663/blake3-py/tree/master/c_impl so my making one would be redundant. I have no interest in building standalone BLAKE3 PyPI packages for Raspberry Pi or Android. My goal was for BLAKE3 to be one of the "included batteries" in Python--which would have meant it would, eventually, be available on the Raspberry Pi and Android builds that way. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Ok, I give up. -- resolution: -> rejected stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: > Given that I don't want to see us gain new vendored copies of > significant but non-critical third party hash code in our tree > (Modules/_blake3/impl/ in PR 31686) for anything but a known > fixed term need (ex: the sha2 libtomcrypt code is gone from > our tree as was clearly going to happen from the start), > the only way I think we should include blake3 support is if > there is already a plan for that code to leave our tree in > the future with a high probability of success. You've said what you want, but not why. It sounds like you are against merging the BLAKE3 PR containing its own impl. Why? -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Jack: I've updated the PR, improving compatibility with the "blake3" package on PyPI. I took your notes, and also looked at the C module you wrote. The resulting commit is here: https://github.com/python/cpython/pull/31686/commits/37ce72b0444ad63fd1989ad36be5f7790e51f4f1 Specifically: * derive_key_context is now a string, which I internally encode into UTF-8. * I added the AUTO member to the module, set to -1. * max_threads may not be zero; it can be >= 1 or AUTO. * I added the reset() method. Some additional thoughts, both on what I did and on what you did: * In your new() method, your error string says "keys must be 32 bytes". I went with "key must be exactly 32 bytes"; the name of the parameter is "key", and I think "exactly" makes the message clearer. * In my new() method, I complain if the derive_key_context is zero-length. In your opinion, is that a good idea, or is that overly fussy? * In your copy() method, you hard-code Blake3Type. It's considered good form to use type(self) here, in case the user is calling copy on a user-created subclass of Blake3Type. * In your copy() method, if the original has a lock created, you create a lock in the copy too. I'm not sure why you bother; I leave the lock member uninitialized, and let the existing logic create the lock in the copy on demand. * In the Blake3_methods array, you list the "update" method twice. I suspect this is totally harmless, but it's unnecessary. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: You make a good point. I filed a separate bug (#46846) suggesting that partial objects should set their own annotations and signature. I agree that objects performing such magic should take care of these details themselves, rather than requiring the inspect module to have workarounds based on deep knowledge of these other modules' inner workings. -- ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: Nobody I've nosied on this issue recently has expressed any opinion on the matter. I'm gonna try one more person: Graham Dumpleton, the maintainer of "wrapt", Python's premier function-wrapping. Graham, care to express any opinions about this issue? Can we fix it without causing widespread wailing and gnashing of teeth? Do you think people are depending on the current how-can-you-describe-it-as-anything-but-broken behavior? -- nosy: +grahamd ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Larry Hastings added the comment: I only did my experiments with the _wrappers.c Christian gave me. If that's not the shipping version of wrapt, and wrapt doesn't exhibit this behavior in 3.10, then that's good. I agree you can wait to address this behavior until it affects code you actually plan to ship. -- ___ Python tracker <https://bugs.python.org/issue45319> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Larry Hastings added the comment: Graham: I'm happy to help. I can write a more elaborate explanation of what's happening, or answer questions, whatever you like. I'm a fan of wrapt so I'd certainly like to see this issue resolved. And, seeing as you're the author and maintainer of wrapt, I assure you you've got more battle-won experience with object proxying than most developers--myself included. You're *absolutely* the right person to ameliorate this issue! -- ___ Python tracker <https://bugs.python.org/issue45319> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Larry Hastings added the comment: (It's also possible your workaround where wrapt explicitly defines an "__annotations__" getset on every child of ObjectProxy is the right fix. I don't know; I don't know the fine points of attribute access, like when does it access getsets on the type, vs getsets on base types, vs setattro, etc.) -- ___ Python tracker <https://bugs.python.org/issue45319> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Larry Hastings added the comment:
This isn't a CPython bug. It's a change in CPython behavior that wrapt needs
to accommodate. In particular, it isn't 'causing a regression for C subclasses
of heap types when the parent class has an "__annotations__" descriptor', nor
do child classes have any difficulty inheriting the descriptor of their parent
classes. That's unsurprising; after all, if I had broken child classes
inheriting the descriptors of their parent classes, a lot more would have
broken than just wrapt.
The problem is in WraptObjectProxy_setattro(). (Which--just to drive my point
home--*is* getting called when you set __annotations__ on one of wrapt's
various proxy objects.) WraptObjectProxy_setattro() proxies setattr calls for
wrapped objects to the original object--if "o" is a wrapt proxy object wrapping
"fn", and you run "o.__annotations__ = x", it should actually execute
"fn.__annotations__ = x" under the covers.
Except WraptObjectProxy_setattro() executes *this* code first, starting at line
1531 in my copy of _wrapped.c:
if (PyObject_HasAttr((PyObject *)Py_TYPE(self), name))
return PyObject_GenericSetAttr((PyObject *)self, name, value);
If the *type* has the attribute, then it doesn't proxy the setattr to the
wrapped object. Instead it does a "generic setattr" on the object itself.
PyObject_HasAttr works by attempting a getattr on the type. If that getattr
call succeeds, PyObject_HasAttr returns true. The type here is FunctionWrapper
(WraptFunctionWrapper_Type). Since we're now looking it up on this type
object, we use the type of the type object, which is "type", to access the
attribute. And getting the "__annotations__" attribute from an object of type
"type" means calling type_get_annotations(), a new descriptor which ensures
that the annotations dict always exists, which means the HasAttr call succeeds
and returns true. In short, this change to the semantics of the
"__annotations__" attribute means wrapt no longer proxies the setattr to the
underlying wrapped object when setting the "__annotations__" attribute on *any*
of its objects.
In my opinion, wrapt needs to accommodate this new behavior. In my testing I
changed the above code to this:
if (!annotations_str) {
annotations_str = PyUnicode_InternFromString("__annotations__");
}
if (PyObject_RichCompareBool(name, annotations_str, Py_NE)
&& PyObject_HasAttr((PyObject *)Py_TYPE(self), name))
return PyObject_GenericSetAttr((PyObject *)self, name, value);
I also declared
static PyObject *annotations_str = NULL;
at the top of the function. With that change in place, the tests now passed.
My hunch is, this approach is more or less what wrapt should do. It *might* be
undersophisticated; it's possible that there are classes out there playing
their own weird descriptor tricks with the "__annotations__" attribute.
Perhaps the fix needs to be on a case-by-case basis, based on the type of the
wrapped object. Anyway this is obviously up to Graham, which is for the best
anyway--he has far more experience than I do with this sort of object proxying
wizardry.
--
resolution: -> third party
stage: test needed -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.org/issue45319>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46930] Iterating over cls.__dict__ in classmethod causes RuntimeError when printing __annotations__
Larry Hastings added the comment: When accessing __annotations__ *in a class without annotations*, and *for the first time*. And your workaround seems reasonable. -- ___ Python tracker <https://bugs.python.org/issue46930> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Right, and I did say "(or BDFL)". Apparently you didn't bother to consult with the BDFL in advance, or at least not in the usual public venues--I haven't found a record of such a conversation on the bpo issue, nor in python-dev. BTW you simultaneously proposed adding SHA3/SHAKE. The total kloc for all this work was over 26k; you didn't mention any discomfort with the size of these patches at the time in public correspondance. In fact, quite the opposite. On 2016/05/28 you said: > I also don't get your obsession with lines of code. The gzip and expat > are far bigger than the KeccakCodePackage. https://mail.python.org/archives/list/python-...@python.org/message/3YHVN2I74UQC36AVY5BGRJJUE4PMU6GX/ -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Jack O'Connor: > Was any of the experimental C extension code [...] useful to you? > I was wondering if it could be possible to copy blake3module.c from > there verbatim. I concede I didn't even look at it. The glue code to mate the library with the CPython runtime wasn't the hard part. And, Python has its own way of working (Argument Clinic etc). I did what seemed easiest, which was to start with CPython's BLAKE2 support and hack it up. Given that I had it working in maybe half a day, this seems reasonable. I should take a peek at your experimental build though, just to confirm I got the interfaces right. > The setup.py build there also has working Windows and NEON support. The CPython build process doesn't use setup.py to build extensions on Windows. I haven't looked recently, but back in the day they were built like any other DLL, using its own "project file". This will require someone to use the MSVS GUI to create a new project, add files, etc etc. I assume they can just use the .asm files for the SIMD extensions so hopefully this will be straightforward. As for NEON support, the technique I used theoretically should Just Work. I look forward to hearing back from someone building on ARM. (I have a cross-compiler setup for building for MiSTer, which is an ARM platform with NEON support. But the cross-compiler is a royal PITA to use... it's a Docker image, and requires a bunch of manual configuration, and I haven't touched any of that in more than a year.) You seem to have done at least a partial code review of my PR, given your comments to follow. I appreciate it! I tried to add you as a "reviewer" but GitHub wouldn't permit it--I assume this is some sort of permissions problem. Still, it'd be nice if you were able to do future code reviews using the GitHub review tool; are you permitted to use that for my PR? > - My derive_key_context parameter requires a string and refuses to > accept bytes. This is consistent with our Rust and C APIs (though the C > API does include a _raw version specifically for bindings, which we're > using here). I was considering going the other way with it actually, requiring bytes. Note that Python has first-class support for hard-coded bytes strings: b = blake3.blake3(derive_key_context=b"My Funny Valentine (1984)") The C interface takes "char *", not a "wchar_t *", and this seemed like the most direct and relatable way to reflect that. But I'm not militant about this, and I'm willing to change the interface to require an actual string (as in Unicode). I note that your C API already dictates that Unicode be encoded as UTF-8, so we can do that, and if the encoding fails the user can deal with it. > - I've included an `AUTO` constant that provides a special value (-1) > for the `max_threads` argument, and I explicitly don't support > `max_threads=0`. I can do that too; again, I prefer the 0 there, but I'm not militant about it. However, it would make sense to me if you had that constant somewhere in the BLAKE3 C .h files, which AFAICT you currently don't. > - I enforce that the `data` arguments are positional-only and that the > other keyword arguments are keyword-only. I think this is consistent > with the rest of hashlib. I suspect hashlib is mostly like that, due to being chock full of legacy code. But I don't see why that's necessary. I think permitting "data" to be a named argument is fine. So unless you have a strong conviction about it--which I bet you don't--I'll leave it as positional-or-keyword. There are rare circumstances where positional-only arguments are useful; this isn't one of them. > - I include a `.reset()` method. I don't mind adding that. > - Unrelated to tests: I haven't made any attempt to zero memory in my > `dealloc` function. But if that's what other hashlib functions do, > then I'm certainly in favor of doing it here too. I inherited that from the BLAKE2 code I carved up to make the BLAKE3 version. And yeah, it made sense to me, so I kept it. Christian Heimes: > GH-31686 is a massive patch set. I'm feeling uncomfortable adding > such much new code for a new hashing algorithm. Did you ask the > Steering Council for approval? I didn't. Like most hashing algorithms, BLAKE3 doesn't allocate memory and doesn't perform any I/O. All it does is meditate on the data you pass in, and write to various pre-allocated fixed-size buffers. As large codebases go this seems pretty harmless, almost inert. The Modules/_blake3/impl directory is about 32kloc. I note that the Modules/_blake2/impl directory you checked in in 2016 is about 21kloc, and you didn't require Steering Council (or BDFL) approval for that. As (former) Steering Council member Barry Warsaw says: JFDI! > The platform detection and compiler flag logic must be added to > co
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Also, for what it's worth: I just ran my checksum benchmarker using a freshly built python a la my PR. Here are my results when hashing 462183782 bytes (dicey-dungeons-linux64.zip): hash algorithm timebytes/sec size hash - -- -- - blake3 0.18976406 2435570699 64 0c72d0a07ba767b75b0c99fed38fda... sha1 0.21692419 2130623562 40 9fb83614394cd3b39e42b1d9f84a3c... sha256 0.26399648 1750719480 64 2320129f2545ff8606d3db1d706d86... sha224 0.28097475 1644929957 56 133b5da8d8b387f2bcfd69b0c73ed8... md4 0.34185237 1351998195 32 dea7585ea9fa4520687bab1dc671858e blake2b 0.53724666 860282275 128 e3653f33858a83b386c2fe865280a1... md5 0.58128106 795112407 32 299440e1968cf8f8abc288bac8c0a4fa sha512_224 0.64589952 715566066 56 413d48b782f114870ef80815540678... sha384 0.64645893 714946859 96 b1c1cd96cef79c15f2171b8aa81304... sha512 0.65424513 706438241 128 e7d0cec3fe8b73d1534a7bdb484176... sha512_256 0.68371638 675987586 64 3f58faba70cea4d6ea8a8371e71bbb... md5-sha1 0.80361958 575127576 72 299440e1968cf8f8abc288bac8c0a4... shake_128 0.84424524 547452041 64 c62a813897b81f67822fc07115deae... blake2s 0.85661793 539544839 64 cb8bd19c6ca446bbf7a8abbec61dc5... sha3_224 0.95759645 482649850 56 6f96d117c7fcbcd802b222854db644... shake_256 1.0152032 455262322 64 2d9f9dafe0ddf792c6407910946845... sha3_256 1.015744455019929 64 cc5d55fe0ac31f6e335da1bc6abaf3... sha3_384 1.3235858 349190644 96 13206910ff231fe51a38fe637ded30... sm3 1.4478934 319211203 64 021cd913540d95b13a03342b54f80d... ripemd160 1.4737549 313609670 40 1a956000b88267ec8fc23327d22548... sha3_512 1.9131832 241578418 128 e84b9f499b013956f6f36c93234ca3... "time" is wall time in seconds. So, yes, BLAKE3 was the fastest hash algorithm available on my machine--2.4GB/sec! (I'm a little surprised by that result actually. My CPU is pretty modern, so I assume it has the SHA1 extensions. And I further assume we're using OpenSSL, and OpenSSL will use those extensions when available. If BLAKE3 is *still* faster that OpenSSL, well! hats off to the BLAKE3 team. And that just goes to show you how useful SIMD extensions are!) -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Okay, so. Here's a PR that adds BLAKE3 support to hashlib. The code was straightforward; I just took the BLAKE2 module and modified it to only have one algorithm. I also copied over the whole C directory tree from BLAKE3, which is totally okay fine by their license. I didn't have to modify it a bit. The tricky part was building it. Building extension modules is done inside setup.py using distutils (or the modern replacement), and in order to get those sexy SIMD implementations I had to compile assembly-language files, or C files with extra flags. What I did works great on my Linux box, and I have my fingers crossed that it'll work on other POSIX platforms. I wrote a lng comment in setup.py explaining what I did and why. Steve: I didn't do anything for Windows support. Do you have time to take a pass at it? Or if you know another core dev who does Windows build stuff, please nosy them. Whoever does the work. I suggest you read https://github.com/BLAKE3-team/BLAKE3/blob/master/c/README.md for a little guidance on how to build BLAKE3 on Windows with SIMD support. Also, I see you now build Python for Windows on ARM! Does this mean Python can use BLAKE3's NEON support? Maybe it's time to find out! Get hyped! Jack and I corresponded last year (or maybe 2020?) about what the API should look like. The idea is, Jack also maintains a BLAKE3 Python extension on pypi, written in Rust. Our goal was to make the two types behave identically, so that it could be like the old stringio / cstringio situation. You can use the built-in one for convenience, but also you can install the Rust version from PyPI for even more performance. Jack, it wouldn't hurt my feelings overly much if you checked my PR to make sure I got the interface right. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Change by Larry Hastings : -- pull_requests: +29805 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/31686 ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: All permutations from 2 lists
On Wed, Mar 2, 2022 at 9:42 PM Avi Gross via Python-list
wrote:
>
> Larry,
>
> i waited patiently to see what others will write and perhaps see if you
> explain better what you need. You seem to gleefully swat down anything
> offered. So I am not tempted to engage.
But then you gave in to the temptation.
> And it is hard to guess as it is not clear what you will do with this.
In the interests of presenting a minimal example I clearly
oversimplified. This is my use case: I get a dict from an outside
source. The dict contains key/value pairs that I need to use to query
a mongodb database. When the values in the dict are all scalar I can
pass the dict directly into the query, e.g.:
self._db_conn[collection_name].find(query). But if any of the values
are lists that does not work. I need to query with something like the
cross product of all the lists. It's not a true product since if a
list is empty it means no filtering on that field, not no filtering on
all the fields. Originally I did not know I could generate a single
query that did that. So I was trying to come up with a way to generate
a list of all the permutations and was going to issue a query for each
individually. Clearly that would become very inefficient if the lists
were long or there were a lot of lists. I then found that I could
specify a list with the "$in" clause, hence my solution.
> def query_lfixer(query):
> for k, v in query.items():
> if type(v)==list:
> query[k] = {"$in": v}
> return query
>
> self._db_conn[collection_name].find(query_lfixer(query))
>
>
> So why did so many of us bother?
Indeed - so why did you bother?
--
https://mail.python.org/mailman/listinfo/python-list
Re: All permutations from 2 lists
On Wed, Mar 2, 2022 at 5:31 PM Joel Goldstick wrote: > > On Wed, Mar 2, 2022 at 5:07 PM Larry Martell wrote: > > > > On Wed, Mar 2, 2022 at 5:00 PM Cameron Simpson wrote: > > > > > > On 02Mar2022 08:29, Larry Martell wrote: > > > >On Tue, Mar 1, 2022 at 7:32 PM Rob Cliffe > > > >wrote: > > > >> I think itertools.product is what you need. > > > >> Example program: > > > >> > > > >> import itertools > > > >> opsys = ["Linux","Windows"] > > > >> region = ["us-east-1", "us-east-2"] > > > >> print(list(itertools.product(opsys, region))) > > > > > > > >This does not work if region = []. I wrote in question that either > > > >list could be empty. > > > > > > What do you want to get if a list is empty? You haven't said. My > > > personal expectation would be an empty result. > > > > > > Alternatively, if you expect an empty list to imply some single default > > > the the experession: > > > > > > the_list or (the_default,) > > > > > > might be of use. > > > > I've solved the issue. > > -- > > https://mail.python.org/mailman/listinfo/python-list > > Would you be so kind as to show the results of your solution? I posted it at 10:49am Eastern time. -- https://mail.python.org/mailman/listinfo/python-list
Re: All permutations from 2 lists
On Wed, Mar 2, 2022 at 5:00 PM Cameron Simpson wrote: > > On 02Mar2022 08:29, Larry Martell wrote: > >On Tue, Mar 1, 2022 at 7:32 PM Rob Cliffe wrote: > >> I think itertools.product is what you need. > >> Example program: > >> > >> import itertools > >> opsys = ["Linux","Windows"] > >> region = ["us-east-1", "us-east-2"] > >> print(list(itertools.product(opsys, region))) > > > >This does not work if region = []. I wrote in question that either > >list could be empty. > > What do you want to get if a list is empty? You haven't said. My > personal expectation would be an empty result. > > Alternatively, if you expect an empty list to imply some single default > the the experession: > > the_list or (the_default,) > > might be of use. I've solved the issue. -- https://mail.python.org/mailman/listinfo/python-list
Re: All permutations from 2 lists
On Wed, Mar 2, 2022 at 10:26 AM Antoon Pardon wrote:
>
>
>
> Op 2/03/2022 om 15:58 schreef Larry Martell:
> > On Wed, Mar 2, 2022 at 9:37 AM Antoon Pardon wrote:
> >>
> >>>>> If one list is empty I want just the other list. What I am doing is
> >>>>> building a list to pass to a mongodb query. If region is empty then I
> >>>>> want to query for just the items in the os list. I guess I can test
> >>>>> for the lists being empty, but I'd like a solution that handles that
> >>>>> as down the road there could be more than just 2 lists.
> >>>> How about the following: Keep a list of your lists you want to permute
> >>>> over.
> >>>> Like the following:
> >>>>
> >>>> permutation_elements = [["Linux","Windows"],["us-east-1", "us-east-2"]]
> >>>>
> >>>> permutation = itertools.product(*permutation_elements)
> >>>>
> >>>> If you don't include the empty list, you will get more or less what you
> >>>> seem to want.
> >>> But I need to deal with that case.
> >> What does that mean? How does using the above method to produce the
> >> permutations
> >> you want, prevent you from dealing with an empty list however you want
> >> when you
> >> encounter them? Just don't add them to the permutation_elements.
> > I need to know what items are in which position. If sometimes the
> > regions are in one index and sometimes in another will not work for
> > me.
>
> I am starting to suspect you didn't think this through. What you are telling
> here
> contradicts what you told earlier that if either list was empty, you just
> wanted
> the other list. Because then you wouldn't know what items were in that list.
>
> The only solution I can see now is that if a list is empty, you either add
> [None] or
> [""] to the permutation_elements (whatever suits you better) and then use
> itertools.product
I found a way to pass this directly into the query:
def query_lfixer(query):
for k, v in query.items():
if type(v)==list:
query[k] = {"$in": v}
return query
self._db_conn[collection_name].find(query_lfixer(query))
--
https://mail.python.org/mailman/listinfo/python-list
Re: All permutations from 2 lists
On Wed, Mar 2, 2022 at 9:37 AM Antoon Pardon wrote:
>
>
>
> Op 2/03/2022 om 15:29 schreef Larry Martell:
> > On Wed, Mar 2, 2022 at 9:10 AM Antoon Pardon wrote:
> >> Op 2/03/2022 om 14:44 schreef Larry Martell:
> >>> On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon
> >>> wrote:
> >>>> Op 2/03/2022 om 14:27 schreef Larry Martell:
> >>>>> On Tue, Mar 1, 2022 at 7:21 PM<2qdxy4rzwzuui...@potatochowder.com>
> >>>>> wrote:
> >>>>>> On 2022-03-01 at 19:12:10 -0500,
> >>>>>> Larry Martellwrote:
> >>>>>>
> >>>>>>> If I have 2 lists, e.g.:
> >>>>>>>
> >>>>>>> os = ["Linux","Windows"]
> >>>>>>> region = ["us-east-1", "us-east-2"]
> >>>>>>>
> >>>>>>> How can I get a list of tuples with all possible permutations?
> >>>>>>>
> >>>>>>> So for this example I'd want:
> >>>>>>>
> >>>>>>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows",
> >>>>>>> "us-east-1"), "Windows", "us-east-2')]
> >>>>>>>
> >>>>>>> The lists can be different lengths or can be 0 length. Tried a few
> >>>>>>> different things with itertools but have not got just what I need.
> >>>>>> [(o, r) for o in os for r in region]
> >>>>> This does not work if region = []. I wrote in my question that either
> >>>>> list could be empty.
> >>>> What do you mean it doesn't work? The result seems to be an empty list,
> >>>> which IMO is a perfectly valid result.
> >>>>
> >>>> All possible permutations over two collections where one collection is
> >>>> empty, should IMO give you an empty collection.
> >>> If one list is empty I want just the other list. What I am doing is
> >>> building a list to pass to a mongodb query. If region is empty then I
> >>> want to query for just the items in the os list. I guess I can test
> >>> for the lists being empty, but I'd like a solution that handles that
> >>> as down the road there could be more than just 2 lists.
> >> How about the following: Keep a list of your lists you want to permute
> >> over.
> >> Like the following:
> >>
> >> permutation_elements = [["Linux","Windows"],["us-east-1", "us-east-2"]]
> >>
> >> permutation = itertools.product(*permutation_elements)
> >>
> >> If you don't include the empty list, you will get more or less what you
> >> seem to want.
> > But I need to deal with that case.
>
> What does that mean? How does using the above method to produce the
> permutations
> you want, prevent you from dealing with an empty list however you want when
> you
> encounter them? Just don't add them to the permutation_elements.
I need to know what items are in which position. If sometimes the
regions are in one index and sometimes in another will not work for
me.
--
https://mail.python.org/mailman/listinfo/python-list
Re: All permutations from 2 lists
On Wed, Mar 2, 2022 at 9:10 AM Antoon Pardon wrote:
>
> Op 2/03/2022 om 14:44 schreef Larry Martell:
> > On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon wrote:
> >>
> >> Op 2/03/2022 om 14:27 schreef Larry Martell:
> >>> On Tue, Mar 1, 2022 at 7:21 PM<2qdxy4rzwzuui...@potatochowder.com>
> >>> wrote:
> >>>> On 2022-03-01 at 19:12:10 -0500,
> >>>> Larry Martell wrote:
> >>>>
> >>>>> If I have 2 lists, e.g.:
> >>>>>
> >>>>> os = ["Linux","Windows"]
> >>>>> region = ["us-east-1", "us-east-2"]
> >>>>>
> >>>>> How can I get a list of tuples with all possible permutations?
> >>>>>
> >>>>> So for this example I'd want:
> >>>>>
> >>>>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows",
> >>>>> "us-east-1"), "Windows", "us-east-2')]
> >>>>>
> >>>>> The lists can be different lengths or can be 0 length. Tried a few
> >>>>> different things with itertools but have not got just what I need.
> >>>> [(o, r) for o in os for r in region]
> >>> This does not work if region = []. I wrote in my question that either
> >>> list could be empty.
> >> What do you mean it doesn't work? The result seems to be an empty list,
> >> which IMO is a perfectly valid result.
> >>
> >> All possible permutations over two collections where one collection is
> >> empty, should IMO give you an empty collection.
> > If one list is empty I want just the other list. What I am doing is
> > building a list to pass to a mongodb query. If region is empty then I
> > want to query for just the items in the os list. I guess I can test
> > for the lists being empty, but I'd like a solution that handles that
> > as down the road there could be more than just 2 lists.
>
> How about the following: Keep a list of your lists you want to permute over.
> Like the following:
>
> permutation_elements = [["Linux","Windows"],["us-east-1", "us-east-2"]]
>
> permutation = itertools.product(*permutation_elements)
>
> If you don't include the empty list, you will get more or less what you
> seem to want.
But I need to deal with that case.
--
https://mail.python.org/mailman/listinfo/python-list
Re: All permutations from 2 lists
On Wed, Mar 2, 2022 at 8:54 AM Joel Goldstick wrote:
>
> On Wed, Mar 2, 2022 at 8:46 AM Larry Martell wrote:
> >
> > On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon wrote:
> > >
> > >
> > > Op 2/03/2022 om 14:27 schreef Larry Martell:
> > > > On Tue, Mar 1, 2022 at 7:21 PM<2qdxy4rzwzuui...@potatochowder.com>
> > > > wrote:
> > > >> On 2022-03-01 at 19:12:10 -0500,
> > > >> Larry Martell wrote:
> > > >>
> > > >>> If I have 2 lists, e.g.:
> > > >>>
> > > >>> os = ["Linux","Windows"]
> > > >>> region = ["us-east-1", "us-east-2"]
> > > >>>
> > > >>> How can I get a list of tuples with all possible permutations?
> > > >>>
> > > >>> So for this example I'd want:
> > > >>>
> > > >>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows",
> > > >>> "us-east-1"), "Windows", "us-east-2')]
> > > >>>
> > > >>> The lists can be different lengths or can be 0 length. Tried a few
> > > >>> different things with itertools but have not got just what I need.
> > > >> [(o, r) for o in os for r in region]
> > > > This does not work if region = []. I wrote in my question that either
> > > > list could be empty.
> > >
> > > What do you mean it doesn't work? The result seems to be an empty list,
> > > which IMO is a perfectly valid result.
> > >
> > > All possible permutations over two collections where one collection is
> > > empty, should IMO give you an empty collection.
> >
> > If one list is empty I want just the other list. What I am doing is
> > building a list to pass to a mongodb query. If region is empty then I
> > want to query for just the items in the os list. I guess I can test
> > for the lists being empty, but I'd like a solution that handles that
> > as down the road there could be more than just 2 lists.
> > --
> > https://mail.python.org/mailman/listinfo/python-list
>
> Does this help you out:
>
> >>> [(o,r) for o in opsys for r in region or "x"]
> [('Linux', 'x'), ('Window', 'x')]
That doesn't work if opsys = [] - either list could be empty.
--
https://mail.python.org/mailman/listinfo/python-list
Re: All permutations from 2 lists
On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon wrote:
>
>
> Op 2/03/2022 om 14:27 schreef Larry Martell:
> > On Tue, Mar 1, 2022 at 7:21 PM<2qdxy4rzwzuui...@potatochowder.com> wrote:
> >> On 2022-03-01 at 19:12:10 -0500,
> >> Larry Martell wrote:
> >>
> >>> If I have 2 lists, e.g.:
> >>>
> >>> os = ["Linux","Windows"]
> >>> region = ["us-east-1", "us-east-2"]
> >>>
> >>> How can I get a list of tuples with all possible permutations?
> >>>
> >>> So for this example I'd want:
> >>>
> >>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows",
> >>> "us-east-1"), "Windows", "us-east-2')]
> >>>
> >>> The lists can be different lengths or can be 0 length. Tried a few
> >>> different things with itertools but have not got just what I need.
> >> [(o, r) for o in os for r in region]
> > This does not work if region = []. I wrote in my question that either
> > list could be empty.
>
> What do you mean it doesn't work? The result seems to be an empty list,
> which IMO is a perfectly valid result.
>
> All possible permutations over two collections where one collection is
> empty, should IMO give you an empty collection.
If one list is empty I want just the other list. What I am doing is
building a list to pass to a mongodb query. If region is empty then I
want to query for just the items in the os list. I guess I can test
for the lists being empty, but I'd like a solution that handles that
as down the road there could be more than just 2 lists.
--
https://mail.python.org/mailman/listinfo/python-list
Re: All permutations from 2 lists
On Tue, Mar 1, 2022 at 7:32 PM Rob Cliffe wrote:
>
> I would not use `os` as an identifier, as it is the name of an important
> built-in module.
This is part of a much larger data structure, I created a simplified
example. It is not actually called os.
> I think itertools.product is what you need.
> Example program:
>
> import itertools
> opsys = ["Linux","Windows"]
> region = ["us-east-1", "us-east-2"]
> print(list(itertools.product(opsys, region)))
This does not work if region = []. I wrote in question that either
list could be empty.
> Output:
>
> [('Linux', 'us-east-1'), ('Linux', 'us-east-2'), ('Windows',
> 'us-east-1'), ('Windows', 'us-east-2')]
>
> itertools.product returns an iterator (or iterable, I'm not sure of the
> correct technical term).
> If you only want to use the result once you can write e.g.
>
> for ops, reg in itertools.product(opsys, region):
> etc.
>
> If you need it more than once, you can convert it to a list (or tuple),
> as above.
> Best wishes
> Rob Cliffe
>
> On 02/03/2022 00:12, Larry Martell wrote:
> > If I have 2 lists, e.g.:
> >
> > os = ["Linux","Windows"]
> > region = ["us-east-1", "us-east-2"]
> >
> > How can I get a list of tuples with all possible permutations?
> >
> > So for this example I'd want:
> >
> > [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows",
> > "us-east-1"), "Windows", "us-east-2')]
> >
> > The lists can be different lengths or can be 0 length. Tried a few
> > different things with itertools but have not got just what I need.
> >
> > TIA!
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: All permutations from 2 lists
On Tue, Mar 1, 2022 at 7:21 PM <2qdxy4rzwzuui...@potatochowder.com> wrote:
>
> On 2022-03-01 at 19:12:10 -0500,
> Larry Martell wrote:
>
> > If I have 2 lists, e.g.:
> >
> > os = ["Linux","Windows"]
> > region = ["us-east-1", "us-east-2"]
> >
> > How can I get a list of tuples with all possible permutations?
> >
> > So for this example I'd want:
> >
> > [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows",
> > "us-east-1"), "Windows", "us-east-2')]
> >
> > The lists can be different lengths or can be 0 length. Tried a few
> > different things with itertools but have not got just what I need.
>
> [(o, r) for o in os for r in region]
This does not work if region = []. I wrote in my question that either
list could be empty.
--
https://mail.python.org/mailman/listinfo/python-list
All permutations from 2 lists
If I have 2 lists, e.g.:
os = ["Linux","Windows"]
region = ["us-east-1", "us-east-2"]
How can I get a list of tuples with all possible permutations?
So for this example I'd want:
[("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows",
"us-east-1"), "Windows", "us-east-2')]
The lists can be different lengths or can be 0 length. Tried a few
different things with itertools but have not got just what I need.
TIA!
--
https://mail.python.org/mailman/listinfo/python-list
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: I emailed the Pydantic and FastAPI guys and didn't hear back. Given what you found on their issue trackers, I think it's unlikely that they care a lot about this issue (but were too busy to reply). It's far more likely that they don't care. Doing a little research (git blame), it looks like the "follow the wrapped chain to find the original signature" work was done by Nick Coghlan about nine years ago; he touched both functools.update_wrapper and the inspect module. The only other people to touch the code recently are Yuri and Batuhan. I've nosied Nick and Batuhan (already looped in Yuri), just to ping them and see if they have any strong opinions. If nobody has anything remarkable to say about it, honestly we probably *can* just merge your patch, Ofey. I see your name is now marked with a star; are you now authorized to contribute to CPython? (Note that I only glanced at your patch so far; if we were going to merge this bugfix I would of course first do a full review.) -- nosy: +BTaskaya, ncoghlan ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28824] os.environ should preserve the case of the OS keys ?
Change by Larry Hastings : -- nosy: -larry, loewis ___ Python tracker <https://bugs.python.org/issue28824> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46847] functools.update_wrapper doesn't understand partial objects and annotations
New submission from Larry Hastings : functools.update_wrapper currently copies over every attribute listed in the "assigned" parameter, which defaults to WRAPPER_ASSIGNMENTS, which means it copies the wrapped function's __annotations__ to the wrapper. This is slightly wrong if the wrapper occludes an annotated parameter: def foo(a: int, b: str, c: float): print(a, b, c) import functools foo_a = functools.partial(foo, 3) functools.update_wrapper(foo_a, foo) print(foo_a.__annotations__) In this case, foo_a.__annotations__ contains an annotation for a parameter named "a", even though foo_a doesn't have a parameter named "a". This problem occurred to me just after I filed #46846; the two issues are definitely related. -- components: Library (Lib) messages: 413898 nosy: larry, rhettinger priority: normal severity: normal stage: test needed status: open title: functools.update_wrapper doesn't understand partial objects and annotations type: behavior versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46847> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46846] functools.partial objects should set __signature__ and _annotations__
New submission from Larry Hastings : I ran across an interesting bug in issue #46761. If you call functools.update_wrapper on a functools.partial object, inspect.signature will return the wrong (original) signature for the partial object. We're still figuring that one out. And, of course, it's telling that the bug has been there for a long time. I suspect this isn't something that has inconvenienced a lot of people. But: I suggest that it's time functools.partial participated in signature stuff. Specifically, I think functools.partial should generate a new and correct __signature__ for the partial object. And I propose it should also generate a new and correct __annotations__ for the partial, by removing all entries for parameters that are filled in by the partial object. Right now inspect.signature has special support for functools.partial objects. It finds the underlying function, and . Which means there's code in both modules that has to understand the internals of partial objects. Just from a code hygiene perspective, it'd be better if all that logic lived under functools. I wonder if functools.partial objects should generally do a better job of impersonating the original function. Should they adopt the same __name__? __file__? __qualname__? My intuition is, it'd be nice if it did. But I might be forgetting something important. (I suspect everything I said about functools.partial also applies to functools.partialmethod.) -- components: Library (Lib) messages: 413897 nosy: larry, rhettinger priority: normal severity: normal stage: test needed status: open title: functools.partial objects should set __signature__ and _annotations__ type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46846> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: Okay, so, I considered the problem for a while, and I have a reasonable theory about what follow_wrapper_chains was for in the first place. If you have a generic decorator, like functools.cache(), it usually looks like this: def my_decorator(fn): def generic_version(*args, **kwargs): args, kwargs = do_something(args, kwargs) return fn(*args, **kwargs) return generic_version @my_decorator def add_five(i): return i+5 If you take the signature of add_five, you'd get (*args, **kwargs), because that's the signature of the wrapper function returned by the decorator. The decorator doesn't change the parameters of the function, but because of how decorators work it can occlude the proper function signature. In that instance, follow_wrapper_chains does the right thing, and as a result you get a precise function signature. (Of course, it would do the wrong thing if your hand-written decorator *also* behaved like a partial application, adding in its own hard-coded arguments, so that the resulting function signature changed.) Still, obviously it's doing the wrong thing when it comes to functools.partial() functions. My suspicion is that I'm the rare individual who actually uses update_wrapper on a functools.partial object. So maybe we have the situation here where, yeah, it's a bug, and we can fix it without causing further breakage. Maybe we can loop in someone who works on a popular runtime function introspection library (FastAPI, Pydantic) to see if they have any take on it. -- ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: Ofey, I appreciate your enthusiasm, but you should probably slow down. Fixing the bug is probably going to be the easy part here. But we're not to that stage quite yet. First, we need to determine * why the code behaves like this--is this behavior a genuine bug, or is it actually a bugfix for some worse behavior? * will fixing the bug cause problems for Python users? and if so, can we still fix the bug while mitigating the damage to people who are unfortunately depending on the bug? The next step is not to write a bugfix for this exact behavior, it's to determine why the code is the way it is. If it was genuinely just a mistake, and we can simply fix it and people will thank us, then we may have a use for your patch. But, generally, people who work on Python are smart, and they don't tend to commit dumb mistakes, so we can't simply assume it's a simple bug and fix it. -- ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46807] Wrong class __annotations__ when field name and type are equal
Larry Hastings added the comment: Yeah, it's the same behavior. In that class, you have defined Str to be "", then you reference Str in the annotation, and its value is "". Whatever you set Str to in the example in [21], that's the value of the annotation. GvR closed the previous report as "not a bug", so I'm gonna do that here too. -- ___ Python tracker <https://bugs.python.org/issue46807> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: Yury, Ka-Ping, can you guys shed any light on this? Your names are still on inspect.py. -- nosy: +ping, yselivanov ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Just checking--I can liberally pull code from https://github.com/BLAKE3-team/BLAKE3 yes? -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46783] Add a new feature to enumerate(iterable, start=0) built-in function
Change by Larry Hastings : -- components: -Argument Clinic nosy: -larry ___ Python tracker <https://bugs.python.org/issue46783> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: I thought someone volunteered to do it--if that's not happening, I could take a look at it next week. Shouldn't be too hard... unless I have to touch autoconf, which I only barely understand. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
New submission from Larry Hastings : It's considered good hygiene to use functools.update_wrapper() to make your wrapped functions look like the original. However, when using functools.partial() to pre-supply arguments to a function, if you then call functools.update_wrapper() to update that partial object, inspect.signature() returns the *original* function's signature, not the *wrapped* function's signature. To be precise: if you wrap a function with functools.partial() to pre-provide arguments, then immediately call inspect.signature() on that partial object, it returns the correct signature with the pre-filled parameters removed. If you then call functools.update_wrapper() to update the partial from the original function, inspect.signature() now returns the *wrong* signature. I looked into it a little. The specific thing changing inspect.signature()'s behavior is the '__wrapped__' attribute added by functools.update_wrapper(). By default inspect.signature() will unwrap partial objects, but only if it has a '__wrapped__' attribute. This all looks pretty deliberate. And it seems like there was some thought given to this wrinkle; inspect.signature() takes a "follow_wrapper_chains" parameter the user can supply to control this behavior. But the default is True, meaning that by default it unwraps partial objects if they have a '__wrapped__'. I admit I don't have any context for this. Why do we want inspect.signature() to return the wrong signature by default? -- components: Library (Lib) files: update_wrapper.breaks.partial.signature.test.py messages: 413299 nosy: larry priority: normal severity: normal stage: test needed status: open title: functools.update_wrapper breaks the signature of functools.partial objects type: behavior versions: Python 3.10, Python 3.11, Python 3.7, Python 3.8, Python 3.9 Added file: https://bugs.python.org/file50625/update_wrapper.breaks.partial.signature.test.py ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: I assume by "intrinsics" you mean using the GCC SIMD stuff, not like inlining memcpy() or something. My assumption is yes, that's fine, we appear to already be using them for BLAKE2. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: > In setup.py I assume that the target platform of the build is the same as the > current interpreter's platform. If this is included in CPython, it won't be using setup.py, so this isn't a concern. I don't think there's a way to use setup.py to cross-compile, so I'm not sure this ever was a concern. > - Compiling assembly files. AFAICT Python currently ships exactly one assembly file, "Modules/_decimal/libmpdec/vcdiv64.asm", which is only built on Windows. It would be a brave new world of configure.ac hacking to build assembly language files on POSIX platforms. As a first pass I say we merge the reference C implementation. Maybe someday we could add the SIMD assembly language stuff--or use the one built in to OpenSSL (if they ever add BLAKE3). > I assume we don't want to check in the .obj files? Correct, we don't. > - blake3module.c contains an awful lot of gotos to handle allocation failure > cases. Works for me, please keep it. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46314] Stray RESUME opcode for unused lambda
Change by Larry Hastings : -- components: -2to3 (2.x to 3.x conversion tool), Argument Clinic, Build, C API, Cross-Build, Demos and Tools, Distutils, Documentation, Extension Modules, FreeBSD, IDLE, IO, Installation, Library (Lib), Parser, Regular Expressions, SSL, Subinterpreters, Tests, Tkinter, Unicode, Windows, XML, asyncio, ctypes, email, macOS nosy: -larry ___ Python tracker <https://bugs.python.org/issue46314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: So, can we shoot for adding this to 3.11? Jack, do you consider the code is in good shape? I'd be up for shepherding it along in the process. In particular, I can contribute the bindings so BLAKE3 is a first-class citizen of hashlib. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46347] memory leak in PyEval_EvalCodeEx
Larry Hastings added the comment: (Sorry--it'll leak "kwnames", "newargs", and "defaults".) -- ___ Python tracker <https://bugs.python.org/issue46347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46347] memory leak in PyEval_EvalCodeEx
Larry Hastings added the comment: The function will still leak "kwnames" and "default" if creating the "func" object fails. Admittedly that would only happen in a low-memory condition which is a) rare and b) probably only happens just before the interpreter completely dies, so it's not worth addressing during today's mild emergency. -- nosy: +larry ___ Python tracker <https://bugs.python.org/issue46347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43424] Document the `controller.name` field in `webbrowser` module
Change by Larry Hastings : -- components: +Documentation nosy: -larry ___ Python tracker <https://bugs.python.org/issue43424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43424] Document the `controller.name` field in `webbrowser` module
Change by Larry Hastings : -- components: +Library (Lib) -Argument Clinic ___ Python tracker <https://bugs.python.org/issue43424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43424] Document the `controller.name` field in `webbrowser` module
Change by Larry Hastings : -- components: -2to3 (2.x to 3.x conversion tool), Build, C API, Cross-Build, Demos and Tools, Distutils, Documentation, Extension Modules, FreeBSD, IDLE, IO, Installation, Interpreter Core, Library (Lib), Parser, Regular Expressions, SSL, Subinterpreters, Tests, Tkinter, Unicode, Windows, XML, asyncio, ctypes, email, macOS ___ Python tracker <https://bugs.python.org/issue43424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
problem reading a CSV file
Win 10, Chrome, Python 3.10.1 New at python error on open statement Probably simple error but I do not see it. The program is a python example with the file name being changed. I want to experiment with changing the literal file name in the open statement to a variable name later. Larry -- https://mail.python.org/mailman/listinfo/python-list
[issue19838] test.test_pathlib.PosixPathTest.test_touch_common fails on FreeBSD with ZFS
Larry Hastings added the comment: I can confirm that the behavior is fixed in ZFS on Linux. My test case C program now prints "Everything is okay." when run on a ZFS partition on Linux, and test_touch_common from the current tree passes every time. ZFS fixing this was the best possible outcome. I'll go ahead and just close it now--why wait! If somebody confirms that the test still fails on FreeBSD, please open a new issue. Thanks for checking in, Irit! -- stage: patch review -> resolved status: pending -> closed ___ Python tracker <https://bugs.python.org/issue19838> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
python problem
I am new at Python. I have installed Python 3.10.1 and the latest Pycharm. When I attempt to execute anything via Pycharm or the command line, I receive a message it can not find Python. I do not know where Python was loaded or where to find and to update PATH to the program. Larry -- https://mail.python.org/mailman/listinfo/python-list
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Larry Hastings added the comment: (Preferably not using "tox". I don't know that tool.) -- ___ Python tracker <https://bugs.python.org/issue45319> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Larry Hastings added the comment: I got the PR locally, but the command-line you gave me fails, tox can't find python3.10 despite it being on the path. Rather than force me to reverse-engineer and recreate your build environment, can you give me a straightforward command-line or shell script to reproduce the problem? -- ___ Python tracker <https://bugs.python.org/issue45319> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Larry Hastings added the comment: I finally have some bandwidth to look at this--sorry for being a bit slow. I wasn't able to reproduce, because the patch didn't apply cleanly. I downloaded the patch ( https://patch-diff.githubusercontent.com/raw/GrahamDumpleton/wrapt/pull/187.patch ) and ran against current wrapt main, and about 50% of the "hunks" failed to apply. -- ___ Python tracker <https://bugs.python.org/issue45319> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31000] Test failure in resource module on ZFS
Larry Hastings added the comment: FWIW the test still fails in exactly the same way. This was building with main, on Pop!_OS 21.04 64-bit. -- ___ Python tracker <https://bugs.python.org/issue31000> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45020] Freeze all modules imported during startup.
Larry Hastings added the comment: Nope. On Windows, os.path is "ntpath". -- ___ Python tracker <https://bugs.python.org/issue45020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45392] docstring of "type" could use an update
Larry Hastings added the comment: Removing it makes sense to me. Not sure what I was thinking, way back when. Thanks for catching--and volunteering to fix--this! -- ___ Python tracker <https://bugs.python.org/issue45392> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45020] Freeze all modules imported during startup.
Larry Hastings added the comment: > What the two approaches have in common is that they require rebuilding the > python binary whenever you edit any of the changed modules. I heard somewhere > (I'm sorry, I honestly don't recall who said it first, possibly Eric himself) > that Jeethu's approach was rejected because of that. My dim recollection was that Jeethu's approach wasn't explicitly rejected, more that the community was more "conflicted" than "strongly interested", so I lost interest, and nobody else followed up. > I don't understand entirely why Jeethu's prototype had part written in C. My theory: it's easier to serialize C objects from C. It's maybe even slightly helpful? But it made building a pain. And yeah it just doesn't seem necessary. The code generator will be tied to the C representation no matter how you do it, so you might as well write it in a nice high-level language. > I never ran it so I don't know what the generated code looked like, [...] You can see an example of Jeethu's serialized objects here: https://raw.githubusercontent.com/python/cpython/267c93d61db9292921229fafd895b5ff9740b759/Python/frozenmodules.c Yours is generally more readable because you're using the new named structure initializers syntax. Though Jeethu's code is using some symbolic constants (e.g. PyUnicode_1BYTE_KIND) where you're just printing the actual value. > > With that in place, it'd be great to pre-cache all the .py files > > automatically read in at startup. > > *All* the .py files? I think the binary bloat cause by deep-freezing the > entire stdlib would be excessive. I did say "all the .py files automatically read in at startup". In current trunk, there are 32 modules in sys.module at startup (when run non-interactively), and by my count 13 of those are written in Python. If we go with Eric's approach, that means we'd turn those .pyc files into static data. My quick experiment suggests that'd be less than 300k. On my 64-bit Linux system, a default build of current trunk (configure && make -j) yields a 23mb python executable, and a 44mb libpython3.11.a. If I build without -g, they are 4.3mb and 7mb respectively. So this speedup would add another 2.5% to the size of a stripped build. If even that 300k was a concern, the marshal approach would also permit us to compile all the deep-frozen modules into a separate shared library and unload it after we're done. I don't know what the runtime impact of "deep-freeze" is, but it seems like it'd be pretty minimal. You're essentially storing these objects in C static data instead of the heap, which should be about the same. Maybe it'd mean the code objects for the module bodies would stick around longer than they otherwise would? But that doesn't seem like it'd add that much overhead. It's interesting to think about applying these techniques to the entire standard library, but as you suggest that would probably be wasteful. On the other hand: if we made a viable tool that could consume some arbitrary set of .py files and produce a C file, and said C file could then be compiled into a shared library, end users could enjoy this speedup over the subset of the standard library their program used, and perhaps even their own source tree(s). -- ___ Python tracker <https://bugs.python.org/issue45020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45020] Freeze all modules imported during startup.
Larry Hastings added the comment: There should be a boolean flag that enables/disables cached copies of .py files from Lib/. You should be able to turn it off with either an environment variable or a command-line option, and when it's off it skips all the internal cached stuff and uses the normal .py / .pyc machinery. With that in place, it'd be great to pre-cache all the .py files automatically read in at startup. As for changes to the build process: the most analogous thing we have is probably Argument Clinic. For what it's worth, Clinic hasn't been very well integrated into the CPython build process. There's a pseudotarget that runs it for you in the Makefile, but it's only ever run manually, and I'm not sure there's *any* build automation for Windows developers. AFAIK it hasn't really been a problem. But then I'm not sure this is a very good analogy--the workflow for making Clinic changes is very different from people hacking on Lib/*.py. It might be sensible to add a mechanism that checks whether or not the pre-cached modules are current. Store a hash for each cached module and check that they all match. This could then be part of the release process, run from a GitHub hook, etc. -- ___ Python tracker <https://bugs.python.org/issue45020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45020] Freeze all modules imported during startup.
Larry Hastings added the comment: Since nobody's said so in so many words (so far in this thread anyway): the prototype from Jeethu Rao in 2018 was a different technology than what Eric is doing. The "Programs/_freeze_importlib.c" Eric's playing with essentially inlines a .pyc file as C static data. The Jeethu Rao approach is more advanced: instead of serializing the objects, it stores the objects from the .pyc file as pre-initialized C static objects. So it saves the un-marshalling step, and therefore should be faster. To import the module you still need to execute the module body code object though--that seems unavoidable. The python-dev thread covers nearly everything I remember about this. The one thing I guess I never mentioned is that building and working with the prototype was frightful; it had both Python code and C code, and it was fragile and hard to get working. My hunch at the time was that it shouldn't be so fragile; it should be possible to write the converter in Python: read in .pyc file, generate .c file. It might have to make assumptions about the internal structure of the CPython objects it instantiates as C static data, but since we'd ship the tool with CPython this should be only a minor maintenance issue. In experimenting with the prototype, I observed that simply calling stat() to ensure the frozen .py file hadn't changed on disk lost us about half the performance win from this approach. I'm not much of a systems programmer, but I wonder if there are (system-proprietary?) library calls one could make to get the stat info for all files in a single directory all at once that might be faster overall. (Of course, caching this information at startup might make for a crappy experience for people who edit Lib/*.py files while the interpreter is running.) One more observation about the prototype: it doesn't know how to deal with any mutable types. marshal.c can deal with list, dict, and set. Does this matter? ISTM the tree of objects under a code object will never have a reference to one of these mutable objects, so it's probably already fine. Not sure what else I can tell you. It gave us a measurable improvement in startup time, but it seemed fragile, and it was annoying to work with/on, so after hacking on it for a week (at the 2018 core dev sprint in Redmond WA) I put it aside and moved on to other projects. -- ___ Python tracker <https://bugs.python.org/issue45020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44879] How to insert newline characters as normal characters while input()?
Larry Hastings added the comment: This is not a bug, you are asking for programming help. Please don't use the Python issue tracker for programming help. You won't get any, you'll just waste people's time. -- components: -Argument Clinic, FreeBSD, IO, Interpreter Core, Windows nosy: -koobs, paul.moore, prasechen, steve.dower, tim.golden, zach.ware resolution: -> not a bug stage: -> resolved status: open -> closed type: crash -> versions: -Python 3.10, Python 3.11, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue44879> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25782] CPython hangs on error __context__ set to the error itself
Change by Larry Hastings : -- nosy: -larry ___ Python tracker <https://bugs.python.org/issue25782> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: SQLALchemy: update with in clause from kwargs
On Tue, Aug 3, 2021 at 7:26 PM dn via Python-list
wrote:
>
> On 04/08/2021 13.08, Larry Martell wrote:
> > I am trying to write a function that takes kwargs as a param and
> > generates an update statement where the rows to be updated are
> > specified in an in clause.
> >
> > Something like this:
> >
> > def update_by_in(self, **kwargs):
> > filter_group = []
> > for col in kwargs['query_params']:
> > # obviously this line does not work as col is a string,
> > but this is the intent
> > filter_group.append(col.in_(tuple(kwargs['query_params'][col])))
> >
> >
> > self._session.query(self.model_class).filter(*filter_group).update(kwargs['values'])
> >
> > self.update_by_in(
> > **{'query_params': {'companyCode': ['A', 'B', 'C']},
> > 'values': {'portfolioName': 'test'}}
> > )
> >
> > Is there a way to do this? I think I need to use setattr in building
> > up the filter_group list, but I'm not quite sure how to do it.
>
>
> When feeling bamboozled by a problem, particularly when using
> sophisticated tools such as SQLAlchemy, the trick is often to simplify
> the problem.
>
> Step 1 (using the sample data provided)
> Write the query on paper - and as constant-values.
>
> Step 2
> Compare the two input dicts with that requirement.
>
> Step 3
> Work-out the transformation(s) required...
>
>
> One complexity is that the parameter to update_by_in() is formed by
> joining two dicts. However, the function later tries to treat them in
> distinct fashions. Why the join/why not two parameters?
>
> companyCode = ['A', 'B', 'C']
> values = {'portfolioName': 'test'}
>
> leading to:
>
> self.update_by_in( companyCode, values )
>
> and:
>
> def update_by_in(self, company_code, portfolio_type ):
>
>
>
> As to the core of the question-asked, I'm a little confused (which may
> be my fuzzy head). Do you want the update(s) - portrayed as a list -
> like this:
>
> [('A', 'test'), ('B', 'test'), ('C', 'test')]
>
> like this:
>
> [('A', 'portfolioName'), ('B', None), ('C', None)]
>
> or only:
>
> [('A', 'portfolioName')]
>
>
> You will find a friend in the itertools (PSL) library:
>
> import itertools as it
>
> list( it.product( companyCode, values.values() ) )
> [('A', 'test'), ('B', 'test'), ('C', 'test')]
>
> list( it.zip_longest( companyCode, values.values() ) )
> [('A', 'test'), ('B', None), ('C', None)]
>
> list( zip( companyCode, values ) )
> [('A', 'portfolioName')]
>
>
> Now, have we simplified things to the point of being able to more-easily
> code the update and filter?
I appreciate the reply, but it does not address my issue, which was
how to get at the column object. Turned out that was simple enough:
for col in kwargs['query_params']:
attr = getattr(self.model_class, col)
filter_group.append(attr.in_(tuple(kwargs['query_params'][col])))
Unfortunately that is causing something to change the query_params, as
filter group ends up like:
print(type(filter_group[0]))
print(filter_group[0])
dbo."Portfolio"."companyCode" IN (:companyCode_1, :companyCode_2,
:companyCode_3)
Which then fails with:
sqlalchemy.orm.evaluator.UnevaluatableError: Cannot evaluate
clauselist with operator
This has now become more of a sqlalchemy question than a python one.
--
https://mail.python.org/mailman/listinfo/python-list
