Re: Sanitise user input for a script

[Peter J. Holzer via Python-list]

On 2024-08-30 19:18:29 +, Simon Connah via Python-list wrote:
> I need to write a script that will take some user input (supplied on a
> website) and then execute a Python script on a host via SSH. I'm
> curious what the best options are for protecting against malicious
> input in much the smae way as you sanitise SQL to protect against SQL
> injections.

(Aside: Don't "sanitize" SQL. Use placeholders.)


> I could do it either on the website itself or by doing it on the host
> machine.

You will have to do it in the web site.

The SSH manual states:

| If supplied, the arguments will be appended to the command, separated by
| spaces, before it is sent to the server to be executed.

So whether you call 
ssh myhost print_args a b c
or
ssh myhost print_args a "b c"
in both cases exactly the same string will be sent to myhost, and it
won't have any chance to distinguish them.

So you will either have to filter ("sanitize") the arguments or properly
quote them before invoking SSH.

> If someone has any suggestions I'd appreciated it. If you need more
> information then please let me know.

First, if there is any chance that your arguments can contain characters
with meaning to the shell (like an apostrophe in a name), get the
quoting correct. If you can, transmit those arguments in a different way
(e.g. as input, maybe just nul-separated, may as JSON, or whatever). 

That removes the SSH-specific problems. There may still be problems with
the python script on the host.

Then, do all the validation you can on the web server. Reject all
requests which aren't valid. But be sure to check against the relevant
specifications, not your prejudices (You may not think that an
apostrophe in an email address is valid, but it is). Include meaningful
error messages (not just "input invalid"). Helping your legitimate users
is more important than slightly inconveniencing an attacker.

hp


-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: pdb: How to use the 'break' parameter?

[Peter J. Holzer via Python-list]

On 2024-08-22 01:44:35 +, Kevin M. Wilson via Python-list wrote:
> break (Old_MacDonald:23 | name[indx] == 'd', indx = 4), based on the doc spec 
> in python.org (https://docs.python.org/3/library/pdb.html#debugger-commands)
> Cell In[1], line 20
> break (Old_MacDonald:23 | name[indx] == 'd', indx = 4)
>   ^
>   SyntaxError: invalid syntax
> I got one blank white screen when I entered the exact first line of the 
> python.org-->b(reak) [([filename:]lineno | function) [, condition]]

You misunderstood the notation. The | means "or", it is not something
you have to type literally.

So the syntax is:
either a line number (maybe in a different file) or a function name
optionally followed by a condition after a comma

So in your case probably something like:

break Old_MacDonald:23, name[indx] == 'd'

(I'm not sure what "indx = 4" was supposed to do. You can't assign
inside of a condition.)

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[Peter J. Holzer via Python-list]

On 2024-08-03 15:17:11 -0500, o1bigtenor via Python-list wrote:
> One of the tools I need to be able to use is esptools - -  well in the
> devuan world you need to run that on either Devaun 3 or 5 - - - its just
> not available on devuan 4.

Couldn't you just upgrade to Devuan 5, then?


> Tried installing all the tools I need using downloads and .deb installs but
> then I need to have python3.12 and that's also not part of Devuan4.

It seems weird that something would work with the (presumably) older
version of Python in Devuan 3 and the (presumably) newer version of
Python in Devuan 5, but not with the version in Devuan 4.


> Not versed enough to set up a good venv (if that's possible) so that I
> could work in that specific venv and have my cake (and get to eat it too
> (grin!).

You need to install Python first to create a venv. AFAIK there is no way
to set up a venv first and then install Python into it.

Does Devuan have a testing or unstable suite? You might be able to
install a newer Python version from that. If not your best bet is to
install Python from source.

hp


-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Relatively prime integers in NumPy

[Peter J. Holzer via Python-list]

On 2024-07-08 19:09:45 +, Popov, Dmitry Yu via Python-list wrote:
> Does NumPy provide a simple mechanism to identify relatively prime
> integers, i.e. integers which don't have a common factor other than +1
> or -1?

Typing "numpy gcd" into my favourite search engine brings me to 
https://numpy.org/doc/stable/reference/generated/numpy.gcd.html

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Difference method vs attribut = function

[Peter J. Holzer via Python-list]

On 2024-06-28 18:08:54 +0200, Ulrich Goebel via Python-list wrote:
> a class can have methods, and it can have attributes, which can hold a
> function. Both is well known, of course.
> 
> My question: Is there any difference?
> 
> The code snipped shows that both do what they should do. But __dict__
> includes just the method,

The other way around: It includes only the attributes, not the methods.

> while dir detects the method and the
> attribute holding a function. My be that is the only difference?
> 
> 
> class MyClass:
> def __init__(self):
> functionAttribute = None
> 
> def method(self):
> print("I'm a method")
> 
> def function():
> print("I'm a function passed to an attribute")

Here is the other main difference: The object is not passed implicitely
to the function. You have no way to access mc here.

You can create a method on the fly with types.MethodType:

import types
mc.functionAttribute = types.MethodType(function, mc)


> By the way: in my usecase I want to pass different functions to
> different instances of MyClass. It is in the context of a database app
> where I build Getters for database data and pass one Getter per
> instance.

Or in this case, since each function is specific to one instance, you
could just use a closure to capture the object. But that might be
confusing to any future maintainers (e.g. yourself in 6 months), if the
method doesn't actually behave like a method.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Tkinter and astral characters (was: Decoding bytes to text strings in Python 2)

[Peter J. Holzer via Python-list]

On 2024-06-24 01:14:22 +0100, MRAB via Python-list wrote:
> Tkinter in recent versions of Python can handle astral characters, at least
> back to Python 3.8, the oldest I have on my Windows PC.

I just tried modifying
https://docs.python.org/3/library/tkinter.html#a-hello-world-program
to display "Hello World \N{ROCKET}" instead (Python 3.10.12 as included
with Ubuntu 22.04). I don't get a warning or error, but the emoji isn't
displayed either.

I suspect that the default font doesn't include emojis and Tk isn't
smart enough to fall back to a different font (unlike xfce4-terminal
which shows the emoji just fine).

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: in Python: (101 102 103 201 202 203 301 302 303 401 402 403 )

[Peter J. Holzer via Python-list]

On 2024-06-14 06:10:06 -, candycanearter07 via Python-list wrote:
> Phil Carmody  wrote at 12:01 this Thursday (GMT):
> > I'd say you can't beat the verbosity, or lack thereof of just plain 
> > zsh/bash:
> >   $ echo {1,2,3,4}0{1,2,3}
> >   101 102 103 201 202 203 301 302 303 401 402 403
> 
> 
> I /think/ you can replace it with {1...4} and {1...3}? I know there is
> some syntax for "range of numbers" but I can't remember it exactly.

Only two dots, not three:

% echo {1..4}0{1..3}
101 102 103 201 202 203 301 302 303 401 402 403

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lprint = ( Lisp-style printing ( of lists and strings (etc.) ) in Python )

[Peter J. Holzer via Python-list]

On 2024-05-30 21:47:14 -0700, HenHanna via Python-list wrote:
> [('the', 36225), ('and', 17551), ('of', 16759), ('i', 16696), ('a', 15816),
> ('to', 15722), ('that', 11252), ('in', 10743), ('it', 10687)]
> 
> ((the 36225) (and 17551) (of 16759) (i 16696) (a 15816) (to 15722) (that
> 11252) (in 10743) (it 10687))
> 
> 
> i think the latter is easier-to-read, so i use this code
>(by Peter Norvig)

This doesn't work well if your strings contain spaces:

Lprint(
[
["Just", "three", "words"],
["Just", "three words"],
["Just three", "words"],
["Just three words"],
]
)

prints:

((Just three words) (Just three words) (Just three words) (Just three words))

Output is often a compromise between readability and precision.


> def lispstr(exp):
># "Convert a Python object back into a Lisp-readable string."
> if isinstance(exp, list):

This won't work for your example, since you have a list of tuples, not a
list of lists and a tuple is not an instance of a list.

> return '(' + ' '.join(map(lispstr, exp)) + ')'
> else:
> return str(exp)
> 
> def Lprint(x): print(lispstr(x))

I like to use pprint, but it's lacking support for user-defined types. I
should be able to add a method (maybe __pprint__?) to my classes which
handle proper formatting (with line breaks and indentation).

hp
-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: From JoyceUlysses.txt -- words occurring exactly once

[Peter J. Holzer via Python-list]

On 2024-05-30 19:26:37 -0700, HenHanna via Python-list wrote:
> hard to decide what to do with hyphens
>and apostrophes
>  (I'd,  he's,  can't, haven't,  A's  and  B's)

Especially since the same character is used as both an apostrophe and a
closing quotation mark. And while that's pretty unambiguous between to
characters it isn't at the end of a word:

This is Alex’ house.
This type of building is called an ‘Alex’ house.
The sentence ‘We are meeting at Alex’ house’ contains an apostrophe.

(using proper unicode quotation marks. It get's worse if you stick to
ASCII.)

Personally I like to use U+0027 APOSTROPHE as an apostrophe and U+2018
LEFT SINGLE QUOTATION MARK and U+2019 RIGHT SINGLE QUOTATION MARK as
single quotation marks[1], but despite the suggestive names, this is not
the common typographical convention, so your texts are unlikely to make
this distinction.

hp

[1] Which I use rarely, anyway.

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminal Emulator (Posting On Python-List Prohibited)

[Peter J. Holzer via Python-list]

On 2024-05-20 00:26:03 +0200, Roel Schroeven via Python-list wrote:
> Skip Montanaro via Python-list schreef op 20/05/2024 om 0:08:
> > > Modern debian (ubuntu) and fedora block users installing using pip.
> > 
> > Even if you're telling it to install in ~/.local? I could see not allowing
> > to run it as root.
> 
> I assumed pip install --user would work, but no. I tried it (on Debian 12
> (bookworm)):
> 
> > $ pip install --user docopt
> > error: externally-managed-environment
> > 
> > × This environment is externally managed
> > ╰─> To install Python packages system-wide, try apt install
> >     python3-xyz, where xyz is the package you are trying to
> >     install.
> > 
> >     If you wish to install a non-Debian-packaged Python package,
> >     create a virtual environment using python3 -m venv path/to/venv.
> >     Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
> >     sure you have python3-full installed.
> > 
> >     If you wish to install a non-Debian packaged Python application,
> >     it may be easiest to use pipx install xyz, which will manage a
> >     virtual environment for you. Make sure you have pipx installed.
> > 
> >     See /usr/share/doc/python3.11/README.venv for more information.
> > 
> > note: If you believe this is a mistake, please contact your Python
> > installation or OS distribution provider. You can override this, at the
> > risk of breaking your Python installation or OS, by passing
> > --break-system-packages.
> > hint: See PEP 668 for the detailed specification.
> 
> Exactly the same output for sudo pip install.

This message (quoted in all its glory) is too long to be useful. The
important bit is at the end:

> > You can override this, at the risk of breaking your Python
> > installation or OS, by passing --break-system-packages.

(I admit I didn't see this the first time I got this message)

python3 -m pip install --user --break-system-packages 
does indeed install into ~/.local/lib/python3.XX/site-packages.

This inconvenient, but otoh I have accidentally installed packages into
~/.local in the past, so maybe it's good to make that more explicit.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

venvs vs. package management (was: Terminal Emulator (Posting On Python-List Prohibited))

[Peter J. Holzer via Python-list]

On 2024-05-18 20:12:33 +0200, Piergiorgio Sartor via Python-list wrote:
> On 18/05/2024 20.04, Mats Wichmann wrote:
> > So venvs make managing all that pretty convenient. Dunno why everybody's
> > so down on venvs...
> 
> Only people which are *not* using python... :-)
> 
> In my experience, venvs is the only possible
> way to use python properly.

That's very much depends on what you mean by properly.

Personally, I use venvs a lot. But most of the reasons have more to do
with team culture than technical constraints. In a different situation
(e.g. if all our developers used Linux and preferrably the same version)
I could see myself using venvs much less or maybe not at all.

> The dependency nightmare created by python, pip and all the rest
> cannot be resolved otherwise.

That's what package management on Linux is for. Sure, it means that you
won't have the newest version of anything and some packages not at all,
but you don't have to care about dependencies. Or updates.

(Missing packages can be a problem: Is there a script to automatically
generate .deb packages from PyPI? I haven't looked recently ...)

> It seems backward compatibility is a taboo...

I have recently written a script which checks out the newest version of
the project, creates a fresh venv using a requirements.txt without
version numbers and runs the test suite. If there is any action required
(either because a test fails or because there is a newer version of any
dependent package) it will create a ticket in redmine. Oh, and this
script runs on a staging server which has the same Linux distribution
(and hence the same Python version) as the production server.
Seems to work, but that is only necessary because we are using venvs. If
we relied on the distro's package management that would basically be a
non-issue.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminal Emulator (Posting On Python-List Prohibited)

[Peter J. Holzer via Python-list]

On 2024-05-16 19:46:07 +0100, Gordinator via Python-list wrote:
> To be fair, the problem is the fact that they use Windows (but I guess Linux
> users have to deal with venvs, so we're even.

I don't think Linux users have to deal with venvs any more than Windows
users. Maybe even less because many distributions come with a decent
set of Python packages.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminal Emulator

[Peter J. Holzer via Python-list]

On 2024-05-14 22:37:17 +0200, Mirko via Python-list wrote:
> Am 14.05.24 um 19:44 schrieb Gordinator via Python-list:
> > I wish to write a terminal emulator in Python. I am a fairly competent
> > Python user, and I wish to try a new project idea. What references can I
> > use when writing my terminal emulator? I wish for it to be a true
> > terminal emulator as well, not just a Tk text widget or something like
> > that.
> > 
> > If you have any advice, please do let me know!
> 
> 
> Not sure, what you mean with:
> 
> > true terminal emulator as well, not just a Tk text widget or
> > something like that
> If you want to write a GUI terminal, than that *is* a terminal emulator and
> *has* a text widget as its visible core. If you want to write something like
> getty which runs on the virtual terminals (Ctrl+Alt+F*) than that is a
> terminal (not a terminal emulator).

Getty isn't a terminal (unless there is another program of the same
name). It's a small program to set up a serial communication line.
Basically it waits for the modem to connect, sets the relevant
parameters (bit rate, byte width, parity, ...) and then hands over to
login. Of course in the case of a linux console there is no modem and no
serial line involved, so it doesn't have much to do. (Of course this
raises the question whether the Linux console is a terminal or a
terminal emulator ...)

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminal Emulator

[Peter J. Holzer via Python-list]

On 2024-05-14 16:03:33 -0400, Grant Edwards via Python-list wrote:
> On 2024-05-14, Alan Gauld via Python-list  wrote:
> > On 14/05/2024 18:44, Gordinator via Python-list wrote:
> >
> >> I wish to write a terminal emulator in Python. I am a fairly
> >> competent Python user, and I wish to try a new project idea. What
> >> references can I use when writing my terminal emulator? I wish for
> >> it to be a true terminal emulator as well, not just a Tk text
> >> widget or something like that.
> >
> > The first thing is to decide which terminal.
> 
> If you want to make life easier, make it a superset of a terminal that
> already exists in the terminfo database.
> 
> Going with some sort of ANSI terminal will probably provide
> operability even with dumb apps which ignore $TERM and just spit out
> basic ANSI escape sequences.

And if you want to go for a superset, xterm might be one of the more
useful: https://www.xfree86.org/current/ctlseqs.html

> If you really want to break trail, you could invent your own control
> sequences, which means you'll have to write terminfo and/or termcap
> entries as well as the terminal emulator.

Right. A saner model than ANSI and its supersets might be a good idea
conceptionally. But I'd expect quite a few programs to break.


> > A VT100 is very different from a 3270. And even a VT330 is quite
> > different from a VT100 although sharing a common subset of control
> > codes. And if you start looking at graphical terminals things get
> > even more interesting!
> 
> "Intersting" is putting it mildly...

Yup. Also there aren't many programs which use, e.g. xterm's
pixel-graphics capabilities.

OTOH, there is something like domterm[1], which can (theoretically)
display anything a browser can display.

hp


[1] https://domterm.org/index.html

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Dialogs

[Peter J. Holzer via Python-list]

On 2024-05-02 16:34:38 +0200, Loris Bennett via Python-list wrote:
> r...@zedat.fu-berlin.de (Stefan Ram) writes:
> >   Me (indented by 2) and the chatbot (flush left). Lines lengths > 72!
> 
> Is there a name for this kind of indentation, i.e. the stuff you are
> writing not being flush left?

Ramism.

> It is sort of contrary to what I think of as "normal" indentation.

Stefan is well known for doing everything contrary to normal convention.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: xkcd.com/353 ( Flying with Python )

[Peter J. Holzer via Python-list]

On 2024-03-31 12:27:34 -0600, Mats Wichmann via Python-list wrote:
> On 3/30/24 10:31, MRAB via Python-list wrote:
> > On 2024-03-30 11:25, Skip Montanaro via Python-list wrote:
> > > > > https://xkcd.com/1306/
> > > > >   what does  SIGIL   mean?
> > > > 
> > > > I think its' a Perl term, referring to the $/@/# symbols in front of
> > > > identifiers.

[You cut out a lot of context here]

> > I wouldn't consider '@' to be a sigil any more than I would a unary minus.
> 
> Nonetheless, Perl folk do use that term, specifically.

I'm pretty sure he's referring to the use of @ in python to denote a
decorator here. Which is a totally different thing than a Perl sigil.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: xkcd.com/353 ( Flying with Python )

[Peter J. Holzer via Python-list]

On 2024-03-30 17:58:08 +, Alan Gauld via Python-list wrote:
> On 30/03/2024 07:04, Greg Ewing via Python-list wrote:
> > On 30/03/24 7:21 pm, HenHanna wrote:
> >> https://xkcd.com/1306/
> >>   what does  SIGIL   mean?
> > 
> > I think its' a Perl term, referring to the $/@/# symbols in front of
> > identifiers.

Correct (although strictly speaking they are in front of an expression,
not an identifier).

> There seem to be several derivation sources including a fantasy world
> city suspended above a very thin, tall steeple
>
> Personally, I know SIGIL as an opensource EPUB editor!

Well, it's an ordinary English word of Latin origin (sigillum means
literally "small sign") in use since the 15th century. No need to go
hunting for proper names.

> None of them seem to have any direct connection to the xkcd cartoon.

In my opinion the connection to Perl sigils is very direct.

hp


-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Configuring an object via a dictionary

[Peter J. Holzer via Python-list]

On 2024-03-17 17:15:32 +1300, dn via Python-list wrote:
> On 17/03/24 12:06, Peter J. Holzer via Python-list wrote:
> > On 2024-03-16 08:15:19 +, Barry via Python-list wrote:
> > > > On 15 Mar 2024, at 19:51, Thomas Passin via Python-list 
> > > >  wrote:
> > > > I've always like writing using the "or" form and have never gotten bit
> > > 
> > > I, on the other hand, had to fix a production problem that using “or” 
> > > introducted.
> > > I avoid this idiom because it fails on falsy values.
> > 
> > Perl has a // operator (pronounced "err"), which works like || (or),
> > except that it tests whether the left side is defined (not None in
> > Python terms) instead of truthy. This still isn't bulletproof but I've
> > found it very handy.
> 
> 
> So, if starting from:
> 
> def method( self, name=None, ):
> 
>  rather than:
> 
> self.name = name if name else default_value
> 
> ie
> 
> self.name = name if name is True else default_value

These two lines don't have the same meaning (for the reason you outlined
below). The second line is also not very useful.



> the more precise:
> 
> self.name = name if name is not None or default_value
> 
> or:
> 
> self.name = default_value if name is None or name

Those are syntax errors. I think you meant to write "else" instead of
"or".

Yes, exactly. That's the semantic of Perl's // operator.

JavaScript has a ?? operator with similar semantics (slightly
complicated by the fact that JavaScript has two "nullish" values).

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Configuring an object via a dictionary

[Peter J. Holzer via Python-list]

On 2024-03-16 08:15:19 +, Barry via Python-list wrote:
> > On 15 Mar 2024, at 19:51, Thomas Passin via Python-list 
> >  wrote:
> > I've always like writing using the "or" form and have never gotten bit
> 
> I, on the other hand, had to fix a production problem that using “or” 
> introducted.
> I avoid this idiom because it fails on falsy values.

Perl has a // operator (pronounced "err"), which works like || (or),
except that it tests whether the left side is defined (not None in
Python terms) instead of truthy. This still isn't bulletproof but I've
found it very handy.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A Single Instance of an Object?

[Peter J. Holzer via Python-list]

On 2024-03-11 16:53:00 -0400, Ivan "Rambius" Ivanov via Python-list wrote:
> I am refactoring some code and I would like to get rid of a global
> variable. Here is the outline:

...

> The global cache variable made unit testing of the lookup(key) method
> clumsy, because I have to clean it after each unit test. I refactored
> it as:
> 
> class Lookup:
> def __init__(self):
> self.cache = {}
> 
> def lookup(key):
> if key in self.cache:
> return self.cache[key]
> 
> value = None
> 
> cmd = f"mycmd {key}"
> proc = subprocess(cmd, capture_output=True, text=True, check=False)
> if proc.returncode == 0:
> value = proc.stdout.strip()
> else:
> logger.error("cmd returned error")
> 
> self.cache[key] = value
> return value
> 
> Now it is easier to unit test, and the cache is not global. However, I
> cannot instantiate Lookup inside the while- or for- loops in main(),
> because the cache should be only one. I need to ensure there is only
> one instance of Lookup - this is why I made it a global variable, so
> that it is accessible to all functions in that script and the one that
> actually needs it is 4 levels down in the call stack.
[...]
> I am looking for the same behaviour as logging.getLogger(name).
> logging.getLogger("myname") will always return the same object no
> matter where it is called as long as the name argument is the same.
> 
> How would you advise me to implement that?

Just add a dict of Lookup objects to your module:

lookups = {}

def get_lookup(name):
if name not in lookups:
lookups[name] = Lookup()
return lookups[name]

Then (assuming your module is also called "lookup", in all other modules
do

import lookup

lo = lookup.get_lookup("whatever")

...
v = lo.lookup("a key")

In your test cases where you need many different lookup tables use

lo = lookup.get_lookup("test1")
...
lo = lookup.get_lookup("test2")
...
lo = lookup.get_lookup("test3")

hp

PS: You don't have to put that in a separate module but I think it's a
lot cleaner that way.

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Testing (sorry)

[Peter J. Holzer via Python-list]

On 2024-02-19 11:38:54 -0500, Thomas Passin via Python-list wrote:
> On 2/19/2024 9:17 AM, Grant Edwards via Python-list wrote:
> > On 2024-02-19, Thomas Passin  wrote:
> > > > About 24 hours later, all of my posts (and the confirmation e-mails)
> > > > all showed up in a burst at the same time on two different unrelated
> > > > e-mail accounts.
> > > > 
> > > > I still have no clue what was going on...
> > > 
> > > Sometimes a post of mine will not show up for hours or even half a day.
> > > They are all addressed directly to the list.  Sometimes my email
> > > provider sends me a notice that the message bounced.  Those notices say
> > > that the address wasn't available when the transmission was tried.
> 
> Here is a typical bounce message that I get:
> 
> : host mail.python.org[188.166.95.178] said:
> 450-4.3.2
> Service currently unavailable 450 4.3.2

This is a *temporary* error. Your provider's server should retry
delivering the message for a decent amount of time (3 to 7 days is
customary).

Your provider's server may send you a notification that the mail cannot
currently be delivered and that it will keep trying. Such notifications
are usually sent after a much shorter period (a few hours).

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Testing (sorry)

[Peter J. Holzer via Python-list]

[Replying to the list *and* Grant]

On 2024-02-17 19:38:04 -0500, Grant Edwards via Python-list wrote:
> Today I noticed that nothing I've posted to python-list in past 3
> weeks has shown up on the list.

January 29th, AFAICS. And end of december before that.

> I don't know how to troubleshoot this other than sending test
> messages.  Obviously, if this shows up on the list, then I've gotten
> it to work...

This did show up and 3 other test messages with very similar text
as well. 

Also there was a whole flurry of almost but not quite identical messages
from you in the "nan" thread.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using my routines as functions AND methods

[Peter J. Holzer via Python-list]

On 2024-01-03 23:17:34 -0500, Thomas Passin via Python-list wrote:
> On 1/3/2024 8:00 PM, Alan Gauld via Python-list wrote:
> > On 03/01/2024 22:47, Guenther Sohler via Python-list wrote:
> > > Hi,
> > > 
> > > In my cpython i have written quite some functions to modify "objects".
> > > and their python syntax is e.g.\
> > > 
> > > translate(obj, vec). e.g whereas obj is ALWAYS first argument.
^^^
> > 
> > > However, I also want to use these functions as class methods without 
> > > having
> > > to
> > > write the function , twice. When using the SAME function as a methos, the
> > > args tuple must insert/contain "self" in the first location, so i have
> > > written a function to do that:
> > 
> > I'm probably missing something obvious here but can't you
> > just assign your function to a class member?
> > 
> > def myFunction(obj, ...): ...
   ^^^
> > 
> > class MyClass:
> >  myMethod = myFunction
> > 
> > 
> > Then you can call it as
> > 
> > myObject = MyClass()
> > myObject.myMethod()
> > 
> > A naive example seems to work but I haven't tried anything
> > complex so there is probably a catch. But sometimes the simple
> > things just work?
> 
> That works if you assign the function to a class instance, but not if you
> assign it to a class.
> 
> def f1(x):
> print(x)

You omitted the first argument (obj).

That should be 

def f1(obj, x):
print(x)


> f1('The plain function')

> 
> class Class1:
> pass

o = Class1()
f1(o, 'The plain function')

works for me.


> class Class2:
> pass
> 
> c1 = Class1()
> c1.newfunc = f1
> c1.newfunc('f1 assigned to instance') # Works as intended

Now this doesn't work any more (but the OP doesn't want that anyway,
AFAICT).


> Class2.newfunc = f1
> c2 = Class2()
> c2.newfunc('f1 assigned to class')  # Complains about extra argument

But this does.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How/where to store calibration values - written by program A, read by program B

[Peter J. Holzer via Python-list]

On 2023-12-29 09:01:24 -0800, Grant Edwards via Python-list wrote:
> On 2023-12-28, Peter J. Holzer via Python-list  wrote:
> > On 2023-12-28 05:20:07 +, rbowman via Python-list wrote:
> >> On Wed, 27 Dec 2023 03:53:42 -0600, Greg Walters wrote:
> >> > The biggest caveat is that the shared variable MUST exist before it can
> >> > be examined or used (not surprising).
> >> 
> >> There are a few other questions. Let's say config.py contains a variable 
> >> like 'font' that is a user set preference or a calibration value 
> >> calculated by A to keep with the thread title. Assuming both scripts are 
> >> running, how does the change get propagated to B after it is set in A
> >
> > It isn't. The variable is set purely in memory. This is a mechanism to
> > share a value between multiple modules used by the same process, not to
> > share between multiple processes (whether they run the same or different
> > scripts)
> >
> >> and written to the shared file?
> >
> > Nothing is ever written to a file.
> 
> Then how does it help the OP to propogate clibration values from one
> program to another or from one program run to the next run?

It doesn't. See his second mail in this thread, where he explains it in
a bit more detail. I think he might be a bit confused in his
terminology.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How/where to store calibration values - written by program A, read by program B

[Peter J. Holzer via Python-list]

On 2023-12-28 05:20:07 +, rbowman via Python-list wrote:
> On Wed, 27 Dec 2023 03:53:42 -0600, Greg Walters wrote:
> > The biggest caveat is that the shared variable MUST exist before it can
> > be examined or used (not surprising).
> 
> There are a few other questions. Let's say config.py contains a variable 
> like 'font' that is a user set preference or a calibration value 
> calculated by A to keep with the thread title. Assuming both scripts are 
> running, how does the change get propagated to B after it is set in A

It isn't. The variable is set purely in memory. This is a mechanism to
share a value between multiple modules used by the same process, not to
share between multiple processes (whether they run the same or different
scripts)

> and written to the shared file?

Nothing is ever written to a file.

You could of course write python files from a python script (in fact I
do this), but that's not what this pattern is about, AFAICS.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more

[Peter J. Holzer via Python-list]

On 2023-12-22 22:56:45 -0500, Thomas Passin via Python-list wrote:
> In my experience one should always make sure to know what version of Python
> is being used, at least if there is more than one version installed on the
> computer.  Even on Linux using a shebang line can be tricky, because you are
> likely to get the system's version of Python,

You are not "likely" to get the system's version of Python, you get the
version of Python you specify. If you specify "/usr/bin/python3", that's
the system's version of Python. If you specify something else, you get
something else. If you specify "/usr/bin/env python3", you get whatever
the user has in their PATH first.


> and that often is not what you want.  OTOH you don't want to go
> symlinking python3 to some other version of python because then the OS
> system may not work right.  So either you have to specify the Python
> version in the shebang,

This. In my considered opinion installed scripts should work regardless
pf the user's PATH, so they must have the correct interpreter in the
shebang. That specifying the correct shebang pulls in the complete
virtual environment is IMHO a great feature of Python.

I've written a small script "install-python" which basically just copies
a file and adjusts the shebang line.

for the use in Makefiles etc.

> or just specify the right version
> on the command line.  In that case you might as well not have included the
> shebang line at all.

Right. However, that's not how scripts are usually invoked on Unix.
Using /usr/bin/env in the command line is supposed to fix that but of
course it assumes that your interpreter is actually called python3.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How/where to store calibration values - written by program A, read by program B

[Peter J. Holzer via Python-list]

On 2023-12-06 07:23:51 -0500, Thomas Passin via Python-list wrote:
> On 12/6/2023 6:35 AM, Barry Scott via Python-list wrote:
> > Personally I would not use .ini style these days as the format does not 
> > include type of the data.
> 
> Neither does JSON.

Well, it distinguishes between some primitive types (string, number,
boolean, null) and provides two container types (dict/object,
list/array). As long as those types are sufficient, JSON includes them.
If you need anything else, you're on your own.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Peter J. Holzer via Python-list]

On 2023-11-25 08:32:24 -0600, Michael F. Stemper via Python-list wrote:
> On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote:
> > Of course, for serious work, some might suggest avoiding constructs like a
> > list of lists and switch to using modules and data structures [...]
> 
> Those who would recommend that approach do not appear to include Mr.
> Rossum, who said:
>   Avoid overengineering data structures.
  ^^^

The key point here is *over*engineering. Don't make things more
complicated than they need to be. But also don't make them simpler than
necessary.

>   Tuples are better than objects (try namedtuple too though).

If Guido thought that tuples would always be better than objects, then
Python wouldn't have objects. Why would he add such a complicated
feature to the language if he thought it was useless?

The (unspoken?) context here is "if tuples are sufficient, then ..."

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[Peter J. Holzer via Python-list]

On 2023-11-17 07:48:41 -0500, Thomas Passin via Python-list wrote:
> On 11/17/2023 6:17 AM, Peter J. Holzer via Python-list wrote:
> > Oh, and Python (just like Perl) allows you to embed whitespace and
> > comments into Regexps, which helps readability a lot if you have to
> > write long regexps.
> > 
[...]
> > > > > re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)
> > 
> > \b - a word boundary.
> > [0-9]{2,7} - 2 to 7 digits
> > -  - a hyphen-minus
> > [0-9]{2}   - exactly 2 digits
> > -  - a hyphen-minus
> > [0-9]{2}   - exactly 2 digits
> > \b - a word boundary.
> > 
> > Seems quite straightforward to me. I'll be impressed if you can write
> > that in Python in a way which is easier to read.
> 
> And the re.VERBOSE (also re.X) flag can always be used so the entire
> expression can be written line-by-line with comments nearly the same
> as the example above

Yes. That's what I alluded to above.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[Peter J. Holzer via Python-list]

On 2023-11-16 11:34:16 +1300, Rimu Atkinson via Python-list wrote:
> > > Why don't you use re.findall?
> > > 
> > > re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)
> > 
> > I think I can see what you did there but it won't make sense to me - or
> > whoever looks at the code - in future.
> > 
> > That answers your specific question. However, I am in awe of people who
> > can just "do" regular expressions and I thank you very much for what
> > would have been a monumental effort had I tried it.
> 
> I feel the same way about regex. If I can find a way to write something
> without regex I very much prefer to as regex usually adds complexity and
> hurts readability.

I find "straight" regexps very easy to write. There are only a handful
of constructs which are all very simple and you just string them
together. But then I've used regexps for 30+ years, so of course they
feel natural to me.

(Reading regexps may be a bit harder, exactly because they are to
simple: There is no abstraction, so a complicated pattern results in a
long regexp.)

There are some extensions to regexps which are conceptually harder, like
lookahead and lookbehind or nested contexts in Perl. I may need the
manual for those (especially because they are new(ish) and every
language uses a different syntax for them) or avoid them altogether.

Oh, and Python (just like Perl) allows you to embed whitespace and
comments into Regexps, which helps readability a lot if you have to
write long regexps.


> You might find https://regex101.com/ to be useful for testing your regex.
> You can enter in sample data and see if it matches.
> 
> If I understood what your regex was trying to do I might be able to suggest
> some python to do the same thing. Is it just removing numbers from text?

Not "removing" them (as I understood it), but extracting them (i.e. find
and collect them).

> > > re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)

\b - a word boundary.
[0-9]{2,7} - 2 to 7 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
\b - a word boundary.

Seems quite straightforward to me. I'll be impressed if you can write
that in Python in a way which is easier to read.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: xor operator

[Peter J. Holzer via Python-list]

On 2023-11-15 12:26:32 +0200, Dom Grigonis wrote:
> 
> Thank you,
> 
> 
> test2 = [True] * 100 + [False] * 2
> test2i = list(range(100))
> 
> %timeit len(set(test2i)) == 1   # 1.6 µs ± 63.6 ns per loop (mean ± std. dev. 
> of 7 runs, 1,000,000 loops each)
> %timeit all(test2)  # 386 ns ± 9.58 ns per loop (mean ± std. dev. 
> of 7 runs, 1,000,000 loops each)
> 
> test2s = set(test2i)
> %timeit len(test2s) == 1# 46.1 ns ± 1.65 ns per loop (mean ± std. 
> dev. of 7 runs, 10,000,000 loops each)
> 
> If you pre-convert to set it is obviously faster. However, set
> operation is most likely going to be part of the procedure. In which
> case it ends up to be significantly slower.

Obviously, if you convert a list to a set just to count the elements
it's going to be slow. My suggestion was to use the set *instead* of the
list. I don't know whether that's possible in your situation, because
you haven't told us anything about it. All I'm suggesting is taking a
step back and reconsider your choice of data structure.

hp
-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: xor operator

[Peter J. Holzer via Python-list]

On 2023-11-14 00:11:30 +0200, Dom Grigonis via Python-list wrote:
> Benchmarks:
> test1 = [False] * 100 + [True] * 2
> test2 = [True] * 100 + [False] * 2
> 
> TIMER.repeat([
> lambda: xor(test1), # 0.0168
> lambda: xor(test2), # 0.0172
> lambda: xor_ss(test1),  # 0.1392
> lambda: xor_ss(test2),  # 0.0084
> lambda: xor_new(test1), # 0.0116
> lambda: xor_new(test2), # 0.0074
> lambda: all(test1), # 0.0016
> lambda: all(test2)  # 0.0046
> ])
> Your first function is fairly slow.
> Second one deals with short-circuiting, but is super slow on full search.
> 
> `xor_new` is the best what I could achieve using python builtins.
> 
> But builtin `all` has the best performance.

One question worth asking is if a list of bool is the best data
structure for the job. This is essentially a bitmap, and a bitmap is
equivalent to a set of integers. len(s) == 1 is also a fairly quick
operation if s is small. On my system, len(test1s) == 1 (where test1s is
{100, 101}) is about as fast as all(test1) and len(test2s) == 1 (where
test2s is set(range(100))) is about twice as fast as all(test2).

If you are willing to stray from the standard library, you could e.g.
use pyroaring instead of sets: This is about as fast as all(test1)
whether there are two bits set or a hundred.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.12.0 venv not working with psycopg2

[Peter J. Holzer via Python-list]

On 2023-10-02 19:44:12 +0300, אורי via Python-list wrote:
> I have an issue since about 5 months now. Python 3.12.0 venv not working
> with psycopg2 on Windows. I created 2 issues on GitHub but they were
> closed. I checked today with the new Python release but it's still not
> working.
> 
> https://github.com/psycopg/psycopg2/issues/1578
> https://github.com/python/cpython/issues/104830

You wil have to come up with a *minimal* test case which reproduces the
problem. Expecting people to download and test your massive application
is unreasonable.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: path to python in venv

[Peter J. Holzer via Python-list]

On 2023-09-27 20:32:25 -, Jon Ribbens via Python-list wrote:
> On 2023-09-27, Larry Martell  wrote:
> > On Wed, Sep 27, 2023 at 12:42 PM Jon Ribbens via 
> > Python-list wrote:
> >> On 2023-09-27, Larry Martell  wrote:
> >> > lrwxrwxrwx 1 larrymartell larrymartell7 Sep 27 11:21 python -> 
> >> > python3
> >> > lrwxrwxrwx 1 larrymartell larrymartell   16 Sep 27 11:21 python3 -> 
> >> > /usr/bin/python3
[...]
> I'm a bit surprised your symlinks are as shown above though - mine
> link from python to python3.11 to /usr/bin/python3.11, so it wouldn't
> change the version of python used even if I installed a different
> system python version.

That's probably because you created the venvs with "python3.11 -m venv ...".
The symlink points to the command you used to create it:

% python3 -m venv venv
% ll venv/bin/python*
lrwxrwxrwx 1 hjp hjp  7 Aug 29  2022 venv/bin/python -> python3*
lrwxrwxrwx 1 hjp hjp 12 Aug 29  2022 venv/bin/python3 -> /bin/python3*
lrwxrwxrwx 1 hjp hjp  7 Aug 29  2022 venv/bin/python3.10 -> python3*

% python3.10 -m venv venv
% ll venv/bin/python*
lrwxrwxrwx 1 hjp hjp 10 Sep 28 00:45 venv/bin/python -> python3.10*
lrwxrwxrwx 1 hjp hjp 10 Sep 28 00:45 venv/bin/python3 -> python3.10*
lrwxrwxrwx 1 hjp hjp 15 Sep 28 00:45 venv/bin/python3.10 -> /bin/python3.10*

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

dateutil on PyPI (was: PEP668 / pipx and "--editable" installs)

[Peter J. Holzer via Python-list]

On 2023-09-20 13:31:14 +, c.buhtz--- via Python-list wrote:
> Dear Peter,
> 
> maybe we have a missunderstanding.
> 
> Am 20.09.2023 14:43 schrieb Peter J. Holzer via Python-list:
> > > > > "dateutil" is not available from PyPi for Python 3.11
> > 
> > That's quite a curious thing to write if you are aware that dateutil is
> > in fact available from PyPi for Python 3.11.
> 
> Do I miss something here?
> 
> See https://pypi.org/project/dateutils/ and also the open Issue about the
> missing support for Python 3.11

You are missing at least an "s". "dateutils" is not "dateutil".

But dateutils can also be installed from PyPI.

> https://github.com/dateutil/dateutil/issues/1233 ?

So it hasn't been tagged with any version >= 3.10 yet. That doesn't mean
it isn't available. It requires a Python version >= 3.3, but 3.11 is >=
3.3, so that's not a problem.

As I demonstrated, it installs just fine.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs

[Peter J. Holzer via Python-list]

On 2023-09-18 18:56:35 +, c.buhtz--- via Python-list wrote:
> On 2023-09-18 10:16 "Peter J. Holzer via Python-list"
>  wrote:
> > On 2023-09-15 14:15:23 +, c.buhtz--- via Python-list wrote:
> > > I tried to install it via "pipx install -e .[develop]". It's
> > > pyproject.toml has a bug: A missing dependency "dateutil". But
> > > "dateutil" is not available from PyPi for Python 3.11 (the default
> > > in Debian 12). But thanks to great Debian they have a
> > > "python3-dateutil" package. I installed it.
> > 
> > This can be installed via pip:
> 
> I'm aware of this.

You wrote:

> > > "dateutil" is not available from PyPi for Python 3.11

That's quite a curious thing to write if you are aware that dateutil is
in fact available from PyPi for Python 3.11.

> But this is not the question.

I know. That's why I labeled my comment as a "Sidenote".

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs

[Peter J. Holzer via Python-list]

On 2023-09-15 14:15:23 +, c.buhtz--- via Python-list wrote:
> I tried to install it via "pipx install -e .[develop]". It's pyproject.toml
> has a bug: A missing dependency "dateutil". But "dateutil" is not available
> from PyPi for Python 3.11 (the default in Debian 12). But thanks to great
> Debian they have a "python3-dateutil" package. I installed it.

Sidenote:
PyPI does have several packages with "dateutil" in their name. From the
version number (2.8.2) I guess that "python-dateutil" is the one
packaged in Debian 12.

This can be installed via pip:

% lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:Debian GNU/Linux 12 (bookworm)
Release:12
Codename:   bookworm

(dateutil) % pip install python-dateutil
Collecting python-dateutil
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
  247.7/247.7 kB 3.1 MB/s eta 
0:00:00
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, python-dateutil
Successfully installed python-dateutil-2.8.2 six-1.16.0

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Postgresql equivalent of Python's timeit?

[Peter J. Holzer via Python-list]

On 2023-09-17 11:01:43 +0200, Albert-Jan Roskam via Python-list wrote:
>On Sep 15, 2023 19:45, "Peter J. Holzer via Python-list"
> wrote:
> 
>  On 2023-09-15 17:42:06 +0200, Albert-Jan Roskam via Python-list wrote:
>  >    This is more related to Postgresql than to Python, I hope this is
>  ok.
>  >    I want to measure Postgres queries N times, much like Python timeit
>  >    (https://docs.python.org/3/library/timeit.html). I know about
>  EXPLAIN
>  >    ANALYZE and psql \timing, but there's quite a bit of variation in
>  the
>  >    times. Is there a timeit-like function in Postgresql?
> 
>  Why not simply call it n times from Python?
> 
>  (But be aware that calling the same query n times in a row is likely to
>  be
>  unrealistically fast because most of the data will already be in
>  memory.)
> 
>=
>Thanks, I'll give this a shot. Hopefully the caching is not an issue if I
>don't re-use the same database connection.

There is some per-session caching, but the bulk of it is shared between
sessions or even in the operating system. And you wouldn't want to get
rid of these caches either (which you could do by rebooting or - a bit
faster - restarting postgres and dropping the caches
(/proc/sys/vm/drop_caches on Linux), because that would make the
benchmark unrealistically slow (unless you want to establish some
worst-case baseline). During normal operations some data will be cached,
but probably not all of it and it will change depending on workload and
possibly other factors.

I think Avi's advice to wait for a few minutes between repetitions is
good. Of course that means that you can't just time the whole thing but
have to time each query separately and then compute the average. (On the
bright side that also gives you the opportunity to compute standard
deviation, min, max, quantiles, etc.)

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: `time.perf_counter_ns` always a 64-bit int?

[Peter J. Holzer via Python-list]

On 2023-09-15 21:48:37 +, rmlibre--- via Python-list wrote:
> I'd like to capture the output of `time.perf_counter_ns()` as an 8-byte
> timestamp.
> 
> I'm aware that the docs provide an undefined start value for that clock.
> I'm going to assume that means it can't be expected to fit within 8
> bytes.

Theoretically this is true. The reference point could be the switch to
the Gregorian calendar in the Vatican, the begin of the Christian era or
the founding of Babylon, all of which were more than 2**63 seconds ago.
However, using one of these dates would be impractical and defeat the
purpose of the performance counters, which are supposed to be high
resolution, monotonic and independent of political influences. So the
reference point is usually the time the system was booted or something
similar.

> However, it would be rather convenient if it could.

Unless you expect your system to have an uptime in excess of 292 years,
don't worry.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Postgresql equivalent of Python's timeit?

[Peter J. Holzer via Python-list]

On 2023-09-15 17:42:06 +0200, Albert-Jan Roskam via Python-list wrote:
>This is more related to Postgresql than to Python, I hope this is ok.
>I want to measure Postgres queries N times, much like Python timeit
>(https://docs.python.org/3/library/timeit.html). I know about EXPLAIN
>ANALYZE and psql \timing, but there's quite a bit of variation in the
>times. Is there a timeit-like function in Postgresql?

Why not simply call it n times from Python?

(But be aware that calling the same query n times in a row is likely to be
unrealistically fast because most of the data will already be in
memory.)

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Passing info to function used in re.sub

[Peter J. Holzer via Python-list]

On 2023-09-03 18:10:29 +0200, Jan Erik Moström via Python-list wrote:
> I want to replace some text using a regex-pattern, but before creating
> replacement text I need to some file checking/copying etc. My code
> right now look something like this:
> 
> def fix_stuff(m):
>   # Do various things that involves for info
>   # that what's available in m
>   replacement_text = m.group(1) + global_var1 + global_var2
>   return replacement_text
> 
> and the call comes here
> 
> global_var1 = "bla bla"
> global_var2 = "pff"
> 
> new_text = re.sub(im_pattern,fix_stuff,md_text)
> 
> 
> The "problem" is that I've currently written some code that works but
> it uses global variables ... and I don't like global variables. I
> assume there is a better way to write this, but how?

If you use fix_stuff only inside one other function, you could make it
local to that function so that it will capture the local variables of
the outer function:

import re

def demo():

local_var1 = "bla bla"
local_var2 = "pff"

def fix_stuff(m):
# Do various things that involves for info
# that what's available in m
replacement_text = m.group(1) + local_var1 + local_var2
return replacement_text

for md_text in ( "aardvark", "barbapapa", "ba ba ba ba barbara ann"):
new_text = re.sub(r"(a+).*?(b+)", fix_stuff, md_text)
print(md_text, new_text)

demo()

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What sort of exception when a class can't find something?

[Peter J. Holzer via Python-list]

On 2023-08-31 21:32:04 +0100, Chris Green via Python-list wrote:
> What sort of exception should a class raise in __init__() when it
> can't find an appropriate set of data for the parameter passed in to
> the class instantiation?
> 
> E.g. I have a database with some names and address in and have a
> class Person that gets all the details for a person given their
> name.
> 
>  
>  
>  person.Person('Fred')
>  ...
>  ...
> 
> 
> If Fred doesn't exist in the database what sort of exception should
> there be?  Is it maybe a ValueError?

It you are going for a builtin exception, I think KeyError is the most
appropriate: It should be a LookupError, since the lookup failed and a
database is more like a mapping than a sequence.

But it would probably be best to define your own exception for that.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using "textwrap" package for unwrappable languages (Japanese)

[Peter J. Holzer via Python-list]

On 2023-08-30 13:18:25 +, c.buhtz--- via Python-list wrote:
> Am 30.08.2023 14:07 schrieb Peter J. Holzer via Python-list:
> > another caveat: Japanese characters are usually double-width. So
> > (unless your line length is 130 characters for English) you would
> > want to add that line break every 32 characters.
> 
> I don't get your calculation here. Original line length is 130 but for
> "double-with" characters you would break at 32 instead of 65 ?

No, I wrote "*unless* your original line length was 130 characters".

I assumed that you want your line to be 65 latin characters wide since
this is what fits nicely on an A4 (or letter) page with a bit of a
margin on both sides. Or on an 80 character terminal screen or window.
And it's also generally considered to be a good line length for
readability.

But Asian "full width" or "wide" characters are twice as wide, so you
can fit only half as many in a single line. Hence 65 // 2 = 32.

But that was only my assumption. I considered it possible that you
started with 130 characters per line (many terminals back in the day had
a 132 character mode, and that's also approximately the line length in
landscape mode or when using a compressed typeface - so 132 is also a
common length limit, although rarely for text (too wide to read
comfortably) and more for code, tables, etc.), divided that by two and
arrived at 65 Japanese characters per line that way. So I mentioned that
to indicate that I had considered the possibility but concluded that it
probably wasn't what you meant.

(And as usual when I write a short sentence to clarify something
I wind up writing 4 paragraphs clarifying the clarification :-/)

> Then I will do something like this
> 
> unicodedata.east_asian_width(mystring[0])
> 
> W is "wide". But there is also "F" (full-width).
> What is the difference between "wide" and "full-width"?

I'm not an expert on Japanese typography by any means. But they have
some full width variants of latin characters and halfwidth variants of
katakana characters. I assume that the categories 'F' and 'H' are for
those, while "normal" Japanese characters are "W":

>>> unicodedata.east_asian_width("\N{DIGIT ONE}")
'Na'
>>> unicodedata.east_asian_width("\N{FULLWIDTH DIGIT ONE}")
'F'
>>> unicodedata.east_asian_width("\N{KATAKANA LETTER ME}")
'W'
>>> unicodedata.east_asian_width("\N{HALFWIDTH KATAKANA LETTER ME}")
'H'

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using "textwrap" package for unwrappable languages (Japanese)

[Peter J. Holzer via Python-list]

On 2023-08-30 11:32:02 +, c.buhtz--- via Python-list wrote:
> I do use "textwrap" package to wrap longer texts passages. Works well with
> English.
> But the source string used is translated via gettext before it is wrapped.
> 
> Using languages like Japanese or Chinese would IMHO result in unwrapped
> text. Japanese rules do allow to break a line nearly where ever you want.
> 
> How can I handle it with "textwrap"?
> 
> At runtime I don't know which language is really used. So I'm not able to
> decide using "textwrap" or just inserting "\n" every 65 characters.

I don't have a solution but want to add another caveat: Japanese
characters are usually double-width. So (unless your line length is 130
characters for English) you would want to add that line break every 32
characters. (unicodedata.east_asian_width() seems to be the canonical
name to find the width of a character, but it returns a code (like 'W'
or 'Na') not a number.)

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Where is the error?

[Peter J. Holzer via Python-list]

Mostly, error messages got a lot better in Python 3.10, but this one had
me scratching my head for a few minutes.

Consider this useless and faulty script:


r = {
"x": (1 + 2 + 3)
"y": (4 + 5 + 6)
"z": (7 + 8 + 9)
}


Python 3.9 (and earlier) reports:


  File "/home/hjp/tmp/foo", line 3
"y": (4 + 5 + 6)
^
SyntaxError: invalid syntax


This isn't great, but experience with lots of programming languages
tells me that an error is noticed where or after it actually occurs, so
it's easy to see that there is a comma missing just before the "y".

Python 3.10 and 3.11 report:


  File "/home/hjp/tmp/foo", line 2
"x": (1 + 2 + 3)
  ^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?


The error message is now a lot better, of course, but the fact that it
points at the expression *before* the error completely threw me. The
underlined expression is clearly not missing a comma, nor is there an
error before that. My real program was a bit longer of course, so I
checked the lines before that to see if I forgot to close any
parentheses. Took me some time to notice the missing comma *after* the
underlined expression.

Is this "clairvoyant" behaviour a side-effect of the new parser or was
that a deliberate decision?

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug in io.TextIOWrapper?

[Peter J. Holzer via Python-list]

On 2023-06-20 02:15:00 +0900, Inada Naoki via Python-list wrote:
> stream.flush() doesn't mean final output.
> Try stream.close()

After close() the value isn't available any more:

Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import io
>>> buffer = io.BytesIO()
>>> stream = io.TextIOWrapper(buffer, encoding='idna')
>>> stream.write('abc.example.com')
15
>>> stream.close()
>>> buffer.getvalue()
Traceback (most recent call last):
  File "", line 1, in 
ValueError: I/O operation on closed file.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list