Re: Anonymous email users

[Barry Scott via Python-list]

> On 23 Jun 2024, at 06:58, Sebastian Wells via Python-list 
>  wrote:
> 
> The spammers won the spam wars, so even if you have someone's real
> e-mail address, that's no guarantee that you can contact them. You
> certainly wouldn't be able to contact me at my real e-mail address,
> unless you also had my phone number, so you could call me and tell
> me that you sent me an e-mail, and what the subject line was so I
> can find it. I don't even open my e-mail inbox unless there's a
> specific message I'm expecting to find there right now.

My email address is well known and yes I get spam emails.

I use the wonderful python based spambayes software to detect spam and
file into a Junk folder. It works for 99.9% of the emails I get.

I am subscribed to a lot of mailing lists. I just checked and I am getting 
~3,200
emails a month of which less than 200 are spam.

A few years ago the spam count was greater than a 1,000 a month.

I have been using spambayes for a very long time, 20 years I guess at this
point and bayesian categorisation has stood the test of time for me.

For me the spammers have not won, I have the tech to keep ahead of them.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Suggested python feature: allowing except in context maneger

[Barry Scott via Python-list]

> On 13 Jun 2024, at 11:01, Yair Eshel via Python-list  
> wrote:
> 
> I read this is a good place to give some suggestions for features in
> python.

Best place these days is to raise an idea on https://discuss.python.org/

Beware that this idea has come up in the past and was rejected.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Flubbed it in the second interation through the string: range error... HOW?

[Barry Scott via Python-list]

> On 29 May 2024, at 05:38, Kevin M. Wilson via Python-list 
>  wrote:
> 
> The format in this email is not of my making, should someone know, how to do 
> this so that it's a readable script do tell!
> KMW


Your mail program may have a plain-text mode to compose messages in try using 
that.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Making 'compiled' modules work with multiple python versions on Linux

[Barry Scott via Python-list]

> On 29 Mar 2024, at 16:09, Olivier B.  
> wrote:
> 
> It is not a symlink on my system, where i built python myself, but a
> 15KB so file. But it seems to lack lots of python symbols.
> 
> Maybe what i should do is actually make libpython.so a physical copy
> of libpyton311.so before linking to it, so now on any system the
> module would look to load libpython.so, which could be pointing to any
> version. I'll try that next

You do not link against the .so at all. All the symbols you need are defined in 
the
python process that loads the extension. Try without the -lpython and it should
just work.

Barry


> 
> Le ven. 29 mars 2024 à 10:10, Barry  a écrit :
>> 
>> 
>> 
>>> On 28 Mar 2024, at 16:13, Olivier B. via Python-list 
>>>  wrote:
>>> 
>>> But on Linux, it seems that linking to libpython3.so instead of
>>> libpython3.11.so.1.0 does not have the same effect, and results in
>>> many unresolved python symbols at link time
>>> 
>>> Is this functionality only available on Windows?
>> 
>> Python limited API works on linux, but you do not link against the .so on 
>> linux I recall.
>> 
>> You will have missed that libpython3.so is a symlink to libpython3.11.so.10.
>> 
>> Windows build practices do not translate one-to-one to linux, or macOS.
>> 
>> Barry
>> 
>> 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Feature proposal: unittest.mock.NAN

[Barry Scott via Python-list]

Python ideas are discussed here these days: https://discuss.python.org/
Suggest you raise this there in the Ideas category.

Barry


> On 24 Jan 2024, at 17:11, Kerrick Staley via Python-list 
>  wrote:
> 
> I think we should define a unittest.mock.NAN constant that can be used with
> Mock.assert_called_with() to assert that an argument passed to a Mock was
> NaN. NaNs are special in that math.nan != math.nan, so you can't just do
> assert_called_with(math.nan). The naming is meant to parallel
> unittest.mock.ANY.
> 
> Here is a reference implementation:
> 
> class _EqNaN:
>def __eq__(self, other):
>return math.isnan(other)
> 
> NAN = _EqNaN()
> 
> The alternative is that users can just define this EqNaN class themselves
> as needed in test code. I encountered the need to test for a NaN argument
> today and was surprised to find that (as far as I can tell) there is no
> pre-built solution to this in unittest or pytest. It feels like it should
> be included in some standard library.
> 
> - Kerrick
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more

[Barry Scott via Python-list]

> On 7 Jan 2024, at 15:09, Sibylle Koczian via Python-list 
>  wrote:
> 
> Oh, and the two Windows and Python versions are on two different computers.
> 
> Will remove the "/env" from my shebang lines, even if I don't understand 
> what's happening.

Thanks for the details.

Only thing I can think of is that "python" may be defaulting to mean python 2.
If you use "#!/usr/bin/env python3" it may work on both.

Did you creates a py.ini file to configure py.exe?

See if you have %userappdata%\py.ini on either windows 10 or windows 11.
If so what is its contents?

I've tried with and without a py.ini and cannot duplicate what you see.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more

[Barry Scott via Python-list]

> On 2 Jan 2024, at 17:24, Thomas Passin via Python-list 
>  wrote:
> 
> You might learn about this if you happen to read and remember the right part 
> of the Python docs.  Otherwise you have no idea what py.exe is up to nor how 
> it does it.  I would say that most people don't know there's an ini file, let 
> alone what it can do.  Of course this situation isn't unique to py.exe!

On discuss.python.org we will share the link to the python windows docs and
often to this specific section: 
https://docs.python.org/3/using/windows.html#python-launcher-for-windows

To create a py.ini does require a user to understand what %localappdata% means.
Some windows users, reasonable, do not use the CLI and know about that syntax 
for environment variables.

Barry


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more

[Barry Scott via Python-list]

> On 23 Dec 2023, at 03:01, Thomas Passin via Python-list 
>  wrote:
> 
> Not on my system. It may depend on whether Python gets installed to Program 
> Files or to %USERPROFILE%/AppData/Local/Programs/Python.  Python 3.9 is the 
> last verson I installed to Program Files, and that's the version that Windows 
> thinks it should use to run Python files.
> 
> Run the little test program I posted.  That will tell you which version of 
> Python the system wants to use.

I always install for all users and this what I get on my Windows 10 and 11 
systems.
As you can see the shebang lines do what is expected based on the config of 
py.exe.

Maybe it works differently if you install for a single user only,
I do not have such a setup to test with.

Windows 10 output.

K:\shebang>py -0 
 -V:3.13  Python 3.13 (64-bit)
 -V:3.13-32   Python 3.13 (32-bit)
 -V:3.12 *Python 3.12 (64-bit)
 -V:3.12-32   Python 3.12 (32-bit)
 -V:3.11  Python 3.11 (64-bit)
 -V:3.11-32   Python 3.11 (32-bit)
 -V:3.10  Python 3.10 (64-bit)
 -V:3.10-32   Python 3.10 (32-bit)
 -V:3.9   Python 3.9 (64-bit)
 -V:3.9-32Python 3.9 (32-bit)
 -V:3.8   Python 3.8 (64-bit)
 -V:3.8-32Python 3.8 (32-bit)
 -V:3.7   Python 3.7 (64-bit)
 -V:3.7-32Python 3.7 (32-bit)
 -V:3.6   Python 3.6 (64-bit)
 -V:3.6-32Python 3.6 (32-bit)
 -V:3.5   Python 3.5
 -V:3.5-32Python 3.5-32
 -V:3.4   Python 3.4
 -V:3.4-32Python 3.4-32
 -V:2.7   Python 2.7
 -V:2.7-32Python 2.7-32

K:\shebang>type shebang_py2.py 
#!/usr/bin/python2
from __future__ import print_function
import sys
print('I am python %r' % (sys.version_info,))

K:\shebang>py shebang_py2.py 
I am python sys.version_info(major=2, minor=7, micro=17, releaselevel='final', 
serial=0)

K:\shebang>shebang_py2.py
I am python sys.version_info(major=2, minor=7, micro=17, releaselevel='final', 
serial=0)

K:\shebang>type shebang_py3.py 
#!/usr/bin/python3
from __future__ import print_function
import sys
print('I am python %r' % (sys.version_info,))

K:\shebang>py shebang_py3.py 
I am python sys.version_info(major=3, minor=12, micro=1, releaselevel='final', 
serial=0)

K:\shebang>shebang_py3.py
I am python sys.version_info(major=3, minor=12, micro=1, releaselevel='final', 
serial=0)

K:\shebang>type shebang_env_py3.py 
#!/usr/bin/env python3
from __future__ import print_function
import sys
print('I am python %r' % (sys.version_info,))

K:\shebang>py shebang_env_py3.py 
I am python sys.version_info(major=3, minor=12, micro=1, releaselevel='final', 
serial=0)

K:\shebang>shebang_env_py3.py
I am python sys.version_info(major=3, minor=12, micro=1, releaselevel='final', 
serial=0)

K:\shebang>type shebang_env_py3_10.py 
#!/usr/bin/env python3.10
from __future__ import print_function
import sys
print('I am python %r' % (sys.version_info,))

K:\shebang>py shebang_env_py3_10.py 
I am python sys.version_info(major=3, minor=10, micro=11, releaselevel='final', 
serial=0)

K:\shebang>shebang_env_py3_10.py
I am python sys.version_info(major=3, minor=10, micro=11, releaselevel='final', 
serial=0)

K:\shebang>assoc .py 
.py=Python.File

K:\shebang>ftype Python.File 
Python.File="C:\WINDOWS\py.exe" "%L" %*


Windows 11 output


: 11:52:10.36 K:\shebang
: \\BARNSTONE\barry> py -0 
 -V:3.12 *Python 3.12 (64-bit)
 -V:3.12-32   Python 3.12 (32-bit)
 -V:3.11  Python 3.11 (64-bit)
 -V:3.11-32   Python 3.11 (32-bit)
 -V:3.10  Python 3.10 (64-bit)
 -V:3.9   Python 3.9 (64-bit)
 -V:3.9-32Python 3.9 (32-bit)
 -V:3.8   Python 3.8 (64-bit)
 -V:3.8-32Python 3.8 (32-bit)

: 11:52:10.40 K:\shebang
: \\BARNSTONE\barry> type shebang_py2.py 
#!/usr/bin/python2
from __future__ import print_function
import sys
print('I am python %r' % (sys.version_info,))

: 11:52:10.41 K:\shebang
: \\BARNSTONE\barry> py shebang_py2.py 
No suitable Python runtime found
Pass --list (-0) to see all detected environments on your machine
or set environment variable PYLAUNCHER_ALLOW_INSTALL to use winget
or open the Microsoft Store to the requested version.

: 11:52:10.47 K:\shebang
: \\BARNSTONE\barry> type shebang_py3.py 
#!/usr/bin/python3
from __future__ import print_function
import sys
print('I am python %r' % (sys.version_info,))

: 11:52:10.49 K:\shebang
: \\BARNSTONE\barry> py shebang_py3.py 
I am python sys.version_info(major=3, minor=12, micro=0, releaselevel='final', 
serial=0)

: 11:52:10.52 K:\shebang
: \\BARNSTONE\barry> shebang_py3.py
I am python sys.version_info(major=3, minor=12, micro=0, releaselevel='final', 
serial=0)

: 11:52:10.58 K:\shebang
: \\BARNSTONE\barry> type shebang_env_py3.py 
#!/usr/bin/env python3
from __future__ import print_function
import sys
print('I am python %r' % (sys.version_info,))

: 11:52:10.60 K:\shebang
: \\BARNSTONE\barry> py shebang_env_py3.py 
I am python sys.version_info(major=3, minor=12, micro=0, 

Re: making your own DirEntry.

[Barry Scott via Python-list]

> On 23 Dec 2023, at 09:48, Antoon Pardon via Python-list 
>  wrote:
> 
> Because I have functions with DirEntry parameters.

I would duck-type a class I control to be my DirEnrry in this situation.
Would also help you when debugging as you can tell injected DirEntry from 
"real" DirEntry.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How/where to store calibration values - written by program A, read by program B

[Barry Scott via Python-list]

> On 6 Dec 2023, at 09:32, Chris Green via Python-list  
> wrote:
> 
> My requirement is *slightly* more complex than just key value pairs,
> it has one level of hierarchy, e.g.:-
> 
>KEY1:
>  a: v1
>  c: v3
>  d: v4
>KEY2:
>  a: v7
>  b: v5
>  d: v6
> 
> Different numbers of value pairs under each KEY.

JSON will allow you to nest dictionaries.

{
'KEY1': {
'a': v1
'c': v3
'd': v4
}
'KEY2': {
 'a': v7
 'b': v5
 'd': v6
}
}

Personally I would not use .ini style these days as the format does not include 
type of the data.

Also I would not use the ast.literal_eval as it makes debugging errors in the 
data harder.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How/where to store calibration values - written by program A, read by program B

[Barry Scott via Python-list]

> On 5 Dec 2023, at 14:37, Chris Green via Python-list  
> wrote:
> 
> Are there any Python modules aimed specifically at this sort of
> requirement?

I tend to use JSON for this type of thing.
Suggest that you use the options to pretty print the json that is saved so that 
a human can read it.

For example:

import json

config = {
'key2': 42,
'key1': 'value 1',
}

print(json.dumps(config, indent=4, separators=(', ', ': '), sort_keys=True))


% python $T/a.py
{
"key1": "value 1",
"key2": 42
}

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Request: inspect: signature & getfullargspec & getcallargs

[Barry Scott via Python-list]

> On 4 Dec 2023, at 02:29, Dom Grigonis via Python-list 
>  wrote:
> 
> Hello,
> 
> I have a request.
> 
> Would it be possible to include `follow_wrapper_chains` and `skip_bound_arg` 
> arguments to higher level functions of `inspect` module?
> 
> Would exposing them, but setting defaults to what they currently are, be 
> possible?
> 
> I sometimes need:
>   * `getcallargs`, but without `bound_arg`
>   * `getfullargspec` to `follow_wrapper_chains`
>   * `signature` to include `bound_arg`.

I suspect that you need to raise this as an idea on https://discuss.python.org/ 
to get the attention of the core devs.

Barry

> 
> 
> Regards,
> DG
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Silly (maybe) question re imported module(s)

[Barry Scott]

> On 19 May 2023, at 07:44, Grizzy Adams via Python-list 
>  wrote:
> 
> Morning All
> 
> I'm working through the tutorial and running / saving work that I wish to 
> keep 
> and build on, most times I can save and (re)import later with no difference 
> to 
> when typed in console or editor and run with F5 (which saves before it can 
> run) 
> 
> 
> But sometimes saved work (albeit small) when imported does not work any longer
> 
> I assume I have missed something, any pointers to what/how/why please

reimport is not 100% reliable, in simple cases it will work, but as your
code gets more complex it can break.

I rarely use reimport. Usually just exit python and start again.

If you are working on scripts then just run them from the Terminal
will work well with any need to use reimport.

Barry

> 
> 
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Windows Gui Frontend

[Barry Scott]

> On 1 Apr 2023, at 18:47, Igor Korot  wrote:
> 
> I suggest going with wxPython/wxGlade.

I ported all my wxPython code to PyQt and have not regretted it.

wxPython was (its been a while so may not be an issue now) far to
hard to make consistent across OS, my apps run on Linux, macOS and Windows.
I found that PyQt was easier to get working.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python not showing correct version

[Barry Scott]

> On 31 Mar 2023, at 22:02, Eryk Sun  wrote:
> 
> 
> The OP installed the standard Python 3.8 distribution, which does
> install the launcher by default. The launcher can run all installed
> versions, including store app installations. By default it runs the
> highest available version, which will probably be the 3.10 store app
> in the OP's case. To make 3.8 the default without having to remove
> 3.10, set the environment variables "PY_PYTHON=3.8" and
> "PY_PYTHON3=3.8" in the user environment.

I find user environment on windows to be less flexible to work with then
adding a py.ini. On my Windows 11 I added %userprofile%\AppData\Local\py.ini.
To make python 3.8 the default that py.exe uses put this in py.ini:

[defaults]
python=3.8-64
python3=3.8-64

Barry


> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python not showing correct version

[Barry Scott]

> On 31 Mar 2023, at 09:33, Sumeet Firodia  wrote:
> 
> Thanks Barry.
> 
> One more thing is that pip --version also refers to python 3.10
> 
> C:\Users\admin>pip --version
> pip 23.0.1 from 
> C:\Users\admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\pip
>  (python 3.10)

Use this command to ensure that pip matches the python you wish to use:

py -3.8 -m pip --version

Barry

> 
> The issue here is I am trying to use snowpark for which I need python 3.8 but 
> because of this I am not able to proceed with next steps.
> 
> Thanks
> Sumeet
> 
> On Thu, 30 Mar 2023 at 22:45, Barry Scott  <mailto:ba...@barrys-emacs.org>> wrote:
>> 
>> 
>>> On 30 Mar 2023, at 15:17, Sumeet Firodia >> <mailto:ssfiro...@gmail.com>> wrote:
>>> 
>>>> 
>>>> Hi Team,
>>>> 
>>>> I have installed Python 3.8 for Snowpark but when I check the version in
>>>> command prompt it shows me Python 3.10.10.
>>>> 
>>>> C:\Users\admin>python --version
>>>> Python 3.10.10
>> 
>> Try this:
>> 
>> py -3.8
>> 
>> And this to list all version of python installed:
>> 
>> py -0
>> 
>> 
>>>> 
>>>> Also when I try to uninstall 3.10 it says no such version is installed.
>>>> 
>>>> C:\Users\admin>pip uninstall python 3.10.10
>>>> WARNING: Skipping python as it is not installed.
>>>> WARNING: Skipping 3.10.10 as it is not installed.
>> 
>> PIP is for installing python modules not the python program.
>> The python program is installed by running the .exe that you get from 
>> python.org <http://python.org/>, for example.
>> Use the standard Windows method to uninstall a program.
>> Search the web for "windows uninstall software" if you are not sure how to 
>> do this.
>> 
>> Barry
>> 
>>>> 
>>>> Can you please help me here as my snowpark assignment is stuck because of
>>>> this issue.
>>>> 
>>>> Thanks
>>>> Sumeet
>>>> 
>>>> 
>>> -- 
>>> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: built-in pow() vs. math.pow()

[Barry Scott]

> On 30 Mar 2023, at 18:11, Barry Scott  wrote:
> 
> 
> 
>> On 30 Mar 2023, at 10:15, Andreas Eisele  wrote:
>> 
>> I sometimes make use of the fact that the built-in pow() function has an 
>> optional third argument for modulo calculation, which is handy when dealing 
>> with tasks from number theory, very large numbers, problems from Project 
>> Euler, etc. I was unpleasantly surprised that math.pow() does not have this 
>> feature, hence "from math import *" overwrites the built-in pow() function 
>> with a function that lacks functionality. I am wondering for the rationale 
>> of this. Does math.pow() do anything that the built-in version can not do, 
>> and if not, why is it even there?
> 
> Maybe math.pow() aways using IEEE 754(?) float point via the C runtime math 
> routines.
> pow() will give int answer if called with int args.

And the C version of pow only supports 2 arg call.

SYNOPSIS
 #include 

 double
 pow(double x, double y);

 long double
 powl(long double x, long double y);

 float
 powf(float x, float y);


> 
> Barry
> 
> 
>> Thanks in advance for any enlightening comment on this.
>> Best regards, Andreas
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list
>> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python not showing correct version

[Barry Scott]

> On 30 Mar 2023, at 15:17, Sumeet Firodia  wrote:
> 
>> 
>> Hi Team,
>> 
>> I have installed Python 3.8 for Snowpark but when I check the version in
>> command prompt it shows me Python 3.10.10.
>> 
>> C:\Users\admin>python --version
>> Python 3.10.10

Try this:

py -3.8

And this to list all version of python installed:

py -0


>> 
>> Also when I try to uninstall 3.10 it says no such version is installed.
>> 
>> C:\Users\admin>pip uninstall python 3.10.10
>> WARNING: Skipping python as it is not installed.
>> WARNING: Skipping 3.10.10 as it is not installed.

PIP is for installing python modules not the python program.
The python program is installed by running the .exe that you get from 
python.org , for example.
Use the standard Windows method to uninstall a program.
Search the web for "windows uninstall software" if you are not sure how to do 
this.

Barry

>> 
>> Can you please help me here as my snowpark assignment is stuck because of
>> this issue.
>> 
>> Thanks
>> Sumeet
>> 
>> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: built-in pow() vs. math.pow()

[Barry Scott]

> On 30 Mar 2023, at 10:15, Andreas Eisele  wrote:
> 
> I sometimes make use of the fact that the built-in pow() function has an 
> optional third argument for modulo calculation, which is handy when dealing 
> with tasks from number theory, very large numbers, problems from Project 
> Euler, etc. I was unpleasantly surprised that math.pow() does not have this 
> feature, hence "from math import *" overwrites the built-in pow() function 
> with a function that lacks functionality. I am wondering for the rationale of 
> this. Does math.pow() do anything that the built-in version can not do, and 
> if not, why is it even there?

Maybe math.pow() aways using IEEE 754(?) float point via the C runtime math 
routines.
pow() will give int answer if called with int args.

Barry


> Thanks in advance for any enlightening comment on this.
> Best regards, Andreas
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PyGILState_Release called twice in embedded application

[Barry Scott]

> On 23 Mar 2023, at 08:46, Arnaud Loonstra  wrote:
> 
> Hi all,
> 
> I'm running in a crash due to a ResourceWarning (some socket is not closed in 
> a used module) after calling PyGILState_Release.
> 
> I'm running Python in a native thread (so a thread created by C not Python). 
> I'm acquiring the GIL through PyGILState_Ensure before doing any CPYthon 
> calls and releasing when finished using PyGILState_Release.
> 
> This has worked fine. But now I'm using a python module (openai) which 
> doesn't close its socket correctly which results in a ResourceWarning which 
> triggers an assert.
> 
> In the backtrace  (below) I can see PyGILState_Release is called again. (7) 
> while I've already called it (126).
> 
> I can make the crash go away by adding
> 
> import warnings
> warnings.simplefilter("ignore", ResourceWarning)
> 
> to my python code. But I'd rather prevent this from happening in the first 
> place.
> 
> Any suggestion very welcomed cause I'm puzzled.

What 3rd party C extension are you running with?
I'd be surprised if the cpython code was the issue.

Barry
 
> 
> Rg,
> 
> Arnaud
> 
> 1   __pthread_kill_implementation  pthread_kill.c   44 
> 0x76fcaccc
> 2   __pthread_kill_internalpthread_kill.c   78 
> 0x76fcad2f
> 3   __GI_raise raise.c  26 
> 0x76f7bef2
> 4   __GI_abort abort.c  79 
> 0x76f66472
> 5   __assert_fail_base assert.c 92 
> 0x76f66395
> 6   __GI___assert_fail assert.c 101 
> 0x76f74df2
> 7   PyGILState_Release pystate.c1742 
> 0x77abcf9f
> 8   tracemalloc_raw_alloc  _tracemalloc.c   779 
> 0x77afa803
> 9   tracemalloc_raw_malloc _tracemalloc.c   789 
> 0x77afa844
> 10  PyMem_RawMallocobmalloc.c   586 
> 0x779dab8d
> 11  decode_current_locale  fileutils.c  472 
> 0x77ada4ab
> 12  _Py_DecodeLocaleEx fileutils.c  598 
> 0x77adafce
> 13  unicode_decode_locale  unicodeobject.c  3970 
> 0x77a13b65
> 14  PyUnicode_DecodeLocale unicodeobject.c  4012 
> 0x77a18c79
> 15  PyErr_SetFromErrnoWithFilenameObjects  errors.c 772 
> 0x77a94178
> 16  PyErr_SetFromErrnoWithFilenameObject   errors.c 750 
> 0x77a942ce
> 17  posix_path_object_errorposixmodule.c1709 
> 0x77ae4201
> 18  path_object_error  posixmodule.c1719 
> 0x77ae420f
> 19  path_error posixmodule.c1737 
> 0x77ae4221
> 20  posix_do_stat  posixmodule.c2565 
> 0x77af42f7
> 21  os_stat_impl   posixmodule.c2897 
> 0x77af4491
> 22  os_statposixmodule.c.h  71 
> 0x77af45e5
> 23  cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c   443 
> 0x779d2ece
> 24  _PyObject_VectorcallTstate pycore_call.h92 
> 0x7798b579
> 25  PyObject_Vectorcallcall.c   299 
> 0x7798b65e
> 26  _PyEval_EvalFrameDefault   ceval.c  4772 
> 0x77a75453
> 27  _PyEval_EvalFrame  pycore_ceval.h   73 
> 0x77a79887
> 28  _PyEval_Vector ceval.c  6435 
> 0x77a799b1
> 29  _PyFunction_Vectorcall call.c   393 
> 0x7798b215
> 30  _PyObject_VectorcallTstate pycore_call.h92 
> 0x7798b579
> 31  PyObject_CallOneArgcall.c   376 
> 0x7798b6cd
> 32  call_show_warning  _warnings.c  609 
> 0x77a31d24
> 33  warn_explicit  _warnings.c  746 
> 0x77a3223b
> 34  do_warn_warnings.c  947 
> 0x77a3236f
> 35  warn_unicode   _warnings.c  1106 
> 0x77a32414
> 36  _PyErr_WarnFormatV _warnings.c  1126 
> 0x77a32481
> 37  PyErr_ResourceWarning  _warnings.c  1177 
> 0x77a32d38
> 38  sock_finalize  socketmodule.c   5073 
> 0x76840cf3
> 39  PyObject_CallFinalizer object.c 208 
> 0x779d6449
> 40  PyObject_CallFinalizerFromDealloc  object.c 226 
> 0x779d75ac
> 41  subtype_dealloctypeobject.c 1382 
> 0x779ecf81
> 42  _Py_Deallocobject.c 2390 
> 0x779d6c38
> 43  Py_DECREF  object.h 527 
> 0x779c116e
> 44  Py_XDECREF object.h 602 
> 0x779c118d
> 

Re: Is there a more efficient threading lock?

[Barry Scott]

On 25/02/2023 23:45, Jon Ribbens via Python-list wrote:

I think it is the case that x += 1 is atomic but foo.x += 1 is not.


No that is not true, and has never been true.

:>>> def x(a):
:...    a += 1
:...
:>>>
:>>> dis.dis(x)
 1   0 RESUME   0

 2   2 LOAD_FAST    0 (a)
 4 LOAD_CONST   1 (1)
 6 BINARY_OP   13 (+=)
10 STORE_FAST   0 (a)
12 LOAD_CONST   0 (None)
14 RETURN_VALUE
:>>>

As you can see there are 4 byte code ops executed.

Python's eval loop can switch to another thread between any of them.

Its is not true that the GIL provides atomic operations in python.

Barry

--
https://mail.python.org/mailman/listinfo/python-list

Re: file.read Method Documentation (Python 2.7.10)

[Barry Scott]

On 09/01/2023 14:34, Stephen Tucker wrote:

Dear Python-list,

Yes, I know that Python 2.x is no longer supported.

I have found that the documentation for this method is misleading when the
file being read is UTF-8-encoded:

Instead of reading *size* bytes, the method reads *size *UTF-8 byte
*sequences*.

Has this error been corrected in the Python 3.x documentation?


Please read the python 3 docs and let us know if you think its correct now.

Barry

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 03/01/2023 21:24, c.bu...@posteo.jp wrote:

Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:

logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))


But I don't want to make all log levels go to stdout. Just DEBUG and 
INFO. But this would be a workaround.


The main question here is why does Python deciecded to make all logs 
go to stderr?
Maybe I totally misunderstood the intention of logging.info()?! Isn't 
this the "usual applicaton output"?


If not, what messages should go into logging.info()? Can you name me 
some examples?


Example:

write an app that prints the contents of a file.

The application output is the contents of the file.

The logging might be this when all is working:

INFO About to open 
INFO Wrote  bytes from 

The logging might be this when there is a problem:

INFO About to open 
ERROR Failed to open  - 

Does that help?

Barry


--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 03/01/2023 21:24, c.bu...@posteo.jp wrote:

Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:

logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))


But I don't want to make all log levels go to stdout. Just DEBUG and 
INFO. But this would be a workaround.


The main question here is why does Python deciecded to make all logs 
go to stderr?
Maybe I totally misunderstood the intention of logging.info()?! Isn't 
this the "usual applicaton output"?


If not, what messages should go into logging.info()? Can you name me 
some examples?


It is up to you, the designer of an application, to decide how it works.
You will take into account conventions that your users will expect you 
to follow.


If the logging module helps you then use it, but you are not forced by 
logging to
design your app is a particular way. The default behavior of the logging 
module is
a generally useful default, but its only a default. You are free to 
setup logging to

meet your needs.

Barry


--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 04/01/2023 06:46, Chris Angelico wrote:

I've known some systems to have a trigger of "reading on FD 0 flushes
FD 1"


C++ has this feature:

Quote from https://en.cppreference.com/w/cpp/io/cin

"Once |std::cin| is constructed, std::cin.tie() returns ::cout 
, and likewise, 
std::wcin.tie() returns ::wcout 
. This means that any 
formatted input operation on |std::cin| forces a call to std::cout 
.flush() if any characters are 
pending for output."


Barry

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 04/01/2023 02:26, Chris Angelico wrote:

Reading/writing the FD is the same as using stdout (technically you'd
write to fd 1 and read from fd 0), but yes, can use /dev/tty to reach
for the console directly.


I think the logic is more like checking that stdin is a tty then using
the tty it to read and write the tty. This works with stdout redirected.
Also stdin is likely only open with read access.

For example the way to prevent ssh-add from prompting in the terminal is 
this:


$ ssh-add ~/.ssh/id_rsa https://mail.python.org/mailman/listinfo/python-list

Re: Possible re bug

[Barry Scott]

Please please fix you email client so that your replies stay with the
emails you are replying to.

Barry


On 28/12/2022 19:09, Stefan Ram wrote:

Alexander Richert writes:
|print(re.findall(".*","pattern"))
|yields ['pattern',''] which is not what I was expecting.

   The asterisk is "greedy", so I agree that it should consume
   /everything/ including all the empty strings at the end.
   To work around this, one can use '^.*' (no smiley intended).



--
https://mail.python.org/mailman/listinfo/python-list

Re: Does one have to use curses to read single characters from keyboard?

[Barry Scott]

> On 11 Dec 2022, at 18:50, Chris Green  wrote:
> 
> My solution in the end was copied from one I found that was much
> simpler and straightforward than most.  I meant to post this earlier
> but it got lost somewhere:-
> 
>import sys, termios, tty
>#
>#
># Read a single character from teminal, specifically for 'Y/N' 
>#
>fdInput = sys.stdin.fileno()
>termAttr = termios.tcgetattr(0)
>#
>#
># Get a single character, setcbreak rather than setraw meands CTRL/C
>etc. still work
>#
>def getch():
>sys.stdout.flush()
>tty.setcbreak(fdInput)
>ch = sys.stdin.buffer.raw.read(1).decode(sys.stdin.encoding)

Will not work for uncode code points above 255.

This is what happened when I typed € key:

:>>> a.getch()
Traceback (most recent call last):
  File "", line 1, in 
  File "/private/var/folders/ll/08dwwqkx6v9bcd15sh06x14wgn/T/a.py", line 
15, in getch
ch = sys.stdin.buffer.raw.read(1).decode(sys.stdin.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 0: 
unexpected end of data

You would need to have a loop that collected all the utf-8 bytes of a single 
code point.
You can to look at the first byte of know if the utf-8 is 1, 2, 3 or 4 bytes 
for a code point.

Barry

>termios.tcsetattr(fdInput, termios.TCSAFLUSH, termAttr)
>sys.stdout.write(ch)
>return ch
>#
>#
># Get a y or n answer, ignore other characters
>#
>def getyn():
>ch = 'x'
>while ch != 'y' and ch != 'n':
>ch = getch().lower()
>return ch
> 
> So getyn() reads a y or an n, ignores anything else and doesn't wait
> for a return key.  Keyboard input operation is restored to normal
> after doing this. Using tty.setcbreak() rather than tty.setraw() means
> that CTRL/C etc. still work if things go really wrong.

> 
> 
> -- 
> Chris Green
> ·
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: NEO6 GPS with Py PICO with micropython

[Barry Scott]

> On 30 Nov 2022, at 10:58, KK CHN  wrote:
> 
> List,
> 
> Just commented the // gpsModule.readline() in the while loop,  (
> refer the link
> https://microcontrollerslab.com/neo-6m-gps-module-raspberry-pi-pico-micropython/
> )
> 
> 
> while True: # gpsModule.readline() // This line commented out and the "GPS
> not found message disappeared". buff = str(gpsModule.readline()) parts =
> buff.split(',')
> 
> 
> The GPS not found error which appears intermittently in the output python
> console for few seconds ( say 7 to 8 seconds  its printing the lines   "
> GPS data not found" )   now  disappears.
> 
> Any thoughts?  How the above line comment made it vanish the  "GPS data
> not found" error output.

Show the full text of the error that you see. Is it a traceback?

What I would do then is read the code that raised the "GPS data not found"
error and find out why it reports that error.

Barry
p.s. Please reply in line, do not top post.

> 
> Krishane
> 
> On Wed, Nov 30, 2022 at 3:58 AM rbowman  wrote:
> 
>> On Tue, 29 Nov 2022 17:23:31 +0530, KK CHN wrote:
>> 
>> 
>>> When I ran the program I am able to see the output of  latitude and
>>> longitude in the console of thony IDE.  But  between certain intervals
>>> of a few seconds  I am getting the latitude and longitude data ( its
>>> printing GPS data not found ?? ) in the python console.
>> 
>> I would guess the 8 seconds in
>> 
>> timeout = time.time() + 8
>> 
>> is too short. Most GPS receivers repeat a sequence on NMEA sentences and
>> the code is specifically looking for $GPGGA. Add
>> 
>> print(buff)
>> 
>> to see the sentences being received. I use the $GPRMC since I'm interested
>> in the position, speed, and heading. It's a different format but if you
>> only want lat/lon you could decode it in a similar fashion as the $GPGGA.
>> 
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.11.0 installation and Tkinter does not work

[Barry Scott]

> On 23 Nov 2022, at 06:31, Stefan Ram  wrote:
> 
> darkst...@o2online.de writes:
>> I want learn python for 4 weeks and have problems, installing Tkinter. If I=
>> installed 3.11.0 for my windows 8.1 from python.org and type
> 
>  Ok, so you already installed from python.org. I wonder a
>  little about the wording "installing Tkinter". Here,
>  I installed /Python/ from python.org, and this /included/ tkinter.
>  If you have really attempted a separate installation of tkinter,
>  it may help to uninstall and instead install Python /including/ 
>  tkinter.
> 
>>> ImportError: DLL load failed while importing _tkinter: Das angegebene
>>> Modul wurde nicht gefunden.
> 
>  Another possibility of analysis is to watch the Python
>  process using "Process Monitor" (formerly from Sysinternals)
>  under Microsoft® Windows (not to be confused with "Process
>  Explorer"). This program requires some familiarization,
>  but then it can show you in which directories a process is
>  searching for which DLLs. This might help you to find the
>  name of the DLL missing and in which directory it should be.

I think the depends.exe tool from sysintenals will do this as well.
There is a mode where you run a program and it collects the data
for all the DLLs that are used either statically linked or dynamicall
loaded, that is the case for _tkinter.

I have not used in in a very long time but I recall it shows errors
from DLLs that failed to load.

Barry


> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to manage python shebang on mixed systems?

[Barry Scott]

> On 7 Nov 2022, at 09:28, Chris Green  wrote:
> 
> Chris Green  wrote:
>>> 3: with your pseudo "python3" script in place, make all the scripts use 
>>> the "#!/usr/bin/env python3" shebang suggested above.
>>> 
>> Yes, that sounds a good plan to me, thanks Cameron.
>> 
> Doesn't '#!/usr/bin/env python3' suffer from the same problem as
> '#!/usr/bin/python3' in the sense that the env executable might not be
> in /usr/bin?

env is always available as /usr/bin/env - I think its spec'ed in posix that way.

The only reason that things are in /bin are for systems that need a subset of
programs to boot the system to point it can mount /usr. env is not going to be
needed for that use case.

> 
> Wouldn't '#! env python3' be better?

Barry

> 
> -- 
> Chris Green
> ·
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Information about slow execution notebook

[Barry Scott]

> On 1 Nov 2022, at 16:08, nhlanhlah198506  wrote:
> 
> I wish to know why sometimes my notebook won't execute my program And VS code 
> won't connect to kernels. Thank you Nhlanhla Ndwandwe Sent from my Galaxy

You need to provide details on what you do and what happens.
Reminder do not attach screen shots, they are striped in this mailing list.
Cut-n-paste full error message information.

Barry


> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: UTF-8 and latin1

[Barry Scott]

> On 25 Oct 2022, at 11:16, Stefan Ram  wrote:
> 
> r...@zedat.fu-berlin.de (Stefan Ram) writes:
>> You can let Python guess the encoding of a file.
>> def encoding_of( name ):
>> path = pathlib.Path( name )
>> for encoding in( "utf_8", "cp1252", "latin_1" ):
>> try:
>> with path.open( encoding=encoding, errors="strict" )as file:
> 
>  I also read a book which claimed that the tkinter.Text
>  widget would accept bytes and guess whether these are
>  encoded in UTF-8 or "ISO 8859-1" and decode them 
>  accordingly. However, today I found that here it does 
>  accept bytes but it always guesses "ISO 8859-1".

The best you can do is assume that if the text cannot decode as utf-8 it may be 
8859-1.

Barry

> 
>  main.py
> 
> import tkinter
> 
> text = tkinter.Text()
> text.insert( tkinter.END, "AÄäÖöÜüß".encode( encoding='ISO 8859-1' ))
> text.insert( tkinter.END, "AÄäÖöÜüß".encode( encoding='UTF-8' ))
> text.pack()
> print( text.get( "1.0", "end" ))
> 
>  output
> 
> AÄäÖöÜüßAÄäÖöÜüß
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Any PyQt developers here?

[Barry Scott]

There is an active PyQt mailing list that has lots of helpful and knowledgeable 
people on it.

https://www.riverbankcomputing.com/mailman/listinfo/pyqt

Barry


> On 25 Oct 2022, at 18:03, DFS  wrote:
> 
> Having problems with removeRow() on a QTableView object.
> 
> After calling removeRow(), the screen isn't updating.  It's as if the model 
> is read-only, but it's a QSqlTableModel() model, which is not read-only.
> 
> The underlying SQL is straightforward (one table) and all columns are 
> editable.
> 
> None of the editStrategies are working either.
> 
> I tried everything I can think of, including changes to the EditTriggers, but 
> no luck.  HELP!
> 
> FWIW, the same removeRow() code works fine with a QTableWidget.
> 
> ---
> object creation and data loading all works fine
> ---
> #open db connection
> qdb = QSqlDatabase.addDatabase("QSQLITE")
> qdb.setDatabaseName(dbname)
> qdb.open()
> 
> #prepare query and execute to return data
> query = QSqlQuery()
> query.prepare(cSQL)
> query.exec_()
> 
> #set model type and query
> model = QSqlTableModel()
> model.setQuery(query)
>   
> #assign model to QTableView object
> view = frm.tblPostsView
> view.setModel(model)
>   
> #get all data
> while(model.canFetchMore()): model.fetchMore()
> datarows = model.rowCount()
> 
> 
> 
> ---
> iterate selected rows also works fine
> SelectionMode is Extended.
> identical code works for a QTableWidget
> ---
> selected = tbl.selectionModel().selectedRows()
> #reverse sort the selected items to delete from bottom up
> selected = sorted(selected,reverse=True)
> for i,val in enumerate(selected):
>   tbl.model().removeRow(selected[i].row())
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ref-strings in logging messages (was: Performance issue with CPython 3.10 + Cython)

[Barry Scott]

> On 8 Oct 2022, at 11:50, Weatherby,Gerard  wrote:
> 
> Logging does support passing a callable, if indirectly. It only calls __str__ 
> on the object passed if debugging is enabled.
>  
> class Defer:
> 
> def __init__(self,fn):
> self.fn = fn
> 
> def __str__(self):
> return self.fn()
> 
> def some_expensive_function():
> return "hello"
> 
> logging.basicConfig()
> logging.debug(Defer(some_expensive_function))

Oh what a clever hack. Took a few minutes of code reading to see why this works.
You are exploiting the str(msg) that is in class LogRecords getMessage().

```
def getMessage(self):
"""
Return the message for this LogRecord.

Return the message for this LogRecord after merging any user-supplied
arguments with the message.
"""
msg = str(self.msg)
if self.args:
msg = msg % self.args
return msg
```

Barry


>  
>  
> From: Python-list  > on behalf of 
> Barry mailto:ba...@barrys-emacs.org>>
> Date: Friday, October 7, 2022 at 1:30 PM
> To: MRAB mailto:pyt...@mrabarnett.plus.com>>
> Cc: python-list@python.org  
> mailto:python-list@python.org>>
> Subject: Re: Ref-strings in logging messages (was: Performance issue with 
> CPython 3.10 + Cython)
> 
> *** Attention: This is an external email. Use caution responding, opening 
> attachments or clicking on links. ***
> 
> > On 7 Oct 2022, at 18:16, MRAB  wrote:
> >
> > On 2022-10-07 16:45, Skip Montanaro wrote:
> >>> On Fri, Oct 7, 2022 at 9:42 AM Andreas Ames 
> >>> 
> >>> wrote:
> >>> 1. The culprit was me. As lazy as I am, I have used f-strings all over the
> >>> place in calls to `logging.logger.debug()` and friends, evaluating all
> >>> arguments regardless of whether the logger was enabled or not.
> >>>
> >> I thought there was some discussion about whether and how to efficiently
> >> admit f-strings to the logging package. I'm guessing that's not gone
> >> anywhere (yet).
> > Letting you pass in a callable to call might help because that you could 
> > use lambda.
> 
> Yep, that’s the obvious way to avoid expensive log data generation.
> Would need logging module to support that use case.
> 
> Barry
> 
> > --
> > https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!mrESxAj9YCHsdtNAfkNiY-Zf6U3WTIqaNrgBmbw1ELlQy51ilob43dD0ONsqvg4a94MEdOdwomgyqfyABbvRnA$
> >  
> > 
> >
> 
> --
> https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!mrESxAj9YCHsdtNAfkNiY-Zf6U3WTIqaNrgBmbw1ELlQy51ilob43dD0ONsqvg4a94MEdOdwomgyqfyABbvRnA$
>  
> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Changing 'Scripts/*.exe'

[Barry Scott]

> On 1 Oct 2022, at 16:50, Gisle Vanem via Python-list  
> wrote:
> 
> Hello list.
> 
> I'm moved my old Python27 installation from
>  f:\ProgramFiler\Python27  ( == 'ProgramFiles')
> to
>  f:\gv\Python27

The design of Windows installed software makes this hard to do without a lot of 
knowledge about what the installation did.

For example there are registry keys that hold information on where python 2.7 
is installed and there is other software that
will read the registry and expect that information to be correct.

I would have backed the old installation, uninstalled and the reinstalled 
python where you want it to be.

> 
> and now many 'scripts/*.exe' program fails
> to start since the old path to 'Python.exe'
> is wrong.
> 
> E.g. 'Scripts/pip2.exe' has the path
> "f:\programfiler\python27\python.exe" hard-coded
> inside it.
> 
> Is there a easy way to fix this w/o re-installing this
> old Python?

The advice you have had will get you someway to fixing the breakage.
But do not be surprised if more things break as you try different things out.

As I say I would have backed up the files, uninstall and reinstall then put 
back from backup
any files that are not installable anymore.

Why did you move the files from where they where installed and working?

Barry


> 
> -- 
> --gv
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.9.14

[Barry Scott]

> On 14 Sep 2022, at 14:03, ⁨אורי⁩ <⁨u...@speedy.net⁩> wrote:
> 
> Hi,
> 
> Python 3.9.14 has been released on Sept. 6, 2022. As I can see written on
> https://www.python.org/downloads/release/python-3914/:
> 
> According to the release calendar specified in PEP 596, Python 3.9 is now
> in the "security fixes only" stage of its life cycle: the 3.9 branch only
> accepts security fixes and releases of those are made irregularly in
> source-only form until October 2025. Python 3.9 isn't receiving regular bug
> fixes anymore, and binary installers are no longer provided for it. Python
> 3.9.13 was the last full bugfix release of Python 3.9 with binary
> installers.
> 
> 
> Is there a safe way to install a 64-bit version of Python 3.9.14 on Windows?

What is stopping you using 3.10?

If 3.9 is important then I guess you will need to build it for yourself.

Barry

> 
> Thanks,
> Uri.
> אורי
> u...@speedy.net
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why I cannot to install packages?

[Barry Scott]

> On 15 Sep 2022, at 12:31, ⁨נתי שטרן⁩ <⁨nsh...@gmail.com⁩> wrote:
> 
> [image: image.png]

Images are stripped; please copy the text of the error you are seeing and post 
that.

Barry

> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Simple TCP proxy

[Barry Scott]

Morten,

As Chris remarked you need to learn a number of networking, python, system 
performance
and other skills to turn your project into production code.

Using threads does not scale very well. Its uses a lot of memory and raises CPU 
used
just to do the context switches. Also the GIL means that even if you are doing 
blocking
I/O the use of threads does not scale well.

Its rare to see multi threaded code, rather what you see is code that uses 
async I/O.

At its heart async code at the low level is using a kernel interface like epoll 
(or on old
systems select). What epoll allow you to do is wait on a sets of FDs for a 
range of
I/O operations. Like ready to read, ready to write and other activity (like the 
socket
closing).

You could write code to use epoll your self, but while fun to write you need to 
know
a lot about networking and linux to cover all the corner cases.

Libraries like twisted, trio, uvloop and pythons selectors implemented 
production quality
version of the required code with good APIs.

Do not judge these libraries by their size. They are no bloated and only as 
complex as
the problem they are solving requires.

There is a simple example of async code using the python selectors here that 
shows
the style of programming.
https://docs.python.org/3/library/selectors.html#examples 


The issues that you likely need to solve and test for include:
* handling unexpected socket close events.
* buffering and flow control from one socket's read to the another socket's 
write.
  What if one side is reading slower then the other is writing?
* timeout sockets that stop sending data and close them

At some point you will exceed the capacity for one process to handle the load.
The solution we used is to listen on the socket in a parent process and fork
enough child processes to handle the I/O load. This avoids issues with the GIL
and allows you to scale.

But I am still not sure why you need to do anything more the increase the 
backlog
on your listen socket in the main app. Set the backlog to 1,000,000 does that 
fix
your issue? 

You will need on Linux to change kernel limits to allow that size. See man 
listen
for info on what you need to change.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.9 pidfd_open

[Barry Scott]

> On 5 Jul 2022, at 19:47, Weatherby,Gerard  wrote:
> 
> python introduced os.pidfd_open(), which works as documented.
> 
> My development environment, PyCharm, complains about it being undefined.
> 
> Should it be in 
> https://raw.githubusercontent.com/python/cpython/3.9/Lib/os.py ?

When faced with this type of problem I introspect from the Python REPL prompt.

Run python 3.9 and see if os.pidfd_open is defined?

Do not do this inside PyCharm to rule out a problem with PyCharm.

Also you will need to be running a linux kernel + glibc version that is new 
enough to proved this to python.

This is what I see with Fedora 36:

$ python3.9
Python 3.9.13 (main, Jun  9 2022, 00:00:00)
[GCC 12.1.1 20220507 (Red Hat 12.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
:>>> import os
:>>> os.pidfd_open

:>>>

Barry


> 
> --
> Gerard Weatherby | Application Architect
> NMRbox | NAN | Department of Molecular Biology and Biophysics | UConn Health
> 263 Farmington Avenue, Farmington, CT 06030-6406
> uchc.edu
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Building on Windows

[Barry Scott]

> On 2 Jul 2022, at 15:41, Sam Ezeh  wrote:
> 
> To add to this, my process was
> 
> 1. Setup the VM
> 2. Install Git
> 3. Clone CPython
> 4. Run `PCbuild\build.bat -d -e`
> 5. Notice the error, then install visual studio community 2022

Try running this:

call "C:\Program Files\Microsoft Visual 
Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat"

I use that to get at the command line tools.

If that does not work then search the disk for msbuild.exe and add its folder 
to your PATH.

Barry

> 6. Re-run `PCbuild\build.bat -d -e` and see the same error
> 
> I can't directly copy and paste between the VM and my host desktop but
> in this scenario, the error was that single line and I can use paste
> sites where necessary.
> 
> Kind regards,
> Sam Ezeh
> 
> On Sat, 2 Jul 2022 at 15:27, Sam Ezeh  wrote:
>> 
>> I have a Windows virtual machine and I'm following the instructions on
>> the devguide [1] to build Python inside it.
>> 
>> When running `PCbuild\build\bat -e -d` I get "Cannot locate
>> MSBuild.exe on PATH or as MSBUILD variable". I've done a minimal
>> amount of searching [2][3] but I'm not well-acquainted with Windows
>> and don't understand the solutions.
>> 
>> Thanks in advance.
>> 
>> Kind regards,
>> Sam Ezeh
>> 
>> [1]: https://devguide.python.org/compiler/
>> [2]: https://bugs.python.org/issue41213
>> [3]: https://bugs.python.org/issue33675
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Subtract n months from datetime [Why?]

[Barry Scott]

> On 22 Jun 2022, at 17:59, Paulo da Silva 
>  wrote:
> 
> Às 05:29 de 21/06/22, Paulo da Silva escreveu:
> 
> As a general response to some comments ...
> 
> Suppose we need to delete records from a database older than ...
> Today, it's usual to specify days. For example you have to keep some gov 
> papers for 90 days. This seems to come from computers era. In our minds, 
> however, we immediately think 90 days=3 months.
> For example, one may want to delete some files older than 9 months. It's far 
> more intuitive than 270 days.
> When we talk about years it is still going. For example I need to keep my 
> receipts for 5 years because IRS audits.
> Accepting this, it's intuitive, for example, that 3 months before July, 31 is 
> April, 30.
> The same happens for the years. 5 years before February, 29 is February, 28.

The advantage of 30 days, 90 days etc is that a contract or law does not need 
to tell you
how to deal with the problems of calendar months.

As you say in peoples thoughts that 1 month or 3 months etc. But an accounts 
department
will know how to to the number of days till they have to pay up.

Barry


> 
> Again, this is my opinion and that's the way I like it :-)
> Regards
> Paulo
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Logging into single file from multiple modules in python when TimedRotatingFileHandler is used

[Barry Scott]

> On 22 Jun 2022, at 11:06, Lars Liedtke  wrote:
> 
> Could be unrelated and only a part of a solution, but if you are on a unixoid 
> system, you could use logrotate, instead of TimedRotatingFileHandler. 
> logfrotate ensures that the logging service does not realize, its logs have 
> been rotated. So it can log as if nothing has happened.


The process that is writing the file must be told that rotation has happened 
for it to work.
Other wise all the logs keep being write to the original file via the FD that 
the process has.

logrotate's config include how to tell the process the log file needs reopening.

Barry


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to replace characters in a string?

[Barry Scott]

> On 8 Jun 2022, at 18:01, Dave  wrote:
> 
> Hi,
> 
> This is a tool I’m using on my own files to save me time. Basically or most 
> of the tracks were imported with different version iTunes over the years. 
> There are two problems:
> 
> 1.   File System characters are replaced (you can’t have ‘/‘ or ‘:’ in a file 
> name).
ok
> 2.   Smart Quotes were added at some point, these need to replaced.
ok
> 3.   Other character based of name being of a non-english origin.
Why is this a problem? Its only if the chars are confusing/will not compare 
that there is something to fix?
All modern OS allow unicode filenames.

Barry


> 
> If find others I’ll add them.
> 
> I’m using MusicBrainz to do a fuzzy match and get the correct name.
> 
> it’s not perfect, but works for 99% of files which is good enough for me!
> 
> Cheers
> Dave
> 
> 
>> On 8 Jun 2022, at 18:23, Avi Gross via Python-list  
>> wrote:
>> 
>> Dave,
>> 
>> Your goal is to compare titles and there can be endless replacements needed 
>> if you allow the text to contain anything but ASCII.
>> 
>> Have you considered stripping out things instead? I mean remove lots of 
>> stuff that is not ASCII in the first place and perhaps also remove lots of 
>> extra punctuation likesingle quotes or question marks or redundant white 
>> space and compare the sort of skeletons of the two? 
>> 
>> And even if that fails, could you have a measure of how different they are 
>> and tolerate if they were say off by one letter albeit "My desert" matching 
>> "My Dessert" might not be a valid match with one being a song about an arid 
>> environment and the other about food you don't need!
>> 
>> Your seemingly simple need can expand into a fairly complex project. There 
>> may be many ideas on how to deal with it but not anything perfect enough to 
>> catch all cases as even a trained human may have to make decisions at times 
>> and not match what other humans do. We have examples like the TV show 
>> "NUMB3RS" that used a perfectly valid digit 3 to stand for an "E" but yet is 
>> often written when I look it up as NUMBERS. You have obvious cases where 
>> titles of songs may contain composite symbols like "œ" which will not 
>> compare to one where it is written out as "oe" so the idea of comparing is 
>> quite complex and the best you might do is heuristic.
>> 
>> UNICODE has many symbols that are almost the same or even look the same or 
>> maybe in one font versus another. There are libraries of functions that 
>> allow some kinds of comparisons or conversions that you could look into but 
>> the gain for you may not be worth it. Nothing stops a person from naming a 
>> song any way they want and I speak many languages and often see a song 
>> re-titled in the local language and using the local alphabet mixed often 
>> with another.
>> 
>> Your original question is perhaps now many questions, depending on what you 
>> choose. You started by wanting to know how to compare and it is moving on to 
>> how to delete parts or make substitutions or use regular expressions and it 
>> can get worse. You can, for example, take a string and identify the words 
>> within it and create a regular expression that inserts sequences between the 
>> words that match any zero or one or more non-word characters such as spaces, 
>> tabs, punctuation or non-ASCII, so that song titles with the same words in a 
>> sequence match no matter what is between them. The possibilities are endless 
>> but consider some of the techniques that are used by some programs that 
>> parse text and suggest alternate spellings  or even programs like Google 
>> Translate that can take a sentence and then suggest you may mean a slightly 
>> altered sentence with one word changed to fit better. 
>> 
>> You need to decide what you want to deal with and what will be 
>> mis-classified by your program. Some of us have suggested folding the case 
>> of the words but that means asong about a dark skinned person in Poland 
>> called "Black Polish" would match a song about keeping your shoes dark with 
>> "black polish" so I keep repeating it is very hard or frankly impossible, to 
>> catch every case I can imagine and the many I can't!
>> 
>> But the emphasis here is not your overall problem. It is about whether and 
>> how the computer language called python, and perhaps some add-on modules, 
>> can be used to solve each smaller need such as recognizing a pattern or 
>> replacing text. It can do quite a bit but only when the specification of the 
>> problem is exact. 
>> 
>> 
>> 
>> 
>> -Original Message-
>> From: Dave 
>> To: python-list@python.org
>> Sent: Wed, Jun 8, 2022 5:09 am
>> Subject: Re: How to replace characters in a string?
>> 
>> Hi,
>> 
>> Thanks for this! 
>> 
>> So, is there a copy function/method that returns a MutableString like in 
>> objective-C? I’ve solved this problems before in a number of languages like 
>> Objective-C and AppleScript.
>> 
>> Basically there is a set of common 

Re: How to test characters of a string

[Barry Scott]

> On 7 Jun 2022, at 23:24, Dave  wrote:
> 
> Yes, it was probably just a typeo on my part.
> 
> I’ve now fixed the majority of cases but still got two strings that look 
> identical but fail to match, this time (again by 10cc), “I’m Mandy Fly Me”.
> 
> I’m putting money on it being a utf8 problem but I’m stuck on how to handle 
> it. It’s probably the single quote in I’m, although it has worked with other 
> songs.
> 
> Any ideas?

You can use difflib to give you a diff of the two strings:

:>>> print('\n'.join(difflib.unified_diff('abc', 'adc')))
---

+++

@@ -1,3 +1,3 @@

 a
-b
+d
 c
:>>>

The docs talk about lines, but difflib works on sequence. I use it a lot to find
differences within lines.

Barry



> 
> All the Best
> Cheers
> Dave
> 
> Here is the whole function/method or whatever it’s called in Python:
> 
> 
> #
> #   checkMusicFiles
> #
> 
> def checkMusicFiles(theBaseMusicLibraryFolder):
>myArtistDict = []
> 
> #
> #  Loop thru Artists Folder
> #
>myArtistsFoldlerList = getFolderList(theBaseMusicLibraryFolder)
>myArtistCount = 0
>for myArtistFolder in myArtistsFoldlerList:
>print('Artist: ' + myArtistFolder)
> #
> #  Loop thru Albums Folder
> #
>myAlbumList = getFolderList(theBaseMusicLibraryFolder + myArtistFolder)
>for myAlbum in myAlbumList:
>print('Album: ' + myAlbum)
> 
> #
> #  Loop thru Tracks (Files) Folder
> #
>myAlbumPath = theBaseMusicLibraryFolder + myArtistFolder + '/' + 
> myAlbum + '/'
>myFilesList = getFileList(myAlbumPath)
>for myFile in myFilesList:
>myFilePath = myAlbumPath + myFile
>myID3 = eyed3.load(myFilePath)
>if myID3 is None:
>continue
> 
>myArtistName = myID3.tag.artist
>if myArtistName is None:
>continue
> 
>myAlbumName = myID3.tag.album
>if myAlbumName is None:
>continue
> 
>myTitleName = myID3.tag.title
>if myTitleName is None:
>continue
> 
>myCompareFileName = myFile[0:-4]
>if myCompareFileName[0].isdigit() and 
> myCompareFileName[1].isdigit():
>myCompareFileName = myFile[3:-4]
> 
>if myCompareFileName != myTitleName:
>myLength1 = len(myCompareFileName)
>myLength2 = len(myTitleName)
>print('File Name Mismatch - Artist: [' + myArtistName + '] 
>  Album: ['+ myAlbumName + ']  Track: [' + myTitleName + ']  File: [' + 
> myCompareFileName + ']')
>if (myLength1 == myLength2):
>print('lengths match: ',myLength1)
>else:
>print('lengths mismatch: ',myLength1,'  ',myLength2)
> 
>print(' ')
> 
> 
> 
> 
>return myArtistsFoldlerList
> 
> 
> 
> 
> 
> 
>> On 8 Jun 2022, at 00:07, MRAB  wrote:
>> 
>> On 2022-06-07 21:23, Dave wrote:
>>> Thanks a lot for this! isDigit was the method I was looking for and 
>>> couldn’t find.
>>> I have another problem related to this, the following code uses the code 
>>> you just sent. I am getting a files ID3 tags using eyed3, this part seems 
>>> to work and I get expected values in this case myTitleName (Track name) is 
>>> set to “Deadlock Holiday” and myCompareFileName is set to “01 Deadlock 
>>> Holiday” (File Name with the Track number prepended). The is digit test 
>>> works and myCompareFileName is set to  “Deadlock Holiday”, so they should 
>>> match, right?
>> OT, but are you sure about that name? Isn't it "Dreadlock Holiday" (by 10cc)?
>> 
>> [snip]
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tail

[Barry Scott]

> On 8 May 2022, at 17:05, Marco Sulla  wrote:
> 
> I think I've _almost_ found a simpler, general way:
> 
> import os
> 
> _lf = "\n"
> _cr = "\r"
> 
> def tail(filepath, n=10, newline=None, encoding=None, chunk_size=100):
>n_chunk_size = n * chunk_size

Why use tiny chunks? You can read 4KiB as fast as 100 bytes as its typically 
the smaller size the file system will allocate.
I tend to read on multiple of MiB as its near instant.

>pos = os.stat(filepath).st_size

You cannot mix POSIX API with text mode.
pos is in bytes from the start of the file.
Textmode will be in code points. bytes != code points.

>chunk_line_pos = -1
>lines_not_found = n
> 
>with open(filepath, newline=newline, encoding=encoding) as f:
>text = ""
> 
>hard_mode = False
> 
>if newline == None:
>newline = _lf
>elif newline == "":
>hard_mode = True
> 
>if hard_mode:
>while pos != 0:
>pos -= n_chunk_size
> 
>if pos < 0:
>pos = 0
> 
>f.seek(pos)

In text mode you can only seek to a value return from f.tell() otherwise the 
behaviour is undefined.

>text = f.read()

You have on limit on the amount of data read.

>lf_after = False
> 
>for i, char in enumerate(reversed(text)):

Simple use text.rindex('\n') or text.rfind('\n') for speed.

>if char == _lf:
>lf_after == True
>elif char == _cr:
>lines_not_found -= 1
> 
>newline_size = 2 if lf_after else 1
> 
>lf_after = False
>elif lf_after:
>lines_not_found -= 1
>newline_size = 1
>lf_after = False
> 
> 
>if lines_not_found == 0:
>chunk_line_pos = len(text) - 1 - i + newline_size
>break
> 
>if lines_not_found == 0:
>break
>else:
>while pos != 0:
>pos -= n_chunk_size
> 
>if pos < 0:
>pos = 0
> 
>f.seek(pos)
>text = f.read()
> 
>for i, char in enumerate(reversed(text)):
>if char == newline:
>lines_not_found -= 1
> 
>if lines_not_found == 0:
>chunk_line_pos = len(text) - 1 - i +
> len(newline)
>break
> 
>if lines_not_found == 0:
>break
> 
> 
>if chunk_line_pos == -1:
>chunk_line_pos = 0
> 
>return text[chunk_line_pos:]
> 
> 
> Shortly, the file is always opened in text mode. File is read at the end in
> bigger and bigger chunks, until the file is finished or all the lines are
> found.

It will fail if the contents is not ASCII.

> 
> Why? Because in encodings that have more than 1 byte per character, reading
> a chunk of n bytes, then reading the previous chunk, can eventually split
> the character between the chunks in two distinct bytes.

No it cannot. text mode only knows how to return code points. Now if you are in
binary it could be split, but you are not in binary mode so it cannot.

> I think one can read chunk by chunk and test the chunk junction problem. I
> suppose the code will be faster this way. Anyway, it seems that this trick
> is quite fast anyway and it's a lot simpler.

> The final result is read from the chunk, and not from the file, so there's
> no problems of misalignment of bytes and text. Furthermore, the builtin
> encoding parameter is used, so this should work with all the encodings
> (untested).
> 
> Furthermore, a newline parameter can be specified, as in open(). If it's
> equal to the empty string, the things are a little more complicated, anyway
> I suppose the code is clear. It's untested too. I only tested with an utf8
> linux file.
> 
> Do you think there are chances to get this function as a method of the file
> object in CPython? The method for a file object opened in bytes mode is
> simpler, since there's no encoding and newline is only \n in that case.

State your requirements. Then see if your implementation meets them.

Barry

> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tail

[Barry Scott]

> On 7 May 2022, at 22:31, Chris Angelico  wrote:
> 
> On Sun, 8 May 2022 at 07:19, Stefan Ram  wrote:
>> 
>> MRAB  writes:
>>> On 2022-05-07 19:47, Stefan Ram wrote:
>> ...
 def encoding( name ):
   path = pathlib.Path( name )
   for encoding in( "utf_8", "latin_1", "cp1252" ):
   try:
   with path.open( encoding=encoding, errors="strict" )as file:
   text = file.read()
   return encoding
   except UnicodeDecodeError:
   pass
   return "ascii"
 Yes, it's potentially slow and might be wrong.
 The result "ascii" might mean it's a binary file.
>>> "latin-1" will decode any sequence of bytes, so it'll never try
>>> "cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong
>>> anyway because the file could contain 0x80..0xFF, which aren't supported
>>> by that encoding.
>> 
>>  Thank you! It's working for my specific application where
>>  I'm reading from a collection of text files that should be
>>  encoded in either utf_8, latin_1, or ascii.
>> 
> 
> In that case, I'd exclude ASCII from the check, and just check UTF-8,
> and if that fails, decode as Latin-1. Any ASCII files will decode
> correctly as UTF-8, and any file will decode as Latin-1.
> 
> I've used this exact fallback system when decoding raw data from
> Unicode-naive servers - they accept and share bytes, so it's entirely
> possible to have a mix of encodings in a single stream. As long as you
> can define the span of a single "unit" (say, a line, or a chunk in
> some form), you can read as bytes and do the exact same "decode as
> UTF-8 if possible, otherwise decode as Latin-1" dance. Sure, it's not
> perfectly ideal, but it's about as good as you'll get with a lot of
> US-based servers. (Depending on context, you might use CP-1252 instead
> of Latin-1, but you might need errors="replace" there, since
> Windows-1252 has some undefined byte values.)

There is a very common error on Windows that files and especially web pages that
claim to be utf-8 are in fact CP-1252.

There is logic in the HTML standards to try utf-8 and if it fails fall back to 
CP-1252.

Its usually the left and "smart" quote chars that cause the issue as they code
as an invalid utf-8.

Barry
 

> 
> ChrisA
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tail

[Barry Scott]

> On 7 May 2022, at 14:40, Stefan Ram  wrote:
> 
> Marco Sulla  writes:
>> So there's no way to reliably read lines in reverse in text mode using
>> seek and read, but the only option is readlines?
> 
>  I think, CPython is based on C. I don't know whether
>  Python's seek function directly calls C's fseek function,
>  but maybe the following parts of the C standard also are
>  relevant for Python?

There is the posix API that and the C FILE API.

I expect that the odities you about NUL chars is all about the FILE
API. As far as I know its the posix API that C Python uses and it
does not suffer from issues with binary files.

Barry

> 
> |Setting the file position indicator to end-of-file, as with
> |fseek(file, 0, SEEK_END), has undefined behavior for a binary
> |stream (because of possible trailing null characters) or for
> |any stream with state-dependent encoding that does not
> |assuredly end in the initial shift state.
> from a footnote in a draft of a C standard
> 
> |For a text stream, either offset shall be zero, or offset
> |shall be a value returned by an earlier successful call to
> |the ftell function on a stream associated with the same file
> |and whence shall be SEEK_SET.
> from a draft of a C standard
> 
> |A text stream is an ordered sequence of characters composed
> |into lines, each line consisting of zero or more characters
> |plus a terminating new-line character. Whether the last line
> |requires a terminating new-line character is implementation-defined.
> from a draft of a C standard
> 
>  This might mean that reading from a text stream that is not
>  ending in a new-line character might have undefined behavior
>  (depending on the C implementation). In practice, it might
>  mean that some things could go wrong near the end of such
>  a stream. 
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: convenience

[Barry Scott]

> On 22 Mar 2022, at 18:00, Avi Gross via Python-list  
> wrote:
> 
> An earlier post talked about a method they used for "convenience" in a way 
> they apparently did not understand and many of us educated them, hopefully.
> 
> That made me wonder of teh impact on our code when we use various forms
> of convenience. Is it convenient for us as programmers, other potential 
> readers,
> or a compiler or interpreter?

Using aliases can impose a cost for maintenance. It adds a burden on the reader
to figure what an alias really refers to.

I encountered this:

from time import time as now.

The use of now() was just confusing to maintainers.

I avoid the convenience path as it often makes maintenance harder.


> 
> The example used was something like:
> 
> varname = objectname.varname
> 
> The above clearly requires an existing object. But there is no reason it
> has to be the same name. Consider the somewhat related idea used
> almost always in code:
> 
> import numpy as np
> 
> Both cases make a sort of pointer variable and are a tad shorter to write.
> 
> But what impact does it have on interpreters and compilers?

On the positive side aliases are very useful to improve the speed of
hot-path code.

read = this.that.the_other.read
while condition:
buf = read()
...

As with all optimisations only do this if you can justify it with benchmark 
results.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: virtualenv and make DESTDIR=

[Barry Scott]

> On 5 Mar 2022, at 19:56, Hartmut Goebel  wrote:
> 
> Am 05.03.22 um 17:34 schrieb Barry Scott:
>> Have the RPM install all the pythone code and dependencies and also install 
>> a short script that
>> sets up PYTHONPATH, LD_LIBRARY_PATH, etc and execs the python3 .py.
> The scripts are already created by entry-points. So basically this means to 
> reinvent the wheel. Or did I miss something?
> 
I see what are getting at - I did miss the something.
It is the setup.py that will create your entry point!

For my own understanding I turned one of my PyPI projects into an RPM.

The project source is on https://github.com/barry-scott/CLI-tools.git 
<https://github.com/barry-scott/CLI-tools.git>

First I turned the this into a tarball:

$ git archive --format=tar --prefix=cli-tools-3.1.0/ master >cli-tools-3.1.0.tar
$ gzip cli-tools-3.1.0.tar

Then I created a RPM spec file:

Summary:cli-tools example packaging for python
Name:   cli-tools
Version:3.1.0
Release:1

BuildArch:  noarch
Source: %{name}-%{version}.tar.gz
License:APL2.0
Prefix: /opt/barry-example

BuildRequires:  python3
BuildRequires:  python3-setuptools

Requires:   python3

%description
cli-tools example packaging for python

%prep
# unpack Source tarball
%setup

%build
cd Source
python3 setup_ssh_wait.py build

%install
cd Source
python3 setup_ssh_wait.py install --prefix=%{prefix} --root=%{buildroot}

# need to add the site-packages folder to the sys.path as
# setup.py assumes it is installing into the systems python library
cat <fix_bin.py

python3 fix_bin.py %{buildroot}%{prefix}/bin/ssh-wait 
%{buildroot}%{prefix}/opt/barry-example/lib

%files
/opt/barry-example/*

%changelog

Then only thing I needed to fix up was that the entry point code needs to have 
the
lib folder in /opt/barry-example added to sys.path. That is what the fix_bin.py 
is
doing.

I then use rpmbuild to create the RPM.

$ rpmbuild -bb cli-tools.spec

To use mock the steps are:

$ rpmbuild -bs cli-tools.spec
$ mock --rebuild --root fedora-35-x86_64 ../SRPMS/cli-tools-3.1.0.arc.rpm

Then I install the built file:

$ dnf install /var/lib/mock/fedora-35-x86_64/result/cli-tools-3.1.0-1.noarch.rpm

And then I can run the command:

$ /opt/barry-example/bin/ssh-wait
Usage: ssh-wait  

ssh-wait will wait for the service to be available
on .

--verbose
- print progress messages
--nowait
- check one and exit 0 if can connect 1 if cannot
--wait-limit=
- exit 1 if service is not avaiable within  seconds.
  defaults to 600.
--service=
- check the service . Can be a service
  name of a port number. Default is ssh.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: virtualenv and make DESTDIR=

[Barry Scott]

> On 5 Mar 2022, at 19:56, Hartmut Goebel  wrote:
> 
> Am 05.03.22 um 17:34 schrieb Barry Scott:
>> Have the RPM install all the pythone code and dependencies and also install 
>> a short script that
>> sets up PYTHONPATH, LD_LIBRARY_PATH, etc and execs the python3 .py.
> The scripts are already created by entry-points. So basically this means to 
> reinvent the wheel. Or did I miss something?
> 

 For example I assume you need something like this:

#!/bin/bash
export PYTHONPATH=/opt//lib
exec /usr/bin/python3 /opt//lib/main.py "$@"

I'm assuming you put all the python code into /opt//lib
and any runnable command into /opt//bin

Barry

> -- 
> Schönen Gruß 
> Hartmut Goebel 
> Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer
> Information Security Management, Security Governance, Secure Software 
> Development
> Goebel Consult, Landshut 
> http://www.goebel-consult.de <http://www.goebel-consult.de/>
> Blog: 
> https://www.goe-con.de/blog/eilt-petition-auf-weact-gegen-vds-jetzt-unterschreiben
>  
> <https://www.goe-con.de/blog/eilt-petition-auf-weact-gegen-vds-jetzt-unterschreiben>
>  
> Kolumne: 
> https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-11-in-troja-nichts-neues
>  
> <https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-11-in-troja-nichts-neues>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: virtualenv and make DESTDIR=

[Barry Scott]

> On 5 Mar 2022, at 19:56, Hartmut Goebel  wrote:
> 
> Am 05.03.22 um 17:34 schrieb Barry Scott:
>> Have the RPM install all the pythone code and dependencies and also install 
>> a short script that
>> sets up PYTHONPATH, LD_LIBRARY_PATH, etc and execs the python3 .py.
> The scripts are already created by entry-points. So basically this means to 
> reinvent the wheel. Or did I miss something?
> 
Are you saying that the only thing you want from the venv is a script that sets 
up PYTHONPATH?

That a trivia thing to write and far easier then battling with packaging a venv 
I'd expect.

Did I miss something?

Barry

> -- 
> Schönen Gruß 
> Hartmut Goebel 
> Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer
> Information Security Management, Security Governance, Secure Software 
> Development
> Goebel Consult, Landshut 
> http://www.goebel-consult.de <http://www.goebel-consult.de/>
> Blog: 
> https://www.goe-con.de/blog/eilt-petition-auf-weact-gegen-vds-jetzt-unterschreiben
>  
> <https://www.goe-con.de/blog/eilt-petition-auf-weact-gegen-vds-jetzt-unterschreiben>
>  
> Kolumne: 
> https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-11-in-troja-nichts-neues
>  
> <https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-11-in-troja-nichts-neues>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: virtualenv and make DESTDIR=

[Barry Scott]

> On 5 Mar 2022, at 16:59, Marco Sulla  wrote:
> 
> On Sat, 5 Mar 2022 at 17:36, Barry Scott  wrote:
>> Note: you usually cannot use pip when building an RPM with mock as the 
>> network is disabled inside the build for
>> security reasons.
> 
> Can't he previously download the packages and run pip on the local packages?
> 

I guess that can work. In this case I would install with pip install --user and 
move the
files out of .local into an appropriate place in /opt/.

At work I package a lot of packages for python and what we do is get the 
sources, 
review them then turn them into an RPM using the setup.py to do the file 
placement.

We do not use the wheels from PyPI ever. They may not match the claimed sources.
Which I have seen a few times; the version on PyPI can contain helpful patches 
that
are not in the sources.

Barry


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: virtualenv and make DESTDIR=

[Barry Scott]

> On 4 Mar 2022, at 13:03, Hartmut Goebel  wrote:
> 
> Hi,
> 
> How can I make installing a virtual environment honor DESTDIR? How can I 
> install a virtual environment in $(DESTDIR)$(PREFIX), which behaves as being 
> set-up in $(PREFIX)? (Of course, this virtual environment can not be used. My 
> aim is to ship it as part of a rpm package)
> 
> In Makefiles is good practice to honor DESTDIR in the "install" target, like 
> this
> 
> install:
> install -t $(DESTDIR)$(PREFIX)/bin build/bin/my-tool
> 
> Now when running
> 
> python3 -m venv $(DESTDIR)$(PREFIX)
> 
> all paths in this virtual environment refer to $(DESTDIR)$(PREFIX) instead of 
> just $$(PREFIX)
> 
> Any ideas?
> 
> 
> Background:
> 
> More about DESTDIR: https://www.gnu.org/prep/standards/html_node/DESTDIR.html
> 
> Following Redhat's commendations, I want to install my (somewhat complex) 
> software into /opt/my-software. To make it easier for users to use the 
> software, my idea was to setup a virtual environment in /opt/my-software. 
> Thus users can easily use /opt/my-software/bin/python and have the library 
> provided by my-software available. My Software also includes some scripts, 
> which will also reside in /opt/my-software/bin and refer to 
> /opt/my-software/bin/python. This will avoid to require users to set up 
> PYTHONPATH when thy want to use MY Software.

If you are packaging the code then I do not see the need to use a venv at all.

Have the RPM install all the pythone code and dependencies and also install a 
short script that
sets up PYTHONPATH, LD_LIBRARY_PATH, etc and execs the python3 .py.

Note: you usually cannot use pip when building an RPM with mock as the network 
is disabled inside the build for
security reasons.

I package two of my projects this was for Fedora as RPMs.

Barry


> 
> 
> -- 
> Schönen Gruß
> Hartmut Goebel
> Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer
> Information Security Management, Security Governance, Secure Software 
> Development
> 
> Goebel Consult, Landshut
> http://www.goebel-consult.de
> 
> Blog: https://www.goe-con.de/blog/why-a-pki-is-barely-trustworthy
> Kolumne: 
> https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-11-in-troja-nichts-neues
>  
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Getting Syslog working on OSX Monterey

[Barry Scott]

> On 4 Mar 2022, at 21:23, Peter J. Holzer  wrote:
> 
> On 2022-02-28 22:05:05 +0000, Barry Scott wrote:
>> On 28 Feb 2022, at 21:41, Peter J. Holzer  wrote:
>>> On 2022-02-27 22:16:54 +, Barry wrote:
>>>> I have always assumed that if I want a logger syslog handler that I would 
>>>> have
>>>> to implement it myself. So far I have code that uses syslog directly and 
>>>> have
>>>> not written that code yet.
>>> 
>>> What do you mean by using syslog directly? The syslog(3) library
>>> function also just sends messages to a "syslog listener" (more commonly
>>> called a syslog daemon) - at least on any unix-like system I'm familiar
>>> with (which doesn't include MacOS). It will, however, always use the
>>> *local* syslog daemon - AFAIK there is no standard way to open a remote
>>> connection (many syslog daemons can be configured to forward messages to
>>> a remote server, however).
>> 
>> I'm re-reading the code to check on what I'm seeing. (Its been a long
>> time since I last look deeply at this code).
>> 
>> You can write to /dev/log if you pass that to
>> SysLogHandler(address='/dev/log'), but the default is to use a socket
>> to talk to a network listener on localhost:514. There are no deamons
>> listening on port 514 on my Fedora systems or mac OS.
> 
> If you are saying that SysLogHandler should use a system specific
> default (e.g. "/dev/log" on Linux) instead of UDP port 514 everywhere, I
> agree 99 % (the remaining 1 % is my contrarian alter ego arguing that
> that's really the distribution maintainer's job since a Linux
> distribution might use some other socket).
> 
> If you are saying it should use the libc syslog routines, I disagree for
> at least two reasons: a) they are OS specific, b) you can't configure the
> destination. So that would remove useful functionality.
> 
> In any case it seems strange to me that you want to rewrite it just to
> avoid passing a single parameter to the constructor (or - more likely -
> adding a single line to a config file).

What I am used to is a setup where programs use syslog() to log and
a dameon like rsyslog is responsible for routing the logs to local files
and/or remote systems. On a modern linux I think the route is
syslog() -> journald. And you can have syslog() -> journald -> rsyslogd.

Using the syslog() function means that any platform/distro details are
hidden from the user of syslog() and as is the case of macOS it
"just works". (I assume, not checked, that the write to the socket does not work
because Apple is not implementing the syslog protocol from the RFC).

> 
>> What you do not see used in the SyslogHandler() is the import syslog
>> and hence its nor using openlog() etc from syslog API.
> 
> For good reasons. The C syslog API is missing important functionality.

What are you think about being missing? Just curious.

As an aside if I had the need to log into a system log mechanism I'd be
looking to use the jounald API so that I can use structured logging on linux
systems.

Barry


> 
>hp
> 
> -- 
>   _  | Peter J. Holzer| Story must make more sense than reality.
> |_|_) ||
> | |   | h...@hjp.at |-- Charles Stross, "Creative writing
> __/   | http://www.hjp.at/ |   challenge!"
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Getting Syslog working on OSX Monterey

[Barry Scott]

> On 3 Mar 2022, at 03:01, Philip Bloom  wrote:
> 
> I'm probably asking on the wrong list, and probably should bother wherever 
> apple's ASL experts live for changes in monterey.   Guess nobody else is 
> seeing this?
> 
> The same exact code is working just fine on OSX Big Sur, but on OSX Monterey 
> it doesn't work at all.  Users that haven't updated are having the program 
> produce logs as it has for years through logging.handlers.SysLogHandler.  Mac 
> OSX definitely has a listening socket at '/var/run/syslog' which shows up in 
> Console.app.
> 
> Apologies, Barry.  I'm not quite understanding your responses.  

This works:

syslog.openlog(logoption=syslog.LOG_PID, facility=syslog.LOG_USER)
syslog.syslog(syslog.LOG_NOTICE, 'QQQ test log')

I see the "QQQ test log" when I run syslog command.

And I can reproduce that logging.handlers.SysLogHandler does not work.
I added debug to the SysLogHandler() code and see that it does indeed write 
into the /var/run/syslog socket.
I suspect that macOS /var/run/syslog does not implement the RFC that this code 
depends on, but that a wild guess.

I suggest that you write your own handler that uses syslog.syslog() and use 
that.

Barry

> 
> When we say OSX has no listener for Syslog, what is the Apple System Log and 
> the general output to Console.app then?  I thought that was the local syslog 
> on OSX and had, for years, been behaving as such when using 
> logging.handlers.SysLogHandler with a config in /etc/asl/ to define how it 
> should be routed and the rollover/cleanup frequency.  
> 
> Thanks for replying, just having trouble understanding.
> 
> 
> On Mon, Feb 28, 2022 at 2:07 PM Barry Scott  <mailto:ba...@barrys-emacs.org>> wrote:
> 
> 
> > On 28 Feb 2022, at 21:41, Peter J. Holzer  > <mailto:hjp-pyt...@hjp.at>> wrote:
> > 
> > On 2022-02-27 22:16:54 +, Barry wrote:
> >> If you look at the code of the logging modules syslog handle you will see 
> >> that
> >> it does not use syslog. It’s assuming that it can network to a syslog 
> >> listener.
> >> Such a listener is not running on my systems as far as I know.
> >> 
> >> I have always assumed that if I want a logger syslog handler that I would 
> >> have
> >> to implement it myself. So far I have code that uses syslog directly and 
> >> have
> >> not written that code yet.
> > 
> > What do you mean by using syslog directly? The syslog(3) library
> > function also just sends messages to a "syslog listener" (more commonly
> > called a syslog daemon) - at least on any unix-like system I'm familiar
> > with (which doesn't include MacOS). It will, however, always use the
> > *local* syslog daemon - AFAIK there is no standard way to open a remote
> > connection (many syslog daemons can be configured to forward messages to
> > a remote server, however).
> 
> I'm re-reading the code to check on what I'm seeing. (Its been a long
> time since I last look deeply at this code).
> 
> You can write to /dev/log if you pass that to
> SysLogHandler(address='/dev/log'), but the default is to use a socket
> to talk to a network listener on localhost:514. There are no deamons
> listening on port 514 on my Fedora systems or mac OS.
> 
> That is not what you would expect as the default if you are using the C
> API.
> 
> What you do not see used in the SyslogHandler() is the import syslog
> and hence its nor using openlog() etc from syslog API.
> 
> Barry
> 
> 
> 
> >hp
> > 
> > -- 
> >   _  | Peter J. Holzer| Story must make more sense than reality.
> > |_|_) ||
> > | |   | h...@hjp.at <mailto:h...@hjp.at> |-- Charles Stross, 
> > "Creative writing
> > __/   | http://www.hjp.at/ <http://www.hjp.at/> |   challenge!"
> > -- 
> > https://mail.python.org/mailman/listinfo/python-list 
> > <https://mail.python.org/mailman/listinfo/python-list>
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list 
> <https://mail.python.org/mailman/listinfo/python-list>
> 
> 
> -- 
> Philip Bloom
> Director, Services Engineering
> AppLovin Corporation
> M: (786)-338-1439   
>  <https://www.linkedin.com/company/applovin>  <https://twitter.com/AppLovin>  
> <https://facebook.com/AppLovin>  <https://www.instagram.com/applovin/>
>  <https://www.applovin.com/>
> 
> 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Barry Scott]

> On 3 Mar 2022, at 13:24, computermaster360  
> wrote:
> 
> I want to make a little survey here.
> 
> Do you find the for-else construct useful?

No. I always have to look up what condition the else fires on.

> Have you used it in
> practice?

No.

> Do you even know how it works, or that there is such a thing
> in Python?

Yes (when I read the docs) and yes.

> I have used it maybe once. My issue with this construct is that
> calling the second block `else` doesn't make sense; a much more
> sensible name would be `then`.

There was a thread on this list asking for alternative syntax for the
for: else: that was not taken up. Something like nobreak: instead of else: 

> 
> Now, imagine a parallel universe, where the for-else construct would
> have a different behavior:
> 
>for elem in iterable:
>process(elem)
>else:
># executed only when the iterable was initially empty
>print('Nothing to process')
> 
> Wouldn't this be more natural? I think so. Also, I face this case much
> more often than having detect whether I broke out of a loop early
> (which is what the current for-else construct is for).

> 
> Now someone may argue that it's easy to check whether the iterable is
> empty beforehand. But is it really? What if it's an iterator?
> Then one would have to resort to using a flag variable and set it in
> each iteration of the loop. An ugly alternative would be trying to
> retrieve
> the first element of the iterable separately, in a try block before
> the for-loop, to find out whether the iterable is empty. This would of
> course
> require making an iterator of the iterable first (since we can't be
> sure it is already an iterator), and then -- if there are any elements
> -- processing
> the first element separately before the for-loop, which means
> duplicating the loop body. You can see the whole thing gets really
> ugly really quickly...
> 
> What are your thoughts? Do you agree? Or am I just not Dutch enough...?

Barry

> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Getting Syslog working on OSX Monterey

[Barry Scott]

> On 28 Feb 2022, at 21:41, Peter J. Holzer  wrote:
> 
> On 2022-02-27 22:16:54 +, Barry wrote:
>> If you look at the code of the logging modules syslog handle you will see 
>> that
>> it does not use syslog. It’s assuming that it can network to a syslog 
>> listener.
>> Such a listener is not running on my systems as far as I know.
>> 
>> I have always assumed that if I want a logger syslog handler that I would 
>> have
>> to implement it myself. So far I have code that uses syslog directly and have
>> not written that code yet.
> 
> What do you mean by using syslog directly? The syslog(3) library
> function also just sends messages to a "syslog listener" (more commonly
> called a syslog daemon) - at least on any unix-like system I'm familiar
> with (which doesn't include MacOS). It will, however, always use the
> *local* syslog daemon - AFAIK there is no standard way to open a remote
> connection (many syslog daemons can be configured to forward messages to
> a remote server, however).

I'm re-reading the code to check on what I'm seeing. (Its been a long
time since I last look deeply at this code).

You can write to /dev/log if you pass that to
SysLogHandler(address='/dev/log'), but the default is to use a socket
to talk to a network listener on localhost:514. There are no deamons
listening on port 514 on my Fedora systems or mac OS.

That is not what you would expect as the default if you are using the C
API.

What you do not see used in the SyslogHandler() is the import syslog
and hence its nor using openlog() etc from syslog API.

Barry



>hp
> 
> -- 
>   _  | Peter J. Holzer| Story must make more sense than reality.
> |_|_) ||
> | |   | h...@hjp.at |-- Charles Stross, "Creative writing
> __/   | http://www.hjp.at/ |   challenge!"
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ping script

[Barry Scott]

> On 27 Feb 2022, at 13:53, Byung-Hee HWANG  wrote:
> 
> simple ping check script with python3 (Python 3.9.2)
> tested under Debian 11 Bullseye:
> 
> soyeomul@penguin:~/gitlab/test$ ./fping.py localhost
> ok
> soyeomul@penguin:~/gitlab/test$ ./fping.py python.org
> ok
> soyeomul@penguin:~/gitlab/test$ ./fping.py python3.org
> python3.org: No address associated with hostname
> something is wrong...

This is correct python3.org  is only setup for email.
Use the host and dig commands to check for yourself.

Compare

$ host python.org 

with

$ host python3.org

And compare:

$ dig -t A python.org 
$ dig -t MX python.org 

with

$ dig -t A python3.org 
$ dig -t MX python3.org 

Barry

> soyeomul@penguin:~/gitlab/test$ 
> 
> Signed-off-by: Byung-Hee HWANG 
> ---
> fping.py | 31 +++
> 1 file changed, 31 insertions(+)
> create mode 100755 fping.py
> 
> diff --git a/fping.py b/fping.py
> new file mode 100755
> index 000..ccc1e58
> --- /dev/null
> +++ b/fping.py
> @@ -0,0 +1,31 @@
> +#!/usr/bin/env python3
> +# -*- coding: utf-8 -*-
> +
> +from subprocess import PIPE, Popen
> +from os import path
> +import sys
> +
> +"""
> +REFERENCE:
> +
> +"""
> +
> +if not path.isfile('/usr/bin/fping'):
> +sys.exit("you first install fping, then try again...")
> +
> +def check(xyz):
> +cmd = "fping %s" % (xyz)
> +try_ = Popen(cmd, stdout=PIPE, shell=True)
> +output = try_.communicate()[0].decode("utf-8")
> +
> +return output
> +
> +
> +if __name__ == "__main__":
> +xyz = sys.argv[1]
> +if "alive" in check(xyz):
> +print("ok")
> +else:
> +print("something is wrong...")
> +
> +# 2022-02-27, GNU Emacs 27.1 (Debian 11 Bullseye)
> -- 
> 2.30.2
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: One-liner to merge lists?

[Barry Scott]

> On 22 Feb 2022, at 09:30, Chris Angelico  wrote:
> 
> On Tue, 22 Feb 2022 at 20:24, Frank Millman  > wrote:
>> 
>> Hi all
>> 
>> I think this should be a simple one-liner, but I cannot figure it out.
>> 
>> I have a dictionary with a number of keys, where each value is a single
>> list -
>> 
> d = {1: ['aaa', 'bbb', 'ccc'], 2: ['fff', 'ggg']}
>> 
>> I want to combine all values into a single list -
>> 
> ans = ['aaa', 'bbb', 'ccc', 'fff', 'ggg']
>> 
>> I can do this -
>> 
> a = []
> for v in d.values():
>> ...   a.extend(v)
>> ...
> a
>> ['aaa', 'bbb', 'ccc', 'fff', 'ggg']
>> 
>> I can also do this -
>> 
> from itertools import chain
> a = list(chain(*d.values()))
> a
>> ['aaa', 'bbb', 'ccc', 'fff', 'ggg']
> 
>> 
>> Is there a simpler way?
>> 
> 
> itertools.chain is a good option, as it scales well to arbitrary
> numbers of lists (and you're guaranteed to iterate over them all just
> once as you construct the list). But if you know that the lists aren't
> too large or too numerous, here's another method that works:
> 
 sum(d.values(), [])
> ['aaa', 'bbb', 'ccc', 'fff', 'ggg']
> 
> It's simply adding all the lists together, though you have to tell it
> that you don't want a numeric summation.

If you code is performance sensitive do not use sum() as it creates lots of tmp 
list that are deleted.

I have an outstanding ticket at work to replace all use of sum() on lists as 
when we profiled it
stands out as a slow operation. We have a lots of list of list that we need to 
flatten.

Barry


> 
> ChrisA
> -- 
> https://mail.python.org/mailman/listinfo/python-list 
> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to check if there is internet?

[Barry Scott]

> On 25 Feb 2022, at 18:07, Abdur-Rahmaan Janhangeer  
> wrote:
> 
> Normally people put Python in the scripting category.
> I learnt typed languages like C++ and Java at first.
> People who learn languages like these tend to put
> Python in a non-serious category. The post was
> more ironic than litteral.

I think that python is a serious language, as I do C and C++.
What is key to a problem is which language is best to solve the
problem (given you have a free choice).

My day job is writing and maintaining a large python application.
Python allows us to use best engineering technics.

Python is used because it out performs C/C++/etc in the speed that
we can implement new features and generate revenue from them.

> 
> After 5 years of following the mailing lists i realised
> that there is more than the eyes meet. My hobby has
> always been language engineering / compiler theory
> and just recently i realised that Python is pretty serious about
> language features.
> 
> I have been following language feature proposals from various
> languages. Some decide to avoid Python's route,
> but others have been trying hard to catch up with Python.
> One gleaming example is the switch case. JS recently proposed pattern
> matching, referencing Python and explaining why the proposal
> is a cool treatment of the usecase.
> 
> As a side note, if by scripting we mean OS commands,
> then Python started as a sysadmin language.

It was started as a research language to teach children programming.
Python's simplicity, that we benefit from to this day, comes from that original 
purpose.

And indeed one of the pleasures of the UK PYCON is seeing children present
their projects to a room full of adult programers. Seeing a 6 year old present 
their
working project is wonderful to behold.

Barry


> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: venv and executing other python programs

[Barry Scott]

> On 15 Feb 2022, at 12:36, Martin Di Paola  wrote:
> 
> I did a few experiments in my machine. I created the following foo.py
> 
>  import pandas
>  print("foo")
> 
> Now "pandas" is installed under Python 3 outside the venv. I can run it 
> successfully calling "python3 foo.py".
> 
> If I add the shebang "#!/usr/bin/env python3" (notice the 3), I can also run 
> it as "./foo.py".

> 
> Calling it as "python foo.py" or using the shebang "#!/usr/bin/env python" 
> does not work and it makes sense since "pandas" is installed
> only for python 3.
> 
> New I create a virtual env with "python3 -m venv xxx" and activate it.
> 
> Once inside I can run foo.py in 4 different ways:
> 
> - python foo.py
> - python3 foo.py
> - ./foo.py (using the shebang "#!/usr/bin/env python")
> - ./foo.py (using the shebang "#!/usr/bin/env python3")
> 
> Now all of that was with "pandas" installed outside of venv but not inside.
> 
> I did the same experiments with another library, "cryptonita" which it is not 
> installed outside but inside and I was able to executed in 4 different ways 
> too (inside the venv of course):
> 
> - python foo.py
> - python3 foo.py
> - ./foo.py (using the shebang "#!/usr/bin/env python")
> - ./foo.py (using the shebang "#!/usr/bin/env python3")

If you have activated the venv then any script that uses /usr/bin/env will use 
executables from the venv
bin folder.

I'm seeing the following in the venv I made on my mac:

 % ls venv.tmp/bin
Activate.ps1dmgbuild*   modulegraph*pyi-archive_viewer* 
pyinstaller*python3.10@
activateds_store*   pip*pyi-bindepend*  
pylupdate5* pyuic5*
activate.cshmacho_dump* pip3*   pyi-grab_version*   
pyrcc5* wheel*
activate.fish   macho_find* pip3.10*pyi-makespec*   
python@
colour-print*   macho_standalone*   py2applet*  pyi-set_version*
python3@

You can see that "python", "python3" and "python3.10" are present.

So it does not help to change the /usr/bin/env to use python3.
Indeed its a feature that if you use /usr/bin/env you obviously want to use the
executable from the activated venv.

I avoid all these issues by not activating the venv. Python has code to know
how to use the venv libraries that are installed in it when invoked. It does 
not 
depend on the activate script being run.

Barry


> 
> Do you have a particular context where you are having troubles? May be there 
> is something else going on...
> 
> Thanks,
> Martin.
> 
> On Tue, Feb 15, 2022 at 06:35:18AM +0100, Mirko via Python-list wrote:
>> Hi,
>> 
>> I have recently started using venv for my hobby-programming. There
>> is an annoying problem. Since venv modifies $PATH, python programs
>> that use the "#!/usr/bin/env python" variant of the hashbang often
>> fail since their additional modules aren't install inside in venv.
>> 
>> How to people here deal with that?
>> 
>> Please note: I'm not interested in discussing whether the
>> env-variant is good or bad. ;-) It's not that *I* use it, but
>> several progs in /usr/bin/.
>> 
>> Thanks for your time.
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: venv and executing other python programs

[Barry Scott]

> On 15 Feb 2022, at 05:35, Mirko via Python-list  
> wrote:
> 
> Hi,
> 
> I have recently started using venv for my hobby-programming. There
> is an annoying problem. Since venv modifies $PATH, python programs
> that use the "#!/usr/bin/env python" variant of the hashbang often
> fail since their additional modules aren't install inside in venv.

Do you mean that your python code is running another python program
via subprocess?

You would need to provide a PATH that does not include the venv so that env
works as expected.

Or are you running the program from the command line after activating the
venv?

I create the venv and use it to run the program using
 /bin/python program.py without activating the venv

> How to people here deal with that?
> 
> Please note: I'm not interested in discussing whether the
> env-variant is good or bad. ;-) It's not that *I* use it, but
> several progs in /usr/bin/.

At least for Fedora it does not allow packages to install programs
that use /usr/bin/env because of issues like you are seeing.

Barry

> 
> Thanks for your time.
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data unchanged when passing data to Python in multiprocessing shared memory

[Barry Scott]

> On 1 Feb 2022, at 23:40, Jen Kris  wrote:
> 
> Barry, thanks for your reply.  
> 
> On the theory that it is not yet possible to pass data from a non-Python 
> language to Python with multiprocessing.shared_memory, I bypassed the problem 
> by attaching 4 bytes to my FIFO pipe message from NASM to Python:
> 
> byte_val = v[10:14]
> 
> where v is the message read from the FIFO.  Then:
> 
> breakup = int.from_bytes(byte_val, "big")
> print("this is breakup " + str(breakup))
> 
> Python prints:  this is breakup 32894
> 
> Note that I had to switch from little endian to big endian.  Python is little 
> endian by default, but in this case it's big endian.  
> 
> However, if anyone on this list knows how to pass data from a non-Python 
> language to Python in multiprocessing.shared_memory please let me (and the 
> list) know.  

By using the struct module you can control for endian of the data.

Barry

> 
> Thanks.  
> 
> 
> Feb 1, 2022, 14:20 by ba...@barrys-emacs.org:
> 
> On 1 Feb 2022, at 20:26, Jen Kris via Python-list  
> wrote:
> 
> I am using multiprocesssing.shared_memory to pass data between NASM and 
> Python. The shared memory is created in NASM before Python is called. Python 
> connects to the shm: shm_00 = 
> shared_memory.SharedMemory(name='shm_object_00',create=False). 
> 
> I have used shared memory at other points in this project to pass text data 
> from Python back to NASM with no problems. But now this time I need to pass a 
> 32-bit integer (specifically 32,894) from NASM to Python. 
> 
> First I convert the integer to bytes in a C program linked into NASM:
> 
> unsigned char bytes[4]
> unsigned long int_to_convert = 32894;
> 
> bytes[0] = (int_to_convert >> 24) & 0xFF;
> bytes[1] = (int_to_convert >> 16) & 0xFF;
> bytes[2] = (int_to_convert >> 8) & 0xFF;
> bytes[3] = int_to_convert & 0xFF;
> memcpy(outbuf, bytes, 4);
> 
> where outbuf is a pointer to the shared memory. On return from C to NASM, I 
> verify that the first four bytes of the shared memory contain what I want, 
> and they are 0, 0, -128, 126 which is binary   1000 
> 0110, and that's correct (32,894). 
> 
> Next I send a message to Python through a FIFO to read the data from shared 
> memory. Python uses the following code to read the first four bytes of the 
> shared memory:
> 
> byte_val = shm_00.buf[:4]
> print(shm_00.buf[0])
> print(shm_00.buf[1])
> print(shm_00.buf[2])
> print(shm_00.buf[3])
> 
> But the bytes show as 40 39 96 96, which is exactly what the first four bytes 
> of this shared memory contained before I called C to overwrite them with the 
> bytes 0, 0, -128, 126. So Python does not see the updated bytes, and 
> naturally int.from_bytes(byte_val, "little") does not return the result I 
> want. 
> 
> I know that Python refers to shm00.buf, using the buffer protocol. Is that 
> the reason that Python can't see the data that has been updated by another 
> language? 
> 
> So my question is, how can I alter the data in shared memory in a non-Python 
> language to pass back to Python?
> 
> Maybe you need to use a memory barrier to force the data to be seen by 
> another cpu?
> Maybe use shm lock operation to sync both sides?
> Googling I see people talking about using stdatomic.h for this.
> 
> But I am far from clear what you would need to do.
> 
> Barry
> 
> Thanks,
> 
> Jen
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Barry Scott]

> On 27 Jan 2022, at 07:46, Tony Flury  wrote:
> 
> 
> On 26/01/2022 22:41, Barry wrote:
>> 
>> 
>> Run python and your code under a debugger and check the ref count of the 
>> object as you step through the code.
>> 
>> Don’t just step through your code but also step through the C python code.
>> That will allow you to see how this works at a low level.
>> Setting a watch point on the ref count will allow you run the code and just 
>> break as the ref count changes.
>> 
>> That is what I do when a see odd c api behaviour.
>> 
>> Barry
> 
> 
> Thanks - I have tried a few times on a few projects to run a debugger in 
> mixed language mode and never had any success.
> 
> I will have to try again.

You mean debugging Python and C/C++? In this case your python code is a simple 
test script and it's C that you care about.
Should not be difficult. I tend to use linux as my lead debug platform as its 
the easiest to work with. But Windows and macOS
also have very good debuggers.

Barry

> 
> 
>>> As posted in the original message - immediately before the call to the C 
>>> function/method sys.getrefcount reports the count to be 2 (meaning it is 
>>> actually a 1).
>>> 
>>> Inside the C function the ref count is incremented and the Py_REFCNT macro 
>>> reports the count as 3 inside the C function as expected (1 for the name in 
>>> the Python code, 1 for the argument as passed to the C function, and 1 for 
>>> the increment), so outside the function one would expect the ref count to 
>>> now be 2 (since the reference caused by calling the function is then 
>>> reversed).
>>> 
>>> However - Immediately outside the C function and back in the Python code 
>>> sys.getrefcount reports the count to be 2 again - meaning it is now really 
>>> 1. So that means that the refcount has been decremented twice in-between 
>>> the return of the C function and the execution of the immediate next python 
>>> statement. I understand one of those decrements - the parameter's ref count 
>>> is incremented on the way in so the same object is decremented on the way 
>>> out (so that calls don't leak references) but I don't understand where the 
>>> second decrement is coming from.
>>> 
>>> Again there is nothing in the Python code that would cause that decrement - 
>>> the decrement behavior is in the Python runtime.
>>> 
> -- 
> Anthony Flury
> email :anthony.fl...@btinternet.com
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 
>>> -- 
>>> Anthony Flury
>>> email :anthony.fl...@btinternet.com
> 
> -- 
> Anthony Flury
> email : anthony.fl...@btinternet.com
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to test for maildir 'folder' in Python?

[Barry Scott]

> On 22 Jan 2022, at 21:26, Chris Green  wrote:
> 
> I have a script that walks a quite deep tree of mail messages to find
> and archive old messages.  I'm trying to convert it from mbox to
> maildir (as I now store my mail in maildir format).
> 
> So I need to test whether a point I have reached in the hierarchy is a
> maildir mailbox or not.  Using mbox format it's easy because 'folders'
> are directories and mailboxes are files.  However with maildir the
> 'folders' have directories within them so the simple tree walking goes
> down a level too far and finds 'folders' which aren't mailboxes called
> 'cur', 'new' and 'tmp'.
> 
> Is there any 'ready made' way in python to tell whether a directory is
> a maildir mailbox?  If not I suppose I'll simply have to check if
> there are 'cur', 'new' and 'tmp' directories within the directory
> which may or may not be a maildir.

You do not need to walk the tree.

The structure is 
Maildir/cur, new, tmp - The INBOX
Maildir//cur, new, tmp

The encoding prefixed with a "." and uses a "." between folder name parts.

An example from my Maildir is ".Python.Users/" for a folder that client
shows as Python/Users

You can see this by doing:

 ls -a ~/Maildir

Barry




> 
> -- 
> Chris Green
> ·
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Barry Scott]

> On 19 Jan 2022, at 10:57, Tony Flury via Python-list  
> wrote:
> 
> I am writing a C extension module for an AVL tree, and I am trying to ensure 
> reference counting is done correctly. I was having a problem with the 
> reference counting so I worked up this little POC of the problem, and I hope 
> someone can explain this.

The ref counting in the C API not easy to get right. Sometimes you must inc 
some times you must not inc.
There are a lot of libraries that take that problem away from you.

For example I maintain PyCXX that is a C++ library that allows your drive the C 
API without needed to worry about ref counting.
There are lots of other libraries as well that aim to do the same thing.

http://cxx.sourceforge.net/ 

Barry




> 
> Extension function :
> 
>   static PyObject *_Node_test_ref_count(PyObject *self)
>   {
>printf("\nIncrementing ref count for self - just for the hell
>   of it\n");
>printf("\n before self has a ref count of %ld\n", Py_REFCNT(self));
>Py_INCREF(self);
>printf("\n after self has a ref count of %ld\n", Py_REFCNT(self));
>fflush(stdout);
>return self;
>   }
> 
> As you can see this function purely increments the reference count of the 
> instance.
> 
> /Note: I understand normally this would be the wrong this to do, but this is 
> a POC of the issue, not live code. In the live code I am attaching a 2nd 
> nodes to each other, and the live code therefore increments the ref-count for 
> both objects - so even if the Python code deletes it's reference the 
> reference count for the instance should still be 1 in order to ensure it 
> doesn't get garbage collected./
> 
> This function is exposed as the test_ref method.
> 
> This is the test case :
> 
> def test_000_009_test_ref_count(self):
> node = _Node("Hello")
> self.assertEqual(sys.getrefcount(node), 2)
> node.test_ref()
> self.assertEqual(sys.getrefcount(node), 3)
> 
> The output of this test case is :
> 
> test_000_009_test_ref_count (__main__.TestNode) ...
> Incrementing ref count for self - just for the hell of it
> 
>  before self has a ref count of 2
> 
>  after self has a ref count of 3
> FAIL
> 
> ==
> FAIL: test_000_009_test_ref_count (__main__.TestNode)
> --
> Traceback (most recent call last):
>   File "/home/tony/Development/python/orderedtree/tests/test_orderedtree.py", 
> line 62, in test_000_009_test_ref_count
> self.assertEqual(sys.getrefcount(node), 3)
> AssertionError: 2 != 3
> 
> So I understand why the first assert will be true - when the 
> sys.getrefcount() function is called the ref count is incremented temporarily 
> (as a borrowed reference), so there are now two references - the 'node' 
> variable, and the borrowed reference in the function call.
> 
> We then call the 'test_ref' method, and again that call causes a borrowed 
> reference (hence the ref count being 2 initially within the method). The 
> 'test_ref' method increments the reference of the instance - as you can see 
> from the output we now have a ref count of 3 - (that count is the 'node' 
> variable in the test case, the borrowed reference due to the method call, and 
> the artificial increment from the 'ref_test' method).
> 
> When the 'ref_test' method exits I would expect the ref count of the instance 
> to now be 2 (one for the 'node' variable, and one as a result of the 
> artificial increment increment').
> 
> I would therefore expect the 2nd assertEqual in the test case to succeed. - 
> in this case the borrowed reference within sys.getfrefcount() should cause 
> the count to be 3.
> 
> As you see though that 2nd assertEqual fails - suggesting that the refcount 
> of 'node' is actually only 1 when the 'test_ref' method exits.
> 
> Can someone explain why the 'test_ref' method fails to change the refcount of 
> the 'node' instance.
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why There Is No Python Compressed Archive or Binaries ?

[Barry Scott]

> On 17 Jan 2022, at 19:53, Sina Mobasheri  wrote:
> 
> Consider scenario that I want run python 3.10 in CentOS 8, I think last 
> python version in CentOS repository is 3.6, if I use epel I can get 3.8 so 
> ..., I think (correct me if I'm wrong ) the only way that I can run python 
> 3.10 is to compile it manually, which is need to know what dependencies 
> python needs for compilation ... (different distribution different packages, 
> which packages for what, you can see that it is intimidating for beginners 
> like me)
> It's useful to just use wget  tar.xz>, unzipped, ser path and ta-da you have cpython 3.10 in CentOS 8

Are there features in python 3.10 that are missing from 3.6 that you need?
If not then the easy thing to do is use 3.6.

You could build a docker containers you based on fedora 35 that has the python 
3.10
and the python libs that you need installed and use that.

You would use podman and its tools to build and run the container on Centos 8.

For my work I'm planning to use 3.6 on Centos 8 as that is one thing I can 
avoid packaging
and maintaining myself.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Who wrote Py_UNREACHABLE?

[Barry Scott]

> On 2 Jan 2022, at 16:17, Marco Sulla  wrote:
> 
> #if defined(RANDALL_WAS_HERE)
> #  define Py_UNREACHABLE() \
>Py_FatalError( \
>"If you're seeing this, the code is in what I thought was\n" \
>"an unreachable state.\n\n" \
>"I could give you advice for what to do, but honestly, why\n" \
>"should you trust me?  I clearly screwed this up.  I'm writing\n" \
>"a message that should never appear, yet I know it will\n" \
>"probably appear someday.\n\n" \
>"On a deep level, I know I'm not up to this task.\n" \
>"I'm so sorry.\n" \
>"https://xkcd.com/2200;)
> #elif defined(Py_DEBUG)
> #  define Py_UNREACHABLE() \
>Py_FatalError( \
>"We've reached an unreachable state. Anything is possible.\n" \
>"The limits were in our heads all along. Follow your dreams.\n" \
>"https://xkcd.com/2200;)

use git blame to find out the commit then use git log to read the commit 
message.

Barry


> 
> etc
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: recover pickled data: pickle data was truncated

[Barry Scott]

> On 1 Jan 2022, at 16:13, Marco Sulla  wrote:
> 
> I agree with Barry. You can create a folder or a file with
> pseudo-random names. I recommend you to use str(uuid.uuid4())

At work and personally I use iso-8601 timestamps to make the files unique and 
easy to
find out when they where created.

:>>> t = datetime.datetime.now()
:>>> t
datetime.datetime(2022, 1, 2, 12, 34, 1, 267935)
:>>> t.strftime('%Y-%m-%dT%H-%M-%S')
'2022-01-02T12-34-01'
:>>>

That is good enough as long as you create the files slower than once a second.

Oh and yes use JSON, it is far better as a way of exchanging data than pickle.
Easy to read and check, can be processes in many languages.

Barry


> 
> On Sat, 1 Jan 2022 at 14:11, Barry  wrote:
>> 
>> 
>> 
>>> On 31 Dec 2021, at 17:53, iMath  wrote:
>>> 
>>> 在 2021年12月30日星期四 UTC+8 03:13:21， 写道：
> On Wed, 29 Dec 2021 at 18:33, iMath  wrote:
> But I found the size of the file of the shelve data didn't change much, 
> so I guess the data are still in it , I just wonder any way to recover my 
> data.
 I agree with Barry, Chris and Avi. IMHO your data is lost. Unpickling
 it by hand is a harsh work and maybe unreliable.
 
 Is there any reason you can't simply add a semaphore to avoid writing
 at the same time and re-run the code and regenerate the data?
>>> 
>>> Thanks for your replies! I didn't have a sense of adding a semaphore on 
>>> writing to pickle data before, so  corrupted the data.
>>> Since my data was colleted in the daily usage, so cannot re-run the code 
>>> and regenerate the data.
>>> In order to avoid corrupting my data again and the complicity of using  a 
>>> semaphore, now I am using json text to store my data.
>> 
>> That will not fix the problem. You will end up with corrupt json.
>> 
>> If you have one writer and one read then may be you can use the fact that a 
>> rename is atomic.
>> 
>> Writer does this:
>> 1. Creat new json file in the same folder but with a tmp name
>> 2. Rename the file from its tmp name to the public name.
>> 
>> The read will just read the public name.
>> 
>> I am not sure what happens in your world if the writer runs a second time 
>> before the data is read.
>> 
>> In that case you need to create a queue of files to be read.
>> 
>> But if the problem is two process racing against each other you MUST use 
>> locking.
>> It cannot be avoided for robust operations.
>> 
>> Barry
>> 
>> 
>>> --
>>> https://mail.python.org/mailman/listinfo/python-list
>> 
>> --
>> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: recover pickled data: pickle data was truncated

[Barry Scott]

> On 26 Dec 2021, at 13:44, Marco Sulla  wrote:
> 
> Use a semaphore.
> 
> On Sun, 26 Dec 2021 at 03:30, iMath  wrote:
>> 
>> Normally, the shelve data should be read and write by only one process at a 
>> time, but unfortunately it was simultaneously read and write by two 
>> processes, thus corrupted it. Is there any way to recover all data in it ? 
>> Currently I just get "pickle data was truncated" exception after reading a 
>> portion of the data?

You have lost the data in that case.

You will need to do what Marco suggests and lock access to the file.
How you do that depends your OS. If is unix OS then its likely you
will want to use fcntl.flock().

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Uninstall old python version(Python 3.9.7)

[Barry Scott]

> On 25 Dec 2021, at 06:13, Công Huy  wrote:
> 
>   I had uninstalled Python 3.9.7 before but it wasn't uninstalled
>   completely. I found it still in my computer and when I click "uninstall",
>   it sent me a board "No Python 3.9 installation was detected". Of course,
>   it won't be an issue if it defaulted my main Python version on my computer
>   and I can't use some libraries I installed like numpy, matplotlib,
>   networkx,... To solve my python homework. I found some help on the
>   internet but it wasn't improved. I don't know how to fix it. Please help
>   me.

You seem to saying that you need 3.9, but you have removed it.
I'm guessing there is more going on then you have explained.
Otherwise simply reinstall python 3.9.

Barry

p.s. You signature contains an advert for outlook.
You might want to get rid of that.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?

[Barry Scott]

> On 26 Dec 2021, at 13:48, Marco Sulla  wrote:
> 
> I have to use _PyObject_GC_IS_TRACKED(). It can't be used unless you
> define Py_BUILD_CORE. I want to avoid this. What macro or function can
> substitute it?

Why is this needed by your code? Surely the GC does its thing as an 
implementation detail of python.

Barry


> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Selection sort

[Barry Scott]

> On 24 Dec 2021, at 14:22, vani arul  wrote:
> 
> Hello,
> I am trying write a code.Can some help me find the error in my code.
> Thanks!
> 
> 
> def selectionsort(arr):
>   # le=len(arr)
>for b in range(0,len(arr)-1):
>pos=b
>for a in range(b+1,len(arr)-1):
>if arr[b]>arr[a+1]:
>arr[b],arr[a+1]=arr[a+1],arr[b]
>return arr
> 
> arr=[3,5,9,8,2,6]
> print(selectionsort(arr))

What are you expecting the code to do?

Barry



> --

> 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Py_TRASHCAN_SAFE_BEGIN/END in C extension?

[Barry Scott]

> On 22 Dec 2021, at 08:14, Marco Sulla  wrote:
> 
> Yes, it's deprecated, but I need it for Python 3.7, since there was
> yet no Py_TRASHCAN_BEGIN / END

Hopefully the bug report will makes clear what you have to do in your code to 
get things working.

Barry

> 
> On Tue, 21 Dec 2021 at 23:22, Barry  wrote:
>> 
>> 
>> 
>> On 21 Dec 2021, at 22:08, Marco Sulla  wrote:
>> 
>> In Python 3.7, must Py_TRASHCAN_SAFE_BEGIN - Py_TRASHCAN_SAFE_END be
>> used in a C extension?
>> 
>> I'm asking because in my C extension I use them in the deallocator
>> without problems, but users signalled me that they segfault in Python
>> 3.7 on Debian 10. I checked and this is true.
>> 
>> 
>> I searched the web for Py_TRASHCAN_SAFE_BEGIN
>> And that quickly lead me to this bug.
>> 
>> https://bugs.python.org/issue40608.
>> 
>> That gives lots of clues for what might be the problem.
>> It seems that is a deprecated api.
>> 
>> Barry
>> 
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>> 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Barry Scott]

> On 8 Dec 2021, at 17:11, Jen Kris  wrote:
> 
> I started this post on November 29, and there have been helpful comments 
> since then from Barry Scott, Cameron Simpson, Peter Holzer and Chris 
> Angelico.  Thanks to all of you.  
> 
> I've found a solution that works for my purpose, and I said earlier that I 
> would post the solution I found. If anyone has a better solution I would 
> appreciate any feedback. 
> 
> To recap, I'm using a pair of named pipes for IPC between C and Python.  
> Python runs as a child process after fork-execv.  The Python program 
> continues to run concurrently in a while True loop, and responds to requests 
> from C at intervals, and continues to run until it receives a signal from C 
> to exit.  C sends signals to Python, then waits to receive data back from 
> Python.  My problem was that C was blocked when Python started. 
> 
> The solution was twofold:  (1) for Python to run concurrently it must be a 
> multiprocessing loop (from the multiprocessing module), and (2) Python must 
> terminate its write strings with \n, or read will block in C waiting for 
> something that never comes.  The multiprocessing module sidesteps the GIL; 
> without multiprocessing the GIL will block all other threads once Python 
> starts. 
> 
> Originally I used epoll() on the pipes.  Cameron Smith and Barry Scott 
> advised against epoll, and for this case they are right.  Blocking pipes work 
> here, and epoll is too much overhead for watching on a single file 
> descriptor. 
> 
> This is the Python code now:
> 
> #!/usr/bin/python3
> from multiprocessing import Process

You already have feedback that multiprocessing is not required.

> import os
> 
> print("Python is running")
> 
> child_pid = os.getpid()
> print('child process id:', child_pid)
> 
> def f(a, b):
> 
> print("Python now in function f")
> 
> pr = os.open('/tmp/Pipe_01', os.O_RDONLY)
> print("File Descriptor1 Opened " + str(pr))
> pw = os.open('/tmp/Pipe_02', os.O_WRONLY)
> print("File Descriptor2 Opened " + str(pw))
> 
> while True:
> 
> v = os.read(pr,64)
> print("Python read from pipe pr")
> print(v)
> 
> if v == b'99':
> os.close(pr)
> os.close(pw)
> print("Python is terminating")
> os._exit(os.EX_OK)
> 
> if v != "Send child PID":
> os.write(pw, b"OK message received\n")

The \n should not be required as UDS (unix domain sockets) are message based 
not a stream of bytes.

> print("Python wrote back")
> 
> if __name__ == '__main__':
> a = 0
> b = 0
> p = Process(target=f, args=(a, b,))
> p.start()
> p.join()
> 
> The variables a and b are not currently used in the body, but they will be 
> later. 
> 
> This is the part of the C code that communicates with Python:
> 
> fifo_fd1 = open(fifo_path1, O_WRONLY);
> fifo_fd2 = open(fifo_path2, O_RDONLY);
> 
> status_write = write(fifo_fd1, py_msg_01, sizeof(py_msg_01));
> if (status_write < 0) perror("write");
You continue on after the error, exit would be better.
> 
> status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
> if (status_read < 0) perror("read");
> printf("C received message 1 from Python\n");
> printf("%.*s",(int)buf_len, fifo_readbuf);

The length of the data read is in status_read not buf_len.

> 
> status_write = write(fifo_fd1, py_msg_02, sizeof(py_msg_02));

How much data did you put into py_msg_01 buffer?
Is it a C string? If so you want to write sizeof(py_msg_02)-1 to avoid sending 
the trailing NUL (0)
of the C string.

> if (status_write < 0) perror("write");
If it failed exit until you have figured out the error handling.
> 
> status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
> if (status_read < 0) perror("read");
> printf("C received message 2 from Python\n");
> printf("%.*s",(int)buf_len, fifo_readbuf);

The length of the data read is in status_read not buf_len.

At no point do I see in this C code the need for a \n in the data sent between 
python and C.

> 
> // Terminate Python multiprocessing
> printf("C is sending exit message to Python\n");
> status_write = write(fifo_fd1, py_msg_03, 2);

Would be better to not be using "2" for the length to write here.
If py_msg_03 only has the exit msg in it the use sizeof(py_msg_03).
If its a C string the subtract 1 for the trailing NUL (0).

> 
> printf("C is closi

Re: Python child process in while True loop blocks parent

[Barry Scott]

> On 6 Dec 2021, at 21:05, Jen Kris  wrote:
> 
> Here is what I don't understand from what you said.  "The child process is 
> created with a single thread—the one that called fork()."  To me that implies 
> that the thread that called fork() is the same thread as the child process.  
> I guess you're talking about the distinction between logical threads and 
> physical threads.  

The thread that called fork is cloned into a new thread in the new process.
It has a clone of the processor registers of the thread, stack memory segment 
of that thread,
which means that it has all the stack variables and stack frames from the 
parent process's thread.


> But the main issue is your suggestion that I should call fork-execv from the 
> thread that runs the main C program, not from a separate physical pthread.  
> That would certainly eliminate the overhead of creating a new pthread. 

Forget about physical threads, they only matter when you are performance tuning 
your code.

> I am working now to finish this, and I will try your suggestion of calling 
> fork-execv from the "main" thread.  When I reply back next I can give you a 
> complete picture of what I'm doing. 
> 
> Your comments, and those of Peter Holzer and Chris Angelico, are most 
> appreciated. 

Barry


> 
> 
> Dec 6, 2021, 10:37 by ba...@barrys-emacs.org:
> 
> On 6 Dec 2021, at 17:09, Jen Kris via Python-list  
> wrote:
> 
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread." From the Linux man pages 
> https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is 
> created with a single thread—the one that called fork()."
> 
> You just quoted the evidence!
> 
> All new processes on unix (may all OS) only ever have one thread when they 
> start.
> The thread-id of the first thread is the same as the process-id and referred 
> to as the main thread.
> 
> I have a one-core one-thread instance at Digital Ocean available running 
> Ubuntu 18.04. I can fork and create a new process on it, but it doesn't 
> create a new thread because it doesn't have one available.
> 
> 
> By that logic it can only run one process...
> 
> It has one hardware core that support one hardware thread.
> Linux can create as many software threads as it likes.
> You may also want to see "Forking vs Threading" 
> (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), 
> "Fork vs Thread" 
> (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux 
> process and thread" (https://zliu.org/post/linux-process-and-thread) ("This 
> means that to create a normal process fork() is used that further calls 
> clone() with appropriate arguments while to create a thread or LWP, a 
> function from pthread library calls clone() with relvant flags. So, the main 
> difference is generated by using different flags that can be passed to 
> clone() funciton(to be exact, it is a system call"). 
> 
> You may be confused by the fact that threads are called light-weight 
> processes.
> 
> No Peter and I are not confused.
> 
> Or maybe I'm confused :)
> 
> Yes you are confused.
> 
> If you have other information, please let me know. Thanks.
> 
> Please get the book I recommended, or another that covers systems programming 
> on unix, and have a read.
> 
> Barry
> 
> Jen
> 
> 
> Dec 5, 2021, 18:08 by hjp-pyt...@hjp.at:
> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
> The C program creates two threads (using pthreads), one for itself and
> one for the child process. On creation, the second pthread is pointed
> to a C program that calls fork-execv to run the Python program. That
> way Python runs on a separate thread. 
> 
> I think you have the relationship between processes and threads
> backwards. A process consists of one or more threads. Fork creates a new
> process and therefore also a new thread.
> 
> hp
> 
> -- 
> _ | Peter J. Holzer | Story must make more sense than reality.
> |_|_) | |
> | | | h...@hjp.at | -- Charles Stross, "Creative writing
> __/ | http://www.hjp.at/ | challenge!"
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: threading and multiprocessing deadlock

[Barry Scott]

> On 5 Dec 2021, at 23:50, Johannes Bauer  wrote:
> 
> Hi there,
> 
> I'm a bit confused. In my scenario I a mixing threading with
> multiprocessing. Threading by itself would be nice, but for GIL reasons
> I need both, unfortunately. I've encountered a weird situation in which
> multiprocessing Process()es which are started in a new thread don't
> actually start and so they deadlock on join.
> 
> I've created a minimal example that demonstrates the issue. I'm running
> on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
> ([GCC 10.3.0] on linux).
> 
> Here's the code:

I suggest that you include the threading.current_thread() in your messages.
Then you can see which thread is saying what.

Barry

> 
> 
> import time
> import multiprocessing
> import threading
> 
> def myfnc():
>   print("myfnc")
> 
> def run(result_queue, callback):
>   result = callback()
>   result_queue.put(result)
> 
> def start(fnc):
>   def background_thread():
>   queue = multiprocessing.Queue()
>   proc = multiprocessing.Process(target = run, args = (queue, 
> fnc))
>   proc.start()
>   print("join?")
>   proc.join()
>   print("joined.")
>   result = queue.get()
>   threading.Thread(target = background_thread).start()
> 
> start(myfnc)
> start(myfnc)
> start(myfnc)
> start(myfnc)
> while True:
>   time.sleep(1)
> 
> 
> What you'll see is that "join?" and "joined." nondeterministically does
> *not* appear in pairs. For example:
> 
> join?
> join?
> myfnc
> myfnc
> join?
> join?
> joined.
> joined.
> 
> What's worse is that when this happens and I Ctrl-C out of Python, the
> started Thread is still running in the background:
> 
> $ ps ax | grep minimal
> 370167 pts/0S  0:00 python3 minimal.py
> 370175 pts/2S+ 0:00 grep minimal
> 
> Can someone figure out what is going on there?
> 
> Best,
> Johannes
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Barry Scott]

> On 6 Dec 2021, at 17:09, Jen Kris via Python-list  
> wrote:
> 
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread."  From the Linux man pages 
> https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is 
> created with a single thread—the one that called fork()." 

You just quoted the evidence!

All new processes on unix (may all OS) only ever have one thread when they 
start.
The thread-id of the first thread is the same as the process-id and referred to 
as the main thread.

> 
> I have a one-core one-thread instance at Digital Ocean available running 
> Ubuntu 18.04.  I can fork and create a new process on it, but it doesn't 
> create a new thread because it doesn't have one available. 


By that logic it can only run one process...

It has one hardware core that support one hardware thread.
Linux can create as many software threads as it likes.

> You may also want to see "Forking vs Threading" 
> (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), 
> "Fork vs Thread" 
> (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux 
> process and thread" (https://zliu.org/post/linux-process-and-thread) ("This 
> means that to create a normal process fork() is used that further calls 
> clone() with appropriate arguments while to create a thread or LWP, a 
> function from pthread library calls clone() with relvant flags. So, the main 
> difference is generated by using different flags that can be passed to 
> clone() funciton(to be exact, it is a system call"). 
> 
> You may be confused by the fact that threads are called light-weight 
> processes. 

No Peter and I are not confused.

> 
> Or maybe I'm confused :)

Yes you are confused.

> 
> If you have other information, please let me know.  Thanks. 

Please get the book I recommended, or another that covers systems programming 
on unix, and have a read.

Barry

> 
> Jen
> 
> 
> Dec 5, 2021, 18:08 by hjp-pyt...@hjp.at:
> 
>> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
>> 
>>> The C program creates two threads (using pthreads), one for itself and
>>> one for the child process.  On creation, the second pthread is pointed
>>> to a C program that calls fork-execv to run the Python program.  That
>>> way Python runs on a separate thread. 
>>> 
>> 
>> I think you have the relationship between processes and threads
>> backwards. A process consists of one or more threads. Fork creates a new
>> process and therefore also a new thread.
>> 
>> hp
>> 
>> -- 
>> _  | Peter J. Holzer| Story must make more sense than reality.
>> |_|_) ||
>> | |   | h...@hjp.at |-- Charles Stross, "Creative writing
>> __/   | http://www.hjp.at/ |   challenge!"
>> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Barry Scott]

> On 1 Dec 2021, at 16:01, Jen Kris  wrote:
> 
> Thanks for your comment re blocking.  
> 
> I removed pipes from the Python and C programs to see if it blocks without 
> them, and it does.
> It looks now like the problem is not pipes.

Ok.

> I use fork() and execv() in C to run Python in a child process, but the 
> Python process blocks

Use strace on the parent process to see what is happening.
You will need to use the option to follow subprocesses so that you can see what 
goes on in the python process.

See man strace and the --follow-forks and --output-separately options.
That will allow you to find the blocking system call that your code is making.

> because fork() does not create a new thread, so the Python global interpreter 
> lock (GIL) prevents the C program from running once Python starts.

Not sure why you think this.

>   So the solution appears to be run Python in a separate thread, which I can 
> do with pthread create.
>   See "Thread State and the Global Interpreter Lock" 
> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock
>  
> 
>  and the sections below that "Non-Python created threads" and "Cautions about 
> fork()." 

I take it you mean that in the parent you think that using pthreads will affect 
python after the exec() call?
I does not. After exec() the process has one main thread create by the kernel 
and a new address space as defined by the /usr/bin/python.
The only state that in inherited from the parent are open file descriptors, the 
current working directory and security state like UID, GID.

> I'm working on that today and I hope all goes well :) 

You seem to be missing background information on how processes work.
Maybe "Advanced Programming in the UNIX Environment" would be helpful?

https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk
 

 

It's a great book and covers a wide range of Unix systems programming topics.

Have you created a small C program that just does the fork and exec of a python 
program to test out your assumptions?
If not I recommend that you do.

Barry


> 
> 
> 
> Nov 30, 2021, 11:42 by ba...@barrys-emacs.org:
> 
> 
>> On 29 Nov 2021, at 22:31, Jen Kris > > wrote:
>> 
>> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl 
>> set, but no event loop to handle it yet.  I'm putting that in now with a 
>> pipe write in Python-- as Cameron pointed out that is the likely source of 
>> blocking on C.  The pipes are opened as rdwr in Python because that's 
>> nonblocking by default.  The child will become more complex, but not in a 
>> way that affects polling.  And thanks for the tip about the c-string 
>> termination. 
>> 
> 
> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY 
> access.
> 
> You may find
> 
> man 2 open
> 
> useful to understand in detail what is behind os.open().
> 
> Barry
> 
> 
> 
>> 
>> 
>> Nov 29, 2021, 14:12 by ba...@barrys-emacs.org 
>> :
>> 
>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list > > wrote:
>> 
>> I have a C program that forks to create a child process and uses execv to 
>> call a Python program. The Python program communicates with the parent 
>> process (in C) through a FIFO pipe monitored with epoll(). 
>> 
>> The Python child process is in a while True loop, which is intended to keep 
>> it running while the parent process proceeds, and perform functions for the 
>> C program only at intervals when the parent sends data to the child -- 
>> similar to a daemon process. 
>> 
>> The C process writes to its end of the pipe and the child process reads it, 
>> but then the child process continues to loop, thereby blocking the parent. 
>> 
>> This is the Python code:
>> 
>> #!/usr/bin/python3
>> import os
>> import select
>> 
>> #Open the named pipes
>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>> Why open rdwr if you are only going to read the pipe?
>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>> Only need to open for write.
>> 
>> ep = select.epoll(-1)
>> ep.register(pr, select.EPOLLIN)
>> 
>> Is the only thing that the child does this:
>> 1. Read message from pr
>> 2. Process message
>> 3. Write result to pw.
>> 4. Loop from 1
>> 
>> If so as Cameron said you do not need to worry about the poll.
>> Do you plan for the child to become more complex?
>> 
>> while True:
>> 
>> events = ep.poll(timeout=2.5, maxevents=-1)
>> #events = ep.poll(timeout=None, maxevents=-1)
>> 
>> print("child is looping")
>> 
>> for fileno, event in events:
>> print("Python fileno")
>> 

Re: Python child process in while True loop blocks parent

[Barry Scott]

> On 29 Nov 2021, at 22:31, Jen Kris  wrote:
> 
> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl set, 
> but no event loop to handle it yet.  I'm putting that in now with a pipe 
> write in Python-- as Cameron pointed out that is the likely source of 
> blocking on C.  The pipes are opened as rdwr in Python because that's 
> nonblocking by default.  The child will become more complex, but not in a way 
> that affects polling.  And thanks for the tip about the c-string termination. 
> 

flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY 
access.

You may find

man 2 open

useful to understand in detail what is behind os.open().

Barry



> 
> 
> Nov 29, 2021, 14:12 by ba...@barrys-emacs.org:
> 
> On 29 Nov 2021, at 20:36, Jen Kris via Python-list  
> wrote:
> 
> I have a C program that forks to create a child process and uses execv to 
> call a Python program. The Python program communicates with the parent 
> process (in C) through a FIFO pipe monitored with epoll(). 
> 
> The Python child process is in a while True loop, which is intended to keep 
> it running while the parent process proceeds, and perform functions for the C 
> program only at intervals when the parent sends data to the child -- similar 
> to a daemon process. 
> 
> The C process writes to its end of the pipe and the child process reads it, 
> but then the child process continues to loop, thereby blocking the parent. 
> 
> This is the Python code:
> 
> #!/usr/bin/python3
> import os
> import select
> 
> #Open the named pipes
> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
> Why open rdwr if you are only going to read the pipe?
> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
> Only need to open for write.
> 
> ep = select.epoll(-1)
> ep.register(pr, select.EPOLLIN)
> 
> Is the only thing that the child does this:
> 1. Read message from pr
> 2. Process message
> 3. Write result to pw.
> 4. Loop from 1
> 
> If so as Cameron said you do not need to worry about the poll.
> Do you plan for the child to become more complex?
> 
> while True:
> 
> events = ep.poll(timeout=2.5, maxevents=-1)
> #events = ep.poll(timeout=None, maxevents=-1)
> 
> print("child is looping")
> 
> for fileno, event in events:
> print("Python fileno")
> print(fileno)
> print("Python event")
> print(event)
> v = os.read(pr,64)
> print("Pipe value")
> print(v)
> 
> The child process correctly receives the signal from ep.poll and correctly 
> reads the data in the pipe, but then it continues looping. For example, when 
> I put in a timeout:
> 
> child is looping
> Python fileno
> 4
> Python event
> 1
> Pipe value
> b'10\x00'
> The C code does not need to write a 0 bytes at the end.
> I assume the 0 is from the end of a C string.
> UDS messages have a length.
> In the C just write 2 byes in the case.
> 
> Barry
> child is looping
> child is looping
> 
> That suggests that a while True loop is not the right thing to do in this 
> case. My question is, what type of process loop is best for this situation? 
> The multiprocessing, asyncio and subprocess libraries are very extensive, and 
> it would help if someone could suggest the best alternative for what I am 
> doing here. 
> 
> Thanks very much for any ideas. 
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Eventfd with epoll BlockingIOError

[Barry Scott]

> On 25 Nov 2021, at 22:29, jenk...@tutanota.com wrote:
> 
> Thanks very much for your reply.  
> 
> I am now getting a single event returned in Python, but it's not the right 
> event, as I'll explain below. 
> 
> I rearranged the Python code based on your comments:
> 
> #!/usr/bin/python3
> import sys
> import os
> import select
> 
> print("Inside Python")
> 
> event_fd = int(sys.argv[3])
> 
> print("Eventfd received by Python")
> print(event_fd)
> 
> event_write_value = 100
> 
> ep = select.epoll(-1)
> ep.register(event_fd, select.EPOLLIN | select.EPOLLOUT )
> 
> os.set_blocking(event_fd, False)
> 
> #__
> 
> print("Starting poll loop")
> 
> for fd_event in ep.poll():
> print("Python fd_event")
> print(fd_event)
> fd_received = fd_event[0]
> event_received = fd_event[1]
> 
> You advised to leave off select.EPOLLOUT from the line ep.register(event_fd, 
> select.EPOLLIN | select.EPOLLOUT ) -- which makes sense because I'm not 
> waiting for that event -- but without it both processes freeze in the for 
> loop (below print("Starting poll loop")) so we never receive an EPOLLIN 
> event.  So I included it, and here is the screen output from gdb:
> 
> Inside Python
> Eventfd received by Python
> 5
> Everything OK in Python
> Starting poll loop
> Python fd_event
> (5, 4)
> Writing to Python
> 5 Received from Python
> 8 Writing to Python
> Failed epoll_wait Bad file descriptor
> 5 Received from Python
> 8 Writing to Python
> Failed epoll_wait Bad file descriptor
> 5 Received from Python
> -1time taken 0.000629
> Failed to close epoll file descriptor
> Unlink_shm status: Bad file descriptor
> fn() took 0.000717 seconds to execute
> [Inferior 1 (process 26718) exited normally]
> (gdb) q
> 
> The Python fd_event tuple is 5, 4 -- 5 is the correct file descriptor and 4 
> is an EPOLLOUT event, which is not what I want. 
> 
> The eventfd is created in C as nonblocking:
> 
> int eventfd_initialize() {
>   int efd = eventfd(0, EFD_NONBLOCK);

So it is not a semaphore as EFD_SEMAPHORE is not set.
Thus you intend to send arbitrary 64 int values and know that the value of 0

>   return efd; }
> 
> When C writes it calls epoll_wait:
> 
> ssize_t epoll_write(int event_fd, int epoll_fd, struct epoll_event * 
> event_struc, int action_code)
> {
>int64_t ewbuf[1];

The type is wrong its uint64_t.

>ewbuf[0] = (int64_t)action_code;
>int maxevents = 1;
>int timeout = -1;
> 
>fprintf(stdout, " Writing to Python \n%d", event_fd);
> 
> write(event_fd, , 8);

> 
> if (epoll_wait(epoll_fd, event_struc, maxevents, timeout) == -1)

Why is this here? Surely you need the python code to unblock because 
action_code is != 0.

> {
> fprintf(stderr, "Failed epoll_wait %s\n", strerror(errno));
> }
> 
> ssize_t rdval = read(event_fd, , 8);   

Why are you reading here? That is what you expect the python code to do.

Do you want to know if python has read the value written? If so then wait for 
the fd to be writeable...

> 
> fprintf(stdout, " Received from Python \n%ld", rdval);

How do you know if the value read is from python?
Why isn't it the value you wrote above?

If you want two ways communications you need 2 eventfd's I'd guess.
One to allow you to send a >0 64 bit int to python.
One to allow python to send a >0 64 bit int to C.

--

What is the problem you are solving?

Is eventfd even the right solution to that problem?

Maybe you would be better off using Unix Domain Sockets that allow
you to send and receive a block of bytes.

Barry



> 
> return 0;
> }
> 
> The C side initializes its epoll this way:
> 
> int epoll_initialize(int efd, int64_t * output_array)
> {
>   struct epoll_event ev = {};
>   int epoll_fd = epoll_create1(0);
> 
>   struct epoll_event * ptr_ev = 
>   
>   if(epoll_fd == -1)
>   {
> fprintf(stderr, "Failed to create epoll file descriptor\n");
> return 1;
>   }
> 
>   ev.events = EPOLLIN | EPOLLOUT;
>   ev.data.fd = efd; //was 0
> 
>   if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, efd, ) == -1)
>   {
>   fprintf(stderr, "Failed to add file descriptor to epoll\n");
>   close(epoll_fd);
>   return 1;
>   }
> 
>   output_array[0] = epoll_fd;
>   output_array[1] = (int64_t)ptr_ev; //
> 
>   return 0;
> }
> 
> Technically C is not waiting for an EPOLLIN event, but again without it both 
> processes freeze unless either C or Python includes both events.  So that 
> appears to be where the problem is. 
> 
> The Linux epoll man page says, "epoll_wait waits for I/O events, blocking the 
> calling thread if no events are currently available."   
> https://man7.org/linux/man-pages/man7/epoll.7.html 
> .  That may be the clue 
> to why both processes freeze when I poll on only one event in each one. 
> 
> Thanks for any ideas based on this update, and thanks again for your earlier 
> reply. 
> 
> Jen
> 
> 
> -- 
> Sent with Tutanota, the secure & ad-free mailbox. 
> 
> 
> 
> Nov 

Re: pyinstaller wrong classified as Windows virus

[Barry Scott]

> On 25 Nov 2021, at 09:20, Ulli Horlacher  
> wrote:
> 
> When I compile my programs with pyinstaller, Windows classifies them as
> virus and even deletes them!

Microsoft will fix the malware detection if you provide the info they need.

Submit false positive info to: 
https://www.microsoft.com/security/portal/submission/submit.aspx 


I have done this for a false positive in the past and they do resolve the false 
positive.

Barry


> 
> pyinstaller.exe --onefile --noconsole -i fex.ico fextasy.py
> 187 INFO: PyInstaller: 4.7
> 187 INFO: Python: 3.10.0
> 218 INFO: Platform: Windows-10-10.0.19041-SP0
> 218 INFO: wrote P:\W10\fextasy.spec
> (...)
> 14392 INFO: Copying 0 resources to EXE
> 14392 INFO: Emedding manifest in EXE
> 14392 INFO: Updating manifest in P:\W10\dist\fextasy.exe
> 14533 INFO: Updating resource type 24 name 1 language 0
> 14579 INFO: Appending PKG archive to EXE
> 18836 INFO: Building EXE from EXE-00.toc completed successfully.
> 
> https://fex.flupp.org/fop/ylds7Y9d/X-20211125101112.png
> 
> What can I do?
> 
> Rebooting does not help :-}
> 
> -- 
> Ullrich Horlacher  Server und Virtualisierung
> Rechenzentrum TIK 
> Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
> Allmandring 30aTel:++49-711-68565868
> 70569 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Eventfd with epoll BlockingIOError

[Barry Scott]

> On 24 Nov 2021, at 22:42, Jen via Python-list  wrote:
> 
> I have a C program that uses fork-execv to run Python 3.10 in a child 
> process, and I am using eventfd with epoll for IPC between them.  The eventfd 
> file descriptor is created in C and passed to Python through execv.  Once the 
> Python child process starts I print the file descriptor to verify that it is 
> correct (it is).  
> 
> In this scenario C will write to the eventfd at intervals and Python will 
> read the eventfd and take action based on the value in the eventfd.  But in 
> the Python while True loop I get "BlockingIOError: [Errno 11] Resource 
> temporarily unavailable" then with each new read it prints "Failed epoll_wait 
> Bad file descriptor." 
> 
> This is the Python code:
> 
> #!/usr/bin/python3
> import sys
> import os
> import select
> 
> print("Inside Python")
> 
> event_fd = int(sys.argv[3])
> 
> print("Eventfd received by Python")
> print(event_fd)
> 
> ep = select.epoll(-1)
> ep.register(event_fd, select.EPOLLIN | select.EPOLLOUT)

This says tell me if I can read or write to the event_fd.
write will be allowed until the kernel buffers are full.

Usually you only add EPOLLOUT if you have data to write.
In this case do not set EPOLLOUT.

And if you know that you will never fill the kernel buffers then you
do not need to bother polling for write.

> 
> event_write_value = 100
> 
> while True:
> 
> print("Waiting in Python for event")
> ep.poll(timeout=None, maxevents=- 1)

You have to get the result of the poll() and process the list of entries that 
are returned.

You must check that POLLIN is set before attempting the read.


> v = os.eventfd_read(event_fd)

Will raise EWOULDBLOCK because there is no data available to read.

Here is the docs from python:

poll.poll([timeout]) 

Polls the set of registered file descriptors, and returns a possibly-empty list 
containing (fd, event) 2-tuples for the descriptors that have events or errors 
to report. fd is the file descriptor, and event is a bitmask with bits set for 
the reported events for that descriptor — POLLIN for waiting input, POLLOUT to 
indicate that the descriptor can be written to, and so forth. An empty list 
indicates that the call timed out and no file descriptors had any events to 
report. If timeout is given, it specifies the length of time in milliseconds 
which the system will wait for events before returning. If timeout is omitted, 
negative, or None 
,
 the call will block until there is an event for this poll object.

You end up with code like this:

for fd_event in ep.poll():
fd, event == fd_event
if (event) != 0 and fd == event_fd:
v = os.eventfd_read(event_fd)

> 
> if v != 99:
> print("found")
> print(v)
> os.eventfd_write(event_fd, event_write_value)
> 
> if v == 99:
> os.close(event_fd)
> 
> This is the C code that writes to Python, then waits for Python to write back:
> 
> ssize_t epoll_write(int event_fd, int epoll_fd, struct epoll_event * 
> event_struc, int action_code)
> {
>int64_t ewbuf[1];
>ewbuf[0] = (int64_t)action_code;
>int maxevents = 1;
>int timeout = -1;
> 
>fprintf(stdout, " Writing to Python \n%d", event_fd);
> 
>write(event_fd, , 8);
> 
> if (epoll_wait(epoll_fd, event_struc, maxevents, timeout) == -1)
> {
> fprintf(stderr, "Failed epoll_wait %s\n", strerror(errno));
> }
> 
> ssize_t rdval = read(event_fd, , 8);   
> 
> fprintf(stdout, " Received from Python \n%ld", rdval);
> 
> return 0;
> }
> 
> This is the screen output when I run with gdb:
> 
>   Inside Python
> Eventfd received by Python
> 5
> Waiting in Python for event
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.10/runpy.py", line 196, in 
> _run_module_as_main
> return _run_code(code, main_globals, None,
>   File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
> exec(code, run_globals)
>   File "/opt/P01_SH/NPC_CPython.py", line 36, in 
> v = os.eventfd_read(event_fd)
> BlockingIOError: [Errno 11] Resource temporarily unavailable

Expected as there you have not checked that there is data to read.
Check for POLLIN being set.

> Writing to Python
> 5 Received from Python
> 8 Writing to Python
> Failed epoll_wait Bad file descriptor
> 5 Received from Python
> 8 Writing to Python
> Failed epoll_wait Bad file descriptor
> 5 Received from Python
> -1time taken 0.000548
> Failed to close epoll file descriptor
> Unlink_shm status: Bad file descriptor
> fn() took 0.000648 seconds to execute
> [Inferior 1 (process 12618) exited normally]
> (gdb)
> 
> So my question is why do I get "BlockingIOError: [Errno 11] Resource 
> temporarily unavailable" and "Failed epoll_wait Bad file descriptor" from 
> Python? 

If your protocol is not trivia you should implement a state machine to know 
what to do at each event.

Barry

> 
> -- 
> Sent with Tutanota, the secure & ad-free mailbox. 
> -- 

Re: Using astype(int) for strings with thousand separator

[Barry Scott]

> On 14 Nov 2021, at 15:41, Mahmood Naderan via Python-list 
>  wrote:
> 
> Hi
> 
> While reading a csv file, some cells have values like '1,024' which I mean 
> they contains thousand separator ','. Therefore, when I want to process them 
> with 
> 
>   row = df.iloc[0].astype(int)

remove the "," from the sting seems the simplest things to do.

Use string's replace() to remove the comma.

Barry

> 
> I get the following error
> 
>   ValueError: invalid literal for int() with base 10: '1,024'
> 
> 
> How can I fix that? Any idea?
> 
> 
> 
> Regards,
> Mahmood
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unable to compile my C Extension on Windows: unresolved external link errors

[Barry Scott]

Sorry iPad sent the message before it was complete...

> On 14 Nov 2021, at 10:38, Marco Sulla  wrote:
> 
> Okay, now the problem seems to be another: I get the same "unresolved
> external link" errors, but only for internal functions.
> 
> This seems quite normal. The public .lib does not expose the internals
> of Python.
> The strange fact is: why can I compile it on Linux and MacOS? Their
> external libraries expose the internal functions?

Windows is not Linux is not macOS,
The toolchain on each OS has its own strengths, weaknesses and quirks.

On Windows DLLs only allow access to the symbols that are explicitly listed to 
be access.
On macOS .dynlib and Unix .so its being extern that does this.

> 
> Anyway, is there a way to compile Python on Windows in such a way that
> I get a shared library that exposes all the functions?

Yes you can do your own build of python that exposes the symbols you want.
But that build will be private to you and will not allow others to use your work
(on the assumption that they will not use your private build of python).

Maybe you could copy the code that you want and add it to your code?
Change any conflicting symbols of course.

Barry

> 
> On Sat, 13 Nov 2021 at 12:17, Marco Sulla  
> wrote:
>> 
>> . Sorry, the problem is I downloaded the 32 bit version of VS
>> compiler and 64 bit version of Python..
>> 
>> On Sat, 13 Nov 2021 at 11:10, Barry Scott  wrote:
>>> 
>>> 
>>> 
>>>> On 13 Nov 2021, at 09:00, Barry  wrote:
>>>> 
>>>> 
>>>> 
>>>>> On 12 Nov 2021, at 22:53, Marco Sulla  
>>>>> wrote:
>>>>> 
>>>>> It seems that on Windows it doesn't find python3.lib,
>>>>> even if I put it in the path. So I get the `unresolved external link`
>>>>> errors.
>>>> 
>>>> I think you need the python310.lib (not sure of file name) to get to the 
>>>> internal symbols.
>>> 
>>> Another thing that you will need to check is that the symbols you are after 
>>> have been
>>> exposed in the DLL at all. Being external in the source is not enough they 
>>> also have to
>>> listed in the .DLL's def file ( is that the right term?) as well.
>>> 
>>> If its not clear yet, you are going to have to read a lot or source code 
>>> and understand
>>> the tool chain used on Windows to solve this.
>>> 
>>> 
>>>> 
>>>> You can use the objdump(?) utility to check that the symbols are in the 
>>>> lib.
>>>> 
>>>> Barry
>>> 
>>> Barry
>>> 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unable to compile my C Extension on Windows: unresolved external link errors

[Barry Scott]

> On 13 Nov 2021, at 09:00, Barry  wrote:
> 
> 
> 
>> On 12 Nov 2021, at 22:53, Marco Sulla  wrote:
>> 
>> It seems that on Windows it doesn't find python3.lib,
>> even if I put it in the path. So I get the `unresolved external link`
>> errors.
> 
> I think you need the python310.lib (not sure of file name) to get to the 
> internal symbols.

Another thing that you will need to check is that the symbols you are after 
have been
exposed in the DLL at all. Being external in the source is not enough they also 
have to
listed in the .DLL's def file ( is that the right term?) as well.

If its not clear yet, you are going to have to read a lot or source code and 
understand
the tool chain used on Windows to solve this.


> 
> You can use the objdump(?) utility to check that the symbols are in the lib.
> 
> Barry

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Definitive guide for Regex

[Barry Scott]

> On 1 Oct 2021, at 10:58, Shaozhong SHI  wrote:
> 
> Hi, Barry,
> 
> In cases of automating checking, validation and producing reports in the 
> context of data quality control and giving specific feedback to production 
> teams, regex is perhaps the only way.
> 
> Perhaps, we can give each element of data specifications a name, that are 
> associated with a regex value, so that we can automate checking and reporting 
> on data sets.  We can report on which row of records meet specification and 
> requirements and which one is not.  And, report on which cell needs to be 
> corrected should a row is found not meeting specification and requirements.
> 
> What do you think?

It depends a lot of the details of that you have to validate. There is not 
enough to guess at a design.
It may well be that uses regex's is a good way to do it.

Barry


> 
> Regards,
> 
> David
> 
> On Thu, 30 Sept 2021 at 22:02, Barry Scott  <mailto:ba...@barrys-emacs.org>> wrote:
> 
> 
> > On 30 Sep 2021, at 19:35, dn via Python-list  > <mailto:python-list@python.org>> wrote:
> > 
> > On 01/10/2021 06.16, Barry Scott wrote:
> >> 
> >> 
> >>> On 30 Sep 2021, at 12:29, Shaozhong SHI  >>> <mailto:shishaozh...@gmail.com>> wrote:
> >>> 
> >>> Dear All,
> >>> 
> >>> I am trying to look for a definitive guide for Regex in Python.
> >>> Can anyone help?
> >> 
> >> Have you read the python docs for the re module?
> > 
> > 
> > I learned from Jeffrey Friedl's book "Mastering Regular Expressions",
> > but that was in a land far away, last century, and under a different
> > language (and the original version - I see it's now up to its third
> > edition).
> > 
> > Despite their concise exercise of power (and the fact that in my
> > Python-life I've never been put into a corner where I absolutely must
> > use one), I'm no longer a fan...
> 
> Agreed, regex is the last tool I reach for in python code.
> I find I use split() a lot to break up strings for processing.
> But there are cases where a regex is the best tool for a particular job
> and I then use the re module. But it costs in maintainability.
> 
> I speak as the author of a regex engine and know how to write scary
> regex's when the need arises.
> 
> Barry
> 
> 
> > -- 
> > Regards,
> > =dn
> > -- 
> > https://mail.python.org/mailman/listinfo/python-list 
> > <https://mail.python.org/mailman/listinfo/python-list>
> > 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list 
> <https://mail.python.org/mailman/listinfo/python-list>

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Definitive guide for Regex

[Barry Scott]

> On 30 Sep 2021, at 19:35, dn via Python-list  wrote:
> 
> On 01/10/2021 06.16, Barry Scott wrote:
>> 
>> 
>>> On 30 Sep 2021, at 12:29, Shaozhong SHI  wrote:
>>> 
>>> Dear All,
>>> 
>>> I am trying to look for a definitive guide for Regex in Python.
>>> Can anyone help?
>> 
>> Have you read the python docs for the re module?
> 
> 
> I learned from Jeffrey Friedl's book "Mastering Regular Expressions",
> but that was in a land far away, last century, and under a different
> language (and the original version - I see it's now up to its third
> edition).
> 
> Despite their concise exercise of power (and the fact that in my
> Python-life I've never been put into a corner where I absolutely must
> use one), I'm no longer a fan...

Agreed, regex is the last tool I reach for in python code.
I find I use split() a lot to break up strings for processing.
But there are cases where a regex is the best tool for a particular job
and I then use the re module. But it costs in maintainability.

I speak as the author of a regex engine and know how to write scary
regex's when the need arises.

Barry


> -- 
> Regards,
> =dn
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Definitive guide for Regex

[Barry Scott]

> On 30 Sep 2021, at 12:29, Shaozhong SHI  wrote:
> 
> Dear All,
> 
> I am trying to look for a definitive guide for Regex in Python.
> Can anyone help?

Have you read the python docs for the re module?

Barry

> 
> Regards,
> 
> David
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: basic auth request

[Barry Scott]

> On 22 Aug 2021, at 12:03, Chris Angelico  wrote:
> 
> On Sun, Aug 22, 2021 at 8:30 PM Barry Scott  <mailto:ba...@barrys-emacs.org>> wrote:
>> 
>> 
>> 
>> On 22 Aug 2021, at 10:37, Chris Angelico  wrote:
>> 
>> When it comes to security, one thing I'm very curious about is why we
>> don't have any sort of certificate renewal verification. My browser
>> could retain the certificates of some web site (or of all web sites,
>> even - they're not THAT large), and if the site presents a different
>> cert, it could show the previously retained one and challenge the
>> server "prove that you're the same guy". This proof would consist of
>> the latest cert, signed by the older cert's key (or possibly a chain
>> that can construct such a proof, which would allow the server to
>> simply retain each new cert signed by the one previous cert, forming a
>> line - or a tree if necessary). My suspicion is that it'd add little
>> above simply having a valid cert, but if people are paranoid, surely
>> that's a better place to look?
>> 
>> 
>> The web site proves it owners the hostname and/or IP address using its 
>> certificate.
>> You use your trust store to show that you can trust that certificate.
>> 
>> The fact that a certificate changes is not a reason to stop trusting a site.
>> 
>> So it does not add anything.
>> 
>> The pain point in PKI is revocation. The gold standard is for a web site to 
>> use OCSP stapling.
>> But that is rare sadly. And because of issues with revocation lists, 
>> (privacy, latency, need to
>> fail open on failiure, DoD vector, etc) this is where the paranoid should 
>> look.
>> 
> 
> Fair point. Let me give you a bit of context.
> 
> Recently, the owner/operator of a site (I'll call it
> https://demo.example/ <https://demo.example/> ) died. Other people, who have 
> been using the
> site extensively, wish for it to continue. If the domain registration
> expires, anyone can reregister it, and can then generate a completely
> new certificate for the common name "demo.example", and web browsers
> will accept that. The old cert may or may not have expired, but it
> won't be revoked.
> 
> As far as I can tell, a web browser with default settings will happily
> accept the change of ownership. It won't care that the IP address,
> certificate, etc, have all changed. It just acknowledges that some CA
> has signed some certificate with the right common name. And therein is
> the vulnerability. (NOTE: I'm not saying that this is a real and
> practical vulnerability - this is theoretical only, and a focus for
> the paranoid.)
> 
> This is true even if the old cert were one of those enhanced
> certificates that some CAs try to upsell you to ("Extended Validation"
> and friends). Even if, in the past, your bank was secured by one of
> those certs, your browser will still accept a perfectly standard cert
> next time. Which, in my opinion, renders those (quite pricey)
> certificates no more secure than something from Let's Encrypt that has
> no validation beyond ownership of DNS.
> 
> Of course, you can pin a certificate. You can ask your browser to warn
> you if it's changed *at all*. But since certs expire, that's highly
> impractical, hence wondering why we don't have a system for using the
> old cert to prove ownership of the new one.
> 
> So how is a web browser supposed to distinguish between (a) normal
> operation in which certs expire and are replaced, and (b) legit or
> non-legit ownership changes? (Of course the browser can't tell you
> whether the ownership change is legit, but out-of-band info can help
> with that.)
> 
> Or does it really matter that little?

Only if this threat model matters to you or your organisation.
Personal its low down of the threats I watch out for.

The on-line world and the real-world are the same here.

If a business changes hands then do you trust the new owners?

Nothing we do with PKI certificates will answer that question.

For web sites that we care a lot about, like banks, we trust that
the site owners take care to protect that site. This includes making
sure that they do not lose control of its DNS name and certificates.

One of the biggest issues for PKI in recent years has been
Certificate Authorities (CA) that issued certificates for web sites without
checking that ownership. In these cases it lead to the browsers
removing those CA's from the trust stores and also putting extra
rules on all CA's to do a higher quality job.

Barry

> 
> ChrisA
> -- 
> https://mail.python.org/mailman/listinfo/python-list 
> <https://mail.python.org/mailman/listinfo/python-list>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: basic auth request

[Barry Scott]

> On 22 Aug 2021, at 10:37, Chris Angelico  wrote:
> 
> When it comes to security, one thing I'm very curious about is why we
> don't have any sort of certificate renewal verification. My browser
> could retain the certificates of some web site (or of all web sites,
> even - they're not THAT large), and if the site presents a different
> cert, it could show the previously retained one and challenge the
> server "prove that you're the same guy". This proof would consist of
> the latest cert, signed by the older cert's key (or possibly a chain
> that can construct such a proof, which would allow the server to
> simply retain each new cert signed by the one previous cert, forming a
> line - or a tree if necessary). My suspicion is that it'd add little
> above simply having a valid cert, but if people are paranoid, surely
> that's a better place to look?

The web site proves it owners the hostname and/or IP address using its 
certificate.
You use your trust store to show that you can trust that certificate.

The fact that a certificate changes is not a reason to stop trusting a site.

So it does not add anything.

The pain point in PKI is revocation. The gold standard is for a web site to use 
OCSP stapling.
But that is rare sadly. And because of issues with revocation lists, (privacy, 
latency, need to
fail open on failiure, DoD vector, etc) this is where the paranoid should look.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: basic auth request

[Barry Scott]

On Tuesday, 17 August 2021 10:20:37 BST Robin Becker wrote:
> While porting an ap from python2.7 to python3 I see this
> 
>   base64string = base64.b64encode('%s:%s' % (wsemail, wspassword))
>   request.add_header("Authorization", "Basic %s" % base64string)
> 
> in python3.x I find this works
> 
>   base64string = base64.b64encode(('%s:%s' % (wsemail,
> wspassword)).encode('ascii')).decode('ascii')
> request.add_header("Authorization", "Basic %s" % base64string)
> 
> but I find the conversion to and from ascii irksome. Is there a more direct
> way to create the basic auth value?

base64 works on BYTES not UNICODE that is why you need to convert to BYTES.

Its an important detail that you must handle. The py2 code meant that you
only see errors if you have a value in your string that is outside the ASCII
range.

> As an additional issue I find I have no clear idea what encoding is allowed
> for the components of a basic auth input. --

You will want to read this: 
https://datatracker.ietf.org/doc/html/rfc7617#section-2.1
It talks about a "charset" auth-param, then seems to say that only allowed 
value is
utf-8 and you most have the unicode Normalization Form C ("NFC").

Oh and if you have the freedom avoid Basic Auth as its not secure at all.

> Robin Becker

Barry



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Cyclic imports

[Barry Scott]

On Monday, 16 August 2021 16:13:47 BST Dan Stromberg wrote:
> Hi folks.
> 
> I'm working on a large codebase that has at least one cyclic import.
> 
> In case I end up needing to eliminate the cyclic imports, is there any sort
> of tool that will generate an import graph and output Just the cycles?
> 
> I tried pyreverse, but it produced too big a graph to be very useful; it
> showed all internal imports, not just the cycles.

I wrote this code to track down a cycling import.
Note  it handles import module, but not from module import.
You would need to make a (simple) edit to add that.

 py_import_time.py ---
#!/usr/bin/python3
import sys
import pathlib

class PyImportTree:
def __init__( self, main_module, python_path ):
self.main_module = main_module
self.python_path = python_path

self.all_modules = {}
self.loadTree( self.main_module )

self.all_being_imported = set()
self.problem_imports = 0

def loadTree( self, module_name ):
all_imports = self.allImports( module_name )
if all_imports is None:
return

self.all_modules[ module_name ] = all_imports

for module in all_imports:
if module not in self.all_modules:
self.loadTree( module )

def findModule( self, module_name ):
for folder in self.python_path:
abs_path = pathlib.Path( folder ) / ('%s.py' % (module_name,))
if abs_path.exists():
return abs_path

return None

def allImports( self, module_name ):
all_imports = []
filename = self.findModule( module_name )
if filename is None:
print( 'Error: Cannot find module %s' % (module_name,), 
file=sys.stderr )
return None

with open( str(filename), 'r' ) as f:
for line in f:
words = line.strip().split()
if words[0:1] == ['import']:
all_imports.append( words[1] )

return all_imports

def printTree( self ):
self.__printTree( self.main_module, 0 )
if self.problem_imports > 0:
print( '%d problem imports' % (self.problem_imports,), 
file=sys.stderr )

def __printTree( self, module_name, indent ):
if module_name not in self.all_modules:
return

if module_name in self.all_being_imported:
print( '%s%s' % ('>   '*indent, module_name) )
self.problem_imports += 1
return

print( '%s%s' % ('-   '*indent, module_name) )

self.all_being_imported.add( module_name )

for module in self.all_modules[ module_name ]:
self.__printTree( module, indent+1 )

self.all_being_imported.remove( module_name )

if __name__ == '__main__':
sys.setrecursionlimit( 30 )
sys.exit( PyImportTree( sys.argv[1], sys.argv[2:] ).printTree() )



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Optimizing Small Python Code

[Barry Scott]

> On 24 Jun 2021, at 16:58, Avi Gross via Python-list  
> wrote:
> 
> Now a has a length of 53!
> 
> It now looks like this:
> 
> b'x\x9c3\xe4R\x00\x03\x03.#8\x0bB\x1br\x19c\x88(\x18q\x99p!q\xc1\x00\xa6\xd1\x98\xcb\x14S\x03\x9a\n\x13.3\x82j
>  \xb4\t\x94\x86\x99\t\x00\xdc\x87\x14\xb7'
> 
> So the shorter cheat program might now be:

The data is smaller, but at the cost of the code that knows how to decompress 
it.

I'd expect that by using compression you have increased memory use as this
amount of data is far smaller than code to decompress it.

Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Interpreter Bug

[Barry Scott]

> On 2 Jun 2021, at 10:34, Alice Braim  wrote:
> 
>   Good morning-
> 
> 
> 
>   I am having some very serious issues with Python, and I was hoping you
>   could help?
> 
>   I downloaded both Python and PyCharm, and the 2 do not seem to be working.
>   Every time I select Python as an interpreter, the whole thing crashes.
>   Obviously, I went onto repairs, but even then it told me there was a
>   `fatal error'. I tried uninstalling and restarting multiple times, and
>   nothing improved. I've tried redownloading loads of times, no improvement.
>   I have the latest version (3.9.5) and there definitely seems to be
>   something going on at your end.

Details please: Which OS which version of Python etc.

What does crashes mean exactly?

Was it PyCharm that crashes or python that crashes?

Barry


> 
>   Any ideas?
> 
> 
> 
>   ~Alexa
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: 3.7.6 struggles a bit

[Barry Scott]

> On 2 Jun 2021, at 18:18, Luke  wrote:
> 
>   When i wrote a program, i tried to open it but it struggles to open.

I'm guessing you are a Windows user.

Does this help? https://docs.python.org/3/faq/windows.html 


Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Definition of "property"

[Barry Scott]

> On 30 May 2021, at 17:57, Irv Kalb  wrote:
> 
> I am doing some writing (for an upcoming book on OOP), and I'm a little 
> stuck.  
> 
> I understand what a "property" is, how it is used and the benefits, but 
> apparently my explanation hasn't made the light bulb go on for my editor.  
> The editor is asking for a definition of property.  I've looked at many 
> articles on line and a number of books, and I haven't found an appropriate 
> one yet.
> 
> I have written some good examples of how it works, but I agree that a 
> definition up front would be helpful.  I have tried a number of times, but my 
> attempts to define it have not been clear.  Perhaps the best I've found so 
> far is from the Python documentation:  
> 
> A property object has getter, setter, and deleter methods usable as 
> decorators that create a copy of the property with the corresponding accessor 
> function set to the decorated function. 
> 
> But I'm hoping that someone here can give me a more concise (one or two 
> sentence) definition of the word "property".   
> 
> (I would like to avoid going through the whole derivation with the property 
> function, as that would distract from the points that I am trying to make.) 

How does this sound?

An object is the combination of behaviour and state.

Classes define the object.
Methods allow the control of behaviour.
Properties hold the state.

The use of getter functions allows a property's value to be calculated.
The use of setting functions allows a property change to update the state an 
object.

The python property mechanism allows the getting and setter to be hidden from
the API as that the user of the object can see the propery as a simple 
attribute of
an object.

Barry



> 
> Thanks in advance,
> 
> Irv
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: f-strings and internationalisation.

[Barry Scott]

> On 24 May 2021, at 19:30, Antoon Pardon  wrote:
> 
> I have now come across several occasion where an author advice the use
> of f-strings above the %-formatting and the format method. However it
> seems these authors were only thinking about rather straight forward
> english communication.
> 
> So what if you want your application to work with multiple languages.
> Can that be done with f-strings?

No it cannot be done. This is because a translation can reorder
the parts of the string is drastic ways that f'strings' does not allow for.

You need to use this style:

_('This %(arg1)s and %(arg2)s') % {'arg1': value_arg1, 'arg2': 
value_arg2}

or
_('This {arg1} and {arg2}').format(arg1=value_args, arg2=values_arg2)

A translator would be free to swap arg1 and arg2 order in a translation.

See https://docs.python.org/3/library/gettext.html for more details.
And https://www.mattlayman.com/blog/2015/i18n/ looks useful as well.

Then you can use use the I18N gettext tools to make a .pot and .po files.

Barry
 


> 
> -- 
> Antoon Pardon.
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Standarize TOML?

[Barry Scott]

> On 15 May 2021, at 23:39, Jason C. McDonald  wrote:
> 
> During the Steering Committee presentation at PyCon, it was mentioned
> that no one has formally proposed TOML be added to the standard library
> (emphasis on formal). THe joke went forth that there would be a flood
> of proposals to that end.
> 
> So, just to kick this off while the thought is still fresh in a bunch of
> people's minds: **should we add a TOML parser to the standard library**?
> 
> The main reason this matters is to help encourage adoption of the now
> PEP-standardized pyproject.toml. A few projects have cited the lack of
> a standardized TOML implementation in the standard library as a reason
> not to adopt pyproject.toml...and the topic thus became weirdly
> political.
> 
> I understand that Brett Cannon intends to bring this up at the next
> language summit, but, ah, might as well put the community two-cents in
> now, hey?
> 
> I, for one, feel like this is obvious.

I think the python ideas list is a better place to have this discussion.

Barry


> 
> -- 
> Jason C. McDonald (CodeMouse92)
> Author | Speaker | Hacker | Time Lord
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list