Re: [PATCH v3 07/16] iotests/297: Don't rely on distro-specific linter binaries

[John Snow]

On Mon, Oct 4, 2021 at 4:17 AM Hanna Reitz  wrote:

> On 22.09.21 21:53, John Snow wrote:
> > (This email just explains python packaging stuff. No action items in
> > here. Skim away.)
> >
> > On Fri, Sep 17, 2021 at 5:43 AM Hanna Reitz  > > wrote:
> >
> > On 16.09.21 06:09, John Snow wrote:
> > > 'pylint-3' is another Fedora-ism. Use "python3 -m pylint" or
> > "python3 -m
> > > mypy" to access these scripts instead. This style of invocation
> will
> > > prefer the "correct" tool when run in a virtual environment.
> > >
> > > Note that we still check for "pylint-3" before the test begins
> > -- this
> > > check is now "overly strict", but shouldn't cause anything that was
> > > already running correctly to start failing.
> > >
> > > Signed-off-by: John Snow  > >
> > > Reviewed-by: Vladimir Sementsov-Ogievskiy
> > mailto:vsement...@virtuozzo.com>>
> > > Reviewed-by: Philippe Mathieu-Daudé  > >
> > > ---
> > >   tests/qemu-iotests/297 | 45
> > --
> > >   1 file changed, 26 insertions(+), 19 deletions(-)
> >
> > I know it sounds silly, but to be honest I have no idea if replacing
> > `mypy` by `python3 -m mypy` is correct, because no mypy documentation
> > seems to suggest it.
> >
> >
> > Right, I don't think it's necessarily documented that you can do this.
> > It just happens to be a very convenient way to invoke the same script
> > without needing to know *where* mypy is. You let python figure out
> > where it's going to import mypy from, and it handles the rest.
> >
> > (This also makes it easier to use things like mypy, pylint etc with an
> > explicitly specified PYTHON interpreter. I don't happen to do that in
> > this patch, but ... we could.)
> >
> >  From what I understand, that’s generally how Python “binaries” work,
> > though (i.e., installed as a module invokable with `python -m`,
> > and then
> > providing some stub binary that, well, effectively does this, but
> > kind
> > of in a weird way, I just don’t understand it), and none of the
> > parameters seem to be hurt in this conversion, so:
> >
> >
> > Right. Technically, any python package can ask for any number of
> > executables to be installed, but the setuptools packaging ecosystem
> > provides a way to "generate" these based on package configuration. I
> > use a few in our own Python packages. If you look in python/setup.cfg,
> > you'll see stuff like this:
> >
> > [options.entry_points]
> > console_scripts =
> > qom = qemu.qmp.qom:main
> > qom-set = qemu.qmp.qom:QOMSet.entry_point
> > qom-get = qemu.qmp.qom:QOMGet.entry_point
> > qom-list = qemu.qmp.qom:QOMList.entry_point
> > qom-tree = qemu.qmp.qom:QOMTree.entry_point
> > qom-fuse = qemu.qmp.qom_fuse:QOMFuse.entry_point [fuse]
> > qemu-ga-client = qemu.qmp.qemu_ga_client:main
> > qmp-shell = qemu.qmp.qmp_shell:main
> >
> > These entries cause those weird little binary wrapper scripts to be
> > generated for you when the package is *installed*. So our packages
> > will put 'qmp-shell' and 'qom-tree' into your $PATH*.The stuff to the
> > right of the equals sign is just a pointer to a function that can be
> > executed that implements the CLI command. qemu.qmp.qmp_shell points to
> > the module to import, and ':main' points to the function to run.
> >
> > The last bit of this is that many, though not all (and there's zero
> > requirement this has to be true), python packages that implement CLI
> > commands will often have a stanza in their __init__.py module that
> > says something like this:
> >
> > if __name__ == '__main__':
> > do_the_command_line_stuff()
> >
> > Alternatively, a package can include a literal __main__.py file that
> > python knows to check for, and this module is the one that gets
> > executed for `python3 -m mypackage` invocations. This is what mypy does.
> >
> > Those are the magical blurbs that allow you to execute a module as if
> > it were a script by running "python3 -m mymodule" -- that hooks
> > directly into that little execution stanza. For python code
> > distributed as packages, that's the real reason to have that little
> > magic stanza -- it provides a convenient way to run stuff without
> > needing to write the incredibly more tedious:
> >
> > python3 -c "from mypy.__main__ import console_entry; console_entry();"
> >
> > ... which is quite a bit more porcelain too, depending on how they
> > re/factor the code inside of the package.
> >
> > Seeing as how mypy explicitly includes a __main__.py file:
> > https://github.com/python/mypy/blob/master/mypy/__main__.py
> > , I am
> > taking it as a given that they are fully aware of invoking mypy in
> > this fashion, and believe it safe to rely on.
>
> Wow, thanks a lot for this 

Re: [PATCH v3 07/16] iotests/297: Don't rely on distro-specific linter binaries

[Hanna Reitz]

On 22.09.21 21:53, John Snow wrote:
(This email just explains python packaging stuff. No action items in 
here. Skim away.)


On Fri, Sep 17, 2021 at 5:43 AM Hanna Reitz > wrote:


On 16.09.21 06:09, John Snow wrote:
> 'pylint-3' is another Fedora-ism. Use "python3 -m pylint" or
"python3 -m
> mypy" to access these scripts instead. This style of invocation will
> prefer the "correct" tool when run in a virtual environment.
>
> Note that we still check for "pylint-3" before the test begins
-- this
> check is now "overly strict", but shouldn't cause anything that was
> already running correctly to start failing.
>
> Signed-off-by: John Snow mailto:js...@redhat.com>>
> Reviewed-by: Vladimir Sementsov-Ogievskiy
mailto:vsement...@virtuozzo.com>>
> Reviewed-by: Philippe Mathieu-Daudé mailto:phi...@redhat.com>>
> ---
>   tests/qemu-iotests/297 | 45
--
>   1 file changed, 26 insertions(+), 19 deletions(-)

I know it sounds silly, but to be honest I have no idea if replacing
`mypy` by `python3 -m mypy` is correct, because no mypy documentation
seems to suggest it.


Right, I don't think it's necessarily documented that you can do this. 
It just happens to be a very convenient way to invoke the same script 
without needing to know *where* mypy is. You let python figure out 
where it's going to import mypy from, and it handles the rest.


(This also makes it easier to use things like mypy, pylint etc with an 
explicitly specified PYTHON interpreter. I don't happen to do that in 
this patch, but ... we could.)


 From what I understand, that’s generally how Python “binaries” work,
though (i.e., installed as a module invokable with `python -m`,
and then
providing some stub binary that, well, effectively does this, but
kind
of in a weird way, I just don’t understand it), and none of the
parameters seem to be hurt in this conversion, so:


Right. Technically, any python package can ask for any number of 
executables to be installed, but the setuptools packaging ecosystem 
provides a way to "generate" these based on package configuration. I 
use a few in our own Python packages. If you look in python/setup.cfg, 
you'll see stuff like this:


[options.entry_points]
console_scripts =
    qom = qemu.qmp.qom:main
    qom-set = qemu.qmp.qom:QOMSet.entry_point
    qom-get = qemu.qmp.qom:QOMGet.entry_point
    qom-list = qemu.qmp.qom:QOMList.entry_point
    qom-tree = qemu.qmp.qom:QOMTree.entry_point
    qom-fuse = qemu.qmp.qom_fuse:QOMFuse.entry_point [fuse]
    qemu-ga-client = qemu.qmp.qemu_ga_client:main
    qmp-shell = qemu.qmp.qmp_shell:main

These entries cause those weird little binary wrapper scripts to be 
generated for you when the package is *installed*. So our packages 
will put 'qmp-shell' and 'qom-tree' into your $PATH*.The stuff to the 
right of the equals sign is just a pointer to a function that can be 
executed that implements the CLI command. qemu.qmp.qmp_shell points to 
the module to import, and ':main' points to the function to run.


The last bit of this is that many, though not all (and there's zero 
requirement this has to be true), python packages that implement CLI 
commands will often have a stanza in their __init__.py module that 
says something like this:


if __name__ == '__main__':
    do_the_command_line_stuff()

Alternatively, a package can include a literal __main__.py file that 
python knows to check for, and this module is the one that gets 
executed for `python3 -m mypackage` invocations. This is what mypy does.


Those are the magical blurbs that allow you to execute a module as if 
it were a script by running "python3 -m mymodule" -- that hooks 
directly into that little execution stanza. For python code 
distributed as packages, that's the real reason to have that little 
magic stanza -- it provides a convenient way to run stuff without 
needing to write the incredibly more tedious:


python3 -c "from mypy.__main__ import console_entry; console_entry();"

... which is quite a bit more porcelain too, depending on how they 
re/factor the code inside of the package.


Seeing as how mypy explicitly includes a __main__.py file: 
https://github.com/python/mypy/blob/master/mypy/__main__.py 
, I am 
taking it as a given that they are fully aware of invoking mypy in 
this fashion, and believe it safe to rely on.


Wow, thanks a lot for this detailed explanation!

There will be a quiz later.
(There will not be a quiz.)


I’m ready to fail any test on Python so one day I can get a “Officially 
knows nothing about Python” badge.


Hanna

Re: [PATCH v3 07/16] iotests/297: Don't rely on distro-specific linter binaries

[John Snow]

(This email just explains python packaging stuff. No action items in here.
Skim away.)

On Fri, Sep 17, 2021 at 5:43 AM Hanna Reitz  wrote:

> On 16.09.21 06:09, John Snow wrote:
> > 'pylint-3' is another Fedora-ism. Use "python3 -m pylint" or "python3 -m
> > mypy" to access these scripts instead. This style of invocation will
> > prefer the "correct" tool when run in a virtual environment.
> >
> > Note that we still check for "pylint-3" before the test begins -- this
> > check is now "overly strict", but shouldn't cause anything that was
> > already running correctly to start failing.
> >
> > Signed-off-by: John Snow 
> > Reviewed-by: Vladimir Sementsov-Ogievskiy 
> > Reviewed-by: Philippe Mathieu-Daudé 
> > ---
> >   tests/qemu-iotests/297 | 45 --
> >   1 file changed, 26 insertions(+), 19 deletions(-)
>
> I know it sounds silly, but to be honest I have no idea if replacing
> `mypy` by `python3 -m mypy` is correct, because no mypy documentation
> seems to suggest it.
>
>
Right, I don't think it's necessarily documented that you can do this. It
just happens to be a very convenient way to invoke the same script without
needing to know *where* mypy is. You let python figure out where it's going
to import mypy from, and it handles the rest.

(This also makes it easier to use things like mypy, pylint etc with an
explicitly specified PYTHON interpreter. I don't happen to do that in this
patch, but ... we could.)

>
>  From what I understand, that’s generally how Python “binaries” work,
> though (i.e., installed as a module invokable with `python -m`, and then
> providing some stub binary that, well, effectively does this, but kind
> of in a weird way, I just don’t understand it), and none of the
> parameters seem to be hurt in this conversion, so:
>
>
Right. Technically, any python package can ask for any number of
executables to be installed, but the setuptools packaging ecosystem
provides a way to "generate" these based on package configuration. I use a
few in our own Python packages. If you look in python/setup.cfg, you'll see
stuff like this:

[options.entry_points]
console_scripts =
qom = qemu.qmp.qom:main
qom-set = qemu.qmp.qom:QOMSet.entry_point
qom-get = qemu.qmp.qom:QOMGet.entry_point
qom-list = qemu.qmp.qom:QOMList.entry_point
qom-tree = qemu.qmp.qom:QOMTree.entry_point
qom-fuse = qemu.qmp.qom_fuse:QOMFuse.entry_point [fuse]
qemu-ga-client = qemu.qmp.qemu_ga_client:main
qmp-shell = qemu.qmp.qmp_shell:main

These entries cause those weird little binary wrapper scripts to be
generated for you when the package is *installed*. So our packages will put
'qmp-shell' and 'qom-tree' into your $PATH*.The stuff to the right of the
equals sign is just a pointer to a function that can be executed that
implements the CLI command. qemu.qmp.qmp_shell points to the module to
import, and ':main' points to the function to run.

The last bit of this is that many, though not all (and there's zero
requirement this has to be true), python packages that implement CLI
commands will often have a stanza in their __init__.py module that says
something like this:

if __name__ == '__main__':
do_the_command_line_stuff()

Alternatively, a package can include a literal __main__.py file that python
knows to check for, and this module is the one that gets executed for
`python3 -m mypackage` invocations. This is what mypy does.

Those are the magical blurbs that allow you to execute a module as if it
were a script by running "python3 -m mymodule" -- that hooks directly into
that little execution stanza. For python code distributed as packages,
that's the real reason to have that little magic stanza -- it provides a
convenient way to run stuff without needing to write the incredibly more
tedious:

python3 -c "from mypy.__main__ import console_entry; console_entry();"

... which is quite a bit more porcelain too, depending on how they
re/factor the code inside of the package.

Seeing as how mypy explicitly includes a __main__.py file:
https://github.com/python/mypy/blob/master/mypy/__main__.py, I am taking it
as a given that they are fully aware of invoking mypy in this fashion, and
believe it safe to rely on.

There will be a quiz later.
(There will not be a quiz.)

Reviewed-by: Hanna Reitz 
>

Thanks!

--js

*If your $PATH is configured to include, say, ~/.local/bin/, that is. On
Fedora, this is where executable shims for python get deposited when you do
"pip3 install --user somepackage". Python virtual environments will
configure your $PATH to include packages installed in that venv, etc etc
etc. Not every distro is configured such that pip packages are on the PATH,
so you can't rely on them being there when writing other scripts, usually.
Invoking the python interpreter directly seems far more flexible.

Re: [PATCH v3 07/16] iotests/297: Don't rely on distro-specific linter binaries

[Hanna Reitz]

On 16.09.21 06:09, John Snow wrote:

'pylint-3' is another Fedora-ism. Use "python3 -m pylint" or "python3 -m
mypy" to access these scripts instead. This style of invocation will
prefer the "correct" tool when run in a virtual environment.

Note that we still check for "pylint-3" before the test begins -- this
check is now "overly strict", but shouldn't cause anything that was
already running correctly to start failing.

Signed-off-by: John Snow 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Philippe Mathieu-Daudé 
---
  tests/qemu-iotests/297 | 45 --
  1 file changed, 26 insertions(+), 19 deletions(-)


I know it sounds silly, but to be honest I have no idea if replacing 
`mypy` by `python3 -m mypy` is correct, because no mypy documentation 
seems to suggest it.


From what I understand, that’s generally how Python “binaries” work, 
though (i.e., installed as a module invokable with `python -m`, and then 
providing some stub binary that, well, effectively does this, but kind 
of in a weird way, I just don’t understand it), and none of the 
parameters seem to be hurt in this conversion, so:


Reviewed-by: Hanna Reitz 

[PATCH v3 07/16] iotests/297: Don't rely on distro-specific linter binaries

[John Snow]

'pylint-3' is another Fedora-ism. Use "python3 -m pylint" or "python3 -m
mypy" to access these scripts instead. This style of invocation will
prefer the "correct" tool when run in a virtual environment.

Note that we still check for "pylint-3" before the test begins -- this
check is now "overly strict", but shouldn't cause anything that was
already running correctly to start failing.

Signed-off-by: John Snow 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Philippe Mathieu-Daudé 
---
 tests/qemu-iotests/297 | 45 --
 1 file changed, 26 insertions(+), 19 deletions(-)

diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
index b3a56a3a29..01dd8147d5 100755
--- a/tests/qemu-iotests/297
+++ b/tests/qemu-iotests/297
@@ -82,8 +82,11 @@ def run_linters():
 env['PYTHONPATH'] += os.pathsep + qemu_module_path
 except KeyError:
 env['PYTHONPATH'] = qemu_module_path
-subprocess.run(('pylint-3', '--score=n', '--notes=FIXME,XXX', *files),
-   env=env, check=False)
+subprocess.run(
+('python3', '-m', 'pylint', '--score=n', '--notes=FIXME,XXX', *files),
+env=env,
+check=False,
+)
 
 print('=== mypy ===')
 sys.stdout.flush()
@@ -94,23 +97,27 @@ def run_linters():
 # must not both define the __main__ module).
 env['MYPYPATH'] = env['PYTHONPATH']
 for filename in files:
-p = subprocess.run(('mypy',
-'--warn-unused-configs',
-'--disallow-subclassing-any',
-'--disallow-any-generics',
-'--disallow-incomplete-defs',
-'--disallow-untyped-decorators',
-'--no-implicit-optional',
-'--warn-redundant-casts',
-'--warn-unused-ignores',
-'--no-implicit-reexport',
-'--namespace-packages',
-filename),
-   env=env,
-   check=False,
-   stdout=subprocess.PIPE,
-   stderr=subprocess.STDOUT,
-   universal_newlines=True)
+p = subprocess.run(
+(
+'python3', '-m', 'mypy',
+'--warn-unused-configs',
+'--disallow-subclassing-any',
+'--disallow-any-generics',
+'--disallow-incomplete-defs',
+'--disallow-untyped-decorators',
+'--no-implicit-optional',
+'--warn-redundant-casts',
+'--warn-unused-ignores',
+'--no-implicit-reexport',
+'--namespace-packages',
+filename,
+),
+env=env,
+check=False,
+stdout=subprocess.PIPE,
+stderr=subprocess.STDOUT,
+universal_newlines=True
+)
 
 if p.returncode != 0:
 print(p.stdout)
-- 
2.31.1