[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-05 Thread Brett Cannon 
On Sat., Jun. 5, 2021, 07:38 Robin Becker,  wrote:

> On 04/06/2021 19:29, Steve Dower wrote:
> ...
> >
> > While we were investigating the reliably broken behaviour, we figured
> that related behaviour was
> > *unreliably* broken on all platforms.
> >
> > Specifically, if you imported a module through a relative path (resolved
> to CWD), changed the CWD, cleared the module
> > cache, and reimported, you would still get the original module, contrary
> to the intent of getting the newly resolved
> > module. ("Correct" code would have also cleared the import path caches,
> not just sys.modules.) This was because the
> > module info was cached using the relative path, and so would be used
> later even though its absolute path had changed.
> >
> > Someone reported this change in 3.8 and we decided to revert it in
> existing released, because the bugfix was for an
> > obscure enough situation that it really wasn't worth breaking any
> existing versions. For unreleased versions, it's
> > better to fix it so that imports will be more reliable in the future.
>
>
> Unfortunately we are using these relative imports to implement plugins for
> an xml language processor so this is likely
> not just an accident of ordering the failing tests. I agree we have to get
> a plugin module a bit more directly using the
> import internals. I still think we might need to control sys.path as
> presumably the plugin code could have imports of
> parallel modules.
>
> I'm beginning to understand that messing about with
> sys.path/modules/meta_path etc etc is a lot harder than it used to be.
>

How so? The key details have not changed since Python 3.4 when module specs
were introduced. And meta path, etc. are 2.7 concepts.


> Incidentally I find that I am able to import a namespace module directly;
> this code doesn't crash (at least in python >= 3.6
>
> ###
> #timpA.py don't run with any meaningful directory A
> import importlib
> def main():
>  module = importlib.import_module('A')
>  print('module=%r' % module)
>
> if __name__=='__main__':
>  main()
> ###
>
> in python 3.6-3.9 this prints out "module=" in
> 3.10.0b1 it prints
> "module= at 0x7f1c8a22fe20>)>".
> I thought I had to use pkgutils or pkg_resources to mess with these
> new-fangled packages.
>

Nope, that has never been required since the concept of namespace packages
were introduced to import.


> Likewise I find that the python statement 'import A' also doesn't fail and
> A is  a namespace.


Yep, that's expected.

I feel quite stupid now :(
>

As soon as you step outside the norm there are a lot of details to
understand; import is very much following the "Complex is better than
complicated" zen. So no need to feel stupid, there's a lot to keep track of
when you start to dive into the details.

-Brett

--
> Robin Becker
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KMJAQUGSXINVBLQOMREURUFN4LVCEIVX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KQFXVNNCYYOEBZJTNRXTT7RQBDRTR5LC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-05 Thread Robin Becker 

On 04/06/2021 19:29, Steve Dower wrote:
...


While we were investigating the reliably broken behaviour, we figured that related behaviour was 
*unreliably* broken on all platforms.


Specifically, if you imported a module through a relative path (resolved to CWD), changed the CWD, cleared the module 
cache, and reimported, you would still get the original module, contrary to the intent of getting the newly resolved 
module. ("Correct" code would have also cleared the import path caches, not just sys.modules.) This was because the 
module info was cached using the relative path, and so would be used later even though its absolute path had changed.


Someone reported this change in 3.8 and we decided to revert it in existing released, because the bugfix was for an 
obscure enough situation that it really wasn't worth breaking any existing versions. For unreleased versions, it's 
better to fix it so that imports will be more reliable in the future.



Unfortunately we are using these relative imports to implement plugins for an xml language processor so this is likely 
not just an accident of ordering the failing tests. I agree we have to get a plugin module a bit more directly using the 
import internals. I still think we might need to control sys.path as presumably the plugin code could have imports of 
parallel modules.


I'm beginning to understand that messing about with sys.path/modules/meta_path 
etc etc is a lot harder than it used to be.

Incidentally I find that I am able to import a namespace module directly; this 
code doesn't crash (at least in python >= 3.6

###
#timpA.py don't run with any meaningful directory A
import importlib
def main():
module = importlib.import_module('A')
print('module=%r' % module)

if __name__=='__main__':
main()
###

in python 3.6-3.9 this prints out "module=" in 3.10.0b1 
it prints
"module=)>".
I thought I had to use pkgutils or pkg_resources to mess with these new-fangled 
packages.

Likewise I find that the python statement 'import A' also doesn't fail and A is 
 a namespace. I feel quite stupid now :(
--
Robin Becker
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KMJAQUGSXINVBLQOMREURUFN4LVCEIVX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-04 Thread Steve Dower 

On 6/3/2021 7:42 PM, Senthil Kumaran wrote:

On Thu, Jun 03, 2021 at 07:08:06PM +0100, Robin Becker wrote:

The regression may well be a platform issue. I am by no means an expert at
building python; I followed a recipe from the ARCH PKGBUILD of some time


I meant the change in the diff we were suspecting was supposed to be
"Windows" specific. But interesting that it was observed in ARCH. The
non-windows piece of changes were probably responsible


While we were investigating the reliably broken behaviour, we figured 
that related behaviour was *unreliably* broken on all platforms.


Specifically, if you imported a module through a relative path (resolved 
to CWD), changed the CWD, cleared the module cache, and reimported, you 
would still get the original module, contrary to the intent of getting 
the newly resolved module. ("Correct" code would have also cleared the 
import path caches, not just sys.modules.) This was because the module 
info was cached using the relative path, and so would be used later even 
though its absolute path had changed.


Someone reported this change in 3.8 and we decided to revert it in 
existing released, because the bugfix was for an obscure enough 
situation that it really wasn't worth breaking any existing versions. 
For unreleased versions, it's better to fix it so that imports will be 
more reliable in the future.


Based on Robin's original post, it looks like this is exactly the 
scenario we wanted to fix, and it's probably exposed an underlying issue 
in that test suite.


Unfortunately, import caching has changed a lot since Python 2.7, so 
it's very unlikely that one approach will be able to do reliable 
reloading across all of those versions. Your best bet since 3.3 is to 
use the importlib module's methods (but your *best* bet overall is to 
run separate test suites in their own process so you can avoid the CWD 
changes).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RDIMZA2KVWZSYUVLWQUZT4TE5BH6QWOV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-03 Thread Cameron Simpson 
On 03Jun2021 17:06, Bernat Gabor  wrote:
>On Thu, Jun 3, 2021 at 5:05 PM Larry Hastings  
>wrote:
>> On 6/3/21 4:20 AM, Chris Johns wrote:
>> Might be out of context here, but IMHO "." shouldn't be assumed to be 
>> the current directory anyway.
>>
>> As someone who has ported python to a system where it isn't, these 
>> assumptions tend to cause problems.
>>
>> That sounds miserable.  What does "." signify on such a system, if not the
>> current directory?
>
>The only other thing that would make sense is a folder in the current
>working directory named ".", not?

I can't speak for Chris Johns, and we're off topic for this thread.

That said, I dislike "." in sys.path, particularly near the front. In 
fact my own "invoke this module's main" wrapper scripts strip it out. I 
remain personally of the opinion that it is a security nightmare, making 
imports _subject_ to the current directory (which might be anywhere, 
include a directory owned by a hostile user). So i dislike it as a 
default.

I know Robin becker is testing specific behaviour - I don't oppose being 
_able_ to put "." in sys.path (though I think a concrete absolute path 
is a saner choice).

So for Bernat and Larry: not systems where "." doesn't mean the working 
directory, but definitely in situations where you want a more secure 
loading of modules (i.e. only from where I expect, not destabilised by a 
cd).

Cheers,
Cameron Simpson 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YMV4LUW5LYFKJYEIHVM5GTCC6POM6BYF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-03 Thread Senthil Kumaran 
On Thu, Jun 03, 2021 at 07:08:06PM +0100, Robin Becker wrote:
> The regression may well be a platform issue. I am by no means an expert at
> building python; I followed a recipe from the ARCH PKGBUILD of some time

I meant the change in the diff we were suspecting was supposed to be
"Windows" specific. But interesting that it was observed in ARCH. The
non-windows piece of changes were probably responsible

> You guys can certainly develop language changes faster than us maintainers
> can use them; the dead hand of past usage is a real handicap.

I was thinking in terms of Bug-Fix (https://bugs.python.org/issue43105)
that is causing an interesting behavior change (still not certain to
call this a regression).

> I have managed a smaller example which breaks for me in arch linux latest.
> 
> The self contained code is here
> 
> https://www.reportlab.com/ftp/timport-310b1.py
> 
> I switched to using importlib.import_module as I think that can now always be 
> used in the versions we support.

Thanks for this report, Robin. 

I will reference this in the issue43105, which has probably has the
context to this.

-- 
Senthil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DGR6DEWZJRN5IDOZPC4LAHP5J3EIQ55X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-03 Thread Robin Becker 

On 03/06/2021 18:14, Senthil Kumaran wrote:

On Thu, Jun 03, 2021 at 09:55:57AM -0700, Guido van Rossum wrote:

Maybe this?

04732ca993 bpo-43105: Importlib now resolves relative paths when creating
module spec objects from file locations (GH-25121)


Likely!. But
https://github.com/python/cpython/commit/04732ca993fa077a8b9640cc77fb2f152339585a
was supposed to a platform specific bug fix.

I didn't a sense that reportlab regression detected was platform
specific one.  The behavior change is an interesting one though.

The regression may well be a platform issue. I am by no means an expert at building python; I followed a recipe from the 
ARCH PKGBUILD of some time past so perhaps there are problems from my incompetence.



More interesting thing for me was developers running large test suites
between alpha and beta releases, and reporting bugs! :-)

You guys can certainly develop language changes faster than us maintainers can use them; the dead hand of past usage is 
a real handicap.


I have managed a smaller example which breaks for me in arch linux latest.

The self contained code is here

https://www.reportlab.com/ftp/timport-310b1.py

I switched to using importlib.import_module as I think that can now always be 
used in the versions we support.

To generate a test base I used an empty directory and then run (in bash)

for d in A B;do rm -rf $d; mkdir $d;echo "print('+ in module %s' % __name__)" > $d/$(echo $d | tr [:upper:] 
[:lower:]).py; done



or
$  for d in A B;do
>rm -rf $d
>mkdir $d
>echo "print('+ in module %s' % __name__)" \
> > $d/$(echo $d | tr [:upper:] [:lower:]).py
> done

to create two simple modules in dirs A & B.

If I run python timport-310b1.py in 3.10.0a7 I occasionally see a problem, but in 3.10.0b1 or b2 I see the problem every 
time; the TB ends like this



print(recursiveImport(mn, debug=1))
  File "/home/robin/devel/reportlab/REPOS/timpbug/tests/timport-310b1.py", line 
36, in recursiveImport
return importlib.import_module(modulename)
  File 
"/home/robin/devel/reportlab/.py310/lib/python3.10/importlib/__init__.py", line 
126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 1050, in _gcd_import
  File "", line 1027, in _find_and_load
  File "", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'a'


The occasional problem with 3.10.0a7 make me think some random element is at 
work perhaps hashing stuff.
--
Robin Becker
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UJKVOAR6BUDOIFRSUGQ3THFA2DYG47PQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-03 Thread Senthil Kumaran 
On Thu, Jun 03, 2021 at 09:55:57AM -0700, Guido van Rossum wrote:
> Maybe this?
> 
> 04732ca993 bpo-43105: Importlib now resolves relative paths when creating
> module spec objects from file locations (GH-25121)

Likely!. But
https://github.com/python/cpython/commit/04732ca993fa077a8b9640cc77fb2f152339585a
was supposed to a platform specific bug fix.

I didn't a sense that reportlab regression detected was platform
specific one.  The behavior change is an interesting one though.

More interesting thing for me was developers running large test suites
between alpha and beta releases, and reporting bugs! :-)

-- 
Senthil

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GJWGQQGZNUWYIX3UX6T4H5VN7IE4YGN6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-03 Thread Guido van Rossum 
Maybe this?

04732ca993 bpo-43105: Importlib now resolves relative paths when creating
module spec objects from file locations (GH-25121)

On Thu, Jun 3, 2021 at 9:49 AM Senthil Kumaran  wrote:

> On 3 Jun 2021, at 09:31, Robin Becker  wrote:
>
> > ReportLab has quite a large codebase and I think it would be hard to
> > get a concise test of this behaviour change. I would be glad if this
> > is an expected change a7-->b1 and if the use of '.' in this way has
> > become somehow wrong.
>
> To me, this sounds like an excellent candidate for git bisect to help
> and figure out which exact commit caused this behavior change.
>
> Was it possible for you to detect that?
>
> Thank you,
> Senthil
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CW2SAOGIO5EIXU7GJZIN26MLF6AC2HD5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NV4E6CALNMWR7C2J2NE2QABKWXYRZTZN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-03 Thread Senthil Kumaran 
On 3 Jun 2021, at 09:31, Robin Becker  wrote:

> ReportLab has quite a large codebase and I think it would be hard to
> get a concise test of this behaviour change. I would be glad if this
> is an expected change a7-->b1 and if the use of '.' in this way has
> become somehow wrong.

To me, this sounds like an excellent candidate for git bisect to help
and figure out which exact commit caused this behavior change.

Was it possible for you to detect that?

Thank you,
Senthil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CW2SAOGIO5EIXU7GJZIN26MLF6AC2HD5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-03 Thread Bernat Gabor 
The only other thing that would make sense is a folder in the current
working directory named ".", not?

On Thu, Jun 3, 2021 at 5:05 PM Larry Hastings  wrote:

>
> On 6/3/21 4:20 AM, Chris Johns wrote:
>
> Might be out of context here, but IMHO "." shouldn't be assumed to be the 
> current directory anyway.
>
> As someone who has ported python to a system where it isn't, these 
> assumptions tend to cause problems.
>
>
> That sounds miserable.  What does "." signify on such a system, if not the
> current directory?
>
>
> */arry*
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LVOYZGSOJ3OGDX7OUTDXRQVXZ7YGGPS7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DZRKL3Q7JHT2RQ3UW3E724VSPTWPCS4L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-03 Thread Larry Hastings 


On 6/3/21 4:20 AM, Chris Johns wrote:

Might be out of context here, but IMHO "." shouldn't be assumed to be the 
current directory anyway.

As someone who has ported python to a system where it isn't, these assumptions 
tend to cause problems.



That sounds miserable.  What does "." signify on such a system, if not 
the current directory?



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LVOYZGSOJ3OGDX7OUTDXRQVXZ7YGGPS7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

2021-06-03 Thread Chris Johns 
Might be out of context here, but IMHO "." shouldn't be assumed to be the 
current directory anyway.

As someone who has ported python to a system where it isn't, these assumptions 
tend to cause problems.

Cheers

Chris 

Sent from my iPhone

> On 3 Jun 2021, at 09:31, Robin Becker  wrote:
> 
> I have a failing test case in 3.10.0b1 for code that uses changes to 
> sys.path and attempts to load unpackaged modules by changing directory and 
> placing '.' into sys.path; the code works as expected in 2.7.18, 3.6-3.10.0a7.
> 
> I printed out related variables in the a7 and b1 runs and find that sys.path 
> is identical (modulo version name changes) and that os.getcwd() is as 
> expected and that os.path.isfile('./%s.py' % modulename) is True.
> 
> In a7 exec('%s as m', NS) works, but in b1 I get ModuleNotFoundError.
> 
> I can fix the problem by changing
> 
>sys.path.insert(0,'.')
> 
> into
> 
>sys.path.insert(0,os.getcwd())
> 
> The behaviour of '.' in sys.path has always been slightly mysterious, but is 
> this a real bug and have we just been lucky for 20 years?
> 
> I have placed some details here
> 
> https://www.reportlab.com/ftp/import-bug-310b1.txt
> 
> ReportLab has quite a large codebase and I think it would be hard to get a 
> concise test of this behaviour change. I would be glad if this is an expected 
> change a7-->b1 and if the use of '.' in this way has become somehow wrong.
> 
> I do know that during the tests the errant code is executed with different 
> values for the target directory; the first usage seems to succeed.
> -- 
> Robin Becker
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/DE3MDGB2JGOJ3X4NWEGJS26BK6PJUPKW/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/664GGOGFJZ3SQ7PWLA6BYP4A27TSFZ63/
Code of Conduct: http://python.org/psf/codeofconduct/