[issue32642] add support for path-like objects in sys.path

[Eryk Sun]

Eryk Sun  added the comment:

> I've got in mind a PyListObject subclass with calls to PyOS_FSPath 
> before insert, append, extend and __getitem__.

The sys module doesn't prevent rebinding sys.path. Most code I think is careful 
to update sys.path. But surely some code replaces it with a new list instead of 
updating via insert(), append(), extend(), or updating a slice such as 
`sys.path[:] = new_path`. For example, ModifiedInterpreter.transfer_path() in 
Lib/idlelib/pyshell.py rebinds sys.path. It doesn't have to, but it does.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

On 26.03.2022 08:59, Nick Coghlan wrote:
> 
> The import system is already complex, so I think the hesitation about 
> allowing arbitrary Path-like objects is warranted. (For example: are 
> importlib's caching semantics really valid for *arbitrary* path-like objects? 
> An object can be path-like without being immutable)
> 
> Coercion on input (as Noam suggests) would have a much lower risk of unwanted 
> side effects, as any dynamic behaviour would be resolved at insertion time.

This is not only about the import system. Lots of Python code out there
manipulates sys.path or reads sys.path for various reasons and does not
expect Path objects as list members, since only strings and bytes
are allowed:

https://docs.python.org/3/library/sys.html#sys.path

Conversion to strings sounds like a good way to get the best out of
both worlds.

I'm just curious on how this would work. You'd like have to create a
list subclass for use with sys.path which applies the conversion
whenever a non-string member gets added. Or perhaps add helper methods
to Path objects to safely add their value to sys.path.

--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Noam Cohen]

Noam Cohen  added the comment:

that's why pre-insert conversion was suggested.
path-like objects could still be added to sys.path, while converting to str 
upon insert, preserving sys.path's bytes and strings only policy.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

pkgutil just skips non-string elements in sys.path.

for dir in search_path:
if not isinstance(dir, str):
continue

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I think you are trying to solve a wrong problem.

> This wasn't obvious because Path objects appear as strings in normal debug 
> output, etc.

How is it?

>>> pathlib.Path('/usr/lib')
PosixPath('/usr/lib')
>>> [pathlib.Path('/usr/lib')]
[PosixPath('/usr/lib')]

I think the problem is with the debug output. Always use repr() for it, 
otherwise you will trick yourself.

> but the corresponding imports weren't working as they should

It would be easier to catch if it was never working with non-string. I think we 
should deprecate any non-string elements in sys.path.

As I shown above some code only works correctly with strings anyway.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Noam Cohen]

Noam Cohen  added the comment:

So I've got in mind a PyListObject subclass with calls to PyOS_FSPath before 
insert, append, extend and __getitem__.

But I feel like this is an overkill; also, extend function might be trickier to 
implement.
Do any of you have better idea?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Nick Coghlan]

Nick Coghlan  added the comment:

The import system is already complex, so I think the hesitation about allowing 
arbitrary Path-like objects is warranted. (For example: are importlib's caching 
semantics really valid for *arbitrary* path-like objects? An object can be 
path-like without being immutable)

Coercion on input (as Noam suggests) would have a much lower risk of unwanted 
side effects, as any dynamic behaviour would be resolved at insertion time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Noam Cohen]

Noam Cohen  added the comment:

You've got a point.
But in my opinion path-like object usage should be intuitive and users should 
not worry about converting it into string in some of the places.

What do you feel about sub-classing list for sys.path and converting path-like 
objects upon setting items / reading?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Are there any problems with converting a Path to string before adding it to 
sys.path? You do this one time, and any users of sys.path will not need to 
bother about conversion. It is better even for performance.

Otherwise we will need to revise and update every code that uses sys.path, and 
many bugs will left in third-party code for years. For example, in unittest the 
code

if not top_level_dir in sys.path:
sys.path.insert(0, top_level_dir)

should be replaced with more cumbersome

for path in sys.path:
if os.fsdecode(path) == top_level_dir:
break
else:
sys.path.insert(0, top_level_dir)

In multiprocessing the code

sys_path=sys.path.copy()
try:
i = sys_path.index('')
except ValueError:
pass
else:
sys_path[i] = process.ORIGINAL_DIR

should be replaced with

sys_path=sys.path.copy()
for i, path in enumerate(sys_path):
if os.fsdecode(path) == '':
sys_path[i] = process.ORIGINAL_DIR
break

It is just two examples. I did not review the whole stdlib, and there should be 
more third-party code.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Noam Cohen]

Change by Noam Cohen :


--
nosy: +ncohen
nosy_count: 8.0 -> 9.0
pull_requests: +30196
pull_request: https://github.com/python/cpython/pull/32118

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Chih-Hsuan Yen]

Change by Chih-Hsuan Yen :


--
nosy:  -yan12125

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jason R. Coombs]

Jason R. Coombs  added the comment:

I'm in support of adding Path support for sys.path, but I also agree with Eric, 
there are innumerable consumers of sys.path beyond importlib. and since 
pathlib.Path isn't a str, it would likely introduce incompatibility. On the 
other hand, users introducing Path objects to sys.path could be warned that 
although importlib supports Path objects, other consumers may not, and that 
support for it in importlib isn't endorsement of the use of those types and the 
consequences aren't necessarily supported.

As an aside, it's too bad a Path object couldn't have been a str subclass (as 
it is for [path](https://pypi.org/project/path), which would have made problems 
like this one much safer to solve.

--
nosy: +jaraco

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Éric Araujo]

Éric Araujo  added the comment:

I’m not an import expert but would have misgivings about having fancy types in 
sys.path too!  It seems so fundamental, and used from C and Python, with a 
simple interface of a direct list (plus importers and finders etc), that I 
would understand it pure strings were required and not Paths (and not all 
path-likes).

--
nosy: +eric.araujo
versions: +Python 3.11 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Roundup Robot]

Change by Roundup Robot :


--
nosy: +python-dev
nosy_count: 6.0 -> 7.0
pull_requests: +21106
pull_request: https://github.com/python/cpython/pull/22000

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Brett Cannon]

Brett Cannon  added the comment:

This seems fine to me and I can't think of any negatives. Can anyone think of 
why this might be a bad idea? (Messing with how sys.path is used is just so 
fundamental I'm being paranoid. :)

--
nosy: +eric.snow, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

srry I opened another issue bpo-33099

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Brett Cannon]

Brett Cannon  added the comment:

Please open a separate issue for the test_poplib issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

it seems to me like the issue in my tests is that some SSL thing is failing?, 
anyone have any experience with this?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

https://pastebin.com/q4FKnPZH

the trace for the test_poplib

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

I'm having issues with my local changes for my PR, I'm unsure why my local 
machine takes a seemingly infinite amount of time on test_poplib, so again I 
think something to do with my local environment is causing issues again, any 
help would be appreciated...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

The issue was resolved by updating my version of the rest of the package 
apparently and remaking the whole thing, must have been some outdated stuff on 
my end causing the issue

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

I've been stuck on "test_poplib" for over 149661 sec now, that's about 41 
hours... I don't think this is working correctly, although I'm unsure  what 
test_poplib is doing that has been affected by what I've changed here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Brett Cannon]

Brett Cannon  added the comment:

'make regen-all' should do it (but so will just running 'make -s -j' like
any old build of CPython).

On Mon, Mar 5, 2018, 12:43 Jay Yin,  wrote:

>
> Jay Yin  added the comment:
>
> I'm unsure how to regenerate the files that interact with the code for
> sys.path as travisCI states
> "
> Generated files not up to date
>  M Python/importlib_external.h
> "
> since my code changes some of how the importing is handled
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

I'm unsure how to regenerate the files that interact with the code for sys.path 
as travisCI states
"
Generated files not up to date
 M Python/importlib_external.h
"
since my code changes some of how the importing is handled

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Change by Jay Yin :


--
keywords: +patch
pull_requests: +5484
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

https://github.com/python/cpython/blob/3c34aad4e7a95913ec7db8e5e948a8fc69047bf7/Lib/importlib/_bootstrap_external.py#L1069-L1090

those are the particular class and lines I'm referring to

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

ok, so I found the PathFinder class you referenced, just making sure, this 
issue pertains to changing "self.path"'s usage and declaration to be a 
path-like object instead of the hard coded 'sys', 'path' returns? or is that 
part of it already?, also since this uses a tuple to keep track of the parent 
path, is there functionality in the path-like objects to find parent paths?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

Thanks for the reply

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Chih-Hsuan Yen]

Chih-Hsuan Yen  added the comment:

> what file(s) is/are the sys.path code located in?

If I understand it correctly, sys.path is handled in 
importlib._bootstrap_external.PathFinder.find_spec(). I can patch PathFinder 
for handling path-like objects: 
https://github.com/yan12125/cpython/commit/e3fd473b54cbb368533e651fd896bbc813a43924

Here's an example usage:

# t.py
import pathlib
import sys

sys.path.append(pathlib.Path('foo'))

import s

# foo/s.py
print(123)

--
nosy: +yan12125

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

what file(s) is/are the sys.path code located in?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

This looks a lot like https://bugs.python.org/issue32446, I'd like to tackle 
this, if we are going through with it.

--
nosy: +jayyin11043

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Chris Jerdonek]

New submission from Chris Jerdonek :

This issue is to suggest enhancing sys.path to recognize path-like objects, per 
PEP 519.

I recently ran into an issue where a path was getting added to sys.path, but 
the corresponding imports weren't working as they should, even though sys.path 
showed the path as being present. Eventually I tracked this down to the path 
being a pathlib.Path object rather than a string. This wasn't obvious because 
Path objects appear as strings in normal debug output, etc.

The sys.path docs currently say this:

> Only strings and bytes should be added to sys.path; all other data types are 
> ignored during import.

--
components: Library (Lib)
messages: 310560
nosy: brett.cannon, chris.jerdonek
priority: normal
severity: normal
status: open
title: add support for path-like objects in sys.path
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com