[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Łukasz Langa]

Change by Łukasz Langa :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[miss-islington]

miss-islington  added the comment:


New changeset 84494db41902774ea6ac72e5b308b429850bbf71 by Miss Islington (bot) 
in branch '3.10':
bpo-41737: expand doc for NotADirectoryError (GH-27471)
https://github.com/python/cpython/commit/84494db41902774ea6ac72e5b308b429850bbf71


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset b5f026112768eb0a06622263bdea86d7d85981c5 by Miss Islington (bot) 
in branch '3.9':
bpo-41737: expand doc for NotADirectoryError (GH-27471) (GH-27577)
https://github.com/python/cpython/commit/b5f026112768eb0a06622263bdea86d7d85981c5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 5.0 -> 6.0
pull_requests: +26082
pull_request: https://github.com/python/cpython/pull/27576

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset f7c23a99cd4f8179b6ba2cffaeb78b852c0f6488 by andrei kulakov in 
branch 'main':
bpo-41737: expand doc for NotADirectoryError (GH-27471)
https://github.com/python/cpython/commit/f7c23a99cd4f8179b6ba2cffaeb78b852c0f6488


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[miss-islington]

Change by miss-islington :


--
pull_requests: +26083
pull_request: https://github.com/python/cpython/pull/27577

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Eryk Sun]

Eryk Sun  added the comment:

> On MacOS, I get Not a directory error on all 3

Confirmation on Linux and macOS suffices for "most POSIX systems", based on 
market share for desktops and servers. It would be nice to also confirm this 
for FreeBSD, AIX, HP-UX, and Solaris, but I suppose it isn't necessary. A 
system that strictly conforms to POSIX will behave the same for those 
operations.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Andrei Kulakov]

Andrei Kulakov  added the comment:

Eryk: On MacOS, I get Not a directory error on all 3:

NotADirectoryError: [Errno 20] Not a directory: 'spam'

NotADirectoryError: [Errno 20] Not a directory: 'spam/'

NotADirectoryError: [Errno 20] Not a directory: 'spam/eggs'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Eryk Sun]

Eryk Sun  added the comment:

> It should be emphasized that it may happen on a **file operation**

I think it's adequately covered by "attempts to open or traverse a 
non-directory file". The reader should know that opening 
"/path/to/file/somename.txt" requires traversing the components in the path. So 
if "file" isn't a directory, raising NotADirectoryError should be expected in 
POSIX.

---

If someone can verify the behavior on common non-Linux POSIX systems such as 
macOS, FreeBSD, and OpenBSD, then the wording could be narrowed down to "on 
most POSIX platforms" instead of "on some platforms". 

For example, given "spam" is a regular file in the current directory, check 
os.open('spam', os.O_DIRECTORY); os.open('spam/', 0); and os.open('spam/eggs', 
0).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Danny Lin]

Danny Lin  added the comment:

It should be emphasized that it may happen on a **file operation**... So at 
least be something like: "On some platforms, it may also be raised if a file 
operation involves an attempt to open or traverse a non-directory file as if it 
were a directory."

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Andrei Kulakov]

Andrei Kulakov  added the comment:

Eryk: sounds good, updated the PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Eryk Sun]

Eryk Sun  added the comment:

I'd prefer a generic wording regarding the platform, and overall simpler 
phrasing without examples: "On some platforms, it may also be raised if an 
operation attempts to open or traverse a non-directory file as if it were a 
directory." 

The latter doesn't apply to Windows generally. Attempting to traverse a 
non-directory fails with ERROR_PATH_NOT_FOUND (mapped to C errno ENOENT), and 
attempting to open a non-directory with a path that has a trailing slash fails 
with ERROR_INVALID_NAME (mapped to C errno EINVAL).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Danny Lin]

Danny Lin  added the comment:

@Andrei Kulakov: That statement is mainly for illustration that such behavior 
may vary across platforms. I use a rather vague statement as I'm not totally 
sure it applies if (and only if) running on POSIX. A more accurate statement is 
welcome as long as it's proven true.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Andrei Kulakov]

Andrei Kulakov  added the comment:

Danny: then it would be probably more useful to say "On POSIX systems, it will 
be raised when ...".

I'm ambivalent on whether this is needed.

Eryk: wdyt?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Danny Lin]

Danny Lin  added the comment:

@Andrei Kulakov: I was commenting on the previous version. The revised version 
(f2ae30b0de3c4ba1f16fc2a430cf22b447c062ed) seems ok to me.

Another question: would it be better if we add "on some platforms" for the part 
that the error may raise on a file operation?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Andrei Kulakov]

Andrei Kulakov  added the comment:

(in fact, there is one example (for f-string exception) but the overall pattern 
is to just have reference type descriptions)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Andrei Kulakov]

Andrei Kulakov  added the comment:

Danny: that would be inconsistent with the rest of the doc, it's written as a 
reference list for exceptions and none of them have examples currently.. It 
also seems fairly clear without an example.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Danny Lin]

Danny Lin  added the comment:

> I've put up a PR that expands the docs for NotADirectoryError here: 
> https://github.com/python/cpython/pull/27471/files

Thank you.

Wouldn't it be more clear if an example is provided? Like:

(e.g. `/path/to` does not exist when running `open('/path/to/file')`)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Andrei Kulakov]

Andrei Kulakov  added the comment:

I've put up a PR that expands the docs for NotADirectoryError here: 

https://github.com/python/cpython/pull/27471/files

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Andrei Kulakov]

Change by Andrei Kulakov :


--
nosy: +andrei.avk
nosy_count: 3.0 -> 4.0
pull_requests: +25990
pull_request: https://github.com/python/cpython/pull/27471

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Eryk Sun]

Eryk Sun  added the comment:

Regarding documentation, builtin open() and the io and os modules generally do 
not provide information about platform-specific errors. But documenting the 
behavior for the NotADirectoryError exception itself may be useful considering 
it applies to many POSIX functions that access filepaths, such as stat, open, 
mkdir, rmdir, unlink, and rename.

Regarding behavior, I don't see anything reasonable that can or should be done 
for POSIX. In Windows, it should be possible to know whether FileNotFoundError 
is due to a bad path prefix (ERROR_PATH_NOT_FOUND, 3) or a missing file 
(ERROR_FILE_NOT_FOUND, 2), but io.FileIO currently only uses standard C errno, 
which maps both of these cases to ENOENT. So one behavior that *can* be fixed 
in this situation is to get the actual Windows error code from MSVC _doserrno, 
such that the raised FileNotFoundError would have the relevant Windows error 
code in its winerror attribute.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Danny Lin]

Danny Lin  added the comment:

> Really? It's not obvious that finding a file would involve directory 
> operations?

Not in some senses, and that's possibly why Windows does not raise 
NotADirectoryError in such case.

I agree that it's a platform-specific implementation detail. However, there are 
lots of platform-specific implementation details written in documentation 
elsewhere already, especially OS related modules such as os and os.path. I 
don't think mentioning platform-specific implementation details for subclasses 
of OSError would be any less reasonable than that.

Adding such notice for NotADirectoryError helps people who want a EAFP style 
code for nonexistent file by preventing them getting trapped writing a 
cross-platform code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Irit Katriel]

Irit Katriel  added the comment:

> I don't think a general developer would expect that 
> open('/path/to/file/somename.txt') implies a directory operation, and it also 
> doesn't on Windows.


Really? It's not obvious that finding a file would involve directory operations?

In what sense does it even matter whether you expect a directory operation to 
happen? The contract is that if the open() fails you get an OSError. The 
documentation doesn't say which OSError, and that is in fact a 
platform-specific implementation detail.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Danny Lin]

Danny Lin  added the comment:

I don't think a general developer would expect that 
open('/path/to/file/somename.txt') implies a directory operation, and it also 
doesn't on Windows.

I suggest that a further notice be added to NotADirectoryError, such as:

Raised when a directory operation (such as os.listdir()) is requested on 
something which is not a directory. Corresponds to errno ENOTDIR. In some 
filesystem such as POSIX, NotADirectoryError (ENOTDIR) is raised when 
attempting to open a path whose ancestor is not a directory.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Irit Katriel]

Irit Katriel  added the comment:

Hi Danny, 

I'm not saying that OSError and FileNotFoundError are equivalent. I'm saying 
that the open() API, as documented, raises OSError when opening the file fails.

The way to check whether a file exists is to use os.path.exists(path) or 
os.path.isfile(path). 

I don't quite follow your last point - opening a file is indeed a file 
operation, but it contains within it a directory operation (finding the file).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Danny Lin]

Danny Lin  added the comment:

By writing "except FileNotFoundError:", the intention is to catch an error when 
the file being opened is not found, and don't catch an error for other cases, 
such as an existing file without adequate permission. Writing "except OSError:" 
catches just too much cases including unwanted ones.

As they are not equivalent, you cannot say that the latter is the "correct" way 
for the former.

And you totally omitted the argument for the inadequate documentation for 
NotADirectoryError. It says NotADirectoryError is raises when a DIRECTORY 
operation is requested, and does not cover the case of opening a file.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Irit Katriel]

Change by Irit Katriel :


--
keywords: +patch
pull_requests: +21774
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/22818

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Irit Katriel]

Irit Katriel  added the comment:

The documentation for open says "If the file cannot be opened, an OSError is 
raised."

NotADirectoryError and FileNotFoundError are both OSErrors.

So the correct way to write Danny's code snippet, according to the 
documentation, is:

try:
open('/path/to/file/somename.txt')
except OSError:
"""do something"""

--
components: +IO
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file in a fake directory

[Terry J. Reedy]

Change by Terry J. Reedy :


--
title: Improper NotADirectoryError when opening a file under a fake directory 
-> Improper NotADirectoryError when opening a file in a fake directory

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file under a fake directory

[Danny Lin]

Danny Lin  added the comment:

I'm not so familiar about the spec. If such behavior is confirmed due to 
implementation difference across OSes, and it's also not desirable to change 
the mapping of the OS error to Python exception, we can simplify left it as-is.

However, this behavior difference can potentially cause cross-platform 
compatibility. For example, the code:

try:
open('/path/to/file/somename.txt')
except FileNotFoundError:
"""do something"""

would work on Windows but break on Linux (or POSIX?).

The current (3.8) description for exception NotADirectoryError is:

Raised when a directory operation (such as os.listdir()) is requested on 
something which is not a directory. Corresponds to errno ENOTDIR.

According to this, a user probably won't expect to receive a NotADirectoryError 
for open('/path/to/file/somename.txt'), as this doesn't seem like a directory 
operation at all, unless he is expert enough to know about the implication of 
errno ENOTDIR.

I think a note should at least be added to the documentation if we are not 
going to change the behavior.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file under a fake directory

[Eryk Sun]

Eryk Sun  added the comment:

> if NotADirectoryError should be raised, it should mention '/path/to/file' 
> rather then '/path/to/file/somename.txt'.

POSIX specifies that C open() should set errno to ENOTDIR when an existing path 
prefix component is neither a directory nor a symlink to a directory [1]. What 
you propose isn't possible to implement reliably unless the filesystem is 
locked or readonly, so it should be handled by the application instead of by 
the system or standard libraries.

> FileNotFoundError: [Errno 2] No such file or directory: 
> 'C:\\path\\chrome\\to\\file\\somename.txt'

Windows specifies that this case should fail as follows: "[i]f 
Link.File.FileType is not DirectoryFile, the operation MUST be failed with 
STATUS_OBJECT_PATH_NOT_FOUND" [2] (see Phase 6 in the linked pseudocode). 

The Windows API maps this status code to ERROR_PATH_NOT_FOUND (3), which is 
distinct from ERROR_FILE_NOT_FOUND (2). However, the C runtime maps both of 
these system error codes to POSIX ENOENT. All isn't lost, however, because it 
also saves the OS error in _doserrno. io.FileIO could preset _doserrno to 0, 
and if it's non-zero after calling _wopen, use its value with 
PyErr_SetExcFromWindowsErrWithFilenameObject instead of calling 
PyErr_SetFromErrnoWithFilenameObject.

---

[1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html
[2] 
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/8ada5fbe-db4e-49fd-aef6-20d54b748e40

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41737] Improper NotADirectoryError when opening a file under a fake directory

[Danny Lin]

New submission from Danny Lin :

On Linux (tested on Ubuntu 16.04), if "/path/to/file" is an existing file, the 
code:

open('/path/to/file/somename.txt')

raises NotADirectoryError: [Errno 20] Not a directory: 
'/path/to/file/somename.txt'

On Windows, similar code:

open(r'C:\path\to\file\somename.txt')

raises FileNotFoundError: [Errno 2] No such file or directory: 
'C:\\path\\chrome\\to\\file\\somename.txt'

I think the behavior on Linux is not correct. The user probably cares about the 
existence of the file to be opened, rather than whether its ancestor 
directories are valid.

OTOH, if NotADirectoryError should be raised, it should mention '/path/to/file' 
rather then '/path/to/file/somename.txt'. But what if '/path/to' or '/path' is 
actually a file? Should it be '/path/to' or '/path' instead for the same reason?

--
messages: 376505
nosy: danny87105
priority: normal
severity: normal
status: open
title: Improper NotADirectoryError when opening a file under a fake directory
type: behavior
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com