[issue12797] io.FileIO and io.open should support openat

[Eryk Sun]

Eryk Sun added the comment:

Марк, os.open added dir_fd support in 3.3, which is implemented on POSIX 
systems by calling openat. The dir_fd parameter is available for many os 
functions. This is discussed in section 1.5, Files and Directories [1].

It would be nice if we could support dir_fd on Windows as well, but we'd have 
to bypass the CRT and Windows API to use the native NT API instead, such as 
NtCreateFile [2]. The kernel has supported opening a file relative to a 
directory handle since it was release in 1993 (NT 3.1). All named kernel 
objects are referenced using an OBJECT_ATTRIBUTES [3] data structure. 
ObjectName -- a path with up to 32768 UTF-16 characters -- is relative to the 
RootDirectory handle if non-NULL. This is how paths relative to the process 
working directory are implemented, but changing the working directory isn't 
thread safe. 

[1]: https://docs.python.org/3/library/os.html#files-and-directories
[2]: https://msdn.microsoft.com/en-us/library/ff566424
[3]: https://msdn.microsoft.com/en-us/library/ff557749

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Марк Коренберг]

Марк Коренберг added the comment:

But... os.openat() is still missing... why status is closed() ?!

--
nosy: +mmarkk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

See #13424 for a doc request about this.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 Updated patch:
 * checks for long overflow
 * raises original exception if opener returns null
 * makes it explicit that opener must return an open file descriptor.

This looks good to me. You just need to add a versionchanged attribute
in the documentation.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Roundup Robot]

Roundup Robot devn...@psf.upfronthosting.co.za added the comment:

New changeset 0d64d9ac2b78 by Ross Lagerwall in branch 'default':
Issue #12797: Added custom opener parameter to builtin open() and FileIO.open().
http://hg.python.org/cpython/rev/0d64d9ac2b78

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc amaur...@gmail.com added the comment:

Is an open file descriptor correct in English? I'd have written an opened 
file descriptor instead (in 5 places).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Charles-François Natali]

Charles-François Natali neolo...@free.fr added the comment:

 Is an open file descriptor correct in English? I'd have written an
 opened file descriptor instead (in 5 places).

open is correct.
For example, you say the store is open, not the store is opened: open is 
an adjective, whereas opened is the past participe.
See 
http://www.usingenglish.com/forum/ask-teacher/15771-open-opened-welcome-welcomed.html

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

 I'd have written an opened file descriptor instead (in 5 places).

Yes, 'open' is an adjective as well as a verb, and the correct one in 
this context. Something that has been opened, such as a sealed jar or 
envelope, might have been re-closed, but it is no longer the same as 
never-opened.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Ross Lagerwall]

Ross Lagerwall rosslagerw...@gmail.com added the comment:

Thanks (and for the English lesson ;-) )

--
assignee:  - rosslagerwall
resolution:  - fixed
stage: needs patch - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Here is my quick review:
- shouldn't the opener also get the third open() argument (although it 
currently seems to always be 0o666)?
- when fdobj is NULL, you shouldn't override the original error
- PyLong_AsLong can fail (if the opener returns too large an int), you should 
check for that

Thank you!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Benjamin Peterson]

Benjamin Peterson benja...@python.org added the comment:

Also, the documentation should indicate what exactly is supposed to be returned 
by opener.

--
nosy: +benjamin.peterson

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Ross Lagerwall]

Ross Lagerwall rosslagerw...@gmail.com added the comment:

Updated patch:
* checks for long overflow
* raises original exception if opener returns null
* makes it explicit that opener must return an open file descriptor.

I don't think that mode should be passed in since it is not specified in the 
parameters to open() (and always defaults to 0o666 anyway). Specifying the file 
mode should be left to the opener if needed.

--
Added file: http://bugs.python.org/file23562/opener_v2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Charles-François Natali]

Charles-François Natali neolo...@free.fr added the comment:

 Thanks. Although, on second thought, I'm not sure whether Amaury's 
 idea (allowing a custom opener) is not better... Thoughts?

+1.
This would also address issues #12760 and #12105.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Ross Lagerwall]

Ross Lagerwall rosslagerw...@gmail.com added the comment:

What would you envisage the API for the custom opener to look like?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 What would you envisage the API for the custom opener to look like?

The same as os.open(), I would say.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Ross Lagerwall]

Ross Lagerwall rosslagerw...@gmail.com added the comment:

Before I implement it properly, is this the kind of api that's desired?


import os
import io

class MyOpener:
def __init__(self, dirname):
self.dirfd = os.open(dirname, os.O_RDONLY)

def open(self, path, flags, mode):
return os.openat(self.dirfd, path, flags, mode)

myop = MyOpener(/tmp)
f = open(testfile, w, opener=myop.open)
f.write(hello)


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 Before I implement it properly, is this the kind of api that's desired?

Yes, although I think most people would use a closure instead of a dedicated 
class.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Ross Lagerwall]

Ross Lagerwall rosslagerw...@gmail.com added the comment:

The attached patch adds the opener keyword + tests.

--
Added file: http://bugs.python.org/file23550/opener.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Ross Lagerwall]

Ross Lagerwall rosslagerw...@gmail.com added the comment:

I guess that would make it more general...

I'll play around with it for a bit. It mustn't become too hard to use though 
since the original point was to simplify the opening of files :-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 Attached is a patch which adds dirfd= as a keyword argument.

Thanks. Although, on second thought, I'm not sure whether Amaury's idea 
(allowing a custom opener) is not better... Thoughts?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Ross Lagerwall]

Ross Lagerwall rosslagerw...@gmail.com added the comment:

Attached is a patch which adds dirfd= as a keyword argument.

--
keywords: +patch
Added file: http://bugs.python.org/file23272/i12797.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

I prefer a new parameter either at the end of the arglist or possibly keyword 
only. The idea for both variations is to let typical users ignore the option, 
which would be hard to do if it is part of the prime parameter. The idea for 
keyword only is that we might want to add other rarely used but useful options. 
They have no natural order, and having say, 8 positional params is pretty 
wretched. (I have worked with such APIs.)

--
nosy: +terry.reedy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Éric Araujo]

Changes by Éric Araujo mer...@netwok.org:


--
nosy: +eric.araujo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Antoine Pitrou]

New submission from Antoine Pitrou pit...@free.fr:

Right now it is painful to integrate openat() with the normal IO classes. You 
have to figure out the low-level flags yourself (i.e. replicate the logic and 
error handling from the FileIO constructor), then replicate the open() logic 
yourself (because you want to set the name attribute on the FileIO object 
before wrapping it).

Therefore it would be nice if the FileIO constructor and the open() function 
supported openat natively. I see two possibilities:
- allow a (dirfd, name) tuple for the first file argument
- allow an optional dirfd argument at the end of the arglist

--
components: IO, Library (Lib)
messages: 142560
nosy: haypo, neologix, pitrou, rosslagerwall
priority: normal
severity: normal
stage: needs patch
status: open
title: io.FileIO and io.open should support openat
type: feature request
versions: Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Nadeem Vawda]

Changes by Nadeem Vawda nadeem.va...@gmail.com:


--
nosy: +nadeem.vawda

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc amaur...@gmail.com added the comment:

A third idea is to find a way to override the low-level open() function (the 
one that returns a fd).

openat() seems to exist only on Linux, so I'm -1 on adding new parameters to 
support this function only.

--
nosy: +amaury.forgeotdarc

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 A third idea is to find a way to override the low-level open()
 function (the one that returns a fd).

Why not. It would e.g. allow to use CreateFile under Windows (the hg
guys do this in order to change the sharing mode to something more
laxist).

 openat() seems to exist only on Linux, so I'm -1 on adding new
 parameters to support this function only.

openat() is POSIX:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

 allow an optional dirfd argument at the end of the arglist

I prefer this suggestion.

I didn't know openat(). Antoine told me that it can be used, for example, to 
fix security vulnerabilities like #4489.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Ned Deily]

Ned Deily n...@acm.org added the comment:

I believe openat is new to POSIX (mandatory as of POSIX 2008).  For example, 
it's not currently in OS X and apparently was first added to FreeBSD in 8.0.  
So it would have to be checked by configure and documented as 
platform-dependent.

--
nosy: +ned.deily

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 I believe openat is new to POSIX (mandatory as of POSIX 2008).  For
 example, it's not currently in OS X and apparently was first added to
 FreeBSD in 8.0.  So it would have to be checked by configure and
 documented as platform-dependent.

We already have os.openat:
http://docs.python.org/dev/library/os.html#os.openat

This request is to make it easier to use with the high-level IO classes.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12797] io.FileIO and io.open should support openat

[Ned Deily]

Ned Deily n...@acm.org added the comment:

 We already have os.openat
Ah, right.  The comment still applies, though, to future documentation of the 
proposed feature.  +1 on it.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12797
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com