[issue38167] O_DIRECT read fails with 4K mmap buffer

Josh Rosenberg added the comment: Yeah, not a bug. The I/O subsystem was substantially rewritten between Python 2 and Python 3, so you sometimes need to be more explicit about things like buffering, but as you note, once the buffering is correct, the code works; there's nothing to fix.

[issue38167] O_DIRECT read fails with 4K mmap buffer

Paul added the comment: > Problem is you follow it with: > > fo = os.fdopen(fd, 'rb+') > which introduces a Python level of buffering around the kernel unbuffered > file descriptor. You'd need to pass buffering=0 to make os.fdopen avoid > returning a buffered file object, making it: > fo =

[issue38167] O_DIRECT read fails with 4K mmap buffer

Josh Rosenberg added the comment: > I do not believe an unbuffered file uses O_DIRECT. This is why I use > os.open(fpath, os.O_DIRECT). Problem is you follow it with: fo = os.fdopen(fd, 'rb+') which introduces a Python level of buffering around the kernel unbuffered file descriptor.

[issue38167] O_DIRECT read fails with 4K mmap buffer

Paul added the comment: > I agree with Josh. If you want to use O_DIRECT, use an unbuffered file object > and be sure to issue reads of the right size. I do not believe an unbuffered file uses O_DIRECT. This is why I use os.open(fpath, os.O_DIRECT). > Also I'm curious: why are you using

[issue38167] O_DIRECT read fails with 4K mmap buffer

Antoine Pitrou added the comment: Also I'm curious: why are you using O_DIRECT, and furthermore, why are you using it to read into mmap'ed memory? -- ___ Python tracker ___

[issue38167] O_DIRECT read fails with 4K mmap buffer

Antoine Pitrou added the comment: I agree with Josh. If you want to use O_DIRECT, use an unbuffered file object and be sure to issue reads of the right size. -- nosy: +pitrou ___ Python tracker

[issue38167] O_DIRECT read fails with 4K mmap buffer

Paul added the comment: This is the platform that I'm working on as well as the failure. I have a review out for a fix. # uname -a Linux init129-13 3.10.0-957.el7.x86_64 x86_64 x86_64 x86_64 GNU/Linux # python3.7 directread.py Traceback (most recent call last): File "small.py", line 7, in

[issue38167] O_DIRECT read fails with 4K mmap buffer

Josh Rosenberg added the comment: Works just fine for me on 3.7.3 on Ubuntu, reading 4096 bytes. How is it failing for you? Is an exception raised? It does seem faintly dangerous to explicitly use O_DIRECT when you're wrapping it in a buffered reader that doesn't know it has to read in

[issue38167] O_DIRECT read fails with 4K mmap buffer

Change by Paul : -- pull_requests: +15742 pull_request: https://github.com/python/cpython/pull/16131 ___ Python tracker ___ ___

[issue38167] O_DIRECT read fails with 4K mmap buffer

Change by Paul : -- keywords: +patch pull_requests: +15741 stage: -> patch review pull_request: https://github.com/python/cpython/pull/16130 ___ Python tracker ___

[issue38167] O_DIRECT read fails with 4K mmap buffer

New submission from Paul : The following Python3 script fails. import mmap import os fd = os.open(path_to_file, os.O_DIRECT | os.O_RDWR) fo = os.fdopen(fd, 'rb+') m = mmap.mmap(-1, 4096) fo.readinto(m) But it worked for Python2. It also works for any other multiple of 4K. For example: m =