[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Min RK]

Min RK  added the comment:

It appears that connect_read_pipe also doesn't accept pipes returned by 
`os.pipe`. If that's the case, what _does_ ProactorEventLoop.connect_read_pipe 
accept? I haven't been able to find any examples of `connect_read_pipe` that 
work on Windows, and every connect_read_pipe call in the cpython test suite 
appears to be skipped on win32. Should it still be raising NotImplementedError 
on ProactorEventLoop?

I think the error handling could be better (I only get logged errors, nothing I 
can catch/handle). It seems like `connect_read_pipe` itself should raise when 
it fails to register the pipe with IOCP. If that's not feasible, 
connection_lost/transport.close should probably be triggered, but it isn't with 
Python 3.9, at least.

Example that works on posix, but seems to fail with non-catchable errors with 
ProactorEventLoop:

```
import asyncio
import os
import sys

class PipeProtocol(asyncio.Protocol):
def __init__(self):
self.finished = asyncio.Future()

def connection_made(self, transport):
print("connection made", file=sys.stderr)
self.transport = transport

def connection_lost(self, exc):
print("connection lost", exc, file=sys.stderr)
self.finished.set_result(None)

def data_received(self, data):
print("data received", data, file=sys.stderr)
self.handler(data)

def eof_received(self):
print("eof received", file=sys.stderr)
self.finished.set_result(None)

async def test():
r, w = os.pipe()
rf = os.fdopen(r, 'r')
x, p = await asyncio.get_running_loop().connect_read_pipe(PipeProtocol, rf)
await asyncio.sleep(1)
print("writing")
os.write(w, b'asdf')
await asyncio.sleep(2)
print("closing")
os.close(w)
await asyncio.wait([p.finished], timeout=3)
x.close()

if __name__ == "__main__":
asyncio.run(test())
```

--
nosy: +minrk
versions: +Python 3.10, Python 3.11, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy:  -gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Martijn Pieters]

Martijn Pieters  added the comment:

I'm trying to figure out why Windows won't let us do this. I think the reason 
is that sys.std(in|out) filehandles are not opened as pipes, and do not have 
the required OVERLAPPED flag set (see the CreateIoCompletionPort documentation 
at 
https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport; 
it's that function that is used to handle pipes (via IocpProactor.recv -> 
IocpProactor._register_with_iocp -> overlapped.CreateIoCompletionPort).

The solution then would be to create a pipe for a stdio filehandle with the 
flag set.

And that's where my Windows-fu ends, and where I lack the VM and motivation to 
go try that out.

--
nosy: +mjpieters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Gabriel Mesquita Cangussu]

Gabriel Mesquita Cangussu added the comment:

Terry,

About your question, "Isn't there some other way to asynchronously read/file 
files, as opposed to sockets and pipes, on Windows?", that is exactly the 
problem I was trying to overcome, specifically for reading the stdin.

On my research I found both this stackoverflow questions with no valid answers:

http://stackoverflow.com/questions/31510190/aysncio-cannot-read-stdin-on-windows

http://stackoverflow.com/questions/31511563/file-to-socket-adapter-in-python

And I understood that 'pipe' wouldn't mean a 'file', but why can't I use 
'sys.stdin' with 'ProactorEventLoop.connect_read_pipe()', can't 'sys.stdin' be 
a pipe on Windows like it can on Posix?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Steve Dower]

Steve Dower added the comment:

Pipes and file handles are equivalent in Windows, but socket handles are their 
own namespace and have their own functions. Some Python code will switch 
between them automatically to make socket functions work with file descriptors, 
but generally I'd expect stream pipes (as opposed to, IIRC, datagram pipes) to 
be compatible with files but not sockets.

Not entirely sure how that plays into this issue, but it's a bit more 
background.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Terry J. Reedy]

Terry J. Reedy added the comment:

The surprise to me, being on Windows, is that the pipe connection methods 
sometimes work with non-pipes.  The limitations of Windows event loops are 
given in https://docs.python.org/3/library/asyncio-eventloops.html#windows.  
The pipe connection functions are discussed in 
https://docs.python.org/3/library/asyncio-eventloop.html#connect-pipes.  Both 
say that the methods do not work with Windows' SelectorEventLoop.

My understanding is that this is because Windows' select() call does not work 
with pipes -- meaning honest-to-goodness OS pipes.  So I understood "*pipe* is 
file-like object." more as a un-surprising statement of fact than as a 
permissive "'pipe' can be any file-like object and not necessarily a pipe".

If 'pipe' were intended to mean 'file-like', then why use 'pipe'?  But I can 
see how a current unix user would understand that sentence the other way.  
Perhaps the sentence should read "For SelectorEventLoops (not on Windows), 
*pipe* can also be any file-like object with the appropriate methods." -- 
assuming that this is true on all non-Windows systems.

Isn't there some other way to asynchronously read/file files, as opposed to 
sockets and pipes, on Windows?

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Steve Dower]

Steve Dower added the comment:

> File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 606, in 
> _register_with_iocp
>   _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)

I don't think there's any case where this is going to work on an actual file - 
CreateIoCompletionPort needs a handle and not a file descriptor.

Presumably this line of code normally gets used with something else that 
defines fileno()?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Gabriel Mesquita Cangussu]

New submission from Gabriel Mesquita Cangussu:

The documentation of asyncio specifies that the methods connect_read_pipe and 
connect_write_pipe are available on Windows with the ProactorEventLoop. The 
documentation then says that those methods accept file-like objects on the pipe 
parameter. However, the methods doesn't seem to work with stdio or any disk 
file under Windows. The following example catches this problem:

import asyncio
import sys

class MyProtocol(asyncio.Protocol):
def connection_made(self, transport):
print('connection established')

def data_received(self, data):
print('received: {!r}'.format(data.decode()))

def connection_lost(self, exc):
print('lost connection')

if sys.platform.startswith('win32'):
loop = asyncio.ProactorEventLoop()
else:
loop = asyncio.SelectorEventLoop()
coro = loop.connect_read_pipe(MyProtocol, sys.stdin)
loop.run_until_complete(coro)
loop.run_forever()
loop.close()


This code when executed on Ubuntu have the desired behavior, but under Windows 
10 it gives OSError: [WinError 6] The handle is invalid. The complete output is 
this:

c:\Users\Gabriel\Documents\Python Scripts>python async_pipe.py
connection established
Fatal read error on pipe transport
protocol: <__main__.MyProtocol object at 0x01970EB2FAC8>
transport: <_ProactorReadPipeTransport fd=0>
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\asyncio\proactor_events.py", line 195, in 
_loop_reading
self._read_fut = self._loop._proactor.recv(self._sock, 4096)
  File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 425, in 
recv
self._register_with_iocp(conn)
  File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 606, in 
_register_with_iocp
_overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 6] Identificador inválido
lost connection


I think that the documentation should state that there is no support for disk 
files and stdio with the methods in question and also state what exactly they 
support (an example would be nice). And, of course, better support for watching 
file descriptors on Windows on future Python releases would be nice.

Thank you,
Gabriel

--
components: Windows, asyncio
messages: 264043
nosy: Gabriel Mesquita Cangussu, gvanrossum, haypo, paul.moore, steve.dower, 
tim.golden, yselivanov, zach.ware
priority: normal
severity: normal
status: open
title: ProactorEventLoop doesn't support stdin/stdout nor files with 
connect_read_pipe/connect_write_pipe
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com