[issue41594] Intermittent failures of loop.subprocess_exec() to capture output

2020-08-27 Thread Kyle Meyer 


Kyle Meyer  added the comment:

I should have thought to provide the output of when debug=True is
passed to asyncio.run().  Here it is, with the python on my Debian
system:

$ python3 -V
Python 3.7.3

$ python3 reproducer.py
Iteration:   1 of 100
Failed on iteration 1
logfile: /tmp/asyncio-debug-rqfsxyth.log

$ cat /tmp/asyncio-debug-rqfsxyth.log 
DEBUG: Starting iteration 1
DEBUG: Using selector: EpollSelector
DEBUG: execute program 'printf' stdout=
DEBUG: process 'printf' created: pid 20488
DEBUG: process 20488 exited with returncode 0
INFO: <_UnixSubprocessTransport pid=20488 running> exited with return code 0
DEBUG: Read pipe 8 connected: (<_UnixReadPipeTransport fd=8 polling>, 
>)
DEBUG: process_exited() called
INFO: execute program 'printf': <_UnixSubprocessTransport pid=20488 
returncode=0 stdout=<_UnixReadPipeTransport fd=8 polling>>
DEBUG: pipe_data_received(): fd=1, data=b'ok'
DEBUG: Close <_UnixSelectorEventLoop running=False closed=False debug=True>
DEBUG: Failed on iteration 1

And with a python built from a recent commit (8e19c8be8701):

$ python -V
Python 3.10.0a0

$ python reproducer.py
Iteration:   1 of 100
Failed on iteration 1
logfile: /tmp/asyncio-debug-9eyhuas4.log

$ cat /tmp/asyncio-debug-9eyhuas4.log 
DEBUG: Starting iteration 1
DEBUG: Using selector: EpollSelector
DEBUG: execute program 'printf' stdout=
DEBUG: process 'printf' created: pid 20524
DEBUG: process 20524 exited with returncode 0
INFO: <_UnixSubprocessTransport pid=20524 running> exited with return code 0
DEBUG: Read pipe 8 connected: (<_UnixReadPipeTransport fd=8 polling>, 
>)
DEBUG: process_exited() called
INFO: execute program 'printf': <_UnixSubprocessTransport pid=20524 
returncode=0 stdout=<_UnixReadPipeTransport fd=8 polling>>
DEBUG: pipe_data_received(): fd=1, data=b'ok'
DEBUG: Close <_UnixSelectorEventLoop running=False closed=False debug=True>
DEBUG: Failed on iteration 1

---

It looks like I can work around the issue (i.e. I don't observe any
lost output) by adding this line to the attached script:

diff --git a/reproducer.py b/reproducer.py
index 5e04c36..a462898 100644
--- a/reproducer.py
+++ b/reproducer.py
@@ -25,6 +25,7 @@ async def get_stdout():
 transport, protocol = await loop.subprocess_exec(
 lambda: Protocol(exit_future),
 "printf", "ok", stdin=None, stderr=None)
+await asyncio.ensure_future(transport._wait())
 await exit_future
 transport.close()
 return bytes(protocol.output)

--
versions: +Python 3.10

___
Python tracker 
<https://bugs.python.org/issue41594>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41594] Intermittent failures of loop.subprocess_exec() to capture output

2020-08-19 Thread Kyle Meyer 


New submission from Kyle Meyer :

I've been debugging an intermittent test failure in code that calls
`loop.subprocess_exec` with an `asyncio.SubprocessProtocol` subclass.
Here's a minimal example that I hope captures the issue.  It's based
closely off of the `DateProtocol` [example][1] in the asyncio protocol
documentation.

import asyncio

class Protocol(asyncio.SubprocessProtocol):
def __init__(self, exit_future):
self.exit_future = exit_future
self.output = bytearray()

def pipe_data_received(self, fd, data):
self.output.extend(data)

def process_exited(self):
self.exit_future.set_result(True)

async def get_stdout():
loop = asyncio.get_running_loop()
exit_future = asyncio.Future(loop=loop)
transport, protocol = await loop.subprocess_exec(
lambda: Protocol(exit_future),
"printf", "ok", stdin=None, stderr=None)
await exit_future
transport.close()
return bytes(protocol.output)

The attached script adds some debugging statements to what's above and
then repeats the specified number of calls to
`asyncio.run(get_stdout())`, aborting if `asyncio.run(get_stdout())`
returns an empty byte string rather then the expected `b'ok'`.  With
Python 3.7.3 on a Debian system, I see occasional aborts.  For
example, after two runs of the script with no unexpectedly empty
output, I saw

$ python3 reproducer.py 500
Iteration:  32 of 500
Failed on iteration 32
logfile: /tmp/asyncio-debug-bqehu8f3.log

$ tail /tmp/asyncio-debug-bqehu8f3.log
DEBUG: process_exited() called
DEBUG: Starting iteration 31
DEBUG: Using selector: EpollSelector
DEBUG: pipe_data_received(): fd=1, data=b'ok'
DEBUG: process_exited() called
DEBUG: Starting iteration 32
DEBUG: Using selector: EpollSelector
DEBUG: process_exited() called
DEBUG: pipe_data_received(): fd=1, data=b'ok'
DEBUG: Failed on iteration 32

Based on the debug statements, it looks like `pipe_data_received` does
get called with the output for the run that comes up empty, but only
after `process_exited` is called.

On my first try with a python built from a recent commit (0be7c216e1),
I triggered the failure:

$ python -V
Python 3.10.0a0

$ python reproducer.py 500
Iteration:   8 of 500
Failed on iteration 8
logfile: /tmp/asyncio-debug-m5fba4ga.log

$ tail /tmp/asyncio-debug-m5fba4ga.log
DEBUG: process_exited() called
DEBUG: Starting iteration 7
DEBUG: Using selector: EpollSelector
DEBUG: pipe_data_received(): fd=1, data=b'ok'
DEBUG: process_exited() called
DEBUG: Starting iteration 8
DEBUG: Using selector: EpollSelector
DEBUG: process_exited() called
DEBUG: pipe_data_received(): fd=1, data=b'ok'
DEBUG: Failed on iteration 8

As I'm following the example from the documentation closely, I hope
I'm not committing a silly error that leads to an expected race here.


[1]: 
https://docs.python.org/3/library/asyncio-protocol.html#loop-subprocess-exec-and-subprocessprotocol

--
components: asyncio
files: reproducer.py
messages: 375680
nosy: asvetlov, kyleam, yselivanov
priority: normal
severity: normal
status: open
title: Intermittent failures of loop.subprocess_exec() to capture output
type: behavior
versions: Python 3.7
Added file: https://bugs.python.org/file49406/reproducer.py

___
Python tracker 
<https://bugs.python.org/issue41594>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26967] argparse: allow_abbrev=False stops -vv from working

2020-02-03 Thread Kyle Meyer 


Change by Kyle Meyer :


--
pull_requests: +17710
pull_request: https://github.com/python/cpython/pull/18337

___
Python tracker 
<https://bugs.python.org/issue26967>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39546] argparse: allow_abbrev=False is ignored for alternative prefix characters

2020-02-03 Thread Kyle Meyer 


Change by Kyle Meyer :


--
keywords: +patch
pull_requests: +17709
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/18337

___
Python tracker 
<https://bugs.python.org/issue39546>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39546] argparse: allow_abbrev=False is ignored for alternative prefix characters

2020-02-03 Thread Kyle Meyer 


New submission from Kyle Meyer :

As of Python v3.8.0 (specifically commit b1e4d1b603), specifying
`allow_abbrev=False` does not disable abbreviation for prefix
characters other than '-'.

--8<---cut here---start->8---
import argparse

parser = argparse.ArgumentParser(prefix_chars='+', allow_abbrev=False)
parser.add_argument('++long')
print(parser.parse_args(['++lo=val']))
--8<---cut here---end--->8---

Observed output (with b1e4d1b603 and current master):

Namespace(long='val')

Expected (and observed with b1e4d1b603^ and 3.7.3):

usage: scratch.py [+h] [++long LONG]
scratch.py: error: unrecognized arguments: ++lo=val


I will follow up with a PR to propose a fix.

--
components: Library (Lib)
messages: 361326
nosy: kyleam
priority: normal
severity: normal
status: open
title: argparse: allow_abbrev=False is ignored for alternative prefix characters
type: behavior
versions: Python 3.8, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue39546>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38449] regression - mimetypes guess_type is confused by ; in the filename

2019-10-11 Thread Kyle Meyer 


Kyle Meyer  added the comment:

I've performed a bisect the issue with the following script:

#!/bin/sh
make -j3 || exit 125
./python <<\EOF || exit 1
import sys
import mimetypes
res = mimetypes.MimeTypes(strict=False).guess_type(";1.tar.gz")
if res[0] is None:
sys.exit(1)
EOF

That points to 87bd2071c7 (bpo-22347: Update mimetypes.guess_type to allow 
proper parsing of URLs (GH-15522), 2019-09-05).  That commit was included in 
3.7.5rc1 when it was cherry picked by 8873bff287.

--
nosy: +kyleam

___
Python tracker 
<https://bugs.python.org/issue38449>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com