On 12 November 2015 at 08:07, Nathaniel Smith <[email protected]> wrote:
> On Wed, Nov 11, 2015 at 10:42 AM, Donald Stufft <[email protected]> wrote:
>> On November 11, 2015 at 1:38:38 PM, Nathaniel Smith ([email protected]) wrote:
>>> > Guaranteeing a clean stdout/stderr is hard: it means you have
>>> to be careful to correctly capture and process the output of every
>>> child you invoke (e.g. compilers), and deal correctly with the
>>> tricky aspects of pipes (deadlocks, sigpipe, ...). And even
>>> then you can get thwarted by accidentally importing the wrong
>>> library into your main process, and discovering that it writes
>>> directly to stdout/stderr on some error condition. And it may
>>> or may not respect your resetting of sys.stdout/sys.stderr
>>> at the python level. So to be really reliable the only thing to
>>> do is to create some pipes and some threads to read the pipes and
>>> do the dup2 dance (but not everyone will actually do this, they'll
>>> just accept corrupted output on errors) and ugh, all of this is
>>> a huge hassle that massively raises the bar on implementing simple
>>> build systems.
>>
>> How is this not true for a worker.py process as well? If the worker process
>> communicates via stdout then it has to make sure it captures the stdout and
>> redirects it before calling into the Python API and then undoes that
>> afterwords. It makes it harder to do incremental output actually because a
>> Python function can’t return in the middle of execution so we’d need to make
>> it some sort of akward generator protocol to make that happen too.
>
> Did you, uh, read the second half of my email? :-) My actual position
> is that we shouldn't even try to get structured incremental output
> from the build system, and should stick with the current approach of
> unstructured incremental output on stdout/stderr. But if we do insist
> on getting structured incremental output, then I described a system
> that's much easier for backends to implement, while leaving it up to
> the frontend to pick whether they want to bother doing complicated
> redirection tricks, and if so then which particular variety of
> complicated redirection trick they like best.
>
> In both approaches, yeah, any kind of incremental output is eventually
> come down to some Python code issuing some sort of function call that
> reports progress without returning, whether that's
> sys.stdout.write(json.dumps(...)) or
> progress_reporter.report_update(...). Between those two options, it's
> sys.stdout.write(json.dumps(...)) that looks more awkward to me.
I think there is some big disconnect in the conversation. AIUI Donald
and Marcus and I are saying that build systems should just use
print("Something happened")
to provide incremental output.
-Rob
--
Robert Collins <[email protected]>
Distinguished Technologist
HP Converged Cloud
_______________________________________________
Distutils-SIG maillist - [email protected]
https://mail.python.org/mailman/listinfo/distutils-sig
