Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1178#issuecomment-46902458
Looks like if an exception is thrown because of casting the output to int
(rather than reading the output of the process itself), then the process exit
code returned is `None` because the process hasn't terminated yet. We should
add a separate except for bad casting, which happens if there is other output
other than the py4j port, and this is the most likely cause of an exception in
the first place. I added this in the latest commit.
I haven't invested a ton of time on figuring out how to use `communicate()`
correctly. I'm not entirely sure if we need it, actually. Because our error
messages actually surface on stdout, and our output messages on stderr, I think
what we have now is sufficient as a fix. In the long term we might want to
avoid using `stdout` to communicate the py4j port entirely, but maybe write it
to a temporary file so Spark doesn't die on us if we echo a seemingly benign
"hello" :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
