Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1178#issuecomment-46902458 Looks like if an exception is thrown because of casting the output to int (rather than reading the output of the process itself), then the process exit code returned is `None` because the process hasn't terminated yet. We should add a separate except for bad casting, which happens if there is other output other than the py4j port, and this is the most likely cause of an exception in the first place. I added this in the latest commit. I haven't invested a ton of time on figuring out how to use `communicate()` correctly. I'm not entirely sure if we need it, actually. Because our error messages actually surface on stdout, and our output messages on stderr, I think what we have now is sufficient as a fix. In the long term we might want to avoid using `stdout` to communicate the py4j port entirely, but maybe write it to a temporary file so Spark doesn't die on us if we echo a seemingly benign "hello" :)
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---