[
https://issues.apache.org/jira/browse/SPARK-13670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179491#comment-15179491
]
Mark Grover commented on SPARK-13670:
-------------------------------------
cc [~vanzin] as fyi.
I ran into this when I was running {{dev/mima}}. It should have failed because
I didn't have the tools jar and launcher.main() did throw an exception but the
subshell ate it all up and the mima testing continued, eventually giving me a
ton of errors since the exclusions list wasn't correctly populated.
Anyways, there are a few ways I can think of fixing it, open to others as well:
1. We could just fix the symptom and have the dev/mima script, explicitly check
for tools directory and tools jar in bash and break, if it's not present.
However, a) that's just fixing the symptom, not the root cause, b) we already
have those checks in launcher code.
2. We could have the command write the output to a file instead of reading it
directly from a subshell. Then, that makes error handling easier.
3. We can have the subshell kill the parent process on error, essentially
running the subshell like so:
{code}
("$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@" || kill
$$)
{code}
The side effect is that any other subshells spawned by the parent, i.e.
spark-class will be terminated as well. Based on a quick look, I didn't see any
other subshells though.
Thoughts? Preferences?
> spark-class doesn't bubble up error from launcher command
> ---------------------------------------------------------
>
> Key: SPARK-13670
> URL: https://issues.apache.org/jira/browse/SPARK-13670
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.0.0
> Reporter: Mark Grover
> Priority: Minor
>
> There's a particular snippet in spark-class
> [here|https://github.com/apache/spark/blob/master/bin/spark-class#L86] that
> runs the spark-launcher code in a subshell.
> {code}
> # The launcher library will print arguments separated by a NULL character, to
> allow arguments with
> # characters that would be otherwise interpreted by the shell. Read that in a
> while loop, populating
> # an array that will be used to exec the final command.
> CMD=()
> while IFS= read -d '' -r ARG; do
> CMD+=("$ARG")
> done < <("$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main
> "$@")
> {code}
> The problem is that the if the launcher Main fails, this code still still
> returns success and continues, even though the top level script is marked
> {{set -e}}. This is because the launcher.Main is run within a subshell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
