Re: Opinion on spark-class script simplification and posix compliance

[Marcelo Vanzin]

On Tue, Jul 28, 2015 at 12:13 PM, Félix-Antoine Fortin <
felix-antoine.for...@calculquebec.ca> wrote:

> The while loop cannot be executed with sh, while the single line can
> be. Since on my system, sh is simply a link on bash, with some options
> activated, I guess this simply means that the while loop syntax is not
> posix compliant. Which spawns two questions:
>
> 1. Would it be useful to make sure Spark scripts are POSIX compliant?
>

I guess if people are trying to run Spark on systems where bash is not
super common it could help. Not sure how common that use case is.
Otherwise, it seems like it would be more noise than actually useful...

Also note that bash, when executed as "/bin/sh", still allows syntax that
other shells do not necessarily understand. A better test would be to try
to run the script with dash or something else that is not bash.


-- 
Marcelo

Opinion on spark-class script simplification and posix compliance

[Félix-Antoine Fortin]

Hi,

Out of curiosity, I have tried to replace the dependence on bash by sh
in the different scripts to launch Spark daemons and jobs. So far,
most scripts work with sh, except "bin/spark-class". The culprit is
the while loop that compose the final command by parsing the output of
launcher library.

After a bit of fiddling, I found that the following loop:
CMD=()
while IFS= read -d '' -r ARG; do
CMD+=("$ARG")
done < <("$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@")

could be replaced by a single line:
CMD=$("$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main
"$@" | xargs -0)

The while loop cannot be executed with sh, while the single line can
be. Since on my system, sh is simply a link on bash, with some options
activated, I guess this simply means that the while loop syntax is not
posix compliant. Which spawns two questions:

1. Would it be useful to make sure Spark scripts are POSIX compliant?
2. Is the simplification of spark-class enough to consider making a
pull-request?

Thanks,
Felix-Antoine Fortin

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org