[jira] [Reopened] (SPARK-9636) Treat $SPARK_HOME as write-only
[
https://issues.apache.org/jira/browse/SPARK-9636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Angerer reopened SPARK-9636:
This is not fixed and there was no reason given why it was closed, so I’ll
reopen it.
> Treat $SPARK_HOME as write-only
> ---
>
> Key: SPARK-9636
> URL: https://issues.apache.org/jira/browse/SPARK-9636
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
>Affects Versions: 1.4.1
> Environment: Linux
>Reporter: Philipp Angerer
>Priority: Minor
> Labels: bulk-closed
>
> when starting spark scripts as user and it is installed in a directory the
> user has no write permissions on, many things work fine, except for the logs
> (e.g. for {{start-master.sh}})
> logs are per default written to {{$SPARK_LOG_DIR}} or (if unset) to
> {{$SPARK_HOME/logs}}.
> if installed in this way, it should, instead of throwing an error, write logs
> to {{/var/log/spark/}}. that’s easy to fix by simply testing a few log dirs
> in sequence for writability before trying to use one. i suggest using
> {{$SPARK_LOG_DIR}} (if set) → {{/var/log/spark/}} → {{~/.cache/spark-logs/}}
> → {{$SPARK_HOME/logs/}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9636) Treat $SPARK_HOME as write-only
[
https://issues.apache.org/jira/browse/SPARK-9636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681403#comment-14681403
]
Philipp Angerer commented on SPARK-9636:
OK, great :)
I see why you think my proposal might be to complex, yet I still think that
“log file relative to binary” is much more surprising in an environment where
log files have certain dedicated places.
{{/var/log/}} is something i really expect a system daemon to use for logs.
{{~/.cache/logs}} is merely the best compromise in absence of a dedicated user
log directoy. (e.g. {{$XDG_USER_DATA_DIR}} and {{$XDG_USER_CONFIG_DIR}} are
clear, but there’s no {{$XDG_USER_STATE_DIR}})
i think all this is a consequence of spark not being a good linux citizen. it
has a {{$SPARK_HOME}}, and relies on it, while there should be a way to run it
split up to sensible directories: {{/usr/share/spark/}} for data
{{/usr/lib/spark/}} for shared libraries, {{/usr/lib/pythonx.x/site-packages/}}
for pyspark, {{/usr/bin/}} for binaries and scripts, {{/etc/spark/}} for
configs, and {{/var/log/spark}} for logfiles.
Treat $SPARK_HOME as write-only
---
Key: SPARK-9636
URL: https://issues.apache.org/jira/browse/SPARK-9636
Project: Spark
Issue Type: Improvement
Components: Input/Output
Affects Versions: 1.4.1
Environment: Linux
Reporter: Philipp Angerer
Priority: Minor
Labels: easyfix
when starting spark scripts as user and it is installed in a directory the
user has no write permissions on, many things work fine, except for the logs
(e.g. for {{start-master.sh}})
logs are per default written to {{$SPARK_LOG_DIR}} or (if unset) to
{{$SPARK_HOME/logs}}.
if installed in this way, it should, instead of throwing an error, write logs
to {{/var/log/spark/}}. that’s easy to fix by simply testing a few log dirs
in sequence for writability before trying to use one. i suggest using
{{$SPARK_LOG_DIR}} (if set) → {{/var/log/spark/}} → {{~/.cache/spark-logs/}}
→ {{$SPARK_HOME/logs/}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9636) Treat $SPARK_HOME as write-only
[
https://issues.apache.org/jira/browse/SPARK-9636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680002#comment-14680002
]
Philipp Angerer commented on SPARK-9636:
quietly
Huh?
Again: The script *does* report the location. I’d perhaps accept your “quietly”
if it would barf heaps of debug messages which drown that one, but the log file
is literally the only line echoed by that script, so this is perhaps the
reference for doing something as explicitly and loudly as possible.
look, this is the full interaction with that script, how it looks now:
{code}
$ /usr/share/apache-spark/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to
/home/user/.cache/spark-logs/spark-user-org.apache.spark.deploy.master.Master-1-hostname.out
$
{code}
Treat $SPARK_HOME as write-only
---
Key: SPARK-9636
URL: https://issues.apache.org/jira/browse/SPARK-9636
Project: Spark
Issue Type: Improvement
Components: Input/Output
Affects Versions: 1.4.1
Environment: Linux
Reporter: Philipp Angerer
Priority: Minor
Labels: easyfix
when starting spark scripts as user and it is installed in a directory the
user has no write permissions on, many things work fine, except for the logs
(e.g. for {{start-master.sh}})
logs are per default written to {{$SPARK_LOG_DIR}} or (if unset) to
{{$SPARK_HOME/logs}}.
if installed in this way, it should, instead of throwing an error, write logs
to {{/var/log/spark/}}. that’s easy to fix by simply testing a few log dirs
in sequence for writability before trying to use one. i suggest using
{{$SPARK_LOG_DIR}} (if set) → {{/var/log/spark/}} → {{~/.cache/spark-logs/}}
→ {{$SPARK_HOME/logs/}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-9636) Treat $SPARK_HOME as write-only
Philipp Angerer created SPARK-9636:
--
Summary: Treat $SPARK_HOME as write-only
Key: SPARK-9636
URL: https://issues.apache.org/jira/browse/SPARK-9636
Project: Spark
Issue Type: Bug
Components: Input/Output
Affects Versions: 1.4.1
Environment: Linux
Reporter: Philipp Angerer
when starting spark scripts as user and it is installed in a directory the user
has no write permissions on, many things work fine, except for the logs (e.g.
for {{start-master.sh}})
logs are per default written to {{$SPARK_LOG_DIR}} or (if unset) to
{{$SPARK_HOME/logs}}.
if installed in this way, it should, instead of throwing an error, write logs
to {{/var/log/spark/}}. that’s easy to fix by simply testing a few log dirs in
sequence for writability before trying to use one. i suggest using
{{$SPARK_LOG_DIR}} (if set) → {{/var/log/spark/}} → {{~/.cache/spark-logs/}} →
{{$SPARK_HOME/logs/}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9636) Treat $SPARK_HOME as write-only
[
https://issues.apache.org/jira/browse/SPARK-9636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14655310#comment-14655310
]
Philipp Angerer commented on SPARK-9636:
everything is more obvious than picing a location relative to the binary ;)
and the location is reported anyway since the {{start-master.sh}} script
outputs {{starting org.apache.spark.deploy.master.Master, logging to
/home/user/.cache/spark-logs/spark-user-org.apache.spark.deploy.master.Master-1-hostname.out}}
about write permissions, mind that i suggest testing them sequentially until
one is found that can be written to. that’s IMHO a more sensible default than
failing, and having to {{grep -i 'log' $SPARK_HOME/sbin/*.sh}} to find that an
environment variable exists, and then retrying with that variable set.
Treat $SPARK_HOME as write-only
---
Key: SPARK-9636
URL: https://issues.apache.org/jira/browse/SPARK-9636
Project: Spark
Issue Type: Improvement
Components: Input/Output
Affects Versions: 1.4.1
Environment: Linux
Reporter: Philipp Angerer
Priority: Minor
Labels: easyfix
when starting spark scripts as user and it is installed in a directory the
user has no write permissions on, many things work fine, except for the logs
(e.g. for {{start-master.sh}})
logs are per default written to {{$SPARK_LOG_DIR}} or (if unset) to
{{$SPARK_HOME/logs}}.
if installed in this way, it should, instead of throwing an error, write logs
to {{/var/log/spark/}}. that’s easy to fix by simply testing a few log dirs
in sequence for writability before trying to use one. i suggest using
{{$SPARK_LOG_DIR}} (if set) → {{/var/log/spark/}} → {{~/.cache/spark-logs/}}
→ {{$SPARK_HOME/logs/}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-9636) Treat $SPARK_HOME as write-only
[
https://issues.apache.org/jira/browse/SPARK-9636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14655310#comment-14655310
]
Philipp Angerer edited comment on SPARK-9636 at 8/5/15 1:04 PM:
everything is more obvious than picking a location relative to the binary ;)
and the location is reported anyway since the {{start-master.sh}} script
outputs {{starting org.apache.spark.deploy.master.Master, logging to
/home/user/.cache/spark-logs/spark-user-org.apache.spark.deploy.master.Master-1-hostname.out}}
about write permissions, mind that i suggest testing them sequentially until
one is found that can be written to. that’s IMHO a more sensible default than
failing, and having to {{grep -i 'log' $SPARK_HOME/sbin/*.sh}} to find that an
environment variable exists, and then retrying with that variable set.
was (Author: angerer):
everything is more obvious than picing a location relative to the binary ;)
and the location is reported anyway since the {{start-master.sh}} script
outputs {{starting org.apache.spark.deploy.master.Master, logging to
/home/user/.cache/spark-logs/spark-user-org.apache.spark.deploy.master.Master-1-hostname.out}}
about write permissions, mind that i suggest testing them sequentially until
one is found that can be written to. that’s IMHO a more sensible default than
failing, and having to {{grep -i 'log' $SPARK_HOME/sbin/*.sh}} to find that an
environment variable exists, and then retrying with that variable set.
Treat $SPARK_HOME as write-only
---
Key: SPARK-9636
URL: https://issues.apache.org/jira/browse/SPARK-9636
Project: Spark
Issue Type: Improvement
Components: Input/Output
Affects Versions: 1.4.1
Environment: Linux
Reporter: Philipp Angerer
Priority: Minor
Labels: easyfix
when starting spark scripts as user and it is installed in a directory the
user has no write permissions on, many things work fine, except for the logs
(e.g. for {{start-master.sh}})
logs are per default written to {{$SPARK_LOG_DIR}} or (if unset) to
{{$SPARK_HOME/logs}}.
if installed in this way, it should, instead of throwing an error, write logs
to {{/var/log/spark/}}. that’s easy to fix by simply testing a few log dirs
in sequence for writability before trying to use one. i suggest using
{{$SPARK_LOG_DIR}} (if set) → {{/var/log/spark/}} → {{~/.cache/spark-logs/}}
→ {{$SPARK_HOME/logs/}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-8760) allow moving and symlinking binaries
Philipp Angerer created SPARK-8760:
--
Summary: allow moving and symlinking binaries
Key: SPARK-8760
URL: https://issues.apache.org/jira/browse/SPARK-8760
Project: Spark
Issue Type: Improvement
Components: PySpark, Spark Shell, Spark Submit, SparkR
Affects Versions: 1.4.0
Reporter: Philipp Angerer
Priority: Minor
you use the following line to determine {{$SPARK_HOME}} in all binaries
{code:none}
export SPARK_HOME=$(cd `dirname $0`/..; pwd)
{code}
however users should be able to override this. also symlinks should be followed:
{code:none}
if [[ -z $SPARK_HOME ]]; then
export SPARK_HOME=$(dirname $(readlink -f $0))
fi
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
