Mark Grover created SPARK-19720:
-----------------------------------
Summary: Redact sensitive information from SparkSubmit console
output
Key: SPARK-19720
URL: https://issues.apache.org/jira/browse/SPARK-19720
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 2.2.0
Reporter: Mark Grover
SPARK-18535 took care of redacting sensitive information from Spark event logs
and UI. However, it intentionally didn't bother redacting the same sensitive
information from SparkSubmit's console output because it was on the client's
machine, which already had the sensitive information on disk (in
spark-defaults.conf) or on terminal (spark-submit command line).
However, it seems now that it's better to redact information from SparkSubmit's
console output as well because orchestration software like Oozie usually expose
SparkSubmit's console output via a UI. To make matters worse, Oozie, in
particular, always sets the {{--verbose}} flag on SparkSubmit invocation,
making the sensitive information readily available in its UI (see
[code|https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L248]
here).
This is a JIRA for tracking redaction of sensitive information from
SparkSubmit's console output.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
