[ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881773#comment-15881773 ]
Apache Spark commented on SPARK-19720: -------------------------------------- User 'markgrover' has created a pull request for this issue: https://github.com/apache/spark/pull/17047 > Redact sensitive information from SparkSubmit console output > ------------------------------------------------------------ > > Key: SPARK-19720 > URL: https://issues.apache.org/jira/browse/SPARK-19720 > Project: Spark > Issue Type: Bug > Components: Spark Submit > Affects Versions: 2.2.0 > Reporter: Mark Grover > > SPARK-18535 took care of redacting sensitive information from Spark event > logs and UI. However, it intentionally didn't bother redacting the same > sensitive information from SparkSubmit's console output because it was on the > client's machine, which already had the sensitive information on disk (in > spark-defaults.conf) or on terminal (spark-submit command line). > However, it seems now that it's better to redact information from > SparkSubmit's console output as well because orchestration software like > Oozie usually expose SparkSubmit's console output via a UI. To make matters > worse, Oozie, in particular, always sets the {{--verbose}} flag on > SparkSubmit invocation, making the sensitive information readily available in > its UI (see > [code|https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L248] > here). > This is a JIRA for tracking redaction of sensitive information from > SparkSubmit's console output. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org