[ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101096#comment-16101096 ]
Diogo Munaro Vieira commented on SPARK-19720: --------------------------------------------- Do you have plans to apply this fix in a Hadoop 2.5 compatible version of Spark? Version 2.2 removed Hadoop 2.5 support > Redact sensitive information from SparkSubmit console output > ------------------------------------------------------------ > > Key: SPARK-19720 > URL: https://issues.apache.org/jira/browse/SPARK-19720 > Project: Spark > Issue Type: Bug > Components: Spark Submit > Affects Versions: 2.2.0 > Reporter: Mark Grover > Assignee: Mark Grover > Fix For: 2.2.0 > > > SPARK-18535 took care of redacting sensitive information from Spark event > logs and UI. However, it intentionally didn't bother redacting the same > sensitive information from SparkSubmit's console output because it was on the > client's machine, which already had the sensitive information on disk (in > spark-defaults.conf) or on terminal (spark-submit command line). > However, it seems now that it's better to redact information from > SparkSubmit's console output as well because orchestration software like > Oozie usually expose SparkSubmit's console output via a UI. To make matters > worse, Oozie, in particular, always sets the {{--verbose}} flag on > SparkSubmit invocation, making the sensitive information readily available in > its UI (see > [code|https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L248] > here). > This is a JIRA for tracking redaction of sensitive information from > SparkSubmit's console output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org