[ https://issues.apache.org/jira/browse/SPARK-20435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-20435: ------------------------------------ Assignee: Apache Spark > More thorough redaction of sensitive information from logs/UI, more unit tests > ------------------------------------------------------------------------------ > > Key: SPARK-20435 > URL: https://issues.apache.org/jira/browse/SPARK-20435 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.2.0 > Reporter: Mark Grover > Assignee: Apache Spark > > SPARK-18535 and SPARK-19720 were works to redact sensitive information (e.g. > hadoop credential provider password, AWS access/secret keys) from event logs > + YARN logs + UI and from the console output, respectively. > While some unit tests were added along with these changes - they asserted > when a sensitive key was found, that redaction took place for that key. They > didn't assert globally that when running a full-fledged Spark app (whether or > YARN or locally), that sensitive information was not present in any of the > logs or UI. Such a test would also prevent regressions from happening in the > future if someone unknowingly adds extra logging that publishes out sensitive > information to disk or UI. > Consequently, it was found that in some Java configurations, sensitive > information was still being leaked in the event logs under the > {{SparkListenerEnvironmentUpdate}} event, like so: > {code} > "sun.java.command":"org.apache.spark.deploy.SparkSubmit ... --conf > spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=secret_password ... > {code} > "secret_password" should have been redacted. > Moreover, previously redaction logic was only checking if the key matched the > secret regex pattern, it'd redact it's value. That worked for most cases. > However, in the above case, the key (sun.java.command) doesn't tell much, so > the value needs to be searched. So the check needs to be expanded to match > against values as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org