[ https://issues.apache.org/jira/browse/SPARK-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383709#comment-16383709 ]
Thomas Graves commented on SPARK-22479: --------------------------------------- [~aash] [~onursatici] this seems to have redacted user names as well as the passwords. We specifically added the User: field to the UI and now its being blocked, which is makes debugging harder. The user name does not seem like something that needs to be redacted by default. what is the reasoning behind that? Note that at least on yarn there are other ways to easily see the username on the UI (like the Resource Paths) so its definitely not a complete solution anyway. > SaveIntoDataSourceCommand logs jdbc credentials > ----------------------------------------------- > > Key: SPARK-22479 > URL: https://issues.apache.org/jira/browse/SPARK-22479 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0 > Reporter: Onur Satici > Assignee: Onur Satici > Priority: Major > Fix For: 2.2.1, 2.3.0 > > > JDBC credentials are not redacted in plans including a > 'SaveIntoDataSourceCommand'. > Steps to reproduce: > {code} > spark-shell --packages org.postgresql:postgresql:42.1.1 > {code} > {code} > import org.apache.spark.sql.execution.QueryExecution > import org.apache.spark.sql.util.QueryExecutionListener > val listener = new QueryExecutionListener { > override def onFailure(funcName: String, qe: QueryExecution, exception: > Exception): Unit = {} > override def onSuccess(funcName: String, qe: QueryExecution, duration: > Long): Unit = { > System.out.println(qe.toString()) > } > } > spark.listenerManager.register(listener) > spark.range(100).write.format("jdbc").option("url", > "jdbc:postgresql:sparkdb").option("password", "pass").option("driver", > "org.postgresql.Driver").option("dbtable", "test").save() > {code} > The above will yield the following plan: > {code} > == Parsed Logical Plan == > SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> > org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), > ErrorIfExists > +- Range (0, 100, step=1, splits=Some(8)) > == Analyzed Logical Plan == > SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> > org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), > ErrorIfExists > +- Range (0, 100, step=1, splits=Some(8)) > == Optimized Logical Plan == > SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> > org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), > ErrorIfExists > +- Range (0, 100, step=1, splits=Some(8)) > == Physical Plan == > ExecutedCommand > +- SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> > org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), > ErrorIfExists > +- Range (0, 100, step=1, splits=Some(8)) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org