[ https://issues.apache.org/jira/browse/SPARK-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15751424#comment-15751424 ]
David Hodeffi commented on SPARK-18829: --------------------------------------- I looked at the source code and you are right, my friend, it prints all of them. I know it sounds really weird, but I never thought that in order to print explain plan out I would have to do it this way. Now It's your decision, closing this issue or update explain() function to return the string back to the user. I am fine with both of them, those I would favor the one that return that String back to client code, so he can handle it by himself. > Printing to logger > ------------------ > > Key: SPARK-18829 > URL: https://issues.apache.org/jira/browse/SPARK-18829 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.6.2 > Environment: ALL > Reporter: David Hodeffi > Priority: Trivial > Labels: easyfix, patch > Original Estimate: 1h > Remaining Estimate: 1h > > I would like to print dataframe.show or df.explain(true) into log file. > right now the code print to standard output without a way to redirect it. > It also cannot be configured on log4j.properties. > My suggestion is to write to the logger and standard output. > i.e > class DataFrame {...... > override def explain(extended: Boolean): Unit = { > val explain = ExplainCommand(queryExecution.logical, extended = extended) > sqlContext.executePlan(explain).executedPlan.executeCollect().foreach { > // scalastyle:off println > r => { > println(r.getString(0)) > logger.debug(r.getString(0)) > } > } > // scalastyle:on println > } > } > def show(numRows: Int, truncate: Boolean): Unit = { > val str =showString(numRows, truncate) > println(str) > logger.debug(str) > } > } -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org