[ https://issues.apache.org/jira/browse/SPARK-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15744464#comment-15744464 ]
David Hodeffi commented on SPARK-18829: --------------------------------------- The issue is it's hard to debug application on yarn. stdout file is **not** the same file as the logger file. option1 : update the function to get arg of type OutputStream and then write string to it but its overkill. option2: simply, go to log4j.properties and configure the logger to print that out. right now option1 / option2 is not possible, that's why I suggested to print it to logger too. changing standard output just because you want to see just this specific log is not is not a good idea, because it will spam the log file with everything that is printed to standard output and you cannot really control that. > Printing to logger > ------------------ > > Key: SPARK-18829 > URL: https://issues.apache.org/jira/browse/SPARK-18829 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.6.2 > Environment: ALL > Reporter: David Hodeffi > Priority: Trivial > Labels: easyfix, patch > Original Estimate: 1h > Remaining Estimate: 1h > > I would like to print dataframe.show or df.explain(true) into log file. > right now the code print to standard output without a way to redirect it. > It also cannot be configured on log4j.properties. > My suggestion is to write to the logger and standard output. > i.e > class DataFrame {...... > override def explain(extended: Boolean): Unit = { > val explain = ExplainCommand(queryExecution.logical, extended = extended) > sqlContext.executePlan(explain).executedPlan.executeCollect().foreach { > // scalastyle:off println > r => { > println(r.getString(0)) > logger.debug(r.getString(0)) > } > } > // scalastyle:on println > } > } > def show(numRows: Int, truncate: Boolean): Unit = { > val str =showString(numRows, truncate) > println(str) > logger.debug(str) > } > } -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org