[
https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768739#comment-15768739
]
Sean Wong commented on SPARK-18933:
-----------------------------------
I found that Loginfo codes in the transformation() or actions() are only shown
in the terminal screen. These log codes are executed during the DAG
construction and before the job submission. But the Loginfo codes in the
compute() function of each RDD are only shown in the stderr file. These log
codes are executed in the process of real job execution. Now I know which part
of Loginfo go to terminal screen and which goes to stderr file. But I still do
not know why these will happen since they all use the Loginfo code and extends
Logging during the class definition.
> Different log output between Terminal screen and stderr file
> ------------------------------------------------------------
>
> Key: SPARK-18933
> URL: https://issues.apache.org/jira/browse/SPARK-18933
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Documentation, Web UI
> Affects Versions: 1.6.3
> Environment: Yarn mode and standalone mode
> Reporter: Sean Wong
> Original Estimate: 612h
> Remaining Estimate: 612h
>
> First of all, I use the default log4j.properties in the Spark conf/
> But I found that the log output(e.g., INFO) is different between Terminal
> screen and stderr File. Some INFO logs exist in both of them. Some INFO logs
> exist in either of them. Why this happens? Is it supposed that the output
> logs are same between the terminal screen and stderr file?
> Then I did a Test. I modified the source code in SparkContext.scala and add
> one line log code "logInfo("This is textFile")" in the textFile function.
> However, after running an application, I found the log "This is textFile"
> shown in the terminal screen. no such log in the stderr file. I am not sure
> if this is a bug. So, hope you can solve this question. Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
