[jira] [Commented] (SPARK-18933) Different log output between Terminal screen and stderr file
[ https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15787331#comment-15787331 ] Sean Owen commented on SPARK-18933: --- stdout and stderr exist for all processes. You can redirect them as you like. > Different log output between Terminal screen and stderr file > > > Key: SPARK-18933 > URL: https://issues.apache.org/jira/browse/SPARK-18933 > Project: Spark > Issue Type: Bug > Components: Deploy, Documentation, Web UI >Affects Versions: 1.6.3 > Environment: Yarn mode and standalone mode >Reporter: Sean Wong > Original Estimate: 612h > Remaining Estimate: 612h > > First of all, I use the default log4j.properties in the Spark conf/ > But I found that the log output(e.g., INFO) is different between Terminal > screen and stderr File. Some INFO logs exist in both of them. Some INFO logs > exist in either of them. Why this happens? Is it supposed that the output > logs are same between the terminal screen and stderr file? > Then I did a Test. I modified the source code in SparkContext.scala and add > one line log code "logInfo("This is textFile")" in the textFile function. > However, after running an application, I found the log "This is textFile" > shown in the terminal screen. no such log in the stderr file. I am not sure > if this is a bug. So, hope you can solve this question. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18933) Different log output between Terminal screen and stderr file
[ https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15786822#comment-15786822 ] Sean Wong commented on SPARK-18933: --- But there is no stderr or stdout file available for driver logs. Only executors have these two files. > Different log output between Terminal screen and stderr file > > > Key: SPARK-18933 > URL: https://issues.apache.org/jira/browse/SPARK-18933 > Project: Spark > Issue Type: Bug > Components: Deploy, Documentation, Web UI >Affects Versions: 1.6.3 > Environment: Yarn mode and standalone mode >Reporter: Sean Wong > Original Estimate: 612h > Remaining Estimate: 612h > > First of all, I use the default log4j.properties in the Spark conf/ > But I found that the log output(e.g., INFO) is different between Terminal > screen and stderr File. Some INFO logs exist in both of them. Some INFO logs > exist in either of them. Why this happens? Is it supposed that the output > logs are same between the terminal screen and stderr file? > Then I did a Test. I modified the source code in SparkContext.scala and add > one line log code "logInfo("This is textFile")" in the textFile function. > However, after running an application, I found the log "This is textFile" > shown in the terminal screen. no such log in the stderr file. I am not sure > if this is a bug. So, hope you can solve this question. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18933) Different log output between Terminal screen and stderr file
[ https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15769604#comment-15769604 ] Sean Owen commented on SPARK-18933: --- I disagree that it's even something that could be improved. Logs are local to machine, yes. You can redirect or reconfigure driver logs / stdout / stderr already. > Different log output between Terminal screen and stderr file > > > Key: SPARK-18933 > URL: https://issues.apache.org/jira/browse/SPARK-18933 > Project: Spark > Issue Type: Bug > Components: Deploy, Documentation, Web UI >Affects Versions: 1.6.3 > Environment: Yarn mode and standalone mode >Reporter: Sean Wong > Original Estimate: 612h > Remaining Estimate: 612h > > First of all, I use the default log4j.properties in the Spark conf/ > But I found that the log output(e.g., INFO) is different between Terminal > screen and stderr File. Some INFO logs exist in both of them. Some INFO logs > exist in either of them. Why this happens? Is it supposed that the output > logs are same between the terminal screen and stderr file? > Then I did a Test. I modified the source code in SparkContext.scala and add > one line log code "logInfo("This is textFile")" in the textFile function. > However, after running an application, I found the log "This is textFile" > shown in the terminal screen. no such log in the stderr file. I am not sure > if this is a bug. So, hope you can solve this question. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18933) Different log output between Terminal screen and stderr file
[ https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768775#comment-15768775 ] Sean Wong commented on SPARK-18933: --- Finally, I have got the answer. It's not the bug but the log system can be improved. The terminal screen output the log from the driver and the stderr file records the log from each executor. The logs in the process of DAG construction are shown in the terminal screen since these Loginfo codes are executed in the driver. The logs in the compute() functions are recorded in the stderr file since these Loginfo codes are executed within each executors. Currently, unlike executor, the dirver does not have the stderr and stdout file, so that the logs are only shown in the terminal screen. I think the log system should also record the driver's logs in the stderr and stdout. I will work on this in the future. > Different log output between Terminal screen and stderr file > > > Key: SPARK-18933 > URL: https://issues.apache.org/jira/browse/SPARK-18933 > Project: Spark > Issue Type: Bug > Components: Deploy, Documentation, Web UI >Affects Versions: 1.6.3 > Environment: Yarn mode and standalone mode >Reporter: Sean Wong > Original Estimate: 612h > Remaining Estimate: 612h > > First of all, I use the default log4j.properties in the Spark conf/ > But I found that the log output(e.g., INFO) is different between Terminal > screen and stderr File. Some INFO logs exist in both of them. Some INFO logs > exist in either of them. Why this happens? Is it supposed that the output > logs are same between the terminal screen and stderr file? > Then I did a Test. I modified the source code in SparkContext.scala and add > one line log code "logInfo("This is textFile")" in the textFile function. > However, after running an application, I found the log "This is textFile" > shown in the terminal screen. no such log in the stderr file. I am not sure > if this is a bug. So, hope you can solve this question. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18933) Different log output between Terminal screen and stderr file
[ https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768739#comment-15768739 ] Sean Wong commented on SPARK-18933: --- I found that Loginfo codes in the transformation() or actions() are only shown in the terminal screen. These log codes are executed during the DAG construction and before the job submission. But the Loginfo codes in the compute() function of each RDD are only shown in the stderr file. These log codes are executed in the process of real job execution. Now I know which part of Loginfo go to terminal screen and which goes to stderr file. But I still do not know why these will happen since they all use the Loginfo code and extends Logging during the class definition. > Different log output between Terminal screen and stderr file > > > Key: SPARK-18933 > URL: https://issues.apache.org/jira/browse/SPARK-18933 > Project: Spark > Issue Type: Bug > Components: Deploy, Documentation, Web UI >Affects Versions: 1.6.3 > Environment: Yarn mode and standalone mode >Reporter: Sean Wong > Original Estimate: 612h > Remaining Estimate: 612h > > First of all, I use the default log4j.properties in the Spark conf/ > But I found that the log output(e.g., INFO) is different between Terminal > screen and stderr File. Some INFO logs exist in both of them. Some INFO logs > exist in either of them. Why this happens? Is it supposed that the output > logs are same between the terminal screen and stderr file? > Then I did a Test. I modified the source code in SparkContext.scala and add > one line log code "logInfo("This is textFile")" in the textFile function. > However, after running an application, I found the log "This is textFile" > shown in the terminal screen. no such log in the stderr file. I am not sure > if this is a bug. So, hope you can solve this question. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org