[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16924 My only concern is that this will cause the SHS to parse these logs more often, but that's really a problem in other parts of the code that need to be made smarter. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16924 ok then this LGTM, but I'm not very confident about this part of code, cc @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16924 Is there any concern regarding checking file length? I guess we have no much choices here to detect file changes? There are comments for why using file length to detect file changes: https://github.com/apache/spark/blob/e26dac5feb02033f980b1e69c9b0ff50869b6f9e/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L62-L72 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16924 > Hadoop doesn't update file length until close and therefor Spark's history server can't detect any changes So the history server detect file changes by checking file length? Is there a better way to do it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16924 In `FsHistoryProvider`, since there is a check for file size, I think it is designed to find updated logs from running applications? https://github.com/apache/spark/blob/e26dac5feb02033f980b1e69c9b0ff50869b6f9e/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L331 Btw, in `EventLoggingListener`, if it writes to local files, seems to me the file length will be updated and so `FsHistoryProvider` can find the updated logs. Seems this looks reasonable to make the non local fs functionally consistent with local fs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/16924 @cloud-fan The History Server has an incomplete applications page as part of its functionality that reads through files still marked .inprogress. Though I'm not a big fan of this use case for the history server, I know that many users utilize it over the actual Web UI. So this is a problem we should fix, @vanzin mind looking at this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16924 I'm confused, how can your customer see a still running application in history server? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user dosoft commented on the issue: https://github.com/apache/spark/pull/16924 This issue has been reported by our customer as following: "Subsequently connecting to spark thriftserver via beeline and running any MR job, it doesnt get reflected in spark history server UI even after the job completion. If we stop and start spark history server, then this job info gets displayed in UI. Shouldnt this UI get auto refreshed with info on completed jobs?" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16924 no, history server is not a good use case, for long running jobs/apps, history server should ONLY detect the change and show the app after it's finished. Can you explain more about your use case for thrift-server? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user dosoft commented on the issue: https://github.com/apache/spark/pull/16924 Just an example of the use case. This issue affects all long-life apps. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16924 so this PR is not for history server, but for thrift server? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user dosoft commented on the issue: https://github.com/apache/spark/pull/16924 @cloud-fan spark thriftserver as long-running application? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16924 what's the use case of this patch? I think it makes sense to show the history entry after the app is finished, I'm not sure why we want the history server to detect it earlier. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16924 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78487/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16924 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16924 **[Test build #78487 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78487/testReport)** for PR 16924 at commit [`b54b18d`](https://github.com/apache/spark/commit/b54b18d8537f3305d7687d47abfaab1242a9284e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16924 **[Test build #78487 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78487/testReport)** for PR 16924 at commit [`b54b18d`](https://github.com/apache/spark/commit/b54b18d8537f3305d7687d47abfaab1242a9284e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/16924 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16924 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78461/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16924 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16924 **[Test build #78461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78461/testReport)** for PR 16924 at commit [`b54b18d`](https://github.com/apache/spark/commit/b54b18d8537f3305d7687d47abfaab1242a9284e). * This patch **fails PySpark pip packaging tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16924 **[Test build #78461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78461/testReport)** for PR 16924 at commit [`b54b18d`](https://github.com/apache/spark/commit/b54b18d8537f3305d7687d47abfaab1242a9284e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16924 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/16924 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/16924 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/16924 add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/16924 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/16924 Could you bring this up to date? @dosoft --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16924 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
