[GitHub] spark issue #22926: [SPARK-25917][Spark UI] memoryMetrics should be Json ign...

2018-11-20 Thread jianjianjiao
Github user jianjianjiao commented on the issue: https://github.com/apache/spark/pull/22926 Thanks @vanzin . I was using 2.3, and with your comment, I found there was one check-in about one month ago that handled this case. Will close this PR, and sorry for the misreporting, I

[GitHub] spark pull request #22926: [SPARK-25917][Spark UI] memoryMetrics should be J...

2018-11-20 Thread jianjianjiao
Github user jianjianjiao closed the pull request at: https://github.com/apache/spark/pull/22926 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22926: [SPARK-25917][Spark UI] memoryMetrics should be Json ign...

2018-11-06 Thread jianjianjiao
Github user jianjianjiao commented on the issue: https://github.com/apache/spark/pull/22926 @AmplabJenkins Could you please find someone to review this? I believe this is a bug in Spark UI. Thanks

[GitHub] spark issue #22926: [SPARK-25917][Spark UI] memoryMetrics should be Json ign...

2018-11-05 Thread jianjianjiao
Github user jianjianjiao commented on the issue: https://github.com/apache/spark/pull/22926 @mccheah, @smurakozi @vanzin Could you please help take a look at this PR? Thanks. --- - To unsubscribe, e-mail

[GitHub] spark pull request #22926: [SPARK-25917][Spark UI] memoryMetrics should be J...

2018-11-01 Thread jianjianjiao
GitHub user jianjianjiao opened a pull request: https://github.com/apache/spark/pull/22926 [SPARK-25917][Spark UI] memoryMetrics should be Json ignored when being none ## What changes were proposed in this pull request? Spark UI's executors page loads forever when

[GitHub] spark issue #22520: [SPARK-25509][Core]Windows doesn't support POSIX permiss...

2018-09-26 Thread jianjianjiao
Github user jianjianjiao commented on the issue: https://github.com/apache/spark/pull/22520 @srowen that makes sense, I will be more patient next time. ^_^. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22520: [SPARK-25509][Core]Windows doesn't support POSIX permiss...

2018-09-25 Thread jianjianjiao
Github user jianjianjiao commented on the issue: https://github.com/apache/spark/pull/22520 @srowen @vanzin tests passed. What should I do now to make it approved to merge? --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #22520: [SPARK-25509][Core]Windows doesn't support POSIX permiss...

2018-09-25 Thread jianjianjiao
Github user jianjianjiao commented on the issue: https://github.com/apache/spark/pull/22520 @srowen Thanks for confirmation. I have sent out new iteration. Could you please authorize testing

[GitHub] spark issue #22520: [SPARK-25509][Core]Windows doesn't support POSIX permiss...

2018-09-25 Thread jianjianjiao
Github user jianjianjiao commented on the issue: https://github.com/apache/spark/pull/22520 @srowen thanks for reviewing this PR, and your comments. 1. have fixed the coding style, thanks. 2. These are the only 2 places using PosixFilePermissions to handle file operations

[GitHub] spark issue #22444: [SPARK-25409][Core]Speed up Spark History loading via in...

2018-09-21 Thread jianjianjiao
Github user jianjianjiao commented on the issue: https://github.com/apache/spark/pull/22444 @squito Yes, you are correct. I was trying to make the applications running during the scan be picked up quicker. It turns out the SPARK-6951 has done great job in achieving

[GitHub] spark pull request #22444: [SPARK-25409][Core]Speed up Spark History loading...

2018-09-21 Thread jianjianjiao
Github user jianjianjiao closed the pull request at: https://github.com/apache/spark/pull/22444 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22444: [SPARK-25409][Core]Speed up Spark History loading via in...

2018-09-21 Thread jianjianjiao
Github user jianjianjiao commented on the issue: https://github.com/apache/spark/pull/22444 @vanzin Really thanks for you suggestions. It becomes much faster loading event logs. from more than 2.5 hours, to 19 minutes, loading 17K event logs, some of them are larger than 10G

[GitHub] spark pull request #22520: [SPARK-25509][Core]Windows doesn't support POSIX ...

2018-09-21 Thread jianjianjiao
GitHub user jianjianjiao opened a pull request: https://github.com/apache/spark/pull/22520 [SPARK-25509][Core]Windows doesn't support POSIX permissions ## What changes were proposed in this pull request? SHS V2 cannot enabled in Windoes, because windows doesn't support

[GitHub] spark pull request #22444: [SPARK-25409][Core]Speed up Spark History loading...

2018-09-17 Thread jianjianjiao
Github user jianjianjiao commented on a diff in the pull request: https://github.com/apache/spark/pull/22444#discussion_r218292773 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -465,20 +475,31 @@ private[history] class

[GitHub] spark issue #22444: [SPARK-25409][Core]Speed up Spark History loading via in...

2018-09-17 Thread jianjianjiao
Github user jianjianjiao commented on the issue: https://github.com/apache/spark/pull/22444 Add @vanzin @steveloughran @squito who made changes to related code. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22444: implement incremental loading and add a flag to l...

2018-09-17 Thread jianjianjiao
GitHub user jianjianjiao opened a pull request: https://github.com/apache/spark/pull/22444 implement incremental loading and add a flag to load incomplete or not ## What changes were proposed in this pull request? 1. Instead of loading all event logs in every loading, load