[GitHub] spark issue #19663: [SPARK-21888][SQL][Hive]add hadoop/hive/hbase/etc config...

2017-11-06 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19663 Please also add [YARN] tag to the PR title, this is actually a yarn problem. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #19663: [SPARK-21888][SQL][Hive]add hadoop/hive/hbase/etc...

2017-11-06 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19663#discussion_r149015858 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -705,6 +705,19 @@ private[spark] class Client

[GitHub] spark issue #18791: [SPARK-21571][Scheduler] Spark history server leaves inc...

2017-11-06 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18791 @ericvandenbergfb please also fix the PR title, thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #19649: [SPARK-22405][SQL] Add more ExternalCatalogEvent

2017-11-05 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19649#discussion_r149009631 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala --- @@ -104,6 +109,8 @@ class

[GitHub] spark pull request #19649: [SPARK-22405][SQL] Add more ExternalCatalogEvent

2017-11-05 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19649#discussion_r149004933 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala --- @@ -158,7 +173,13 @@ abstract class

[GitHub] spark pull request #19649: [SPARK-22405][SQL] Add more ExternalCatalogEvent

2017-11-05 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19649#discussion_r149003803 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala --- @@ -147,7 +154,15 @@ abstract class

[GitHub] spark pull request #19649: [SPARK-22405][SQL] Add more ExternalCatalogEvent

2017-11-02 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/19649 [SPARK-22405][SQL] Add more ExternalCatalogEvent ## What changes were proposed in this pull request? We're building a data lineage tool in which we need to monitor the metadata changes

[GitHub] spark issue #19586: [SPARK-22367][WIP][CORE] Separate the serialization of c...

2017-11-01 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19586 I tend to agree with @cloud-fan , I think you can implement your own serializer out of Spark to be more specialized for your application, that will definitely be more efficient than the built

[GitHub] spark issue #19396: [SPARK-22172][CORE] Worker hangs when the external shuff...

2017-11-01 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19396 Sorry I didn't notice it, will double-check next time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #19396: [SPARK-22172][CORE] Worker hangs when the external shuff...

2017-11-01 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19396 OK, let me merge to master branch. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #19586: [SPARK-22367][WIP][CORE] Separate the serialization of c...

2017-10-31 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19586 Using configurations seems not so elegant, also configuration is application based, how would you turn off/on this feature in the runtime? Sorry I cannot give you a good advice, maybe kryo's

[GitHub] spark issue #19396: [SPARK-22172][CORE] Worker hangs when the external shuff...

2017-10-31 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19396 I'm OK with the current changes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #19586: [SPARK-22367][WIP][CORE] Separate the serialization of c...

2017-10-31 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19586 @ConeyLiu what about the below example, does your implementation support this? ```scala trait Base { val name: String } case class A(name: String) extends Base case class

[GitHub] spark issue #19580: [SPARK-11334][CORE] Fix bug in Executor allocation manag...

2017-10-31 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19580 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #19580: [SPARK-11334][CORE] Fix bug in Executor allocation manag...

2017-10-31 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19580 jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #19580: [SPARK-11334][CORE] Fix bug in Executor allocatio...

2017-10-26 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19580#discussion_r147325260 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -267,6 +267,10 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #19580: [SPARK-11334][CORE] Fix bug in Executor allocatio...

2017-10-26 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19580#discussion_r147304200 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -267,6 +267,10 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #19580: [SPARK-11334][CORE] Fix bug in Executor allocatio...

2017-10-26 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19580#discussion_r147303973 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -678,7 +679,9 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #19580: [SPARK-11334][CORE] Fix bug in Executor allocatio...

2017-10-26 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19580#discussion_r147304306 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -709,7 +712,9 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark issue #19519: [SPARK-21840][core] Add trait that allows conf to be dir...

2017-10-26 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19519 LGTM, merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #19534: [SPARK-22312][CORE] Fix bug in Executor allocation manag...

2017-10-25 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19534 @sitalkedia would you please reopen this PR, I think the second issue I fixed before is not valid anymore, for the first issue the fix is no difference compared to here

[GitHub] spark pull request #11205: [SPARK-11334][Core] Handle maximum task failure s...

2017-10-25 Thread jerryshao
Github user jerryshao closed the pull request at: https://github.com/apache/spark/pull/11205 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #11205: [SPARK-11334][Core] Handle maximum task failure situatio...

2017-10-25 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/11205 Verified again, looks like the 2nd bullet is not valid anymore, I cannot reproduce it in latest master branch, this might have already been fixed in SPARK-13054. So only first issue

[GitHub] spark issue #11205: [SPARK-11334][Core] Handle maximum task failure situatio...

2017-10-25 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/11205 @vanzin , in the current code `stageIdToTaskIndices` cannot be used to track number of running tasks, because this structure doesn't remove task index from itself when task is finished

[GitHub] spark issue #19458: [SPARK-22227][CORE] DiskBlockManager.getAllBlocks now to...

2017-10-25 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19458 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #19458: [SPARK-22227][CORE] DiskBlockManager.getAllBlocks now to...

2017-10-25 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19458 There's a UT failure (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83014/testReport/junit/org.apache.spark.storage/BlockIdSuite/test_bad_deserialization/). @superbobry

[GitHub] spark pull request #19519: [SPARK-21840][core] Add trait that allows conf to...

2017-10-24 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19519#discussion_r146737263 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApplication.scala --- @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19519: [SPARK-21840][core] Add trait that allows conf to...

2017-10-24 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19519#discussion_r146734075 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApplication.scala --- @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #18492: [SPARK-19326] Speculated task attempts do not get...

2017-10-23 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/18492#discussion_r146190420 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -373,8 +373,14 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark issue #19554: [SPARK-22319][Core][BACKPORT-2.2] call loginUserFromKeyt...

2017-10-23 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19554 @sjrand would you please close this PR, it is already merged to branch 2.2. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #19554: [SPARK-22319][Core][BACKPORT-2.2] call loginUserFromKeyt...

2017-10-23 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19554 Thanks, merging to branch 2.2. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #19554: [SPARK-22319][Core][BACKPORT-2.2] call loginUserFromKeyt...

2017-10-22 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19554 ok to test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19554: [SPARK-22319][Core] call loginUserFromKeytab before acce...

2017-10-22 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19554 Can you please add a tag in PR title `[BACKPORT-2.2]`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #19554: [SPARK-22319][Core] call loginUserFromKeytab before acce...

2017-10-22 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19554 ok to test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19534: [SPARK-22312][CORE] Fix bug in Executor allocation manag...

2017-10-22 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19534 @sitalkedia I'm OK with either. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #19519: [SPARK-21840][core] Add trait that allows conf to...

2017-10-22 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19519#discussion_r146154530 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApplication.scala --- @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #19540: [SPARK-22319][Core] call loginUserFromKeytab before acce...

2017-10-22 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19540 @sjrand , can you please create another PR against branch-2.2, it is not auto-mergeable, thanks! --- - To unsubscribe, e-mail

[GitHub] spark issue #19540: [SPARK-22319][Core] call loginUserFromKeytab before acce...

2017-10-22 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19540 LGTM, merging to master and branch 2.2. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19519: [SPARK-21840][core] Add trait that allows conf to be dir...

2017-10-20 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19519 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19540: [SPARK-22319][Core] call loginUserFromKeytab before acce...

2017-10-19 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19540 I think branch 2.2 also has similar issue when fetching resources from remote secure HDFS. --- - To unsubscribe, e-mail

[GitHub] spark issue #19540: [SPARK-22319][Core] call loginUserFromKeytab before acce...

2017-10-19 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19540 ok to test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19540: [SPARK-22319][Core] call loginUserFromKeytab before acce...

2017-10-19 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19540 Thanks for the fix! I didn't test on secure cluster when did glob path support, so I didn't realize such issue

[GitHub] spark issue #19534: [SPARK-22312][CORE] Fix bug in Executor allocation manag...

2017-10-19 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19534 @sitalkedia I have a very old similar PR #11205 , maybe you can refer to it. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #19469: [SPARK-22243][DStreams]spark.yarn.jars reload from confi...

2017-10-19 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19469 @felixcheung As you can see there's bunch of configurations needs to be added here in https://github.com/apache-spark-on-k8s/spark/pull/516, that's why I'm asking a general solutions

[GitHub] spark issue #19519: [SPARK-21840][core] Add trait that allows conf to be dir...

2017-10-19 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19519 @vanzin , how do we leverage this new trait, would you please explain more? Thanks! --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #19534: [SPARK-22312][CORE] Fix bug in Executor allocation manag...

2017-10-19 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19534 @sitalkedia would you please fix the PR title, seems it is broken now. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-19 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19509 LGTM, merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #19469: [SPARK-22243][DStreams]spark.yarn.jars reload from confi...

2017-10-18 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19469 @ssaavedra , yes I think so. with the pull-in of k8s support, I would guess more configurations need to be added to exclusion rule. With current solution, one by one PR doesn't make so sense. We

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-18 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19509 LGTM, just one minor comment. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #19509: [SPARK-22290][core] Avoid creating Hive delegatio...

2017-10-18 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19509#discussion_r145329972 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala --- @@ -347,6 +347,10 @@ package object config

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-18 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19509 I see, thanks for the explanation. I didn't think about such scenario. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #19263: [SPARK-22050][CORE] Allow BlockUpdated events to be opti...

2017-10-17 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19263 @vanzin, do you have other comments? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19469: [SPARK-22243][DStreams]spark.yarn.jars reload from confi...

2017-10-17 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19469 @ChenjunZou did you get a chance to look at my left comment? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-17 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19509 >The effect of this change is that now it's possible to initialize multiple, non-concurrent SparkContext instances in the same JVM. @vanzin , do we support in now? As I remembe

[GitHub] spark issue #19476: [SPARK-22062][CORE] Spill large block to disk in BlockMa...

2017-10-17 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19476 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145013312 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -653,15 +663,34 @@ private[spark] class BlockManager( require

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145011923 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -509,11 +508,10 @@ class BlockManagerSuite extends SparkFunSuite

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145011775 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -684,7 +713,7 @@ private[spark] class BlockManager

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145010440 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -653,15 +663,34 @@ private[spark] class BlockManager( require

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145009567 --- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala --- @@ -662,7 +662,9 @@ private[spark] object SparkConf extends Logging

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145009167 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -653,15 +663,34 @@ private[spark] class BlockManager( require

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-16 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19419 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19396: [SPARK-22172][CORE] Worker hangs when the external shuff...

2017-10-16 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19396 Sorry for the late response. I understand you purpose now. I think such behavior discrepancy is not a big problem. I guess the reason why NM still run with exception is that NM doesn't

[GitHub] spark pull request #19419: [SPARK-22188] [CORE] Adding security headers for ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19419#discussion_r144775941 --- Diff: docs/security.md --- @@ -186,7 +186,54 @@ configure those ports. +### HTTP Security Headers + +Apache Spark can

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r144775481 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1552,4 +1582,65 @@ private[spark] object BlockManager

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r144770999 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1552,4 +1582,65 @@ private[spark] object BlockManager

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r144769226 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1552,4 +1582,65 @@ private[spark] object BlockManager

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r144765076 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1552,4 +1582,65 @@ private[spark] object BlockManager

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r144764761 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -355,11 +355,21 @@ package object config { .doc

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r144763884 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -355,11 +355,21 @@ package object config { .doc

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19419 >/home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/internal/config/package.scala:440:0: Whitespace at end of line Please fix the style is

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19419 ok to test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #19419: [SPARK-22188] [CORE] Adding security headers for ...

2017-10-13 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19419#discussion_r144488504 --- Diff: docs/configuration.md --- @@ -2013,7 +2013,62 @@ Apart from these, the following properties are also available, and may be useful

[GitHub] spark pull request #19464: [SPARK-22233] [core] Allow user to filter out emp...

2017-10-13 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19464#discussion_r144472362 --- Diff: core/src/test/scala/org/apache/spark/FileSuite.scala --- @@ -510,4 +510,86 @@ class FileSuite extends SparkFunSuite with LocalSparkContext

[GitHub] spark pull request #19464: [SPARK-22233] [core] Allow user to filter out emp...

2017-10-13 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19464#discussion_r144472244 --- Diff: core/src/test/scala/org/apache/spark/FileSuite.scala --- @@ -510,4 +510,86 @@ class FileSuite extends SparkFunSuite with LocalSparkContext

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r144456817 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -426,4 +426,11 @@ package object config { .toSequence

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r144453507 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -426,4 +426,11 @@ package object config { .toSequence

[GitHub] spark issue #19476: [SPARK-22062][CORE] Spill large block to disk in BlockMa...

2017-10-12 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19476 @cloud-fan @jiangxb1987 @jinxing64 would you please help to review when you have time, thanks! --- - To unsubscribe, e-mail

[GitHub] spark pull request #19458: [SPARK-22227][CORE] DiskBlockManager.getAllBlocks...

2017-10-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19458#discussion_r144450997 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala --- @@ -100,7 +100,16 @@ private[spark] class DiskBlockManager(conf

[GitHub] spark pull request #19263: [SPARK-22050][CORE] Allow BlockUpdated events to ...

2017-10-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19263#discussion_r19147 --- Diff: docs/configuration.md --- @@ -714,6 +714,13 @@ Apart from these, the following properties are also available, and may be useful

[GitHub] spark pull request #19263: [SPARK-22050][CORE] Allow BlockUpdated events to ...

2017-10-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19263#discussion_r18138 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -41,6 +41,22 @@ package object config { .bytesConf

[GitHub] spark pull request #19419: [SPARK-22188] [CORE] Adding security headers for ...

2017-10-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19419#discussion_r144286844 --- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala --- @@ -79,6 +79,9 @@ private[spark] object JettyUtils extends Logging { val

[GitHub] spark pull request #19419: [SPARK-22188] [CORE] Adding security headers for ...

2017-10-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19419#discussion_r144283398 --- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala --- @@ -89,6 +92,13 @@ private[spark] object JettyUtils extends Logging

[GitHub] spark issue #19458: [SPARK-22227][CORE] DiskBlockManager.getAllBlocks now to...

2017-10-12 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19458 Yes, I agree in any case it should not throw an exception. But in this PR you filtered out temp shuffle/local blocks, do you think this block is valid or not, are they blocks? So I'd

[GitHub] spark issue #19458: [SPARK-22227][CORE] DiskBlockManager.getAllBlocks now to...

2017-10-11 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19458 Instead of filtering out temp blocks, why not adding parsing rule for `TempLocalBlockId` and `TempShuffleBlockId`? That could also solve the problem. Since `DiskBlockManager#getAllFiles` doesn't

[GitHub] spark pull request #19419: [SPARK-22188] [CORE] Adding security headers for ...

2017-10-11 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19419#discussion_r144186220 --- Diff: conf/spark-defaults.conf.template --- @@ -25,3 +25,10 @@ # spark.serializer org.apache.spark.serializer.KryoSerializer

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-11 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19419 @vanzin @tgravescs @ajbozarth what is your opinion on this PR? Is it a necessary fix for Spark? --- - To unsubscribe, e

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-10-11 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r144182701 --- Diff: pom.xml --- @@ -2649,6 +2649,13 @@ + kubernetes --- End diff -- We should also change the sbt

[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...

2017-10-11 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19464 IIUC this issue also existed in `NewHadoopRDD` and `FileScanRDD` (possibly), we'd better also fix them. --- - To unsubscribe

[GitHub] spark pull request #19464: [SPARK-22233] [core] Allow user to filter out emp...

2017-10-11 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19464#discussion_r144181321 --- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala --- @@ -196,7 +196,10 @@ class HadoopRDD[K, V]( // add the credentials here

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-11 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/19476 [SPARK-22062][CORE] Spill large block to disk in BlockManager's remote fetch to avoid OOM ## What changes were proposed in this pull request? In the current BlockManager's

[GitHub] spark issue #19469: [SPARK-22243][DStreams]spark.yarn.jars reload from confi...

2017-10-10 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19469 There's a similar PR #19427 , I was wondering if we can provide a general solution for such issues, like using a configuration to specify all the confs which needs to be reloaded

[GitHub] spark issue #19466: [SPARK-22237] [CORE] Fix spark submit file download for ...

2017-10-10 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19466 Would you please show us an example of how it breaks? The codes here which assigning all resources to local ones might work, but it covers which line is really broken, can you please describe

[GitHub] spark issue #19399: [SPARK-22175][WEB-UI] Add status column to history page

2017-10-09 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19399 I agree with @squito that the criteria to define application's success should be well considered. Here in your current code, only if all the jobs are successful then the application is marked

[GitHub] spark issue #19287: [SPARK-22074][Core] Task killed by other attempt task sh...

2017-10-09 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19287 LGTM, merging to master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #19077: [SPARK-21860][core]Improve memory reuse for heap ...

2017-10-08 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19077#discussion_r143380706 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -116,9 +116,10 @@ private [sql] object

[GitHub] spark issue #19287: [SPARK-22074][Core] Task killed by other attempt task sh...

2017-10-08 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19287 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #19419: [SPARK-22188] [CORE] Adding security headers for ...

2017-10-08 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19419#discussion_r143377794 --- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala --- @@ -79,6 +79,9 @@ private[spark] object JettyUtils extends Logging { val

[GitHub] spark pull request #19419: [SPARK-22188] [CORE] Adding security headers for ...

2017-10-08 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19419#discussion_r143377976 --- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala --- @@ -89,6 +92,9 @@ private[spark] object JettyUtils extends Logging

[GitHub] spark pull request #19419: [SPARK-22188] [CORE] Adding security headers for ...

2017-10-08 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19419#discussion_r143377740 --- Diff: conf/spark-defaults.conf.template --- @@ -25,3 +25,10 @@ # spark.serializer org.apache.spark.serializer.KryoSerializer

<    2   3   4   5   6   7   8   9   10   11   >