[GitHub] spark issue #18025: [SPARK-20889][SparkR] Grouped documentation for AGGREGAT...

2017-06-18 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/18025 very cool, thanks, I guess there's only this last comment https://github.com/apache/spark/pull/18025#discussion_r122631306 --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #18230: [SPARK-19688] [STREAMING] Not to read `spark.yarn.creden...

2017-06-18 Thread saturday-shi
Github user saturday-shi commented on the issue: https://github.com/apache/spark/pull/18230 @jerryshao > "reload" here meanings retrieving back SparkConf from checkpoint file and using this retrieved SparkConf to create SparkContext when restarting streaming application.

[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...

2017-06-18 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/18025#discussion_r122631306 --- Diff: R/pkg/R/stats.R --- @@ -52,22 +52,17 @@ setMethod("crosstab", collect(dataFrame(sct)) }) -#' Calculat

[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...

2017-06-18 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/18025#discussion_r122631024 --- Diff: R/pkg/R/stats.R --- @@ -52,22 +52,17 @@ setMethod("crosstab", collect(dataFrame(sct)) }) -#' Calculat

[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...

2017-06-18 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/18025#discussion_r122630839 --- Diff: R/pkg/R/functions.R --- @@ -361,10 +361,13 @@ setMethod("column", #' #' @rdname corr #' @name corr -#' @family math function

[GitHub] spark pull request #16347: [SPARK-18934][SQL] Writing to dynamic partitions ...

2017-06-18 Thread junegunn
Github user junegunn closed the pull request at: https://github.com/apache/spark/pull/16347 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[GitHub] spark issue #16347: [SPARK-18934][SQL] Writing to dynamic partitions does no...

2017-06-18 Thread junegunn
Github user junegunn commented on the issue: https://github.com/apache/spark/pull/16347 Hive makes sure that the output file is properly sorted by the column specified in `SORT BY` clause by having only one reduce task (output) for each partition. ``` STAGE PLANS: S

[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...

2017-06-18 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/18025#discussion_r122630544 --- Diff: R/pkg/R/functions.R --- @@ -361,10 +361,13 @@ setMethod("column", #' #' @rdname corr #' @name corr -#' @family math function

[GitHub] spark pull request #18320: [SPARK-21093][R] Terminate R's worker processes i...

2017-06-18 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/18320#discussion_r122630135 --- Diff: R/pkg/inst/worker/daemon.R --- @@ -31,7 +31,15 @@ inputCon <- socketConnection( port = port, open = "rb", blocking = TRUE, timeout = c

[GitHub] spark issue #18340: [SPARK-21132] [SQL] DISTINCT modifier of function argume...

2017-06-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18340 cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #18350: [MINOR] Fix some typo of the document

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18350 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark pull request #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18343#discussion_r122625325 --- Diff: core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala --- @@ -175,6 +175,7 @@ class KryoSerializer(conf: SparkConf) kryo

[GitHub] spark pull request #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18343#discussion_r122625202 --- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala --- @@ -141,7 +141,7 @@ private[spark] class HighlyCompressedMapStatus private (

[GitHub] spark issue #18350: [MINOR] Fix some typo of the document

2017-06-18 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18350 Hi @srowen, would you mind take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeE...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18343 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark pull request #18350: [MINOR] Fix some typo of the document

2017-06-18 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/18350 [MINOR] Fix some typo of the document ## What changes were proposed in this pull request? Fix some typo of the document. ## How was this patch tested? Existing tests.

[GitHub] spark issue #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeE...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78239/ Test PASSed. ---

[GitHub] spark issue #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeE...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18343 **[Test build #78239 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78239/testReport)** for PR 18343 at commit [`e2816ec`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #17328: [SPARK-19975][Python][SQL] Add map_keys and map_values f...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17328 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #17328: [SPARK-19975][Python][SQL] Add map_keys and map_values f...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17328 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78242/ Test PASSed. ---

[GitHub] spark issue #17328: [SPARK-19975][Python][SQL] Add map_keys and map_values f...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17328 **[Test build #78242 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78242/testReport)** for PR 17328 at commit [`021b551`](https://github.com/apache/spark/commit/0

[GitHub] spark pull request #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18343#discussion_r122623575 --- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala --- @@ -141,7 +141,7 @@ private[spark] class HighlyCompressedMapStatus private (

[GitHub] spark pull request #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18343#discussion_r122623298 --- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala --- @@ -141,7 +141,7 @@ private[spark] class HighlyCompressedMapStatus private (

[GitHub] spark issue #18346: [SPARK-21134][SQL] Don't collapse codegen-only expressio...

2017-06-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18346 cc @cloud-fan @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishe

[GitHub] spark issue #18269: [SPARK-21056][SQL] Use at most one spark job to list fil...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18269 let's wait @mallman 's response to make sure this patch does fix the problem --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #18269: [SPARK-21056][SQL] Use at most one spark job to l...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18269#discussion_r122622206 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala --- @@ -248,60 +245,94 @@ object InMemoryFileIndex e

[GitHub] spark pull request #18269: [SPARK-21056][SQL] Use at most one spark job to l...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18269#discussion_r122622157 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala --- @@ -248,60 +245,94 @@ object InMemoryFileIndex e

[GitHub] spark pull request #18269: [SPARK-21056][SQL] Use at most one spark job to l...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18269#discussion_r122622031 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala --- @@ -248,60 +245,94 @@ object InMemoryFileIndex e

[GitHub] spark issue #17471: [SPARK-3577] Report Spill size on disk for UnsafeExterna...

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17471 I was just looking though PRs for my curiosity. Please let me leave a gentle ping @sitalkedia. --- If your project is set up for it, you can reply to this email and have your reply appear on Gi

[GitHub] spark pull request #18269: [SPARK-21056][SQL] Use at most one spark job to l...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18269#discussion_r122621941 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala --- @@ -248,60 +245,94 @@ object InMemoryFileIndex e

[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...

2017-06-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18303 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #18092: [SPARK-20640][CORE]Make rpc timeout and retry for shuffl...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18092 **[Test build #78247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78247/testReport)** for PR 18092 at commit [`d31d8da`](https://github.com/apache/spark/commit/d3

[GitHub] spark pull request #18092: [SPARK-20640][CORE]Make rpc timeout and retry for...

2017-06-18 Thread liyichao
Github user liyichao commented on a diff in the pull request: https://github.com/apache/spark/pull/18092#discussion_r122621671 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -1281,6 +1286,61 @@ class BlockManagerSuite extends SparkFunSuite wit

[GitHub] spark pull request #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus...

2017-06-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18343#discussion_r122621682 --- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala --- @@ -141,7 +141,7 @@ private[spark] class HighlyCompressedMapStatus private (

[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...

2017-06-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18303 Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and w

[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18303 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78235/ Test PASSed. ---

[GitHub] spark issue #18346: [SPARK-21134][SQL] Don't collapse codegen-only expressio...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18346 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18303 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18346: [SPARK-21134][SQL] Don't collapse codegen-only expressio...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18346 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18346: [SPARK-21134][SQL] Don't collapse codegen-only expressio...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18346 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78238/ Test PASSed. ---

[GitHub] spark issue #18290: [SPARK-20989][Core] Fail to start multiple workers on on...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18290 LGTM, only one question: are we going to support it? To reuse the same shuffle service across workers or to allow multiple shuffle services on one host? --- If your project is set up for it, you

[GitHub] spark issue #18346: [SPARK-21134][SQL] Don't collapse codegen-only expressio...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18346 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78237/ Test PASSed. ---

[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18303 **[Test build #78235 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78235/testReport)** for PR 18303 at commit [`8c39912`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #18346: [SPARK-21134][SQL] Don't collapse codegen-only expressio...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18346 **[Test build #78238 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78238/testReport)** for PR 18346 at commit [`c2783f4`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #18346: [SPARK-21134][SQL] Don't collapse codegen-only expressio...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18346 **[Test build #78237 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78237/testReport)** for PR 18346 at commit [`4f412b7`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #18346: [SPARK-21134][SQL] Don't collapse codegen-only expressio...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18346 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78236/ Test PASSed. ---

[GitHub] spark issue #18346: [SPARK-21134][SQL] Don't collapse codegen-only expressio...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18346 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18346: [SPARK-21134][SQL] Don't collapse codegen-only expressio...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18346 **[Test build #78236 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78236/testReport)** for PR 18346 at commit [`eead5e1`](https://github.com/apache/spark/commit/e

[GitHub] spark pull request #18092: [SPARK-20640][CORE]Make rpc timeout and retry for...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18092#discussion_r122620995 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -1281,6 +1286,61 @@ class BlockManagerSuite extends SparkFunSuite wi

[GitHub] spark pull request #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18343#discussion_r122620862 --- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala --- @@ -141,7 +141,7 @@ private[spark] class HighlyCompressedMapStatus private (

[GitHub] spark issue #17401: [SPARK-18364][YARN] Expose metrics for YarnShuffleServic...

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17401 gentle ping @ash211. I just wonder if it is active now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request #18324: [SPARK-21045][PYSPARK]Fixed executor blocked beca...

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18324#discussion_r122620720 --- Diff: python/pyspark/worker.py --- @@ -177,8 +180,11 @@ def process(): process() except Exception: try: +

[GitHub] spark issue #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls in cat...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15417 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls in cat...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15417 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78243/ Test FAILed. ---

[GitHub] spark issue #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls in cat...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15417 **[Test build #78243 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78243/testReport)** for PR 15417 at commit [`f255696`](https://github.com/apache/spark/commit/f

[GitHub] spark pull request #18092: [SPARK-20640][CORE]Make rpc timeout and retry for...

2017-06-18 Thread liyichao
Github user liyichao commented on a diff in the pull request: https://github.com/apache/spark/pull/18092#discussion_r122620600 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -1281,6 +1286,59 @@ class BlockManagerSuite extends SparkFunSuite wit

[GitHub] spark issue #17758: [SPARK-20460][SQL] Make it more consistent to handle col...

2017-06-18 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/17758 ok, I'll recheck. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeE...

2017-06-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18343 @wangyum Can you also add a test for this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18323: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18323 **[Test build #78245 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78245/testReport)** for PR 18323 at commit [`7407541`](https://github.com/apache/spark/commit/74

[GitHub] spark pull request #14085: [SPARK-16408][SQL] SparkSQL Added file get Except...

2017-06-18 Thread zenglinxi0615
Github user zenglinxi0615 commented on a diff in the pull request: https://github.com/apache/spark/pull/14085#discussion_r122620464 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/commands.scala --- @@ -113,8 +113,9 @@ case class AddFile(path: String) extends

[GitHub] spark issue #18092: [SPARK-20640][CORE]Make rpc timeout and retry for shuffl...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18092 **[Test build #78246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78246/testReport)** for PR 18092 at commit [`c8e7c64`](https://github.com/apache/spark/commit/c8

[GitHub] spark issue #18092: [SPARK-20640][CORE]Make rpc timeout and retry for shuffl...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18092 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #17758: [SPARK-20460][SQL] Make it more consistent to handle col...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17758 shall we check duplicated columns in write path? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #17395: [SPARK-20065][SS][WIP] Avoid to output empty parquet fil...

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17395 hmmm. @uncleGen, shell we close this for now? reopening when it's ready would welcome. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #17328: [SPARK-19975][Python][SQL] Add map_keys and map_values f...

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17328 +1 for this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark pull request #18324: [SPARK-21045][PYSPARK]Fixed executor blocked beca...

2017-06-18 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/18324#discussion_r122620265 --- Diff: python/pyspark/worker.py --- @@ -177,8 +180,11 @@ def process(): process() except Exception: try: +

[GitHub] spark pull request #18092: [SPARK-20640][CORE]Make rpc timeout and retry for...

2017-06-18 Thread liyichao
Github user liyichao commented on a diff in the pull request: https://github.com/apache/spark/pull/18092#discussion_r122620196 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -1281,6 +1286,59 @@ class BlockManagerSuite extends SparkFunSuite wit

[GitHub] spark issue #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls in cat...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15417 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls in cat...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15417 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78241/ Test FAILed. ---

[GitHub] spark issue #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls in cat...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15417 **[Test build #78241 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78241/testReport)** for PR 15417 at commit [`5436c38`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #17084: [SPARK-18693][ML][MLLIB] ML Evaluators should use weight...

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17084 gentle ping @imatiach-msft . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark issue #18025: [SPARK-20889][SparkR] Grouped documentation for AGGREGAT...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18025 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78244/ Test PASSed. ---

[GitHub] spark issue #18025: [SPARK-20889][SparkR] Grouped documentation for AGGREGAT...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18025 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18025: [SPARK-20889][SparkR] Grouped documentation for AGGREGAT...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18025 **[Test build #78244 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78244/testReport)** for PR 18025 at commit [`6eae126`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeE...

2017-06-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18343 @wangyum Thanks for updating. Can you try to disable kyro and try it again? So we can verify it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHu

[GitHub] spark issue #17681: [SPARK-20383][SQL] Supporting Create [temporary] Functio...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17681 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78240/ Test FAILed. ---

[GitHub] spark issue #17681: [SPARK-20383][SQL] Supporting Create [temporary] Functio...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17681 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark pull request #16766: [SPARK-19426][SQL] Custom coalesce for Dataset

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16766#discussion_r122619579 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -2603,12 +2603,27 @@ class Dataset[T] private[sql]( * current upstrea

[GitHub] spark pull request #16766: [SPARK-19426][SQL] Custom coalesce for Dataset

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16766#discussion_r122619526 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -2603,12 +2603,27 @@ class Dataset[T] private[sql]( * current upstrea

[GitHub] spark issue #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeE...

2017-06-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18343 Because we write/read `hugeBlockSizes` in `writeExternal`/`readExternal`, it seems to me that it is intended to be serialized. So I think removing `transient` should be ok. LGTM cc @cloud-fa

[GitHub] spark issue #17681: [SPARK-20383][SQL] Supporting Create [temporary] Functio...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17681 **[Test build #78240 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78240/testReport)** for PR 17681 at commit [`f6898c4`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #18349: [SPARK-20927][SS] Change some operators in Dataset to no...

2017-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18349 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark issue #18349: [SPARK-20927][SS] Change some operators in Dataset to no...

2017-06-18 Thread ZiyueHuang
Github user ZiyueHuang commented on the issue: https://github.com/apache/spark/pull/18349 @zsxwing Could you please review this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark issue #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeE...

2017-06-18 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/18343 @viirya Yes, I' using `org.apache.spark.serializer.KryoSerializer`, [master branch](https://github.com/apache/spark/tree/ce49428ef7d640c1734e91ffcddc49dbc8547ba7) still has this issue, error logs:

[GitHub] spark pull request #18349: [SPARK-20927][SS] Change some operators in Datase...

2017-06-18 Thread ZiyueHuang
GitHub user ZiyueHuang opened a pull request: https://github.com/apache/spark/pull/18349 [SPARK-20927][SS] Change some operators in Dataset to no-op for a streaming query. ## What changes were proposed in this pull request? Change some operators(persist, unpersist, checkpoi

[GitHub] spark issue #16347: [SPARK-18934][SQL] Writing to dynamic partitions does no...

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16347 gentle ping @junegunn on ^. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeE...

2017-06-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18343 I think this should be addressed before 2.2. I already asked notice of other committers on dev mailing list. --- If your project is set up for it, you can reply to this email and have your reply app

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14957 @xuanyuanking, let's close this and help review #16578 if you agree on the comments above. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #18343: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeE...

2017-06-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18343 @wangyum Are you using kyro serializer? I think it is why you hit this issue. Once you use kyro, I think the `readExternal` in `HighlyCompressedMapStatus` won't be used to deserialize the ob

[GitHub] spark issue #18025: [SPARK-20889][SparkR] Grouped documentation for AGGREGAT...

2017-06-18 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/18025 This is how the doc for column_aggregate_functions looks like (only snapshot of the main parts): ![image](https://user-images.githubusercontent.com/11082368/27269174-85df12fa-5469-11e7

[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18303 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fea

[GitHub] spark issue #18092: [SPARK-20640][CORE]Make rpc timeout and retry for shuffl...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18092 LGTM except 2 minor comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #18092: [SPARK-20640][CORE]Make rpc timeout and retry for...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18092#discussion_r122617592 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -1281,6 +1286,59 @@ class BlockManagerSuite extends SparkFunSuite wi

[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...

2017-06-18 Thread actuaryzhang
Github user actuaryzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/18025#discussion_r122617531 --- Diff: R/pkg/R/stats.R --- @@ -52,22 +52,17 @@ setMethod("crosstab", collect(dataFrame(sct)) }) -#' Calcula

[GitHub] spark issue #18025: [SPARK-20889][SparkR] Grouped documentation for AGGREGAT...

2017-06-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18025 **[Test build #78244 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78244/testReport)** for PR 18025 at commit [`6eae126`](https://github.com/apache/spark/commit/6e

[GitHub] spark pull request #18092: [SPARK-20640][CORE]Make rpc timeout and retry for...

2017-06-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18092#discussion_r122617448 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -1281,6 +1286,59 @@ class BlockManagerSuite extends SparkFunSuite wi

[GitHub] spark issue #13893: [SPARK-14172][SQL] Hive table partition predicate not pa...

2017-06-18 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/13893 ya, this still exists. Let me find some time to resolve this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...

2017-06-18 Thread actuaryzhang
Github user actuaryzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/18025#discussion_r122617405 --- Diff: R/pkg/R/stats.R --- @@ -52,22 +52,17 @@ setMethod("crosstab", collect(dataFrame(sct)) }) -#' Calcula

[GitHub] spark issue #12257: [SPARK-14483][WEBUI] Display user name for each job and ...

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/12257 gentle ping @sarutak on ^ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and w

[GitHub] spark pull request #18347: [SPARK-20599][SS] ConsoleSink should work with (b...

2017-06-18 Thread jaceklaskowski
Github user jaceklaskowski commented on a diff in the pull request: https://github.com/apache/spark/pull/18347#discussion_r122617147 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -465,6 +465,8 @@ case class DataSource(

  1   2   3   >