[GitHub] spark pull request #20437: [SPARK-23270][Streaming][WEB-UI]FileInputDStream ...

2018-01-31 Thread guoxiaolongzte
Github user guoxiaolongzte commented on a diff in the pull request: https://github.com/apache/spark/pull/20437#discussion_r164975752 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -157,7 +157,7 @@ class FileInputDStream[K, V, F

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20450 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86866/ Test FAILed. ---

[GitHub] spark pull request #20451: [SPARK-23146][WIP] Support client mode for Kubern...

2018-01-31 Thread echarles
GitHub user echarles opened a pull request: https://github.com/apache/spark/pull/20451 [SPARK-23146][WIP] Support client mode for Kubernetes cluster backend ## What changes were proposed in this pull request? The changes allow to support Kubernetes resource manager in

[GitHub] spark issue #20448: [SPARK-23203][SQL] make DataSourceV2Relation immutable

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20448 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86865/ Test FAILed. ---

[GitHub] spark issue #20448: [SPARK-23203][SQL] make DataSourceV2Relation immutable

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20448 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20450 **[Test build #86866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86866/testReport)** for PR 20450 at commit

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20450 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20448: [SPARK-23203][SQL] make DataSourceV2Relation immutable

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20448 **[Test build #86865 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86865/testReport)** for PR 20448 at commit

[GitHub] spark issue #20451: [SPARK-23146][WIP] Support client mode for Kubernetes cl...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20451 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20437: [SPARK-23270][Streaming][WEB-UI]FileInputDStream ...

2018-01-31 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/20437#discussion_r164976661 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -157,7 +157,7 @@ class FileInputDStream[K, V, F <:

[GitHub] spark issue #20451: [SPARK-23146][WIP] Support client mode for Kubernetes cl...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20451 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/422/

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20452 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/430/

[GitHub] spark issue #20448: [SPARK-23203][SQL] make DataSourceV2Relation immutable

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20448 **[Test build #86867 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86867/testReport)** for PR 20448 at commit

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20452 **[Test build #86874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86874/testReport)** for PR 20452 at commit

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20452 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20435: [SPARK-23268][SQL]Reorganize packages in data sou...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20435#discussion_r165047744 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceOffset.scala --- @@ -20,14 +20,15 @@ package

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20452 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/429/

[GitHub] spark issue #20448: [SPARK-23203][SQL] make DataSourceV2Relation immutable

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20448 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20454 **[Test build #86875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86875/testReport)** for PR 20454 at commit

[GitHub] spark pull request #20455: [SPARK-23284][SQL] Document several get API of Co...

2018-01-31 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/20455 [SPARK-23284][SQL] Document several get API of ColumnVector's behavior when accessing null slot ## What changes were proposed in this pull request? For some ColumnVector get APIs such as

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20454 **[Test build #86877 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86877/testReport)** for PR 20454 at commit

[GitHub] spark issue #20456: [SPARK-22624][PYSPARK] Expose range partitioning shuffle...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20456 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20419: [SPARK-23032][SQL][FOLLOW-UP]Add codegenStageId in comme...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20419 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20419: [SPARK-23032][SQL][FOLLOW-UP]Add codegenStageId in comme...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20419 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/436/

[GitHub] spark pull request #20454: [SPARK-23202][SQL] Add new DataSourceWriter API: ...

2018-01-31 Thread gengliangwang
GitHub user gengliangwang opened a pull request: https://github.com/apache/spark/pull/20454 [SPARK-23202][SQL] Add new DataSourceWriter API: onDataWriterCommit ## What changes were proposed in this pull request? Currently, the api

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20452 Overall LGTM. One question is, at #19980, we replace `anyNullsSet` with some predicate of `numNulls` (like `numNulls == 0` or `numNulls` > 0) in many places, shall we use `hasNull` now for those

[GitHub] spark issue #20386: [SPARK-23202][SQL] Break down DataSourceV2Writer.commit ...

2018-01-31 Thread gengliangwang
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/20386 @rdblue @cloud-fan @jose-torres thanks for the comments! I was trying to make the API compatible with `onTaskCommit(taskCommit: TaskCommitMessage)` in `FileCommitProtocol` possible.

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20435 **[Test build #86873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86873/testReport)** for PR 20435 at commit

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20435 **[Test build #86879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86879/testReport)** for PR 20435 at commit

[GitHub] spark issue #20455: [SPARK-23284][SQL] Document the behavior of several Colu...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20455 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20455: [SPARK-23284][SQL] Document the behavior of several Colu...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20455 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/434/

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/435/

[GitHub] spark pull request #20456: [SPARK-22624][PYSPARK] Expose range partitioning ...

2018-01-31 Thread xubo245
GitHub user xubo245 opened a pull request: https://github.com/apache/spark/pull/20456 [SPARK-22624][PYSPARK] Expose range partitioning shuffle introduced by spark-22614 ## What changes were proposed in this pull request? Expose range partitioning shuffle introduced

[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...

2018-01-31 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20404 Thanks all for your review, greatly appreciated. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20448: [SPARK-23203][SQL] make DataSourceV2Relation immutable

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20448 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86867/ Test FAILed. ---

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/433/

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20452 **[Test build #86869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86869/testReport)** for PR 20452 at commit

[GitHub] spark issue #20448: [SPARK-23203][SQL] make DataSourceV2Relation immutable

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20448 **[Test build #86876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86876/testReport)** for PR 20448 at commit

[GitHub] spark issue #20164: [SPARK-22971][ML] OneVsRestModel should use temporary Ra...

2018-01-31 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20164 I might be missing something, but the user sets the raw prediction col name. Now it's going to use a different name than what the user set. Isn't it simply an error to apply a second model,

[GitHub] spark issue #20453: [SPARK-23281][SQL] Query produces results in incorrect o...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20453 **[Test build #86870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86870/testReport)** for PR 20453 at commit

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/427/

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20450 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20450 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/428/

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20454 **[Test build #86871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86871/testReport)** for PR 20454 at commit

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20450 **[Test build #86872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86872/testReport)** for PR 20450 at commit

[GitHub] spark pull request #20448: [SPARK-23203][SQL] make DataSourceV2Relation immu...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20448#discussion_r165049675 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala --- @@ -17,36 +17,84 @@ package

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86875/ Test FAILed. ---

[GitHub] spark issue #20455: [SPARK-23284][SQL] Document the behavior of several Colu...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20455 **[Test build #86878 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86878/testReport)** for PR 20455 at commit

[GitHub] spark issue #20455: [SPARK-23284][SQL] Document the behavior of several Colu...

2018-01-31 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20455 Once map support is added later, we should also document `getMap`'s behavior. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20456: [SPARK-22624][PYSPARK] Expose range partitioning shuffle...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20456 **[Test build #86880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86880/testReport)** for PR 20456 at commit

[GitHub] spark issue #20456: [SPARK-22624][PYSPARK] Expose range partitioning shuffle...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20456 **[Test build #86881 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86881/testReport)** for PR 20456 at commit

[GitHub] spark issue #19340: [SPARK-22119][ML] Add cosine distance to KMeans

2018-01-31 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19340 @zhengruifeng yes I understand why the solutions aren't the same, though it depends on whether you think that's what k-_means_ is supposed to do or not. We're not actually maximizing an expectation

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86871/ Test FAILed. ---

[GitHub] spark issue #20448: [SPARK-23203][SQL] make DataSourceV2Relation immutable

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20448 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20448: [SPARK-23203][SQL] make DataSourceV2Relation immutable

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20448 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/431/

[GitHub] spark pull request #20449: [SPARK-23040][CORE]: Returns interruptible iterat...

2018-01-31 Thread advancedxy
Github user advancedxy commented on a diff in the pull request: https://github.com/apache/spark/pull/20449#discussion_r165066153 --- Diff: core/src/main/scala/org/apache/spark/shuffle/BlockStoreShuffleReader.scala --- @@ -104,9 +104,18 @@ private[spark] class

[GitHub] spark issue #20456: [SPARK-22624][PYSPARK] Expose range partitioning shuffle...

2018-01-31 Thread xubo245
Github user xubo245 commented on the issue: https://github.com/apache/spark/pull/20456 @gatorsmile please review it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20453: [SPARK-23281][SQL] Query produces results in incorrect o...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20453 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86870/ Test PASSed. ---

[GitHub] spark issue #20453: [SPARK-23281][SQL] Query produces results in incorrect o...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20453 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19984: [SPARK-22789] Map-only continuous processing exec...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19984#discussion_r165055237 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingRelation.scala --- @@ -75,6 +76,52 @@ case class

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20452 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86869/ Test PASSed. ---

[GitHub] spark issue #11207: [SPARK-12583][Mesos] Mesos shuffle service: Don't delete...

2018-01-31 Thread bbossy
Github user bbossy commented on the issue: https://github.com/apache/spark/pull/11207 @IgorBerman correct, #11272 was a follow up on this and got merged. I still experience issues with the shuffle service on Mesos, but I haven't been able to pin point the cause. ---

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread gengliangwang
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/20454 @cloud-fan @jose-torres @rdblue --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20454 **[Test build #86871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86871/testReport)** for PR 20454 at commit

[GitHub] spark issue #20448: [SPARK-23203][SQL] make DataSourceV2Relation immutable

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20448 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20435: [SPARK-23268][SQL]Reorganize packages in data sou...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20435#discussion_r165048049 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -403,7 +403,7 @@ class MicroBatchExecution(

[GitHub] spark pull request #20449: [SPARK-23040][CORE]: Returns interruptible iterat...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20449#discussion_r165051362 --- Diff: core/src/main/scala/org/apache/spark/shuffle/BlockStoreShuffleReader.scala --- @@ -104,9 +104,18 @@ private[spark] class

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/432/

[GitHub] spark issue #20449: [SPARK-23040][CORE]: Returns interruptible iterator for ...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20449 cc @jiangxb1987 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #11207: [SPARK-12583][Mesos] Mesos shuffle service: Don't delete...

2018-01-31 Thread IgorBerman
Github user IgorBerman commented on the issue: https://github.com/apache/spark/pull/11207 @bbossy thanks. I'm experiencing them too despite the merged PR. I think I have some direction why it fails, here is my setup: 0. v2.2.0 1. Dynamic allocation is on 2. min executors

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20454 **[Test build #86875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86875/testReport)** for PR 20454 at commit

[GitHub] spark issue #20419: [SPARK-23032][SQL][FOLLOW-UP]Add codegenStageId in comme...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20419 **[Test build #86882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86882/testReport)** for PR 20419 at commit

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20452 LGTM. It is fine with me for `hasNull`, to. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20452 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86874/ Test PASSed. ---

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20452 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20372: [SPARK-23249] [SQL] Improved block merging logic for par...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20372 thanks, merging to master/2.3! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/438/

[GitHub] spark pull request #20448: [SPARK-23203][SQL] make DataSourceV2Relation immu...

2018-01-31 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/20448#discussion_r165132718 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala --- @@ -17,36 +17,84 @@ package

[GitHub] spark issue #20456: [SPARK-22624][PYSPARK] Expose range partitioning shuffle...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20456 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86881/ Test PASSed. ---

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20452 **[Test build #86874 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86874/testReport)** for PR 20452 at commit

[GitHub] spark pull request #20452: revert [SPARK-22785][SQL] remove ColumnVector.any...

2018-01-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20452 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20386: [SPARK-23202][SQL] Break down DataSourceV2Writer....

2018-01-31 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/20386#discussion_r165119560 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataSourceWriter.java --- @@ -63,32 +68,42 @@ DataWriterFactory

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20454 **[Test build #86884 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86884/testReport)** for PR 20454 at commit

[GitHub] spark issue #20454: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDa...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20454 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20452 Hi, @cloud-fan . I reopen the JIRA since it's technically reverted. You can resolve that back with a new title like `Renaming anyNullsSet to hasNull` or something. ---

[GitHub] spark issue #20373: [SPARK-23159][PYTHON] Update cloudpickle to v0.4.2 plus ...

2018-01-31 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/20373 Yeah the formatting changes will just make it more complicated for the next update, so they should probably not be included now. cloudpickle#140 looks pretty clean so I'll put that in. ---

[GitHub] spark issue #20452: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSe...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20452 thanks @dongjoon-hyun ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20373: [SPARK-23159][PYTHON] Update cloudpickle to v0.4.2 plus ...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20373 **[Test build #86885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86885/testReport)** for PR 20373 at commit

[GitHub] spark pull request #20386: [SPARK-23202][SQL] Break down DataSourceV2Writer....

2018-01-31 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/20386#discussion_r165131292 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/streaming/writer/StreamWriter.java --- @@ -32,40 +32,44 @@

[GitHub] spark issue #20295: [SPARK-23011] Support alternative function form with gro...

2018-01-31 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/20295 Hey @HyukjinKwon @ueshin could you please take a look at this? Thanks! --- - To unsubscribe, e-mail:

[GitHub] spark pull request #20450: [SPARK-23280][SQL] add map type support to Column...

2018-01-31 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/20450#discussion_r165098764 --- Diff: sql/core/src/main/java/org/apache/spark/sql/vectorized/ColumnVector.java --- @@ -215,10 +215,18 @@ public final ColumnarRow getStruct(int rowId) {

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20450 **[Test build #86872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86872/testReport)** for PR 20450 at commit

  1   2   3   4   5   6   7   >