[GitHub] spark pull request #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream ...

2018-06-12 Thread BryanCutler
GitHub user BryanCutler opened a pull request: https://github.com/apache/spark/pull/21546 [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format for creating from and collecting Pandas DataFrames ## What changes were proposed in this pull request? This changes the calls

[GitHub] spark pull request #21515: [SPARK-24372][build] Add scripts to help with pre...

2018-06-12 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21515#discussion_r194896168 --- Diff: dev/.rat-excludes --- @@ -106,3 +106,4 @@ spark-warehouse structured-streaming/* kafka-source-initial-offset-version-2.1.0.bin

[GitHub] spark pull request #21503: [SPARK-24478][SQL] Move projection and filter pus...

2018-06-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21503#discussion_r194895594 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala --- @@ -17,15 +17,56 @@ package

[GitHub] spark issue #21531: [SPARK-24521][SQL][TEST] Fix ineffective test in CachedT...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21531 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21050 **[Test build #91732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91732/testReport)** for PR 21050 at commit

[GitHub] spark issue #21531: [SPARK-24521][SQL][TEST] Fix ineffective test in CachedT...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21531 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91719/ Test PASSed. ---

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3955/

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/66/

[GitHub] spark issue #21531: [SPARK-24521][SQL][TEST] Fix ineffective test in CachedT...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21531 **[Test build #91719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91719/testReport)** for PR 21531 at commit

[GitHub] spark pull request #21050: [SPARK-23912][SQL]add array_distinct

2018-06-12 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/21050#discussion_r194893615 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1882,3 +1883,134 @@ case class

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3807/ ---

[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...

2018-06-12 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21398 @vanzin @cloud-fan. Going back to the facts. The fix/work-around in Cloudera's distro obviously exists because the distro integrates with Sentry anyway. Removing the location did fix the Spark Job

[GitHub] spark issue #21545: [SPARK-23010][BUILD][FOLLOWUP] Fix java checkstyle failu...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21545 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3806/ ---

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3807/ ---

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/65/

[GitHub] spark issue #21545: [SPARK-23010][BUILD][FOLLOWUP] Fix java checkstyle failu...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21545 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3806/ ---

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3804/ ---

[GitHub] spark pull request #21503: [SPARK-24478][SQL] Move projection and filter pus...

2018-06-12 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21503#discussion_r194875888 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala --- @@ -17,15 +17,56 @@ package

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3954/

[GitHub] spark issue #21545: [SPARK-23010][BUILD][FOLLOWUP] Fix java checkstyle failu...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21545 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21545: [SPARK-23010][BUILD][FOLLOWUP] Fix java checkstyle failu...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21545 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/64/

[GitHub] spark pull request #21503: [SPARK-24478][SQL] Move projection and filter pus...

2018-06-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21503#discussion_r194871032 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala --- @@ -17,15 +17,56 @@ package

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3951/

[GitHub] spark issue #21545: [SPARK-23010][BUILD][FOLLOWUP] Fix java checkstyle failu...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21545 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3953/

[GitHub] spark issue #21545: [SPARK-23010][BUILD][FOLLOWUP] Fix java checkstyle failu...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21545 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21543: [SPARK-24531][TESTS] Replace 2.3.0 version with 2.3.1

2018-06-12 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21543 cc @vanzin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3804/ ---

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3803/ ---

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91731/testReport)** for PR 21366 at commit

[GitHub] spark issue #21545: [SPARK-23010][BUILD][FOLLOWUP] Fix java checkstyle failu...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21545 **[Test build #91730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91730/testReport)** for PR 21545 at commit

[GitHub] spark issue #21504: [SPARK-24479][SS] Added config for registering streaming...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21504 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91715/ Test FAILed. ---

[GitHub] spark issue #21504: [SPARK-24479][SS] Added config for registering streaming...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21504 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21504: [SPARK-24479][SS] Added config for registering streaming...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21504 **[Test build #91715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91715/testReport)** for PR 21504 at commit

[GitHub] spark pull request #21545: [SPARK-23010][BUILD] Fix java checkstyle failure ...

2018-06-12 Thread jiangxb1987
GitHub user jiangxb1987 opened a pull request: https://github.com/apache/spark/pull/21545 [SPARK-23010][BUILD] Fix java checkstyle failure of kubernetes-integration-tests ## What changes were proposed in this pull request? Fix java checkstyle failure of

[GitHub] spark issue #21544: add one supported type missing from the javadoc

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21544 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21544: add one supported type missing from the javadoc

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21544 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21544: add one supported type missing from the javadoc

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21544 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21529: [SPARK-24495][SQL] EnsureRequirement returns wron...

2018-06-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21529#discussion_r194863357 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala --- @@ -679,6 +679,17 @@ class PlannerSuite extends SharedSQLContext

[GitHub] spark pull request #21544: add one supported type missing from the javadoc

2018-06-12 Thread yuj
GitHub user yuj opened a pull request: https://github.com/apache/spark/pull/21544 add one supported type missing from the javadoc ## What changes were proposed in this pull request? The supported java.math.BigInteger type is not mentioned in the javadoc of Encoders.bean()

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_from_arrays function

2018-06-12 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21258 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21543: [SPARK-24531][TESTS] Replace 2.3.0 version with 2.3.1

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21543 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/63/

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-12 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21529 > I think we need to support both. testing different physical operators needs same result, testing something like type coercion mode needs different result. Anyway let's discuss it in the

[GitHub] spark issue #21543: [SPARK-24531][TESTS] Replace 2.3.0 version with 2.3.1

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21543 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21543: [SPARK-24531][TESTS] Replace 2.3.0 version with 2.3.1

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21543 **[Test build #91729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91729/testReport)** for PR 21543 at commit

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/62/

[GitHub] spark pull request #21067: [SPARK-23980][K8S] Resilient Spark driver on Kube...

2018-06-12 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21067#discussion_r194862373 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala --- @@ -67,12 +68,19 @@

[GitHub] spark pull request #21067: [SPARK-23980][K8S] Resilient Spark driver on Kube...

2018-06-12 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21067#discussion_r194862246 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala --- @@ -67,12 +68,19 @@

[GitHub] spark issue #21543: [SPARK-24531][TESTS] Replace 2.3.0 version with 2.3.1

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21543 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3952/

[GitHub] spark issue #21543: [SPARK-24531][TESTS] Replace 2.3.0 version with 2.3.1

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21543 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21529: [SPARK-24495][SQL] EnsureRequirement returns wron...

2018-06-12 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21529#discussion_r194861969 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala --- @@ -679,6 +679,17 @@ class PlannerSuite extends SharedSQLContext {

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21258: [SPARK-23933][SQL] Add map_from_arrays function

2018-06-12 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21258 Thanks! merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21503: [SPARK-24478][SQL] Move projection and filter pus...

2018-06-12 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21503#discussion_r194861645 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala --- @@ -17,15 +17,56 @@ package

[GitHub] spark pull request #21543: [SPARK-24531][TESTS] Replace 2.3.0 version with 2...

2018-06-12 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/21543 [SPARK-24531][TESTS] Replace 2.3.0 version with 2.3.1 ## What changes were proposed in this pull request? The PR updates the 2.3 version tested to the new release 2.3.1. ## How

[GitHub] spark pull request #21067: [SPARK-23980][K8S] Resilient Spark driver on Kube...

2018-06-12 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21067#discussion_r194860793 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KubernetesFeatureConfigStep.scala --- @@ -18,54 +18,87 @@

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3803/ ---

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3950/

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21503: [SPARK-24478][SQL] Move projection and filter pus...

2018-06-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21503#discussion_r194858392 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala --- @@ -17,15 +17,56 @@ package

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91728/testReport)** for PR 21366 at commit

[GitHub] spark issue #21276: [SPARK-24216][SQL] Spark TypedAggregateExpression uses g...

2018-06-12 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21276 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21276: [SPARK-24216][SQL] Spark TypedAggregateExpression...

2018-06-12 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21276 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91727/testReport)** for PR 21366 at commit

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/61/

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-12 Thread mccheah
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21366 Latest change removes the dependency on rxjava. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use level triggering and state...

2018-06-12 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r194856403 --- Diff: pom.xml --- @@ -760,6 +760,12 @@ 1.10.19 test + --- End diff -- I ended up just

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds arrays_zip function to sp...

2018-06-12 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21045 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-12 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21527 I'm totally onboard with exposing it, though I'd say undocumented unless we have some concrete advice on how to set it. I was just explaining my experience so far and why I haven't bothered with

[GitHub] spark pull request #21475: [SPARK-24416] Fix configuration specification for...

2018-06-12 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21475 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds arrays_zip function to sparksql

2018-06-12 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21045 Thanks! merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21475: [SPARK-24416] Fix configuration specification for killBl...

2018-06-12 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21475 merged to master. Thanks @redsanket ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20887: [SPARK-23774][SQL] `Cast` to CHAR/VARCHAR should ...

2018-06-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20887#discussion_r194850161 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -2792,4 +2793,40 @@ class SQLQuerySuite extends QueryTest with

[GitHub] spark issue #21498: [SPARK-24410][SQL][Core] Optimization for Union outputPa...

2018-06-12 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21498 In theory, this should be done in a cost-based style. Changing the way how union combines data will reduce the parallelism. For example, if we union 2 tables each has 5 partitions.

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21508 **[Test build #91725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91725/testReport)** for PR 21508 at commit

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21050 **[Test build #91726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91726/testReport)** for PR 21050 at commit

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/60/

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3949/

[GitHub] spark pull request #21505: [SPARK-24457][SQL] Improving performance of strin...

2018-06-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21505#discussion_r194848108 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -111,6 +113,22 @@ object DateTimeUtils {

[GitHub] spark pull request #21505: [SPARK-24457][SQL] Improving performance of strin...

2018-06-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21505#discussion_r194847824 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/StringToTimestampBenchmark.scala --- @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...

2018-06-12 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/20641 @krcz which one? Btw both job run successfully although might not have been escaped properly. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21529 @mgaido91 Could you help improve the test coverage of joins `org.apache.spark.sql.JoinSuite`? Due to the incomplete test case coverage, we did not discover this at the very beginning. We need

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-12 Thread markhamstra
Github user markhamstra commented on the issue: https://github.com/apache/spark/pull/21527 > we can definitely update the description with more details. Eventually, some of the motivation and advice/suggestions need to get into the main user docs, as well. ---

[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...

2018-06-12 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20636 SGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21536: [MINOR][CORE][TEST] Remove unnecessary sort in UnsafeInM...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21536 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21527 **[Test build #91724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91724/testReport)** for PR 21527 at commit

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-12 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21527 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-12 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21527 we can definitely update the description with more details. Personally I'm not fond of any hardcoded magic number like this that you could override with at least a internal config

[GitHub] spark issue #21536: [MINOR][CORE][TEST] Remove unnecessary sort in UnsafeInM...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21536 **[Test build #91721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91721/testReport)** for PR 21536 at commit

[GitHub] spark issue #21536: [MINOR][CORE][TEST] Remove unnecessary sort in UnsafeInM...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21536 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91721/ Test FAILed. ---

[GitHub] spark pull request #21503: [SPARK-24478][SQL] Move projection and filter pus...

2018-06-12 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21503#discussion_r194841328 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala --- @@ -17,15 +17,56 @@ package

[GitHub] spark pull request #21537: [SPARK-24505][SQL] Convert strings in codegen to ...

2018-06-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21537#discussion_r194841055 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -579,6 +579,22 @@ class

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use level triggering and state...

2018-06-12 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r194840955 --- Diff: pom.xml --- @@ -760,6 +760,12 @@ 1.10.19 test + --- End diff -- These types are not

<    1   2   3   4   5   6   7   8   >