[GitHub] spark issue #21250: [SPARK-23291][SQL][R][BRANCH-2.3] R's substr should not ...

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21250 **[Test build #90325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90325/testReport)** for PR 21250 at commit

[GitHub] spark issue #21249: [SPARK-23291][R][FOLLOWUP] Update SparkR migration note ...

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21249 **[Test build #90326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90326/testReport)** for PR 21249 at commit

[GitHub] spark pull request #20787: [MINOR][DOCS] Documenting months_between directio...

2018-05-07 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20787#discussion_r186458260 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1194,13 +1194,21 @@ case class

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-07 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r186455988 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -28,6 +30,34 @@ import

[GitHub] spark pull request #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to b...

2018-05-07 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21122#discussion_r186452677 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -1354,7 +1354,8 @@ class HiveDDLSuite val

[GitHub] spark issue #20929: [SPARK-23772][SQL][WIP] Provide an option to ignore colu...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20929 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3004/

[GitHub] spark issue #20929: [SPARK-23772][SQL][WIP] Provide an option to ignore colu...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20929 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20929: [SPARK-23772][SQL][WIP] Provide an option to ignore colu...

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20929 **[Test build #90324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90324/testReport)** for PR 20929 at commit

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19602 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3003/

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19602 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19602 **[Test build #90323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90323/testReport)** for PR 19602 at commit

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2018-05-07 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/19602 I rebased this pr and resolved conflicts. cc @cloud-fan @jiangxb1987 Not sure if you have interest on this. Take a look if have time. Thanks ! ---

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-07 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r186445741 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -28,6 +30,34 @@ import

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-07 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r186445302 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -28,6 +30,34 @@ import

[GitHub] spark issue #21248: [SPARK-24191][ML]Example code for Power Iteration Cluste...

2018-05-07 Thread shahidki31
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/21248 cc @mengxr @WeichenXu123 @felixcheung. Can you please verify this patch? --- - To unsubscribe, e-mail:

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21028 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3002/

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21028 **[Test build #90322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90322/testReport)** for PR 21028 at commit

[GitHub] spark issue #21250: [SPARK-23291][SQL][R][BRANCH-2.3] R's substr should not ...

2018-05-07 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21250 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21250: [SPARK-23291][SQL][R][BRANCH-2.3] R's substr shou...

2018-05-07 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21250#discussion_r186438139 --- Diff: docs/sparkr.md --- @@ -663,3 +663,7 @@ You can inspect the search path in R with [`search()`](https://stat.ethz.ch/R-ma - The

[GitHub] spark issue #20787: [MINOR][DOCS] Documenting months_between direction

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20787 **[Test build #90321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90321/testReport)** for PR 20787 at commit

[GitHub] spark pull request #20787: [MINOR][DOCS] Documenting months_between directio...

2018-05-07 Thread aditkumar
Github user aditkumar commented on a diff in the pull request: https://github.com/apache/spark/pull/20787#discussion_r186436718 --- Diff: R/pkg/R/functions.R --- @@ -1906,6 +1906,7 @@ setMethod("atan2", signature(y = "Column"), #' @details #' \code{datediff}:

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186433318 --- Diff: python/pyspark/sql/functions.py --- @@ -1798,6 +1798,22 @@ def create_map(*cols): return Column(jc) +@ignore_unicode_prefix

[GitHub] spark issue #21144: [SPARK-24043][SQL] Interpreted Predicate should initiali...

2018-05-07 Thread bersprockets
Github user bersprockets commented on the issue: https://github.com/apache/spark/pull/21144 @hvanhovell @maropu Is there anything on this PR that I should do? --- - To unsubscribe, e-mail:

[GitHub] spark issue #21252: [SPARK-24193] Sort by disk when number of limit is big i...

2018-05-07 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/21252 I will add a suite tomorrow. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...

2018-05-07 Thread andrusha
Github user andrusha commented on the issue: https://github.com/apache/spark/pull/21238 Seems like it addresses similar problem to https://github.com/apache/spark/pull/21095. It might be worth investigating how to unify both. ---

[GitHub] spark issue #21106: [SPARK-23711][SQL][WIP] Add fallback logic for UnsafePro...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21106 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3001/

[GitHub] spark issue #21106: [SPARK-23711][SQL][WIP] Add fallback logic for UnsafePro...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21106 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-05-07 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21082 Hi @ueshin @hvanhovell can you take a look at this please? Thanks! --- - To unsubscribe, e-mail:

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #90320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90320/testReport)** for PR 21073 at commit

[GitHub] spark issue #16478: [SPARK-7768][SQL] Revise user defined types (UDT)

2018-05-07 Thread metasim
Github user metasim commented on the issue: https://github.com/apache/spark/pull/16478 Does this PR remove the ability to read directly from Tungsten memory, in cases where one wants to avoid the serialization/deserialization costs? IOW, in an `Expression` over a UDT, can one still

[GitHub] spark issue #21236: [SPARK-23935][SQL] Adding map_entries function

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21236 **[Test build #90318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90318/testReport)** for PR 21236 at commit

[GitHub] spark issue #21106: [SPARK-23711][SQL][WIP] Add fallback logic for UnsafePro...

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21106 **[Test build #90319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90319/testReport)** for PR 21106 at commit

[GitHub] spark pull request #20604: [SPARK-23365][CORE] Do not adjust num executors w...

2018-05-07 Thread Ngone51
Github user Ngone51 commented on a diff in the pull request: https://github.com/apache/spark/pull/20604#discussion_r186425765 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1643,7 +1646,10 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark issue #21096: [SPARK-24011][CORE][WIP] cache rdd's immediate parent Sh...

2018-05-07 Thread Ngone51
Github user Ngone51 commented on the issue: https://github.com/apache/spark/pull/21096 Thanks for your opinions @squito @markhamstra . Maybe, I should leave it for now. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21209: [SPARK-24141][CORE] Fix bug in CoarseGrainedSchedulerBac...

2018-05-07 Thread Ngone51
Github user Ngone51 commented on the issue: https://github.com/apache/spark/pull/21209 ping @jiangxb1987 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21252: [SPARK-24193] Sort by disk when number of limit is big i...

2018-05-07 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21252 We can have a test to check the physical plan under different threshold. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21252: [SPARK-24193] Sort by disk when number of limit is big i...

2018-05-07 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21252 The changes looks good to me, but it should also be great to have a test suite to cover this change. Seems we don't have a test suite for the rule `SpecialLimits` ? ---

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186418197 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -235,6 +235,69 @@ case class

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-07 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r186417334 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -28,6 +30,34 @@ import

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-07 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r186417229 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -28,6 +30,34 @@ import

[GitHub] spark issue #21190: [SPARK-22938][SQL][followup] Assert that SQLConf.get is ...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21190 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3000/

[GitHub] spark issue #21190: [SPARK-22938][SQL][followup] Assert that SQLConf.get is ...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21190 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21190: [SPARK-22938][SQL][followup] Assert that SQLConf.get is ...

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21190 **[Test build #90317 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90317/testReport)** for PR 21190 at commit

[GitHub] spark issue #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2999/

[GitHub] spark issue #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21258 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21250: [SPARK-23291][SQL][R][BRANCH-2.3] R's substr shou...

2018-05-07 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21250#discussion_r186413375 --- Diff: docs/sparkr.md --- @@ -663,3 +663,7 @@ You can inspect the search path in R with [`search()`](https://stat.ethz.ch/R-ma - The

[GitHub] spark issue #21252: [SPARK-24193] Sort by disk when number of limit is big i...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21252 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21252: [SPARK-24193] Sort by disk when number of limit is big i...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21252 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2998/

[GitHub] spark issue #18581: [SPARK-21289][SQL][ML] Supports custom line separator fo...

2018-05-07 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18581 Nope not yet, I will try to make it within the next release soon. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21249: [SPARK-23291][R][FOLLOWUP] Update SparkR migration note ...

2018-05-07 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21249 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21106: [SPARK-23711][SQL][WIP] Add fallback logic for UnsafePro...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21106 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21258 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2997/

[GitHub] spark issue #21106: [SPARK-23711][SQL][WIP] Add fallback logic for UnsafePro...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21106 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2996/

[GitHub] spark pull request #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwr...

2018-05-07 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21257#discussion_r186411390 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala --- @@ -207,9 +207,25 @@ case class

[GitHub] spark pull request #21250: [SPARK-23291][SQL][R][BRANCH-2.3] R's substr shou...

2018-05-07 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21250#discussion_r186411141 --- Diff: docs/sparkr.md --- @@ -663,3 +663,7 @@ You can inspect the search path in R with [`search()`](https://stat.ethz.ch/R-ma - The

[GitHub] spark issue #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21258 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186408884 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -405,6 +405,7 @@ object FunctionRegistry {

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186410527 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -235,6 +235,69 @@ case class

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186408382 --- Diff: python/pyspark/sql/functions.py --- @@ -1798,6 +1798,22 @@ def create_map(*cols): return Column(jc)

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186410991 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -1033,6 +1033,17 @@ object functions { @scala.annotation.varargs

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186409077 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -235,6 +235,69 @@ case class

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186408332 --- Diff: python/pyspark/sql/functions.py --- @@ -1798,6 +1798,22 @@ def create_map(*cols): return Column(jc)

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186410288 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -235,6 +235,69 @@ case class

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186410897 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala --- @@ -186,6 +186,37 @@ class ComplexTypeSuite

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186410190 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -235,6 +235,69 @@ case class

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r186409860 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -235,6 +235,69 @@ case class

[GitHub] spark issue #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2995/

[GitHub] spark issue #18581: [SPARK-21289][SQL][ML] Supports custom line separator fo...

2018-05-07 Thread beickhoff
Github user beickhoff commented on the issue: https://github.com/apache/spark/pull/18581 @HyukjinKwon, is there another PR to handle CSV? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #21250: [SPARK-23291][SQL][R][BRANCH-2.3] R's substr shou...

2018-05-07 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21250#discussion_r186410353 --- Diff: docs/sparkr.md --- @@ -663,3 +663,7 @@ You can inspect the search path in R with [`search()`](https://stat.ethz.ch/R-ma - The

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16677 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2994/

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16677 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21258 **[Test build #90312 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90312/testReport)** for PR 21258 at commit

[GitHub] spark issue #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21258 **[Test build #90316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90316/testReport)** for PR 21258 at commit

[GitHub] spark issue #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21258 **[Test build #90314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90314/testReport)** for PR 21258 at commit

[GitHub] spark issue #21252: [SPARK-24193] Sort by disk when number of limit is big i...

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21252 **[Test build #90315 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90315/testReport)** for PR 21252 at commit

[GitHub] spark issue #21106: [SPARK-23711][SQL][WIP] Add fallback logic for UnsafePro...

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21106 **[Test build #90313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90313/testReport)** for PR 21106 at commit

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16677 **[Test build #90311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90311/testReport)** for PR 16677 at commit

[GitHub] spark pull request #21250: [SPARK-23291][SQL][R][BRANCH-2.3] R's substr shou...

2018-05-07 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21250#discussion_r186407992 --- Diff: docs/sparkr.md --- @@ -663,3 +663,7 @@ You can inspect the search path in R with [`search()`](https://stat.ethz.ch/R-ma - The

[GitHub] spark issue #21256: [SPARK-24160][FOLLOWUP] Fix compilation failure

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21256 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21256: [SPARK-24160][FOLLOWUP] Fix compilation failure

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21256 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90300/ Test PASSed. ---

[GitHub] spark issue #21256: [SPARK-24160][FOLLOWUP] Fix compilation failure

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21256 **[Test build #90300 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90300/testReport)** for PR 21256 at commit

[GitHub] spark issue #16478: [SPARK-7768][SQL] Revise user defined types (UDT)

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16478 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90307/ Test FAILed. ---

[GitHub] spark issue #16478: [SPARK-7768][SQL] Revise user defined types (UDT)

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16478 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #16478: [SPARK-7768][SQL] Revise user defined types (UDT)

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16478 **[Test build #90307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90307/testReport)** for PR 16478 at commit

[GitHub] spark issue #21251: [SPARK-10878][core] Fix race condition when multiple cli...

2018-05-07 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21251 cc @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21240: [SPARK-21274][SQL] Add a new generator function replicat...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21240 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90305/ Test PASSed. ---

[GitHub] spark issue #21240: [SPARK-21274][SQL] Add a new generator function replicat...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21240 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21240: [SPARK-21274][SQL] Add a new generator function replicat...

2018-05-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21240 **[Test build #90305 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90305/testReport)** for PR 21240 at commit

[GitHub] spark issue #21252: [SPARK-24193] Sort by disk when number of limit is big i...

2018-05-07 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/21252 @cloud-fan @viirya Thanks for comments. I refined accordingly. Please check~ --- - To unsubscribe, e-mail:

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-07 Thread zheh12
Github user zheh12 commented on the issue: https://github.com/apache/spark/pull/21257 cc @rxin @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #13493: [SPARK-15750][MLLib][PYSPARK] Constructing FPGrowth fail...

2018-05-07 Thread WeichenXu123
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/13493 LGTM! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21258 I prepared this function using the different name from `map` in Presto. Any good idea (e.g. other names or another implemention in `map`) would be appreciated. ---

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_fromarray function

2018-05-07 Thread kiszk
GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/21258 [SPARK-23933][SQL] Add map_fromarray function ## What changes were proposed in this pull request? The PR adds the SQL function `map_fromarray`. The behavior of the function is based on

[GitHub] spark issue #21257: [SPARK-24194] HadoopFsRelation cannot overwrite a path t...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21257 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21257: [SPARK-24194] HadoopFsRelation cannot overwrite a path t...

2018-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21257 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21257: [SPARK-24194] HadoopFsRelation cannot overwrite a...

2018-05-07 Thread zheh12
GitHub user zheh12 opened a pull request: https://github.com/apache/spark/pull/21257 [SPARK-24194] HadoopFsRelation cannot overwrite a path that is also b… ## What changes were proposed in this pull request? When insert overwrite in a parquet table. There will be a error

[GitHub] spark pull request #20095: [SPARK-22126][ML] Added fitMultiple method with d...

2018-05-07 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/20095#discussion_r186381507 --- Diff: mllib/src/main/scala/org/apache/spark/ml/Estimator.scala --- @@ -79,7 +82,52 @@ abstract class Estimator[M <: Model[M]] extends

[GitHub] spark pull request #21236: [SPARK-23935][SQL] Adding map_entries function

2018-05-07 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21236#discussion_r186377811 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -118,6 +118,162 @@ case class

<    1   2   3   4   5   6   7   >