[GitHub] spark issue #21333: [SPARK-23778][CORE] Avoid unneeded shuffle when union ge...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21333 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21329: [SPARK-24277][SQL] Code clean up in SQL module: HadoopMa...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21329 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21329: [SPARK-24277][SQL] Code clean up in SQL module: HadoopMa...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21329 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90639/ Test FAILed. ---

[GitHub] spark issue #21329: [SPARK-24277][SQL] Code clean up in SQL module: HadoopMa...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21329 **[Test build #90639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90639/testReport)** for PR 21329 at commit

[GitHub] spark issue #21333: [SPARK-23778][CORE] Avoid unneeded shuffle when union ge...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21333 **[Test build #90647 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90647/testReport)** for PR 21333 at commit

[GitHub] spark pull request #21333: [SPARK-23778][CORE] Avoid unneeded shuffle when u...

2018-05-15 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/21333 [SPARK-23778][CORE] Avoid unneeded shuffle when union gets an empty RDD ## What changes were proposed in this pull request? When a `union` is invoked on several RDDs of which one is an

[GitHub] spark pull request #21322: [SPARK-24225][CORE] Support closing AutoClosable ...

2018-05-15 Thread JeetKunDoug
Github user JeetKunDoug commented on a diff in the pull request: https://github.com/apache/spark/pull/21322#discussion_r188314886 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -384,15 +385,36 @@ private[spark] class MemoryStore( }

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21332 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21332 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90645/ Test FAILed. ---

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21332 **[Test build #90645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90645/testReport)** for PR 21332 at commit

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21291 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21291 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3235/

[GitHub] spark issue #21322: [SPARK-24225][CORE] Support closing AutoClosable objects...

2018-05-15 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21322 @JeetKunDoug The same issue we discuss above. I think if there's a deserialized version of the variable, it can be not in `MemoryStore` but only serialized bytes in disk store. The reason

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21291 **[Test build #90646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90646/testReport)** for PR 21291 at commit

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/21332 cc @jose-torres As we discussion in #21293, the main difference between us is whether we can reuse current implementation of scheduler and shuffle, but in this part about the

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21332 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21332 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3234/

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21332 **[Test build #90645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90645/testReport)** for PR 21332 at commit

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-15 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21291 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21332: [SPARK-24236][SS] Continuous replacement for Shuf...

2018-05-15 Thread xuanyuanking
GitHub user xuanyuanking opened a pull request: https://github.com/apache/spark/pull/21332 [SPARK-24236][SS] Continuous replacement for ShuffleExchangeExec ## What changes were proposed in this pull request? 1. New RDD named ContinuousShuffleRowRDD 2. New case class

[GitHub] spark issue #21322: [SPARK-24225][CORE] Support closing AutoClosable objects...

2018-05-15 Thread JeetKunDoug
Github user JeetKunDoug commented on the issue: https://github.com/apache/spark/pull/21322 @viirya it seems from my admittedly cursory look at where we use the `cachedValues` reference map that we should be OK in this case - if there's a deserialized version of the variable (the only

[GitHub] spark pull request #21322: [SPARK-24225][CORE] Support closing AutoClosable ...

2018-05-15 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21322#discussion_r188306362 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -384,15 +385,36 @@ private[spark] class MemoryStore( }

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-05-15 Thread zecevicp
Github user zecevicp commented on the issue: https://github.com/apache/spark/pull/21109 Btw, thank you @mgaido91 and @kiszk for the comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...

2018-05-15 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21312 Thanks @HyukjinKwon @BryanCutler @cloud-fan @icexelloss --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...

2018-05-15 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21312 thanks, merging to master/2.3! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces...

2018-05-15 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21312 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21286: [SPARK-24238][SQL] HadoopFsRelation can't append the sam...

2018-05-15 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/21286 Does Spark have a jobID in writing path? Below path is an example in my debugging log: ```

[GitHub] spark pull request #21319: [SPARK-24267][SQL] explicitly keep DataSourceRead...

2018-05-15 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21319#discussion_r188299230 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala --- @@ -22,78 +22,57 @@ import

[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21319 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3233/

[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21319 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21321 cc @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21322: [SPARK-24225][CORE] Support closing AutoClosable ...

2018-05-15 Thread JeetKunDoug
Github user JeetKunDoug commented on a diff in the pull request: https://github.com/apache/spark/pull/21322#discussion_r188295537 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -384,15 +385,36 @@ private[spark] class MemoryStore( }

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-05-15 Thread zecevicp
Github user zecevicp commented on the issue: https://github.com/apache/spark/pull/21109 Well, that is the essence of the contribution: to have a moving window over the data, instead of a fixed block (per equi-join match). To implement a moving window you need something like a queue.

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21321 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21321 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90638/ Test PASSed. ---

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21321 **[Test build #90638 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90638/testReport)** for PR 21321 at commit

[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21319 **[Test build #90644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90644/testReport)** for PR 21319 at commit

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21257 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90636/ Test PASSed. ---

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21257 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21257 **[Test build #90636 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90636/testReport)** for PR 21257 at commit

[GitHub] spark issue #21045: [WIP][SPARK-23931][SQL] Adds zip function to sparksql

2018-05-15 Thread mn-mikke
Github user mn-mikke commented on the issue: https://github.com/apache/spark/pull/21045 @DylanGuedes What about `eval.value`? Example: ``` val evals = children.map(_.genCode(ctx)) val args = ctx.freshName("args") val inputs = evals.zipWithIndex.map { case

[GitHub] spark issue #21289: [SPARK-24240] Add a config to control whether InMemoryFi...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21289 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90637/ Test PASSed. ---

[GitHub] spark pull request #21319: [SPARK-24267][SQL] explicitly keep DataSourceRead...

2018-05-15 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21319#discussion_r188289317 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownOperatorsToDataSource.scala --- @@ -17,48 +17,81 @@

[GitHub] spark issue #21289: [SPARK-24240] Add a config to control whether InMemoryFi...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21289 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21289: [SPARK-24240] Add a config to control whether InMemoryFi...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21289 **[Test build #90637 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90637/testReport)** for PR 21289 at commit

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188287355 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,88 @@ case class

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188286751 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,88 @@ case class

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21109 @zecevicp for instance do we really need `InMemoryUnsafeRowQueue`? why `ExternalAppendOnlyUnsafeRowArray` is not ok? --- - To

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188283661 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,110 @@ case class

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188283550 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,110 @@ case class

[GitHub] spark issue #21165: [Spark-20087][CORE] Attach accumulators / metrics to 'Ta...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21165 **[Test build #90643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90643/testReport)** for PR 21165 at commit

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 Thank you so much for the suggestions! I tried to use IntelliJ a few times but at the end I always return to sbt/terminal/vim after some frustration (mainly due to not being able to configure

[GitHub] spark issue #21293: [SPARK-24237][SS] Continuous shuffle dependency and map ...

2018-05-15 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/21293 @jose-torres Great thanks for you advise and guidance for us! I found the main difference between us is whether we can reuse current implementation of scheduler and shuffle. I marked in your

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-05-15 Thread zecevicp
Github user zecevicp commented on the issue: https://github.com/apache/spark/pull/21109 @mgaido91 Regarding the amount of code, maybe you can suggest how to reduce it? Because I don't see a way... I think the code is well contained (mostly in separate new classes) and is not

[GitHub] spark pull request #21293: [SPARK-24237][SS] Continuous shuffle dependency a...

2018-05-15 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21293#discussion_r188277683 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ContinuousShuffleMapTask.scala --- @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21109 **[Test build #90642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90642/testReport)** for PR 21109 at commit

[GitHub] spark pull request #21293: [SPARK-24237][SS] Continuous shuffle dependency a...

2018-05-15 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21293#discussion_r188273722 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -769,6 +796,43 @@ private[spark] class MapOutputTrackerWorker(conf:

[GitHub] spark issue #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP PARTITIO...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19691 @DazhuangSu are you still working on this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21293: [SPARK-24237][SS] Continuous shuffle dependency a...

2018-05-15 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21293#discussion_r188270290 --- Diff: core/src/main/scala/org/apache/spark/Dependency.scala --- @@ -88,14 +90,53 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] spark issue #21331: [SPARK-24276][SQL] Order of literals in IN should not af...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21331 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3232/

[GitHub] spark pull request #21293: [SPARK-24237][SS] Continuous shuffle dependency a...

2018-05-15 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21293#discussion_r188269208 --- Diff: core/src/main/scala/org/apache/spark/Dependency.scala --- @@ -65,15 +65,17 @@ abstract class NarrowDependency[T](_rdd: RDD[T]) extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188269083 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -434,18 +511,139 @@ case class SortMergeJoinExec(

[GitHub] spark issue #21331: [SPARK-24276][SQL] Order of literals in IN should not af...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21331 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21331: [SPARK-24276][SQL] Order of literals in IN should not af...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21331 **[Test build #90641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90641/testReport)** for PR 21331 at commit

[GitHub] spark issue #18903: [SPARK-21590][SS]Window start time should support negati...

2018-05-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18903 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21286: [SPARK-24238][SQL] HadoopFsRelation can't append the sam...

2018-05-15 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/21286 > After the job is committed, `skip_dir/tab1/_temporary` will be deleted. Then when other jobs attempt to commit, an error will be reported. I see. Yes, that's

[GitHub] spark pull request #21331: [SPARK-24276][SQL] Order of literals in IN should...

2018-05-15 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/21331 [SPARK-24276][SQL] Order of literals in IN should not affect semantic equality ## What changes were proposed in this pull request? When two `In` operators are created with the same list

[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewExecu...

2018-05-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21316 Is it better to update title and the PR description, too? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188266461 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -434,18 +511,139 @@ case class SortMergeJoinExec(

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188265911 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -434,18 +511,139 @@ case class SortMergeJoinExec(

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188265529 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -434,18 +511,139 @@ case class SortMergeJoinExec(

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188264496 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark issue #21246: [SPARK-23901][SQL] Add masking functions

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21246 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90634/ Test PASSed. ---

[GitHub] spark pull request #21319: [SPARK-24267][SQL] explicitly keep DataSourceRead...

2018-05-15 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21319#discussion_r188255505 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownOperatorsToDataSource.scala --- @@ -17,48 +17,81 @@

[GitHub] spark pull request #21319: [SPARK-24267][SQL] explicitly keep DataSourceRead...

2018-05-15 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21319#discussion_r188255611 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownOperatorsToDataSource.scala --- @@ -17,48 +17,81 @@

[GitHub] spark issue #21246: [SPARK-23901][SQL] Add masking functions

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21246 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21319: [SPARK-24267][SQL] explicitly keep DataSourceRead...

2018-05-15 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21319#discussion_r188254452 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala --- @@ -22,78 +22,57 @@ import

[GitHub] spark issue #21246: [SPARK-23901][SQL] Add masking functions

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21246 **[Test build #90634 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90634/testReport)** for PR 21246 at commit

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90635/ Test PASSed. ---

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21028 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21028 **[Test build #90635 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90635/testReport)** for PR 21028 at commit

[GitHub] spark pull request #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwr...

2018-05-15 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/21257#discussion_r188262174 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -163,6 +170,15 @@ class

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21257 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21257 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90632/ Test PASSed. ---

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21257 **[Test build #90632 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90632/testReport)** for PR 21257 at commit

[GitHub] spark issue #21330: [SPARK-22234] Support distinct window functions

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21330 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3231/

[GitHub] spark issue #21330: [SPARK-22234] Support distinct window functions

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21330 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188258797 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark issue #21330: [SPARK-22234] Support distinct window functions

2018-05-15 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/21330 @cloud-fan @jiangxb1987 @cenyuhai Do you think this change makes sense? I can keep working on this. --- - To

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188258635 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark issue #21330: [SPARK-22234] Support distinct window functions

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21330 **[Test build #90640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90640/testReport)** for PR 21330 at commit

[GitHub] spark pull request #21330: [SPARK-22234] Support distinct window functions

2018-05-15 Thread jinxing64
GitHub user jinxing64 opened a pull request: https://github.com/apache/spark/pull/21330 [SPARK-22234] Support distinct window functions ## What changes were proposed in this pull request? This pr proposes to support distinct window functions. After this change, query like below

[GitHub] spark issue #21327: [SPARK-24107][CORE][followup] ChunkedByteBuffer.writeFul...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21327 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90628/ Test PASSed. ---

[GitHub] spark issue #21327: [SPARK-24107][CORE][followup] ChunkedByteBuffer.writeFul...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21327 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21327: [SPARK-24107][CORE][followup] ChunkedByteBuffer.writeFul...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21327 **[Test build #90628 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90628/testReport)** for PR 21327 at commit

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188254469 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188254449 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188254046 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188253781 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188253438 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -97,13 +100,13 @@ object PhysicalOperation extends

<    1   2   3   4   5   >