[GitHub] spark pull request #20543: [SPARK-23357][CORE] 'SHOW TABLE EXTENDED LIKE pat...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20543 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19758: [SPARK-3162][MLlib] Local Tree Training Pt 1: Ref...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19758 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18918: [SPARK-21707][SQL]Improvement a special case for ...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18918 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13143: [SPARK-15359] [Mesos] Mesos dispatcher should han...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13143 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21726: Branch 2.3

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21726 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17422: [SPARK-20087] Attach accumulators / metrics to 'T...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17422 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19510: [SPARK-22292][Mesos] Added spark.mem.max support ...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19510 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18268: [SPARK-21054] [SQL] Reset Command support reset s...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18268 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14653: [SPARK-10931][PYSPARK][ML] PySpark ML Models shou...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14653 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19274: [SPARK-22056][Streaming] Add subconcurrency for K...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19274 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20437: [SPARK-23270][Streaming][WEB-UI]FileInputDStream ...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20437 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20090: [SPARK-22907]Clean broadcast garbage when IOExcep...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20090 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20100: [SPARK-22913][SQL] Improved Hive Partition Prunin...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20100 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16910: [SPARK-19575][SQL]Reading from or writing to a hi...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16910 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21731: Update example to work locally

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21731 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21453: Test branch to see how Scala 2.11.12 performs

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21453 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12951: [SPARK-15176][Core] Add maxShares setting to Pool...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12951 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...

2018-07-18 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21777 btw, it seems this pr exceeds the current timeout... Any way to temporarily make the timeout longer? We always need to configure timeout in the Jenkins-side like https://github.com/apache/spark/

[GitHub] spark pull request #18477: [SPARK-21261][DOCS]SQL Regex document fix

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18477 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21781: [INFRA] Close stale PR

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21781 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21787: [SPARK-24568] Code refactoring for DataType equal...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21787 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21095: [SPARK-23529][K8s] Support mounting hostPath volu...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21095 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19233: [Spark-22008][Streaming]Spark Streaming Dynamic A...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19233 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21240: [SPARK-21274][SQL] Add a new generator function r...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21240 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12904: [SPARK-15125][SQL] Changing CSV data source mappi...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12904 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21455: [SPARK-24093][DStream][Minor]Make some fields of ...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21455 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21812: SPARK UI K8S : this parameter's illustration(spar...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21812 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21533: [SPARK-24195][Core] Bug fix for local:/ path in SparkCon...

2018-07-18 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21533 Please also update the title and PR description because we changed the proposed solution in the middle. --- - To unsubscribe

[GitHub] spark pull request #21638: [SPARK-22357][CORE] SparkContext.binaryFiles igno...

2018-07-18 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21638#discussion_r203589083 --- Diff: core/src/main/scala/org/apache/spark/input/PortableDataStream.scala --- @@ -47,7 +47,7 @@ private[spark] abstract class StreamFileInputFormat[

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-07-18 Thread markhamstra
Github user markhamstra commented on the issue: https://github.com/apache/spark/pull/21589 Thank you, @HyukjinKwon There are a significant number of Spark users who use the Job Scheduler model with a SparkContext shared across many users and many Jobs. Promoting tools and pa

[GitHub] spark pull request #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-07-18 Thread attilapiros
Github user attilapiros commented on a diff in the pull request: https://github.com/apache/spark/pull/21635#discussion_r203594956 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterSource.scala --- @@ -0,0 +1,49 @@ +/* + * License

[GitHub] spark issue #21795: [SPARK-24840][SQL] do not use dummy filter to switch cod...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21795 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21795: [SPARK-24840][SQL] do not use dummy filter to swi...

2018-07-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21795 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21698: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-07-18 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21698 > I see some discussion about making shuffles deterministic, but it proved to be very difficult. Is there a prior discussion on this you can point me to? Is it that even if you used fetch-to-dis

[GitHub] spark issue #20856: [SPARK-23731][SQL] FileSourceScanExec throws NullPointer...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20856 @HyukjinKwon good analysis! Currently Spark is a little messy about what shall be serialized and sent to executors. Sometimes we just send an entire query tree but only read a few propert

[GitHub] spark issue #21698: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21698 > checkpoint can not guarantee that you shall always get the same output ... IIRC we can checkpoint to HDFS? Then it becomes reliable. --- ---

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-18 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21758 @mridulm Sorry I missed that message, now I've updated the comment, we can continue the discussion on that thread. --- - To

[GitHub] spark issue #21698: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-07-18 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21698 > > checkpoint can not guarantee that you shall always get the same output ... > > IIRC we can checkpoint to HDFS? Then it becomes reliable. Sure, thanks for clarify on that.

[GitHub] spark issue #21732: [SPARK-24762][SQL] Aggregator should be able to use Opti...

2018-07-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21732 ping @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@

[GitHub] spark pull request #21813: [SPARK 24424] Support ANSI-SQL compliant syntax f...

2018-07-18 Thread dilipbiswal
GitHub user dilipbiswal opened a pull request: https://github.com/apache/spark/pull/21813 [SPARK 24424] Support ANSI-SQL compliant syntax for GROUPING SET ## What changes were proposed in this pull request? Enhances the parser and analyzer to support ANSI compliant syntax fo

[GitHub] spark issue #21813: [SPARK 24424][SQL] Support ANSI-SQL compliant syntax for...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21813 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21813: [SPARK 24424][SQL] Support ANSI-SQL compliant syntax for...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21813 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified//

[GitHub] spark issue #21813: [SPARK 24424][SQL] Support ANSI-SQL compliant syntax for...

2018-07-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21813 **[Test build #93260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93260/testReport)** for PR 21813 at commit [`b5ada3f`](https://github.com/apache/spark/commit/b5

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-07-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21469 **[Test build #93257 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93257/testReport)** for PR 21469 at commit [`32d0418`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21469 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21469 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93257/ Test PASSed. ---

[GitHub] spark pull request #21782: [SPARK-24816][SQL] SQL interface support repartit...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21782#discussion_r203604973 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala --- @@ -394,6 +394,41 @@ class FilterPushdownBen

[GitHub] spark issue #21700: [SPARK-24717][SS] Split out max retain version of state ...

2018-07-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21700 **[Test build #93256 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93256/testReport)** for PR 21700 at commit [`cf78a2a`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #21700: [SPARK-24717][SS] Split out max retain version of state ...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93256/ Test PASSed. ---

[GitHub] spark issue #21700: [SPARK-24717][SS] Split out max retain version of state ...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21700 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-07-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20838 **[Test build #93253 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93253/testReport)** for PR 20838 at commit [`2c4f15c`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20838 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93253/ Test PASSed. ---

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20838 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21774: [SPARK-24811][SQL]Avro: add new function from_avr...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21774#discussion_r203606284 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala --- @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Sof

[GitHub] spark pull request #21774: [SPARK-24811][SQL]Avro: add new function from_avr...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21774#discussion_r203606677 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala --- @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Sof

[GitHub] spark pull request #21774: [SPARK-24811][SQL]Avro: add new function from_avr...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21774#discussion_r203606783 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala --- @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Sof

[GitHub] spark pull request #21774: [SPARK-24811][SQL]Avro: add new function from_avr...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21774#discussion_r203606818 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/CatalystDataToAvro.scala --- @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Sof

[GitHub] spark pull request #21774: [SPARK-24811][SQL]Avro: add new function from_avr...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21774#discussion_r203606947 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/package.scala --- @@ -36,4 +40,27 @@ package object avro { @scala.annotation.var

[GitHub] spark pull request #21774: [SPARK-24811][SQL]Avro: add new function from_avr...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21774#discussion_r203607394 --- Diff: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroCatalystDataConversionSuite.scala --- @@ -0,0 +1,175 @@ +/* + * Licensed to

[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21774 Since Spark doesn't have a persistent UDF API like Hive UDF, I think this is the best we can do now. In the future we should migrate this to UDF API so that we can register it with a name and use

[GitHub] spark issue #21739: [SPARK-22187][SS] Update unsaferow format for saved stat...

2018-07-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21739 **[Test build #93258 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93258/testReport)** for PR 21739 at commit [`c262e87`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-07-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21451 **[Test build #93255 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93255/testReport)** for PR 21451 at commit [`335e26d`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #21739: [SPARK-22187][SS] Update unsaferow format for saved stat...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21739 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93258/ Test PASSed. ---

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21451 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93255/ Test PASSed. ---

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21451 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21739: [SPARK-22187][SS] Update unsaferow format for saved stat...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21739 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21789: [SPARK-24829][SQL]In Spark Thrift Server, CAST AS...

2018-07-18 Thread zuotingbing
Github user zuotingbing commented on a diff in the pull request: https://github.com/apache/spark/pull/21789#discussion_r203610279 --- Diff: sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/Column.java --- @@ -349,7 +349,7 @@ public void addValue(Type type, Object fie

[GitHub] spark pull request #21608: [SPARK-24626] [SQL] Improve location size calcula...

2018-07-18 Thread Achuth17
Github user Achuth17 commented on a diff in the pull request: https://github.com/apache/spark/pull/21608#discussion_r203613041 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -148,6 +148,19 @@ class StatisticsSuite extends StatisticsCollect

[GitHub] spark issue #21732: [SPARK-24762][SQL] Aggregator should be able to use Opti...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21732 > Non top-level and top-level encoders for Option[Product] have a little difference. Can we treat them the same but at the end of encoder creation, we flatten the `Option[Product]`? ---

[GitHub] spark pull request #21782: [SPARK-24816][SQL] SQL interface support repartit...

2018-07-18 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/21782#discussion_r203613170 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala --- @@ -394,6 +394,41 @@ class FilterPushdownBench

[GitHub] spark pull request #21789: [SPARK-24829][SQL]In Spark Thrift Server, CAST AS...

2018-07-18 Thread zuotingbing
Github user zuotingbing commented on a diff in the pull request: https://github.com/apache/spark/pull/21789#discussion_r203613567 --- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala --- @@ -766,6 +774,14 @@ class Hive

[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21802 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21802 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1112/

[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...

2018-07-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21608 **[Test build #93262 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93262/testReport)** for PR 21608 at commit [`107f4c6`](https://github.com/apache/spark/commit/10

[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.

2018-07-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21802 **[Test build #93261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93261/testReport)** for PR 21802 at commit [`9081e2f`](https://github.com/apache/spark/commit/90

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203614509 --- Diff: core/src/main/scala/org/apache/spark/BarrierTaskInfo.scala --- @@ -0,0 +1,23 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203615062 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -1839,6 +1844,18 @@ abstract class RDD[T: ClassTag]( def toJavaRDD() : JavaRDD[T

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203615271 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -1839,6 +1844,18 @@ abstract class RDD[T: ClassTag]( def toJavaRDD() : JavaRDD[T

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203616384 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ActiveJob.scala --- @@ -60,4 +60,10 @@ private[spark] class ActiveJob( val finished = Array

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203616328 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ActiveJob.scala --- @@ -60,4 +60,10 @@ private[spark] class ActiveJob( val finished = Array

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203616623 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1311,17 +1312,6 @@ class DAGScheduler( }

[GitHub] spark issue #21533: [SPARK-24195][Core] Bug fix for local:/ path in SparkCon...

2018-07-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21533 **[Test build #93259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93259/testReport)** for PR 21533 at commit [`eb46ccf`](https://github.com/apache/spark/commit/e

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203617306 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1349,6 +1339,48 @@ class DAGScheduler( s"longer run

[GitHub] spark issue #21533: [SPARK-24195][Core] Bug fix for local:/ path in SparkCon...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21533 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93259/ Test PASSed. ---

[GitHub] spark issue #21533: [SPARK-24195][Core] Bug fix for local:/ path in SparkCon...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21533 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21732: [SPARK-24762][SQL] Aggregator should be able to use Opti...

2018-07-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21732 At the end of encoder creation? You mean at the end of calling `ExpressionEncoder.apply()`? But it is used both for top-level encoder e.g., `Dataset[Option[Product]]` and non top-level encoder e.g.,

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203618106 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1349,6 +1339,48 @@ class DAGScheduler( s"longer run

[GitHub] spark issue #21803: [SPARK-24849][SQL] Converting a value of StructType to a...

2018-07-18 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21803 @hvanhovell Could you look at the PR please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional co

[GitHub] spark pull request #21814: [SPARK-24858][SQL] Avoid unnecessary parquet foot...

2018-07-18 Thread gengliangwang
GitHub user gengliangwang opened a pull request: https://github.com/apache/spark/pull/21814 [SPARK-24858][SQL] Avoid unnecessary parquet footer reads ## What changes were proposed in this pull request? Currently the same Parquet footer is read twice in the function `buildRe

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203618471 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1386,29 +1418,90 @@ class DAGScheduler( )

[GitHub] spark issue #21803: [SPARK-24849][SQL] Converting a value of StructType to a...

2018-07-18 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21803 > (As I described in the jira) What's this func is used for? @maropu I answered in JIRA, please, look at it. --- - To un

[GitHub] spark issue #21533: [SPARK-24195][Core] Ignore the files with "local" scheme...

2018-07-18 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/21533 @jiangxb1987 Thanks for reminding, rephrase done. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For add

[GitHub] spark pull request #21803: [SPARK-24849][SQL] Converting a value of StructTy...

2018-07-18 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/21803#discussion_r203618974 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala --- @@ -436,6 +436,14 @@ object StructType extends AbstractDataType {

[GitHub] spark issue #21814: [SPARK-24858][SQL] Avoid unnecessary parquet footer read...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21814 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203619016 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1386,29 +1418,90 @@ class DAGScheduler( )

[GitHub] spark issue #21814: [SPARK-24858][SQL] Avoid unnecessary parquet footer read...

2018-07-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21814 **[Test build #93263 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93263/testReport)** for PR 21814 at commit [`5667cc5`](https://github.com/apache/spark/commit/56

[GitHub] spark issue #21814: [SPARK-24858][SQL] Avoid unnecessary parquet footer read...

2018-07-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21814 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1113/

[GitHub] spark issue #21700: [SPARK-24717][SS] Split out max retain version of state ...

2018-07-18 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21700 LGTM! I am merging it! Thank you for all the hard work. And my apologies for not being able to give it time earlier to review it. ---

<    1   2   3   4   5   6