[GitHub] spark issue #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syntax for...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21813 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syntax for...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21813 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21818 **[Test build #93290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93290/testReport)** for PR 21818 at commit

[GitHub] spark issue #18714: [SPARK-20236][SQL] dynamic partition overwrite

2018-07-19 Thread koertkuipers
Github user koertkuipers commented on the issue: https://github.com/apache/spark/pull/18714 @cloud-fan i created [SPARK-24860](https://issues.apache.org/jira/browse/SPARK-24860) for this --- - To unsubscribe,

[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21818 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21818 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request #21818: [SPARK-24860][SQL] Support setting of partitionOv...

2018-07-19 Thread koertkuipers
GitHub user koertkuipers opened a pull request: https://github.com/apache/spark/pull/21818 [SPARK-24860][SQL] Support setting of partitionOverWriteMode in output options for writing DataFrame ## What changes were proposed in this pull request? Besides spark setting

[GitHub] spark pull request #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syn...

2018-07-19 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21813#discussion_r203813530 --- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 --- @@ -400,12 +400,12 @@ hintStatement fromClause :

[GitHub] spark pull request #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syn...

2018-07-19 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21813#discussion_r203813381 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -440,19 +440,36 @@ class Analyzer(

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-07-19 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21320 @mallman Could we move the changes made in `ParquetReadSupport.scala` to a separate PR? Then, we can merge this PR very quickly. ---

[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...

2018-07-19 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21774 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21774: [SPARK-24811][SQL]Avro: add new function from_avr...

2018-07-19 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21774#discussion_r203805849 --- Diff: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala --- @@ -0,0 +1,87 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #21774: [SPARK-24811][SQL]Avro: add new function from_avr...

2018-07-19 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21774#discussion_r203805491 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/CatalystDataToAvro.scala --- @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #21817: [SPARK-24861][SS][test] create corrected temp directorie...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21817 **[Test build #93289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93289/testReport)** for PR 21817 at commit

[GitHub] spark issue #21817: [SPARK-24861][SS] create corrected temp directories in R...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21817 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21817: [SPARK-24861][SS] create corrected temp directories in R...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21817 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21817: [SPARK-24861][SS] create corrected temp directories in R...

2018-07-19 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21817 cc @jose-torres @tdas --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21817: [SPARK-24861][SS] create corrected temp directori...

2018-07-19 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/21817 [SPARK-24861][SS] create corrected temp directories in RateSourceSuite ## What changes were proposed in this pull request? `RateSourceSuite` may leave garbage files under

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93287/ Test FAILed. ---

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21758 **[Test build #93287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93287/testReport)** for PR 21758 at commit

[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21774 **[Test build #93288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93288/testReport)** for PR 21774 at commit

[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21774 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21774 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21774: [SPARK-24811][SQL]Avro: add new function from_avr...

2018-07-19 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21774#discussion_r203796377 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/CatalystDataToAvro.scala --- @@ -0,0 +1,67 @@ +/* + * Licensed to the

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-19 Thread arunmahadevan
Github user arunmahadevan commented on the issue: https://github.com/apache/spark/pull/21721 @jose-torres, addressed initial comments. @tdas, can you also take a look when possible ? --- - To unsubscribe,

[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...

2018-07-19 Thread szyszy
Github user szyszy commented on the issue: https://github.com/apache/spark/pull/20761 Hi @vanzin , @galv Please check my latest fixes based on your previous comments. @squito : Could you please join this review to speed things up? Thanks in advance! ---

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794871 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,185 @@ +/* + * Licensed to

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794896 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,185 @@ +/* + * Licensed to

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794842 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794443 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794460 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794562 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794078 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -306,9 +318,14 @@ private[yarn] class

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794112 --- Diff: docs/running-on-yarn.md --- @@ -121,6 +121,40 @@ To use a custom metrics.properties for the application master and executors, upd Use

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794227 --- Diff: docs/running-on-yarn.md --- @@ -121,6 +121,40 @@ To use a custom metrics.properties for the application master and executors, upd Use

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794300 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -17,7 +17,7 @@ package

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203793998 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -306,9 +318,14 @@ private[yarn] class

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203793921 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -306,9 +318,14 @@ private[yarn] class

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203793839 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,185 @@ +/* + * Licensed to

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203793702 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203793778 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,185 @@ +/* + * Licensed to

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203793589 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,180 @@ +/* + * Licensed to the

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203793303 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,180 @@ +/* + * Licensed to the

[GitHub] spark issue #21546: [SPARK-23030][SQL][PYTHON] Use Arrow stream format for c...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21546 @BryanCutler, this takes longer then I thought. Will complete my review till this week. For clarification, still no objection about merging it in orthogonally with my review. ---

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21758 **[Test build #93287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93287/testReport)** for PR 21758 at commit

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203791726 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,185 @@ +/* + * Licensed to

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203791602 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -249,6 +259,10 @@ private[spark] class Client(

[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...

2018-07-19 Thread szyszy
Github user szyszy commented on the issue: https://github.com/apache/spark/pull/20761 About the ResourceTypeValidator: The reason why this class seems complex is that it is capable of printing "smart" error messages like they are described in the scaladoc. When you define memory

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203791125 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,185 @@ +/* + * Licensed to

[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20761 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20761 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93286/ Test FAILed. ---

[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20761 **[Test build #93286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93286/testReport)** for PR 20761 at commit

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203790741 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,185 @@ +/* + * Licensed to

[GitHub] spark issue #21782: [SPARK-24816][SQL] SQL interface support repartitionByRa...

2018-07-19 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21782 I don't have a good idea, cc @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20761 **[Test build #93286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93286/testReport)** for PR 20761 at commit

[GitHub] spark pull request #21546: [SPARK-23030][SQL][PYTHON] Use Arrow stream forma...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21546#discussion_r203790304 --- Diff: python/pyspark/serializers.py --- @@ -184,27 +184,67 @@ def loads(self, obj): raise NotImplementedError -class

[GitHub] spark issue #21804: [SPARK-24268][SQL] Use datatype.catalogString in error m...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21804 **[Test build #93285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93285/testReport)** for PR 21804 at commit

[GitHub] spark issue #21804: [SPARK-24268][SQL] Use datatype.catalogString in error m...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21804 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21804: [SPARK-24268][SQL] Use datatype.catalogString in error m...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21804 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...

2018-07-19 Thread Achuth17
Github user Achuth17 commented on the issue: https://github.com/apache/spark/pull/21608 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93284/ Test FAILed. ---

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21758 **[Test build #93284 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93284/testReport)** for PR 21758 at commit

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-07-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r203773818 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -659,6 +659,11 @@ private[spark] class BlockManager( * Get block

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21758 **[Test build #93284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93284/testReport)** for PR 21758 at commit

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function

2018-07-19 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21103#discussion_r203746577 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3799,332 @@ object ArrayUnion

[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function

2018-07-19 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21103#discussion_r203741325 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3799,332 @@ object ArrayUnion

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93280/ Test PASSed. ---

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21103 **[Test build #93280 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93280/testReport)** for PR 21103 at commit

[GitHub] spark issue #21803: [SPARK-24849][SQL] Converting a value of StructType to a...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21803 To me yup, but if you are in doubt, I am perfectly okay with waiting some more days and see if other opinions arrive. --- -

[GitHub] spark issue #21798: [SPARK-24836][SQL] New option for Avro datasource - igno...

2018-07-19 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21798 I added new option to `AvroOptions` and documented it. Please, look at the PR one more time. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #21546: [SPARK-23030][SQL][PYTHON] Use Arrow stream forma...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21546#discussion_r203760078 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -3349,20 +3385,20 @@ class Dataset[T] private[sql]( } }

[GitHub] spark issue #21803: [SPARK-24849][SQL] Converting a value of StructType to a...

2018-07-19 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21803 > Thing is, fromDDL was added because it's needed other APIs rather then it's own purpose. The `toDDL` function can be reused from `SHOW CREATE TABLE` too:

[GitHub] spark issue #21653: [SPARK-13343] speculative tasks that didn't commit shoul...

2018-07-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21653 #21729 has been merged @hthuynh2 can you update this one? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #21729: [SPARK-24755][Core] Executor loss can cause task ...

2018-07-19 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21729 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21729: [SPARK-24755][Core] Executor loss can cause task to not ...

2018-07-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21729 +1 I'm going to merge, thanks @hthuynh2 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21798: [SPARK-24836][SQL] New option for Avro datasource - igno...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21798 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21798: [SPARK-24836][SQL] New option for Avro datasource - igno...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21798 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93283/ Test PASSed. ---

[GitHub] spark issue #21798: [SPARK-24836][SQL] New option for Avro datasource - igno...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21798 **[Test build #93283 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93283/testReport)** for PR 21798 at commit

[GitHub] spark issue #21798: [SPARK-24836][SQL] New option for Avro datasource - igno...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21798 **[Test build #93283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93283/testReport)** for PR 21798 at commit

[GitHub] spark issue #21803: [SPARK-24849][SQL] Converting a value of StructType to a...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21803 Ah, I misunderstood then. Thing is, `fromDDL` was added because it's needed other APIs rather then it's own purpose. I wonder how commonly it will be used to be honest. I haven't seen such

[GitHub] spark pull request #21814: [SPARK-24858][SQL] Avoid unnecessary parquet foot...

2018-07-19 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21814 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21815: [SPARK-23731][SQL] Make FileSourceScanExec canonicalizab...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21815 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93281/ Test PASSed. ---

[GitHub] spark issue #21815: [SPARK-23731][SQL] Make FileSourceScanExec canonicalizab...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21815 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21815: [SPARK-23731][SQL] Make FileSourceScanExec canonicalizab...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21815 **[Test build #93281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93281/testReport)** for PR 21815 at commit

[GitHub] spark issue #21814: [SPARK-24858][SQL] Avoid unnecessary parquet footer read...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21814 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21804: [SPARK-24268][SQL] Use datatype.catalogString in ...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21804#discussion_r203746047 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/AbstractDataType.scala --- @@ -145,7 +145,7 @@ abstract class NumericType extends

[GitHub] spark pull request #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-07-19 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21635#discussion_r203744555 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterSource.scala --- @@ -0,0 +1,49 @@ +/* + * Licensed

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21589 > Can we add the methods as experimental and if we will observe some problems in the upcoming releases, we will just remove them? For clarification, I think we could but if there was

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203742139 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -1055,6 +1055,64 @@ class DAGSchedulerSuite extends

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203741986 --- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala --- @@ -627,6 +627,48 @@ class SparkContextSuite extends SparkFunSuite with

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203741697 --- Diff: core/src/main/scala/org/apache/spark/scheduler/WorkerOffer.scala --- @@ -21,4 +21,10 @@ package org.apache.spark.scheduler * Represents

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203741114 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -274,7 +274,9 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

<    1   2   3   4   5   6   >