[GitHub] spark issue #21803: [SPARK-24849][SQL] Converting a value of StructType to a...

2018-07-19 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21803 @hvanhovell Could you look at the PR please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21814: [SPARK-24858][SQL] Avoid unnecessary parquet foot...

2018-07-19 Thread gengliangwang
GitHub user gengliangwang opened a pull request: https://github.com/apache/spark/pull/21814 [SPARK-24858][SQL] Avoid unnecessary parquet footer reads ## What changes were proposed in this pull request? Currently the same Parquet footer is read twice in the function

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203620942 --- Diff: core/src/main/scala/org/apache/spark/scheduler/WorkerOffer.scala --- @@ -21,4 +21,10 @@ package org.apache.spark.scheduler * Represents

[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21802 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21754: [SPARK-24705][SQL] Cannot reuse an exchange opera...

2018-07-19 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21754#discussion_r203630277 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala --- @@ -85,14 +85,20 @@ case class ReusedExchangeExec(override

[GitHub] spark issue #21782: [SPARK-24816][SQL] SQL interface support repartitionByRa...

2018-07-19 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/21782 Teradata has `PARTITION BY RANGE_N` option in CREATE TABLE SQL statement. But SELECT SQL statement not seen in other systems.

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93273/ Test FAILed. ---

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21357: [SPARK-24311][SS] Refactor HDFSBackedStateStoreProvider ...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21357 **[Test build #93278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93278/testReport)** for PR 21357 at commit

[GitHub] spark issue #21533: [SPARK-24195][Core] Bug fix for local:/ path in SparkCon...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21533 **[Test build #93259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93259/testReport)** for PR 21533 at commit

[GitHub] spark issue #21815: [SPARK-23731][SQL] Make FileSourceScanExec canonicalizab...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21815 **[Test build #93271 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93271/testReport)** for PR 21815 at commit

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21584 **[Test build #93272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93272/testReport)** for PR 21584 at commit

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21584 **[Test build #93274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93274/testReport)** for PR 21584 at commit

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93272/ Test FAILed. ---

[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21803: [SPARK-24849][SQL] Converting a value of StructType to a...

2018-07-19 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21803 > is the purpose of this API is to have a int instead of struct Basically, yes. All those methods `simpleString()`, `catalogString()`, `sql()` return `struct< ... : ...>` which is

[GitHub] spark issue #21804: [SPARK-24268][SQL] Use datatype.catalogString in error m...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21804 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21804: [SPARK-24268][SQL] Use datatype.catalogString in error m...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21804 **[Test build #93276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93276/testReport)** for PR 21804 at commit

[GitHub] spark issue #21804: [SPARK-24268][SQL] Use datatype.catalogString in error m...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21804 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21803: [SPARK-24849][SQL] Converting a value of StructType to a...

2018-07-19 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21803 > Does the alternative code I posted above work in that case? > You can use struct.map(field => s"${quoteIdentifier(field.name)} ${field.dataType.sql}").mkString(",") in the application code

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203791602 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -249,6 +259,10 @@ private[spark] class Client(

[GitHub] spark issue #21546: [SPARK-23030][SQL][PYTHON] Use Arrow stream format for c...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21546 @BryanCutler, this takes longer then I thought. Will complete my review till this week. For clarification, still no objection about merging it in orthogonally with my review. ---

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794443 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794460 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794562 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the

[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21774 **[Test build #93288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93288/testReport)** for PR 21774 at commit

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-07-19 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21320 @mallman Could we move the changes made in `ParquetReadSupport.scala` to a separate PR? Then, we can merge this PR very quickly. ---

[GitHub] spark pull request #21818: [SPARK-24860][SQL] Support setting of partitionOv...

2018-07-19 Thread koertkuipers
GitHub user koertkuipers opened a pull request: https://github.com/apache/spark/pull/21818 [SPARK-24860][SQL] Support setting of partitionOverWriteMode in output options for writing DataFrame ## What changes were proposed in this pull request? Besides spark setting

[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93277/ Test PASSed. ---

[GitHub] spark issue #21357: [SPARK-24311][SS] Refactor HDFSBackedStateStoreProvider ...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21357 **[Test build #93278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93278/testReport)** for PR 21357 at commit

[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21733 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203739055 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala --- @@ -50,6 +50,7 @@ private[spark] class TaskDescription( val

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203741986 --- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala --- @@ -627,6 +627,48 @@ class SparkContextSuite extends SparkFunSuite with

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203742139 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -1055,6 +1055,64 @@ class DAGSchedulerSuite extends

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203741697 --- Diff: core/src/main/scala/org/apache/spark/scheduler/WorkerOffer.scala --- @@ -21,4 +21,10 @@ package org.apache.spark.scheduler * Represents

[GitHub] spark issue #21803: [SPARK-24849][SQL] Converting a value of StructType to a...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21803 Ah, I misunderstood then. Thing is, `fromDDL` was added because it's needed other APIs rather then it's own purpose. I wonder how commonly it will be used to be honest. I haven't seen such

[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function

2018-07-19 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21103#discussion_r203741325 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3799,332 @@ object ArrayUnion

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21758 **[Test build #93284 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93284/testReport)** for PR 21758 at commit

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93284/ Test FAILed. ---

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20761 **[Test build #93286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93286/testReport)** for PR 20761 at commit

[GitHub] spark pull request #21546: [SPARK-23030][SQL][PYTHON] Use Arrow stream forma...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21546#discussion_r203790304 --- Diff: python/pyspark/serializers.py --- @@ -184,27 +184,67 @@ def loads(self, obj): raise NotImplementedError -class

[GitHub] spark issue #21782: [SPARK-24816][SQL] SQL interface support repartitionByRa...

2018-07-19 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21782 I don't have a good idea, cc @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203791726 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,185 @@ +/* + * Licensed to

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794112 --- Diff: docs/running-on-yarn.md --- @@ -121,6 +121,40 @@ To use a custom metrics.properties for the application master and executors, upd Use

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794078 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -306,9 +318,14 @@ private[yarn] class

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794300 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -17,7 +17,7 @@ package

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203794227 --- Diff: docs/running-on-yarn.md --- @@ -121,6 +121,40 @@ To use a custom metrics.properties for the application master and executors, upd Use

[GitHub] spark pull request #21774: [SPARK-24811][SQL]Avro: add new function from_avr...

2018-07-19 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21774#discussion_r203796377 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/CatalystDataToAvro.scala --- @@ -0,0 +1,67 @@ +/* + * Licensed to the

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-19 Thread arunmahadevan
Github user arunmahadevan commented on the issue: https://github.com/apache/spark/pull/21721 @jose-torres, addressed initial comments. @tdas, can you also take a look when possible ? --- - To unsubscribe,

[GitHub] spark issue #21817: [SPARK-24861][SS] create corrected temp directories in R...

2018-07-19 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21817 cc @jose-torres @tdas --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21817: [SPARK-24861][SS] create corrected temp directories in R...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21817 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21817: [SPARK-24861][SS] create corrected temp directori...

2018-07-19 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/21817 [SPARK-24861][SS] create corrected temp directories in RateSourceSuite ## What changes were proposed in this pull request? `RateSourceSuite` may leave garbage files under

[GitHub] spark pull request #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syn...

2018-07-19 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21813#discussion_r203813530 --- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 --- @@ -400,12 +400,12 @@ hintStatement fromClause :

[GitHub] spark pull request #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syn...

2018-07-19 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21813#discussion_r203813381 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -440,19 +440,36 @@ class Analyzer(

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203791125 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,185 @@ +/* + * Licensed to

[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20761 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...

2018-07-19 Thread szyszy
Github user szyszy commented on the issue: https://github.com/apache/spark/pull/20761 About the ResourceTypeValidator: The reason why this class seems complex is that it is capable of printing "smart" error messages like they are described in the scaladoc. When you define memory

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203793303 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,180 @@ +/* + * Licensed to the

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-19 Thread szyszy
Github user szyszy commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r203793589 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,180 @@ +/* + * Licensed to the

[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21774 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21774 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21817: [SPARK-24861][SS] create corrected temp directories in R...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21817 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21817: [SPARK-24861][SS][test] create corrected temp directorie...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21817 **[Test build #93289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93289/testReport)** for PR 21817 at commit

[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...

2018-07-19 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21774 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #18714: [SPARK-20236][SQL] dynamic partition overwrite

2018-07-19 Thread koertkuipers
Github user koertkuipers commented on the issue: https://github.com/apache/spark/pull/18714 @cloud-fan i created [SPARK-24860](https://issues.apache.org/jira/browse/SPARK-24860) for this --- - To unsubscribe,

[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21818 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21818 **[Test build #93290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93290/testReport)** for PR 21818 at commit

[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21818 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21733 **[Test build #93277 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93277/testReport)** for PR 21733 at commit

[GitHub] spark pull request #21764: [SPARK-24802] Optimization Rule Exclusion

2018-07-19 Thread maryannxue
Github user maryannxue commented on a diff in the pull request: https://github.com/apache/spark/pull/21764#discussion_r203730125 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -175,6 +182,44 @@ abstract class

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-07-19 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21589 > Unless there is some other compelling reason for introducing this which I have missed; I am -1 on introducing this change. I would like to describe one class of use cases which you don't

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21758 **[Test build #93282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93282/testReport)** for PR 21758 at commit

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203737856 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1349,6 +1339,48 @@ class DAGScheduler( s"longer

[GitHub] spark pull request #21804: [SPARK-24268][SQL] Use datatype.catalogString in ...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21804#discussion_r203746047 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/AbstractDataType.scala --- @@ -145,7 +145,7 @@ abstract class NumericType extends

[GitHub] spark issue #21815: [SPARK-23731][SQL] Make FileSourceScanExec canonicalizab...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21815 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93281/ Test PASSed. ---

[GitHub] spark issue #21815: [SPARK-23731][SQL] Make FileSourceScanExec canonicalizab...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21815 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21729: [SPARK-24755][Core] Executor loss can cause task to not ...

2018-07-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21729 +1 I'm going to merge, thanks @hthuynh2 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21758 **[Test build #93284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93284/testReport)** for PR 21758 at commit

[GitHub] spark issue #21357: [SPARK-24311][SS] Refactor HDFSBackedStateStoreProvider ...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21357 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21357: [SPARK-24311][SS] Refactor HDFSBackedStateStoreProvider ...

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21357 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93278/ Test PASSed. ---

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203738500 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1386,29 +1418,90 @@ class DAGScheduler( )

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r203741114 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -274,7 +274,9 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21653: [SPARK-13343] speculative tasks that didn't commit shoul...

2018-07-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21653 #21729 has been merged @hthuynh2 can you update this one? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #21803: [SPARK-24849][SQL] Converting a value of StructType to a...

2018-07-19 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21803 > Thing is, fromDDL was added because it's needed other APIs rather then it's own purpose. The `toDDL` function can be reused from `SHOW CREATE TABLE` too:

[GitHub] spark pull request #21546: [SPARK-23030][SQL][PYTHON] Use Arrow stream forma...

2018-07-19 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21546#discussion_r203760078 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -3349,20 +3385,20 @@ class Dataset[T] private[sql]( } }

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21758: [SPARK-24795][CORE] Implement barrier execution mode

2018-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21758 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21764: [SPARK-24802] Optimization Rule Exclusion

2018-07-19 Thread maryannxue
Github user maryannxue commented on a diff in the pull request: https://github.com/apache/spark/pull/21764#discussion_r203730652 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -160,6 +160,13 @@ abstract class

<    1   2   3   4   5   6   >