[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/21889 > select id, name.middle, address from temp - Works > select name.middle, address from temp - Fails > select name.middle from temp - Works > select name.middle, id, address from temp -

[GitHub] spark issue #21948: [SPARK-24991][SQL] use InternalRow in DataSourceWriter

2018-08-04 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21948 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21948: [SPARK-24991][SQL] use InternalRow in DataSourceW...

2018-08-04 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21948#discussion_r207725721 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataWriterFactory.java --- @@ -33,7 +33,10 @@ public interface

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21102 cc @ueshin @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #17185: [SPARK-19602][SQL] Support column resolution of f...

2018-08-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17185#discussion_r207725519 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala --- @@ -169,25 +181,50 @@ package object expressions {

[GitHub] spark pull request #21948: [SPARK-24991][SQL] use InternalRow in DataSourceW...

2018-08-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21948#discussion_r207725400 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/MemorySinkV2Suite.scala --- @@ -44,16 +46,16 @@ class MemorySinkV2Suite extends

[GitHub] spark issue #21948: [SPARK-24991][SQL] use InternalRow in DataSourceWriter

2018-08-04 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21948 +1 when tests are passing. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21948: [SPARK-24991][SQL] use InternalRow in DataSourceW...

2018-08-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21948#discussion_r207725389 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousRateStreamSource.scala --- @@ -89,7 +89,8 @@ class

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-08-04 Thread cclauss
Github user cclauss commented on the issue: https://github.com/apache/spark/pull/20838 jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21977: SPARK-25004: Add spark.executor.pyspark.memory limit.

2018-08-04 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21977 Anyone able to reproduce the YARN test failures? When I run the tests, a different set fails and seems unrelated to python. This shouldn't change the behavior of any tests because the environment

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-08-04 Thread cclauss
Github user cclauss commented on the issue: https://github.com/apache/spark/pull/20838 May I please break this PR into pieces as I did in #21959 and #21960 so that we can isolate why tests keep failing? These issues were all discovered using the same (flake8 F821) technique but they

[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21978 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21978 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94232/ Test PASSed. ---

[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21978 **[Test build #94232 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94232/testReport)** for PR 21978 at commit

[GitHub] spark issue #17185: [SPARK-19602][SQL] Support column resolution of fully qu...

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17185 **[Test build #94234 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94234/testReport)** for PR 17185 at commit

[GitHub] spark issue #17185: [SPARK-19602][SQL] Support column resolution of fully qu...

2018-08-04 Thread skambha
Github user skambha commented on the issue: https://github.com/apache/spark/pull/17185 I have addressed the review comments in this commit [here](https://github.com/apache/spark/pull/17185/commits/065687f3b987e254f41279d45f0cced6e42e) @cloud-fan, please take a look.

[GitHub] spark pull request #21984: [SPARK-24772][SQL] Avro: support logical date typ...

2018-08-04 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21984#discussion_r207724930 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala --- @@ -92,7 +92,7 @@ class AvroSerializer(rootCatalystType:

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread ajacques
Github user ajacques commented on the issue: https://github.com/apache/spark/pull/21889 @mallman `select id, name.middle, address from temp` - **Works** `select name.middle, address from temp` - **Fails** ---

[GitHub] spark issue #21977: SPARK-25004: Add spark.executor.pyspark.memory limit.

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21977 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94230/ Test FAILed. ---

[GitHub] spark issue #21977: SPARK-25004: Add spark.executor.pyspark.memory limit.

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21977 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21305: [SPARK-24251][SQL] Add AppendData logical plan.

2018-08-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21305#discussion_r207724886 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala --- @@ -336,4 +337,97 @@ object DataType { case (fromDataType,

[GitHub] spark issue #21977: SPARK-25004: Add spark.executor.pyspark.memory limit.

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21977 **[Test build #94230 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94230/testReport)** for PR 21977 at commit

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94231/ Test PASSed. ---

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21102 **[Test build #94231 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94231/testReport)** for PR 21102 at commit

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16677 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16677 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94227/ Test FAILed. ---

[GitHub] spark issue #21948: [SPARK-24991][SQL] use InternalRow in DataSourceWriter

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21948 **[Test build #94233 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94233/testReport)** for PR 21948 at commit

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16677 **[Test build #94227 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94227/testReport)** for PR 16677 at commit

[GitHub] spark issue #21948: [SPARK-24991][SQL] use InternalRow in DataSourceWriter

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21948 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21948: [SPARK-24991][SQL] use InternalRow in DataSourceWriter

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21948 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21948: [SPARK-24991][SQL] use InternalRow in DataSourceWriter

2018-08-04 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21948 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21948: [SPARK-24991][SQL] use InternalRow in DataSourceW...

2018-08-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21948#discussion_r207723998 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/MemorySinkV2Suite.scala --- @@ -44,16 +46,16 @@ class MemorySinkV2Suite

[GitHub] spark pull request #21948: [SPARK-24991][SQL] use InternalRow in DataSourceW...

2018-08-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21948#discussion_r207723962 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousRateStreamSource.scala --- @@ -89,7 +89,8 @@ class

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-08-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r207723947 --- Diff: core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-08-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r207723925 --- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala --- @@ -0,0 +1,233 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/21889 > Test build #94228 has finished for PR 21889 at commit 92901da. The test failure appears to be unrelated to this PR. Is it just me or has the test suite become flakier in the past

[GitHub] spark issue #22000: [SPARK-25025][SQL] Remove the default value of isAll in ...

2018-08-04 Thread dilipbiswal
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22000 cc @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/21889 > The tests as committed pass for me, but I removed the order by id and I got that error. Are you saying it works with the specific query in my comment? @ajacques Please try this query:

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21889 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21889 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94228/ Test FAILed. ---

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21889 **[Test build #94228 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94228/testReport)** for PR 21889 at commit

[GitHub] spark issue #22000: [SPARK-25025][SQL] Remove the default value of isAll in ...

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22000 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22000: [SPARK-25025][SQL] Remove the default value of isAll in ...

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22000 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94226/ Test PASSed. ---

[GitHub] spark issue #22000: [SPARK-25025][SQL] Remove the default value of isAll in ...

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22000 **[Test build #94226 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94226/testReport)** for PR 22000 at commit

[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21898 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21898 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94225/ Test FAILed. ---

[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21898 **[Test build #94225 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94225/testReport)** for PR 21898 at commit

[GitHub] spark pull request #21305: [SPARK-24251][SQL] Add AppendData logical plan.

2018-08-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21305#discussion_r207723244 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala --- @@ -336,4 +337,97 @@ object DataType { case (fromDataType,

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207723205 --- Diff: core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207723188 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -296,7 +338,7 @@ private[spark] object EventLoggingListener

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207723177 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +103,50 @@ class ExecutorSummary private[spark]( val

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207723173 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +103,50 @@ class ExecutorSummary private[spark]( val

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207723165 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -669,6 +686,34 @@ private[spark] class AppStatusListener(

[GitHub] spark pull request #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21102#discussion_r207723142 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3801,339 @@ object ArrayUnion {

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207723141 --- Diff: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala --- @@ -251,6 +261,214 @@ class EventLoggingListenerSuite

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207723114 --- Diff: core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala --- @@ -217,7 +218,12 @@ class ReplayListenerSuite extends

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207723098 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -691,7 +723,19 @@ private[spark] object JsonProtocol { (json \

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21320 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21320 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94224/ Test PASSed. ---

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21320 **[Test build #94224 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94224/testReport)** for PR 21320 at commit

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207722892 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +103,50 @@ class ExecutorSummary private[spark]( val

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207722887 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +103,50 @@ class ExecutorSummary private[spark]( val

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207722865 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -302,10 +305,10 @@ private class LiveExecutor(val executorId: String,

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207722773 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -155,7 +160,14 @@ private[spark] class EventLoggingListener(

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207722724 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -93,6 +95,9 @@ private[spark] class EventLoggingListener(

[GitHub] spark issue #21999: [WIP][SQL] Flattening nested structures

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21999 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94223/ Test PASSed. ---

[GitHub] spark issue #21999: [WIP][SQL] Flattening nested structures

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21999 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21999: [WIP][SQL] Flattening nested structures

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21999 **[Test build #94223 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94223/testReport)** for PR 21999 at commit

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207722674 --- Diff: core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/21889 > The tests as committed pass for me, but I removed the order by id and I got that error. Are you saying it works with the specific query in my comment? Oh! I didn't notice you changed the

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread ajacques
Github user ajacques commented on the issue: https://github.com/apache/spark/pull/21889 The tests as committed pass for me, but I removed the `order by id` and I got that error. Are you saying it works with the specific query in my comment? ---

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-04 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207722621 --- Diff: core/src/main/scala/org/apache/spark/memory/MemoryManager.scala --- @@ -180,6 +180,26 @@ private[spark] abstract class MemoryManager(

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/21889 > @mallman: I've rebased on top of your changes and pushed. I'm seeing the following: That test passes for me locally. Also, I inspected your branch and could not find any errors in the

[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21978 **[Test build #94232 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94232/testReport)** for PR 21978 at commit

[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21978 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21978 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.

2018-08-04 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21978 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21102 **[Test build #94231 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94231/testReport)** for PR 21102 at commit

[GitHub] spark issue #21977: SPARK-25004: Add spark.executor.pyspark.memory limit.

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21977 **[Test build #94230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94230/testReport)** for PR 21977 at commit

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21102 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21977: SPARK-25004: Add spark.executor.pyspark.memory limit.

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21977 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21977: SPARK-25004: Add spark.executor.pyspark.memory limit.

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21977 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21102 retedt this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21977: SPARK-25004: Add spark.executor.pyspark.memory li...

2018-08-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21977#discussion_r207722434 --- Diff: python/pyspark/worker.py --- @@ -259,6 +260,26 @@ def main(infile, outfile): "PYSPARK_DRIVER_PYTHON are correctly

[GitHub] spark pull request #21977: SPARK-25004: Add spark.executor.pyspark.memory li...

2018-08-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21977#discussion_r207722412 --- Diff: python/pyspark/worker.py --- @@ -259,6 +260,26 @@ def main(infile, outfile): "PYSPARK_DRIVER_PYTHON are correctly

[GitHub] spark pull request #21977: SPARK-25004: Add spark.executor.pyspark.memory li...

2018-08-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21977#discussion_r207722402 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -133,10 +133,17 @@ private[yarn] class

[GitHub] spark issue #21948: [SPARK-24991][SQL] use InternalRow in DataSourceWriter

2018-08-04 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21948 @cloud-fan, thanks for documenting the behavior and removing the default copy. I had a couple of questions, but I think it is close. ---

[GitHub] spark pull request #21948: [SPARK-24991][SQL] use InternalRow in DataSourceW...

2018-08-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21948#discussion_r207722363 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/MemorySinkV2Suite.scala --- @@ -44,16 +46,16 @@ class MemorySinkV2Suite extends

[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-08-04 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/21889 > @mallman: I've rebased on top of your changes and pushed. I'm seeing the following That's the test case that I "unignored". It was passing. There must be some simple explanation. I will

[GitHub] spark pull request #21948: [SPARK-24991][SQL] use InternalRow in DataSourceW...

2018-08-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21948#discussion_r207722340 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousRateStreamSource.scala --- @@ -89,7 +89,8 @@ class

[GitHub] spark pull request #21982: [SPARK-23911][SQL] Add aggregate function.

2018-08-04 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21982 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21982: [SPARK-23911][SQL] Add aggregate function.

2018-08-04 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21982 Thanks! merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94221/ Test FAILed. ---

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21102 **[Test build #94221 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94221/testReport)** for PR 21102 at commit

[GitHub] spark issue #21977: SPARK-25004: Add spark.executor.pyspark.memory limit.

2018-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21977 **[Test build #94229 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94229/testReport)** for PR 21977 at commit

[GitHub] spark issue #21977: SPARK-25004: Add spark.executor.pyspark.memory limit.

2018-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21977 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94229/ Test FAILed. ---

  1   2   3   4   >