[GitHub] spark pull request #19256: [SPARK-21338][SQL]implement isCascadingTruncateTa...

2017-09-17 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19256#discussion_r139302993 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/AggregatedDialect.scala --- @@ -41,4 +41,8 @@ private class AggregatedDialect(dialects:

[GitHub] spark issue #19256: [SPARK-21338][SQL]implement isCascadingTruncateTable() m...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19256 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #16600: [SPARK-19242][SQL] SHOW CREATE TABLE should generate new...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16600 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19256: [SPARK-21338][SQL]implement isCascadingTruncateTable() m...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19256 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81848/ Test FAILed. ---

[GitHub] spark issue #16600: [SPARK-19242][SQL] SHOW CREATE TABLE should generate new...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16600 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81849/ Test FAILed. ---

[GitHub] spark issue #19256: [SPARK-21338][SQL]implement isCascadingTruncateTable() m...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19256 **[Test build #81848 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81848/testReport)** for PR 19256 at commit

[GitHub] spark issue #19218: [SPARK-21786][SQL] The 'spark.sql.parquet.compression.co...

2017-09-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19218 I see. If you set `spark.sql.hive.convertMetastoreParquet` to false, you will also hit the issue for non-partitioned table. ---

[GitHub] spark issue #19218: [SPARK-21786][SQL] The 'spark.sql.parquet.compression.co...

2017-09-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19218 @fjh100456 We have priority for three different inputs. Here, you just consider one of three. Please also add the extra checks. Hopefully, @dongjoon-hyun can help you answer your questions. He

[GitHub] spark pull request #19218: [SPARK-21786][SQL] The 'spark.sql.parquet.compres...

2017-09-17 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19218#discussion_r139302763 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -101,6 +101,19 @@ case class

[GitHub] spark issue #19256: [SPARK-21338][SQL]implement isCascadingTruncateTable() m...

2017-09-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19256 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19256: [SPARK-21338][SQL]implement isCascadingTruncateTable() m...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19256 **[Test build #81848 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81848/testReport)** for PR 19256 at commit

[GitHub] spark issue #16600: [SPARK-19242][SQL] SHOW CREATE TABLE should generate new...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16600 **[Test build #81849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81849/testReport)** for PR 16600 at commit

[GitHub] spark issue #19230: [SPARK-22003][SQL] support array column in vectorized re...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19230 **[Test build #81850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81850/testReport)** for PR 19230 at commit

[GitHub] spark issue #19258: add MockNetCat

2017-09-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19258 Close this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19253: [SPARK-22037][SQL] Collapse Project if it is the ...

2017-09-17 Thread gengliangwang
Github user gengliangwang closed the pull request at: https://github.com/apache/spark/pull/19253 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19253: [SPARK-22037][SQL] Collapse Project if it is the child o...

2017-09-17 Thread gengliangwang
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/19253 @gatorsmile @viirya Thanks. I should close this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #19249: [SPARK-22032][PySpark] Speed up StructType.fromIn...

2017-09-17 Thread maver1ck
Github user maver1ck commented on a diff in the pull request: https://github.com/apache/spark/pull/19249#discussion_r139303509 --- Diff: python/pyspark/sql/types.py --- @@ -619,7 +621,8 @@ def fromInternal(self, obj): # it's already converted by pickler

[GitHub] spark pull request #19254: [MINOR][CORE] Cleanup dead code and duplication i...

2017-09-17 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/19254#discussion_r139304877 --- Diff: core/src/main/java/org/apache/spark/memory/MemoryConsumer.java --- @@ -116,13 +110,7 @@ public void freeArray(LongArray array) { protected

[GitHub] spark pull request #19254: [MINOR][CORE] Cleanup dead code and duplication i...

2017-09-17 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/19254#discussion_r139304891 --- Diff: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java --- @@ -403,6 +391,16 @@ public long spill(long numBytes) throws IOException {

[GitHub] spark issue #19254: [MINOR][CORE] Cleanup dead code and duplication in Mem. ...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19254 **[Test build #3923 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3923/testReport)** for PR 19254 at commit

[GitHub] spark pull request #19254: [MINOR][CORE] Cleanup dead code and duplication i...

2017-09-17 Thread original-brownbear
Github user original-brownbear commented on a diff in the pull request: https://github.com/apache/spark/pull/19254#discussion_r139305647 --- Diff: core/src/main/java/org/apache/spark/memory/MemoryConsumer.java --- @@ -116,13 +110,7 @@ public void freeArray(LongArray array) {

[GitHub] spark pull request #19254: [MINOR][CORE] Cleanup dead code and duplication i...

2017-09-17 Thread original-brownbear
Github user original-brownbear commented on a diff in the pull request: https://github.com/apache/spark/pull/19254#discussion_r139305653 --- Diff: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java --- @@ -403,6 +391,16 @@ public long spill(long numBytes) throws

[GitHub] spark issue #19254: [MINOR][CORE] Cleanup dead code and duplication in Mem. ...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19254 **[Test build #3924 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3924/testReport)** for PR 19254 at commit

[GitHub] spark pull request #19254: [MINOR][CORE] Cleanup dead code and duplication i...

2017-09-17 Thread original-brownbear
Github user original-brownbear commented on a diff in the pull request: https://github.com/apache/spark/pull/19254#discussion_r139307959 --- Diff: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java --- @@ -403,6 +391,16 @@ public long spill(long numBytes) throws

[GitHub] spark pull request #19261: [SPARK-22040] Add current_date function with time...

2017-09-17 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19261#discussion_r139309404 --- Diff: python/pyspark/sql/functions.py --- @@ -793,12 +793,12 @@ def ntile(n): # -- Date/Timestamp functions

[GitHub] spark pull request #19261: [SPARK-22040] Add current_date function with time...

2017-09-17 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19261#discussion_r139309394 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -2508,6 +2508,14 @@ object functions { def current_date(): Column =

[GitHub] spark pull request #19261: [SPARK-22040] Add current_date function with time...

2017-09-17 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19261#discussion_r139309376 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -2508,6 +2508,14 @@ object functions { def current_date(): Column =

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r139312181 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala --- @@ -0,0 +1,130 @@ +/* + *

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-09-17 Thread vkhristenko
Github user vkhristenko commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r139312613 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala --- @@ -0,0 +1,130 @@ +/* + *

[GitHub] spark pull request #18193: [SPARK-15616] [SQL] CatalogRelation should fallba...

2017-09-17 Thread cenyuhai
Github user cenyuhai commented on a diff in the pull request: https://github.com/apache/spark/pull/18193#discussion_r139312866 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala --- @@ -139,6 +138,54 @@ class DetermineTableStats(session: SparkSession)

[GitHub] spark pull request #19255: [WIP][SPARK-22029][PySpark] Add lru_cache to _par...

2017-09-17 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19255#discussion_r139305858 --- Diff: python/pyspark/sql/types.py --- @@ -24,6 +24,7 @@ import re import base64 from array import array +from functools import

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81852 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81852/testReport)** for PR 19249 at commit

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81852/ Test FAILed. ---

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81854/testReport)** for PR 19249 at commit

[GitHub] spark issue #19254: [MINOR][CORE] Cleanup dead code and duplication in Mem. ...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19254 **[Test build #3923 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3923/testReport)** for PR 19254 at commit

[GitHub] spark issue #19254: [MINOR][CORE] Cleanup dead code and duplication in Mem. ...

2017-09-17 Thread original-brownbear
Github user original-brownbear commented on the issue: https://github.com/apache/spark/pull/19254 Test failure looks unrelated (even though it happened twice in a row now): ```sh [info] org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED *** (1 second,

[GitHub] spark pull request #19204: [SPARK-21981][PYTHON][ML] Added Python interface ...

2017-09-17 Thread yanboliang
Github user yanboliang commented on a diff in the pull request: https://github.com/apache/spark/pull/19204#discussion_r139312388 --- Diff: python/pyspark/ml/evaluation.py --- @@ -328,6 +329,86 @@ def setParams(self, predictionCol="prediction", labelCol="label", kwargs

[GitHub] spark pull request #19204: [SPARK-21981][PYTHON][ML] Added Python interface ...

2017-09-17 Thread yanboliang
Github user yanboliang commented on a diff in the pull request: https://github.com/apache/spark/pull/19204#discussion_r139312199 --- Diff: python/pyspark/ml/evaluation.py --- @@ -328,6 +329,86 @@ def setParams(self, predictionCol="prediction", labelCol="label", kwargs

[GitHub] spark pull request #19204: [SPARK-21981][PYTHON][ML] Added Python interface ...

2017-09-17 Thread yanboliang
Github user yanboliang commented on a diff in the pull request: https://github.com/apache/spark/pull/19204#discussion_r139312034 --- Diff: python/pyspark/ml/evaluation.py --- @@ -328,6 +329,86 @@ def setParams(self, predictionCol="prediction", labelCol="label", kwargs

[GitHub] spark pull request #19204: [SPARK-21981][PYTHON][ML] Added Python interface ...

2017-09-17 Thread yanboliang
Github user yanboliang commented on a diff in the pull request: https://github.com/apache/spark/pull/19204#discussion_r139312046 --- Diff: python/pyspark/ml/evaluation.py --- @@ -328,6 +329,86 @@ def setParams(self, predictionCol="prediction", labelCol="label", kwargs

[GitHub] spark pull request #19262: [MINOR][ML] Remove unnecessary default value sett...

2017-09-17 Thread yanboliang
GitHub user yanboliang opened a pull request: https://github.com/apache/spark/pull/19262 [MINOR][ML] Remove unnecessary default value setting for evaluators. ## What changes were proposed in this pull request? Remove unnecessary default value setting for all evaluators, as we

[GitHub] spark issue #19226: [SPARK-21985][PySpark] PairDeserializer is broken for do...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19226 **[Test build #81856 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81856/testReport)** for PR 19226 at commit

[GitHub] spark pull request #19230: [SPARK-22003][SQL] support array column in vector...

2017-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19230#discussion_r139305746 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVector.java --- @@ -16,6 +16,7 @@ */ package

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81852/testReport)** for PR 19249 at commit

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81853/ Test PASSed. ---

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81853/testReport)** for PR 19249 at commit

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81854/ Test PASSed. ---

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81854 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81854/testReport)** for PR 19249 at commit

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19259: [BACKPORT-2.1][SPARK-19318][SPARK-22041][SQL] Docker tes...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19259 **[Test build #81851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81851/testReport)** for PR 19259 at commit

[GitHub] spark issue #19259: [BACKPORT-2.1][SPARK-19318][SPARK-22041][SQL] Docker tes...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19259 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81851/ Test PASSed. ---

[GitHub] spark pull request #19260: [SPARK-22043][PYTHON] Improves error message for ...

2017-09-17 Thread HyukjinKwon
GitHub user HyukjinKwon opened a pull request: https://github.com/apache/spark/pull/19260 [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles ## What changes were proposed in this pull request? This PR proposes to improve error message from:

[GitHub] spark issue #19260: [SPARK-22043][PYTHON] Improves error message for show_pr...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19260 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81855/ Test PASSed. ---

[GitHub] spark issue #19261: [SPARK-22040] Add current_date function with timezone id

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19261 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19261: [SPARK-22040] Add current_date function with time...

2017-09-17 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/19261 [SPARK-22040] Add current_date function with timezone id ## What changes were proposed in this pull request? Add current_date function with timezone id. ## How was this patch

[GitHub] spark pull request #19261: [SPARK-22040] Add current_date function with time...

2017-09-17 Thread jaceklaskowski
Github user jaceklaskowski commented on a diff in the pull request: https://github.com/apache/spark/pull/19261#discussion_r139309272 --- Diff: python/pyspark/sql/functions.py --- @@ -793,12 +793,12 @@ def ntile(n): # -- Date/Timestamp functions

[GitHub] spark pull request #19261: [SPARK-22040] Add current_date function with time...

2017-09-17 Thread jaceklaskowski
Github user jaceklaskowski commented on a diff in the pull request: https://github.com/apache/spark/pull/19261#discussion_r139309246 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -2508,6 +2508,14 @@ object functions { def current_date():

[GitHub] spark pull request #19261: [SPARK-22040] Add current_date function with time...

2017-09-17 Thread jaceklaskowski
Github user jaceklaskowski commented on a diff in the pull request: https://github.com/apache/spark/pull/19261#discussion_r139309261 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -2508,6 +2508,14 @@ object functions { def current_date():

[GitHub] spark pull request #19135: [SPARK-21923][CORE]Avoid calling reserveUnrollMem...

2017-09-17 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/19135#discussion_r139309495 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -190,11 +190,11 @@ private[spark] class MemoryStore( //

[GitHub] spark issue #19254: [MINOR][CORE] Cleanup dead code and duplication in Mem. ...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19254 **[Test build #3924 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3924/testReport)** for PR 19254 at commit

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r139311391 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/SelectedField.scala --- @@ -0,0 +1,144 @@ +/* + * Licensed to the

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r139311726 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala --- @@ -0,0 +1,130 @@ +/* + *

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r139311738 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala --- @@ -0,0 +1,130 @@ +/* + *

[GitHub] spark issue #19262: [MINOR][ML] Remove unnecessary default value setting for...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19262 **[Test build #81857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81857/testReport)** for PR 19262 at commit

[GitHub] spark issue #19230: [SPARK-22003][SQL] support array column in vectorized re...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19230 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19259: [BACKPORT-2.1][SPARK-19318][SPARK-22041][SQL] Doc...

2017-09-17 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/19259 [BACKPORT-2.1][SPARK-19318][SPARK-22041][SQL] Docker test case failure: `SPARK-16625: General data types to be mapped to Oracle` … options in case-sensitive manner. ## What changes were

[GitHub] spark issue #19230: [SPARK-22003][SQL] support array column in vectorized re...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19230 **[Test build #81850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81850/testReport)** for PR 19230 at commit

[GitHub] spark issue #19230: [SPARK-22003][SQL] support array column in vectorized re...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19230 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81850/ Test FAILed. ---

[GitHub] spark pull request #19255: [WIP][SPARK-22029][PySpark] Add lru_cache to _par...

2017-09-17 Thread maver1ck
Github user maver1ck commented on a diff in the pull request: https://github.com/apache/spark/pull/19255#discussion_r139305244 --- Diff: python/pyspark/sql/types.py --- @@ -24,6 +24,7 @@ import re import base64 from array import array +from functools import

[GitHub] spark issue #19232: [SPARK-22009][ML] Using treeAggregate improve some algs

2017-09-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19232 Yeah I wonder if this slows things down for smaller data sets, because of the extra levels and checks, but then again, when the aggregation is small, anything's similarly fast. The default depth is

[GitHub] spark pull request #19249: [SPARK-22032][PySpark] Speed up StructType.fromIn...

2017-09-17 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19249#discussion_r139305674 --- Diff: python/pyspark/sql/types.py --- @@ -483,7 +483,8 @@ def __init__(self, fields=None): self.names = [f.name for f in fields]

[GitHub] spark pull request #19249: [SPARK-22032][PySpark] Speed up StructType.fromIn...

2017-09-17 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19249#discussion_r139305834 --- Diff: python/pyspark/sql/types.py --- @@ -619,7 +621,8 @@ def fromInternal(self, obj): # it's already converted by pickler

[GitHub] spark pull request #19249: [SPARK-22032][PySpark] Speed up StructType.fromIn...

2017-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19249#discussion_r139305954 --- Diff: python/pyspark/sql/types.py --- @@ -619,7 +621,8 @@ def fromInternal(self, obj): # it's already converted by pickler

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81853/testReport)** for PR 19249 at commit

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19249 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19249 LGTM too but hey @maver1ck could you add some comments around the codes and move the worst case benchmarks into the PR description? I guess this wouldn't be too demanding. ---

[GitHub] spark issue #19254: [MINOR][CORE] Cleanup dead code and duplication in Mem. ...

2017-09-17 Thread original-brownbear
Github user original-brownbear commented on the issue: https://github.com/apache/spark/pull/19254 Seems to be broken on Jenkins `master` too https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-sbt-hadoop-2.7/3490/ ---

[GitHub] spark issue #19226: [SPARK-21985][PySpark] PairDeserializer is broken for do...

2017-09-17 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19226 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r139312657 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/QueryPlanConstraints.scala --- @@ -77,20 +77,21 @@ trait

[GitHub] spark pull request #19204: [SPARK-21981][PYTHON][ML] Added Python interface ...

2017-09-17 Thread yanboliang
Github user yanboliang commented on a diff in the pull request: https://github.com/apache/spark/pull/19204#discussion_r139312695 --- Diff: python/pyspark/ml/evaluation.py --- @@ -328,6 +329,86 @@ def setParams(self, predictionCol="prediction", labelCol="label", kwargs

[GitHub] spark issue #19259: [BACKPORT-2.1][SPARK-19318][SPARK-22041][SQL] Docker tes...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19259 **[Test build #81851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81851/testReport)** for PR 19259 at commit

[GitHub] spark pull request #19249: [SPARK-22032][PySpark] Speed up StructType.fromIn...

2017-09-17 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19249#discussion_r139306038 --- Diff: python/pyspark/sql/types.py --- @@ -619,7 +621,8 @@ def fromInternal(self, obj): # it's already converted by pickler

[GitHub] spark pull request #19249: [SPARK-22032][PySpark] Speed up StructType.fromIn...

2017-09-17 Thread maver1ck
Github user maver1ck commented on a diff in the pull request: https://github.com/apache/spark/pull/19249#discussion_r139306166 --- Diff: python/pyspark/sql/types.py --- @@ -619,7 +621,8 @@ def fromInternal(self, obj): # it's already converted by pickler

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread maver1ck
Github user maver1ck commented on the issue: https://github.com/apache/spark/pull/19249 Done. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19259: [BACKPORT-2.1][SPARK-19318][SPARK-22041][SQL] Docker tes...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19259 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19260: [SPARK-22043][PYTHON] Improves error message for show_pr...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19260 **[Test build #81855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81855/testReport)** for PR 19260 at commit

[GitHub] spark issue #19260: [SPARK-22043][PYTHON] Improves error message for show_pr...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19260 **[Test build #81855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81855/testReport)** for PR 19260 at commit

[GitHub] spark issue #19260: [SPARK-22043][PYTHON] Improves error message for show_pr...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19260 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19254: [MINOR][CORE] Cleanup dead code and duplication in Mem. ...

2017-09-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19254 I think it's a problem with a recent change, yeah. Just wait until it can be resolved then we can retest --- - To unsubscribe,

[GitHub] spark pull request #19256: [SPARK-21338][SQL]implement isCascadingTruncateTa...

2017-09-17 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/19256#discussion_r139313120 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/AggregatedDialect.scala --- @@ -41,4 +41,8 @@ private class AggregatedDialect(dialects:

[GitHub] spark issue #19204: [SPARK-21981][PYTHON][ML] Added Python interface for Clu...

2017-09-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19204 **[Test build #81858 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81858/testReport)** for PR 19204 at commit

[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

2017-09-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19252 `TruncateTableCommand` also has similar issues. Could you also fix it in this PR? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19256: [SPARK-21338][SQL]implement isCascadingTruncateTable() m...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19256 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81859/ Test FAILed. ---

[GitHub] spark issue #19226: [SPARK-21985][PySpark] PairDeserializer is broken for do...

2017-09-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19226 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19261: [SPARK-22040] Add current_date function with timezone id

2017-09-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19261 Any other database has such an interface? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19259: [BACKPORT-2.1][SPARK-19318][SPARK-22041][SQL] Docker tes...

2017-09-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19259 Have you run the docker test? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion

2017-09-17 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19249 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19226: [SPARK-21985][PySpark] PairDeserializer is broken for do...

2017-09-17 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19226 Merged to master, branch-2.2 and branch-2.1. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

  1   2   >