[GitHub] spark issue #22447: [SPARK-25450][SQL] PushProjectThroughUnion rule uses the...

2018-09-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22447 Thanks! Merged to master/2.4/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #22475: [SPARK-4502][SQL] Rename to spark.sql.optimizer.nestedSc...

2018-09-19 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22475 Thanks! Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22447: [SPARK-25450][SQL] PushProjectThroughUnion rule u...

2018-09-19 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22447#discussion_r218689985 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PushProjectThroughUnionSuite.scala --- @@ -0,0 +1,56

[GitHub] spark issue #22461: [SPARK-25453][SQL][TEST] OracleIntegrationSuite IllegalA...

2018-09-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22461 add to whitelist --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #22447: [SPARK-25450][SQL] PushProjectThroughUnion rule uses the...

2018-09-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22447 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21451: [SPARK-24296][CORE] Replicate large blocks as a stream.

2018-09-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21451 @squito Thanks for digging it! This PR introduced the failed test case. We have to know whether it exposes any serious bug (if it is not introduced by this PR) and impacts our 2.4

[GitHub] spark pull request #22295: [SPARK-25255][PYTHON]Add getActiveSession to Spar...

2018-09-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22295#discussion_r218309924 --- Diff: python/pyspark/sql/session.py --- @@ -252,6 +253,22 @@ def newSession(self): """ return self.__class__

[GitHub] spark issue #22440: [SPARK-24151][SQL] Case insensitive resolution of CURREN...

2018-09-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22440 Thanks! Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r218307450 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/QueryExecutionSuite.scala --- @@ -16,11 +16,33 @@ */ package

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-17 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r218303899 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -99,10 +99,11 @@ private[spark] object Utils extends Logging

[GitHub] spark pull request #22420: [SPARK-25429][SQL]Use Set improve SparkListenerBu...

2018-09-17 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22420#discussion_r218293704 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala --- @@ -83,7 +83,7 @@ class SQLAppStatusListener

[GitHub] spark pull request #17400: [SPARK-19981][SQL] Respect aliases in output part...

2018-09-17 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17400#discussion_r218261028 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/AliasAwareOutputPartitioning.scala --- @@ -0,0 +1,89 @@ +/* + * Licensed

[GitHub] spark issue #22430: [SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite datatype ...

2018-09-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22430 This is a pure test only PR. Backporting to 2.3 is optional. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22430: [SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite datatype ...

2018-09-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22430 Thanks! Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #22426: [SPARK-25436] Bump master branch version to 2.5.0-SNAPSH...

2018-09-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22426 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22396: [SPARK-23425][SQL][FOLLOWUP] Support wildcards in...

2018-09-15 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22396#discussion_r217897114 --- Diff: docs/sql-programming-guide.md --- @@ -1898,6 +1898,7 @@ working with timestamps in `pandas_udf`s to get the best performance, see

[GitHub] spark issue #22427: [SPARK-25438][SQL][TEST] Fix FilterPushdownBenchmark to ...

2018-09-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22427 cc @rdblue --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20433: [SPARK-23264][SQL] Make INTERVAL keyword optional in INT...

2018-09-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20433 @maropu We need a new SQLConf in Parser. See my example in https://github.com/apache/spark/pull/21922 --- - To unsubscribe

[GitHub] spark issue #22430: [SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite datatype ...

2018-09-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22430 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22426: [SPARK-25436] Bump master branch version to 2.5.0...

2018-09-15 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22426#discussion_r217891517 --- Diff: project/MimaExcludes.scala --- @@ -1202,6 +1206,7 @@ object MimaExcludes { } def excludes(version: String) = version match

[GitHub] spark pull request #22426: [SPARK-25436] Bump master branch version to 2.5.0...

2018-09-15 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22426#discussion_r217889470 --- Diff: project/MimaExcludes.scala --- @@ -1202,6 +1206,7 @@ object MimaExcludes { } def excludes(version: String) = version match

[GitHub] spark pull request #22426: [SPARK-25436] Bump master branch version to 2.5.0...

2018-09-14 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22426#discussion_r217876458 --- Diff: docs/_config.yml --- @@ -14,8 +14,8 @@ include: # These allow the documentation to be updated with newer releases # of Spark

[GitHub] spark issue #22426: [SPARK-25436] Bump master branch version to 2.5.0-SNAPSH...

2018-09-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22426 cc @yhuai @rxin @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22426: [SPARK-25436] Bump master branch version to 2.5.0...

2018-09-14 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/22426 [SPARK-25436] Bump master branch version to 2.5.0-SNAPSHOT ## What changes were proposed in this pull request? In the dev list, we can still discuss whether the next version is 2.5.0

[GitHub] spark issue #21451: [SPARK-24296][CORE] Replicate large blocks as a stream.

2018-09-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21451 https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-branch-2.4-test-maven-hadoop-2.7/21

[GitHub] spark issue #21451: [SPARK-24296][CORE] Replicate large blocks as a stream.

2018-09-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21451 ``` org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 7, localhost, executor 1

[GitHub] spark issue #22395: [SPARK-16323][SQL] Add IntegralDivide expression

2018-09-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22395 Could we check the definition of div in MySQL? Is it the same as the one implemented in this PR? https://dev.mysql.com/doc/refman/8.0/en/arithmetic-functions.html#operator_div

[GitHub] spark issue #22408: [SPARK-25417][SQL] ArrayContains function may return inc...

2018-09-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22408 My general idea is to avoid risky implicit type casting at the beginning. We can relax it in the future, if needed. After all, users can manually cast the types after seeing the reasonable error

[GitHub] spark issue #22408: [SPARK-25417][SQL] ArrayContains function may return inc...

2018-09-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22408 What is the corresponding behavior in Presto? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #22410: [SPARK-25418][SQL] The metadata of DataSource table shou...

2018-09-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22410 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22410: [SPARK-25418][SQL] The metadata of DataSource tab...

2018-09-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22410#discussion_r217603253 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -1309,6 +1312,8 @@ object HiveExternalCatalog

[GitHub] spark pull request #21048: [SPARK-23966][SS] Refactoring all checkpoint file...

2018-09-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21048#discussion_r217407633 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -930,6 +930,13 @@ object SQLConf { .intConf

[GitHub] spark pull request #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in...

2018-09-13 Thread gatorsmile
Github user gatorsmile closed the pull request at: https://github.com/apache/spark/pull/22403 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in...

2018-09-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22403#discussion_r217265214 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala --- @@ -37,6 +38,7 @@ class

[GitHub] spark issue #22407: [SPARK-25416][SQL] ArrayPosition function may return inc...

2018-09-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22407 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22388: Revert [SPARK-24882][SQL] improve data source v2 API fro...

2018-09-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22388 Unable to merge to master. @cloud-fan Could you submit another PR to the master branch? --- - To unsubscribe, e-mail

[GitHub] spark pull request #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in...

2018-09-12 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/22403 [SPARK-25402][SQL][BACKPORT-2.2] Null handling in BooleanSimplification ## What changes were proposed in this pull request? This PR is to fix the null handling in BooleanSimplification

[GitHub] spark pull request #22390: [SPARK-25402][SQL] Null handling in BooleanSimpli...

2018-09-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22390#discussion_r216713161 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -263,10 +263,12 @@ object BooleanSimplification

[GitHub] spark issue #22389: [SPARK-17916][SPARK-25241][SQL][FOLLOW-UP] Fix empty str...

2018-09-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22389 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22390: [SPARK-25402][SQL] Null handling in BooleanSimplificatio...

2018-09-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22390 cc @cloud-fan @adrian-ionescu --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22390: [SPARK-25402][SQL] Null handling in BooleanSimpli...

2018-09-11 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/22390 [SPARK-25402][SQL] Null handling in BooleanSimplification ## What changes were proposed in this pull request? This PR is to fix the null handling in BooleanSimplification. In the rule

[GitHub] spark issue #21433: [SPARK-23820][CORE] Enable use of long form of callsite ...

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21433 @michaelmior Since Spark 2.4 is branch cut, this PR still needs more review. I would revert this PR from branch 2.4 and master first. We can discuss the conf and implementation in the master

[GitHub] spark issue #17650: [SPARK-20350] Add optimization rules to apply Complement...

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17650 I am fixing it now. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #22140: [SPARK-25072][PySpark] Forbid extra value for custom Row

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22140 We are very conservative when backporting the PR to the released version. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #20611: [SPARK-23425][SQL]Support wildcard in HDFS path f...

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20611#discussion_r216426579 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -303,94 +303,44 @@ case class LoadDataCommand

[GitHub] spark issue #17650: [SPARK-20350] Add optimization rules to apply Complement...

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17650 Just FYI, we found a bug in this rule, regarding NULL handling. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #20611: [SPARK-23425][SQL]Support wildcard in HDFS path f...

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20611#discussion_r216413405 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -303,94 +303,44 @@ case class LoadDataCommand

[GitHub] spark issue #22140: [SPARK-25072][PySpark] Forbid extra value for custom Row

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22140 @BryanCutler @HyukjinKwon Thanks for your understanding. Normally, we are very conservative to introduce any potential behavior change to the released version. I just reverted it from

[GitHub] spark pull request #21433: [SPARK-23820][CORE] Enable use of long form of ca...

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21433#discussion_r216213289 --- Diff: core/src/main/scala/org/apache/spark/storage/RDDInfo.scala --- @@ -53,10 +55,16 @@ class RDDInfo( } private[spark] object

[GitHub] spark pull request #21433: [SPARK-23820][CORE] Enable use of long form of ca...

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21433#discussion_r216209988 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -72,6 +72,9 @@ package object config { private[spark] val

[GitHub] spark pull request #21433: [SPARK-23820][CORE] Enable use of long form of ca...

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21433#discussion_r216209901 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -72,6 +72,9 @@ package object config { private[spark] val

[GitHub] spark pull request #21433: [SPARK-23820][CORE] Enable use of long form of ca...

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21433#discussion_r216209470 --- Diff: core/src/main/scala/org/apache/spark/storage/RDDInfo.scala --- @@ -53,10 +55,16 @@ class RDDInfo( } private[spark] object

[GitHub] spark issue #18142: [SPARK-20918] [SQL] Use FunctionIdentifier as function i...

2018-09-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18142 We do not need to follow Hive if Hive does not follow SQL compliance. Our main goal is to follow the mainstream DBMS vendors. BTW, we can enhance our parser to recognize the other

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22377 cc @cloud-fan @MaxGekk --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Conver...

2018-09-09 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/22377 [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a value of StructType to a DDL string ## What changes were proposed in this pull request? Add the version number for the new APIs

[GitHub] spark pull request #21273: [SPARK-17916][SQL] Fix empty string being parsed ...

2018-09-08 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21273#discussion_r216138533 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala --- @@ -164,7 +164,7 @@ class CSVOptions

[GitHub] spark issue #22234: [SPARK-25241][SQL] Configurable empty values when readin...

2018-09-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22234 @MaxGekk Could you take this PR over? I think we need to merge this to Spark 2.4. Users can set the behaviors to the previous one by this new conf `emptyValue`, if needed. Also update

[GitHub] spark issue #22140: [SPARK-25072][PySpark] Forbid extra value for custom Row

2018-09-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22140 @BryanCutler What is the reason to backport this PR? This sounds a behavior change. @xuanyuanking Could you please update the document

[GitHub] spark issue #22361: Revert [SPARK-10399] [SPARK-23879] [SPARK-23762]

2018-09-07 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22361 cc @kiszk @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22361: Revert [SPARK-10399] [SPARK-23879] [SPARK-23762]

2018-09-07 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/22361 Revert [SPARK-10399] [SPARK-23879] [SPARK-23762] ## What changes were proposed in this pull request? When running TPC-DS benchmarks on 2.4 release, we saw more than 10% performance

[GitHub] spark issue #22345: [SPARK-12321][SQL][FOLLOW-UP] Add tests for fromString

2018-09-07 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22345 Thanks! Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRelation ...

2018-09-07 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22270 LGTM Thanks! Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark pull request #22345: [SPARK-12321][SQL][FOLLOW-UP] Add tests for fromS...

2018-09-06 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/22345 [SPARK-12321][SQL][FOLLOW-UP] Add tests for fromString ## What changes were proposed in this pull request? Add test cases for fromString ## How was this patch tested? N/A You

[GitHub] spark issue #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRelation ...

2018-09-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22270 Thank you! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22112 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRelation ...

2018-09-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22270 @dilipbiswal Any update on this PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Rewrite condition when dedupli...

2018-09-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r215402140 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -921,12 +924,18 @@ class Analyzer

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Rewrite condition when dedupli...

2018-09-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r215391569 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -805,10 +807,10 @@ class Analyzer

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Rewrite condition when dedupli...

2018-09-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r215391402 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeMap.scala --- @@ -23,12 +23,14 @@ package

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Rewrite condition when dedupli...

2018-09-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r215391459 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeMap.scala --- @@ -23,12 +23,14 @@ package

[GitHub] spark issue #22336: [SPARK-25306][SQL][FOLLOWUP] Change `test` to `ignore` i...

2018-09-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22336 Sorry,,, I just merged it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22112 The current solution looks good to me for unblocking the Apache 2.4 release. We definitely should continue improving the fix, as what the other reviewers suggested above

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22319 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22234: [SPARK-25241][SQL] Configurable empty values when readin...

2018-09-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22234 Have we documented the behavior changes in the migration guide? If not, can we do it? --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #22234: [SPARK-25241][SQL] Configurable empty values when readin...

2018-09-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22234 Did we introduce any behavior change in https://github.com/apache/spark/pull/21273? Does this PR resolve

[GitHub] spark issue #22171: [SPARK-25177][SQL] When dataframe decimal type column ha...

2018-09-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22171 @vinodkc Could you answer the question from @cloud-fan ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #22112: [SPARK-23243][Core] Fix RDD.repartition() data co...

2018-09-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22112#discussion_r215070653 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1513,37 +1513,34 @@ private[spark] class DAGScheduler

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22112 > So in order to fix that we would need a way to tell the executors to remove that older committed shuffle data @tgravescs It is also hard to implement such a robust solut

[GitHub] spark pull request #21638: [SPARK-22357][CORE] SparkContext.binaryFiles igno...

2018-09-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21638#discussion_r215030825 --- Diff: core/src/main/scala/org/apache/spark/input/PortableDataStream.scala --- @@ -47,7 +47,7 @@ private[spark] abstract class StreamFileInputFormat

[GitHub] spark pull request #20295: [SPARK-23011][SQL][PYTHON] Support alternative fu...

2018-09-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20295#discussion_r214796189 --- Diff: python/pyspark/sql/tests.py --- @@ -4588,6 +4613,80 @@ def test_timestamp_dst(self): result = df.groupby('time').apply(foo_udf

[GitHub] spark pull request #20295: [SPARK-23011][SQL][PYTHON] Support alternative fu...

2018-09-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20295#discussion_r214795846 --- Diff: python/pyspark/sql/tests.py --- @@ -4588,6 +4613,80 @@ def test_timestamp_dst(self): result = df.groupby('time').apply(foo_udf

[GitHub] spark issue #22315: [SPARK-25308][SQL] ArrayContains function may return a e...

2018-09-03 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22315 @dilipbiswal Could we also add the test cases for the other high-order functions, if missing? --- - To unsubscribe, e-mail

[GitHub] spark pull request #22319: [SPARK-25044][SQL][followup] add back UserDefined...

2018-09-03 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22319#discussion_r214784141 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -41,12 +41,16 @@ import

[GitHub] spark pull request #22259: [SPARK-25044][SQL] (take 2) Address translation o...

2018-09-03 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22259#discussion_r214585176 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -47,7 +48,8 @@ case class ScalaUDF

[GitHub] spark issue #21638: [SPARK-22357][CORE] SparkContext.binaryFiles ignore minP...

2018-09-02 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21638 @bomeng Could you submit a follow-up PR to add a test case? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #21638: [SPARK-22357][CORE] SparkContext.binaryFiles igno...

2018-09-02 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21638#discussion_r214581248 --- Diff: core/src/main/scala/org/apache/spark/input/PortableDataStream.scala --- @@ -47,7 +47,7 @@ private[spark] abstract class StreamFileInputFormat

[GitHub] spark pull request #21638: [SPARK-22357][CORE] SparkContext.binaryFiles igno...

2018-09-02 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21638#discussion_r214581076 --- Diff: core/src/main/scala/org/apache/spark/input/PortableDataStream.scala --- @@ -47,7 +47,7 @@ private[spark] abstract class StreamFileInputFormat

[GitHub] spark pull request #22063: [WIP][SPARK-25044][SQL] Address translation of LM...

2018-09-02 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22063#discussion_r214560313 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -40,7 +41,7 @@ import

[GitHub] spark pull request #22048: [SPARK-25108][SQL] Fix the show method to display...

2018-08-31 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22048#discussion_r214490659 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -2794,6 +2794,30 @@ private[spark] object Utils extends Logging

[GitHub] spark pull request #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRe...

2018-08-31 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22270#discussion_r214432388 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -85,12 +85,12 @@ class DataFrameFunctionsSuite extends

[GitHub] spark pull request #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRe...

2018-08-31 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22270#discussion_r214432272 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1464,12 +1465,14 @@ case class

[GitHub] spark issue #22292: [SPARK-25286][CORE] Removing the dangerous parmap

2018-08-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22292 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22300: [SPARK-25296][SQL][TEST] Create ExplainSuite

2018-08-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22300 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22300: [SPARK-25296][SQL][TEST] Create ExplainSuite

2018-08-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22300 @kiszk Could you help move the related test cases in StreamSuite.scala to here? --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22300: [SPARK-25296][SQL][TEST] Create ExplainSuite

2018-08-31 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22300#discussion_r214396672 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/ExplainSuite.scala --- @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #22300: [SPARK-25296][SQL][TEST] Create ExplainSuite

2018-08-31 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/22300 [SPARK-25296][SQL][TEST] Create ExplainSuite ## What changes were proposed in this pull request? Move the output verification of Explain test cases to a new suite ExplainSuite

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-08-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21721 I also agree with @zsxwing . We should revert these two PRs. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-08-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21721 Also ping @rxin @marmbrus . Another related PR is https://github.com/apache/spark/pull/21819. --- - To unsubscribe, e-mail

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-08-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22112 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22276: [SPARK-25242][SQL] make sql config setting fluent

2018-08-29 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22276 We can think about this API change in 3.0 release. Now, Spark 2.4 release is not allowed to break the binary compatibility

<    1   2   3   4   5   6   7   8   9   10   >