Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21821
This PR is majorly for Spark 2.3 branch.
The code changes will be removed from the master branch when
https://github.com/apache/spark/pull/21822 is merged. However, the test cases
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21821
@mgaido91 See the comment
https://github.com/apache/spark/pull/21821#issuecomment-407096818
---
-
To unsubscribe, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21866
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21596
@Fokko Let us wait for the code freeze of Spark 2.4 release?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user gatorsmile closed the pull request at:
https://github.com/apache/spark/pull/21869
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21850
Personally, I do not think we need this extra case.
> If primitive has more opportunities for further optimization.
Could you explain m
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21803
@MaxGekk Please include the test case for SHOW CREATE TABLE. Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21866
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21866
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21869
cc @maryannxue @gengliangwang
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21869
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21863
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21729
For other reviewers, this is merged to master/2.3
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/21869
[SPARK-24891][SQL] Fix HandleNullInputsForUDF rule
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21853
LGTM
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21853
Ideally, this is a perfect fix. We can make it more general to remove all
the unnecessary sorts during the query planning. However, this optimization is
still nice to have since the sorts
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21803
It is nice to have. Actually, I believe we need to fix the bug in `SHOW
CREATE TABLE`, which is widely used
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21863
@dilipbiswal This PR is to fix a message. It is nice to have.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21848
LGTM
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21848#discussion_r204971225
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
---
@@ -403,14 +404,14 @@ object SimplifyConditionals
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21834
@maropu Do we have a log message for users to know the generated where
clauses? If not, could you add one
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21818#discussion_r204969835
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1335,7 +1335,9 @@ object SQLConf {
"overwr
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21822
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21866#discussion_r204967223
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala ---
@@ -56,7 +56,7 @@ private[avro] class AvroFileFormat extends
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21851
LGTM
Thanks! Merged to master
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21834#discussion_r204892163
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -1341,6 +1352,70 @@ class JDBCSuite extends QueryTest
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21866
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21866#discussion_r204890425
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala ---
@@ -56,7 +56,7 @@ private[avro] class AvroFileFormat extends
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21857
@dilipbiswal Also post the design doc in the PR description?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21857
cc @ueshin @maryannxue Please review this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204887497
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
---
@@ -52,7 +52,7 @@ trait CheckAnalysis extends
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204887309
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
---
@@ -182,14 +182,16 @@ case class
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204886904
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -1919,6 +1919,21 @@ class Dataset[T] private[sql](
Except
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21752#discussion_r204885925
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala ---
@@ -717,4 +717,9 @@ class DataFrameAggregateSuite extends
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21752#discussion_r204885339
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -470,8 +470,11 @@ class RelationalGroupedDataset protected
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21752#discussion_r204883667
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -470,8 +470,11 @@ class RelationalGroupedDataset protected
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21752
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21403
cc @maryannxue Review this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20699
This only handles a corner case. This is not what we expect.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21440
@squito Thank you!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21851
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21775
The commit has been tested.
LGTM Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21775
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21850#discussion_r204862351
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
---
@@ -414,6 +414,9 @@ object SimplifyConditionals
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21118
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21863
Could you add a test case?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21861
Thanks for your PR, but we plan to propose new changes in DataSourceV2 APIs
very shortly.
---
-
To unsubscribe, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21803
@MaxGekk Thanks for fixing the issue in `SHOW CREATE TABLE`! Could you open
a JIRA and add the JIRA number to this PR. Also, could you add the failed case
in ShowCreateTableSuite
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21851
update the PR description?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21851
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21848#discussion_r204802956
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1627,6 +1627,8 @@ case class
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21823
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21848
Currently, we are setting the expressions `deterministic ` to false when
they are either having side effect or non-deterministic. We already did it for
Hive UDFs who have stateful tags
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21853
cc @maryannxue
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21839
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21851#discussion_r204531052
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -2145,14 +2145,24 @@ class Analyzer
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21848#discussion_r204524410
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
---
@@ -390,6 +390,7 @@ object SimplifyConditionals
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21848#discussion_r204513622
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
---
@@ -390,6 +390,7 @@ object SimplifyConditionals
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21758
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21823#discussion_r204482993
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/SameResultSuite.scala ---
@@ -58,4 +61,16 @@ class SameResultSuite extends QueryTest
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21805
LGTM
Thanks! Merged to master
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21839
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21764
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21821
@hvanhovell The question is whether `HandleNullInputsForUDF ` is the only
rule that caused the issue. If not, we still need to add an AnalysisBarrier
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21821
cc @hvanhovell
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21123
Please do not merge this PR. The major API changes of dsv2 will be sent
soon.
---
-
To unsubscribe, e-mail: reviews
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21764
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21839
@xuanyuanking Could you add an end-to-end test into
ScriptTransformationSuite to verify the results?
---
-
To unsubscribe, e
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21839#discussion_r204286954
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -450,13 +450,16 @@ object ColumnPruning extends
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21775#discussion_r204285948
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
---
@@ -114,7 +114,10 @@ case class CatalogTablePartition
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21775#discussion_r204285853
--- Diff: docs/sql-programming-guide.md ---
@@ -1843,6 +1843,7 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21764
LGTM pending Jenkins
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21764#discussion_r204278968
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -175,6 +191,41 @@ abstract class Optimizer
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21440
cc @squito @mridulm @markhamstra @jerryshao @vanzin @JoshRosen @rxin
@zsxwing
---
-
To unsubscribe, e-mail: reviews
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21440
Although the code quality is pretty good, I am still afraid it could
introduce some unexpected issues. Is that possible we can introduce a conf to
fall back to the previous implementation? We
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21841
@gengliangwang Could you address the conflicts?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21838
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21764#discussion_r204261317
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -160,6 +160,25 @@ abstract class Optimizer
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21821
@cloud-fan This sounds good to me. @maryannxue Please fix the rule
`HandleNullInputsForUDF`?
---
-
To unsubscribe, e-mail
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21821#discussion_r204248020
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -891,8 +891,9 @@ object DDLUtils {
* Throws exception
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21687#discussion_r204225273
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -695,6 +695,41 @@ abstract class
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21798
LGTM
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21832
LGTM
Thanks! Merged to master
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman I still think we need to split it to two PRs. To resolve the
issues you mentioned above, how about creating a separate PR? Only 10 days left
before the code freeze of Spark 2.4. We plan
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21826#discussion_r204192991
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
---
@@ -442,8 +442,6 @@ case class Or(left: Expression
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21831
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21831
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21832
add to whitelist
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21829
LGTM
Thanks! Merged to master
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21508
cc @maropu Help review this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21831#discussion_r204165401
--- Diff: resource-managers/kubernetes/integration-tests/pom.xml ---
@@ -25,7 +25,7 @@
spark-kubernetes-integration-tests_2.11
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21831
cc @mccheah @ssuchter
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21832
Test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21832#discussion_r204161199
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -606,7 +607,15 @@ private[client] class Shim_v0_13 extends
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21832#discussion_r204157323
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -606,7 +606,15 @@ private[client] class Shim_v0_13 extends
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21832#discussion_r204156800
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/FiltersSuite.scala ---
@@ -72,6 +72,10 @@ class FiltersSuite extends SparkFunSuite
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21832
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20057
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21774
Need to revert this PR since it breaks the build.
spark-master-compile-maven-hadoop-2.6 #7902 (broken since this build
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21805#discussion_r204123221
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala
---
@@ -50,6 +50,8 @@ case class CachedRDDBuilder
901 - 1000 of 14035 matches
Mail list logo
