Github user xuanyuanking closed the pull request at:
https://github.com/apache/spark/pull/20675
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20675
@HeartSaVioR Thanks for your reply, sorry for just seen your comment. Yep,
will keep tracking this feature after we supports shuffled stateful operators
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22918
The `as the spark-25902 mentioned.` in pr description maybe a typo?
SPARK-25892?
---
-
To unsubscribe, e-mail: reviews
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r230316414
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroEncoder.scala ---
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r230316478
--- Diff:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ---
@@ -1374,4 +1377,185 @@ class AvroSuite extends QueryTest
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r230305677
--- Diff:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ---
@@ -1374,4 +1377,185 @@ class AvroSuite extends QueryTest
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r229775793
--- Diff:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ---
@@ -1374,4 +1377,185 @@ class AvroSuite extends QueryTest
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r229770190
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1617,6 +1617,58 @@ case class
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r229768227
--- Diff:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ---
@@ -1374,4 +1377,182 @@ class AvroSuite extends QueryTest
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r229767694
--- Diff:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ---
@@ -1374,4 +1377,182 @@ class AvroSuite extends QueryTest
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r229765028
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroEncoder.scala ---
@@ -0,0 +1,534 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22878#discussion_r229762665
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroEncoder.scala ---
@@ -0,0 +1,534 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22899#discussion_r229609343
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1051,30 +1034,65 @@ class Analyzer
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22899#discussion_r229608676
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1051,30 +1034,65 @@ class Analyzer
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22878
Thanks @gengliangwang and @HyukjinKwon. Done in this commit.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22878
also cc @bdrillard, link this to #21348.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22878
@dongjoon-hyun Thanks for your comment, let me see how to achieve this,
@bdrillard 's commits based on databricks/spark-avro
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/22878
[SPARK-25789][SQL] Support for Dataset of Avro
## What changes were proposed in this pull request?
Please credit to @bdrillard cause this mainly based on his previous work
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21348
@bdrillard Thanks for our reply, of cause I'll request your review and make
sure I understand your implement correctly.
```
Should we go ahead and re-open it and follow through
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22855#discussion_r228716995
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -298,30 +312,40 @@ class KryoDeserializationStream
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22855#discussion_r228716497
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -92,6 +94,18 @@ class KryoSerializer(conf: SparkConf
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22855#discussion_r228716824
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoSerializerBenchmark.scala
---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22855#discussion_r228716844
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala ---
@@ -33,6 +33,7 @@ import org.apache.spark.serializer.KryoTest
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21348
The jira SPARK-25789 guide me here, thanks for @bdrillard your great job,
we also meet the requirement on supporting dataset of avro during Structure
Streaming. I'm adapting your code
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21348#discussion_r228563440
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1799,3 +1805,65 @@ case class
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22795#discussion_r227021073
--- Diff: python/pyspark/sql/functions.py ---
@@ -3023,6 +3023,42 @@ def pandas_udf(f=None, returnType=None,
functionType=None
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22746
Thanks all reviewers! Sorry for still having some mistake in new doc and
I'll keep checking on this.
---
-
To unsubscribe
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22746#discussion_r226288610
--- Diff: docs/sql-data-sources-other.md ---
@@ -0,0 +1,114 @@
+---
+layout: global
+title: Other Data Sources
+displayTitle: Other
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22746
@kiszk Great thanks for all the detailed check, addressed in 17995f9. Also
double checked by grep the typo for each error you found
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22746#discussion_r226226005
--- Diff: docs/sql-data-sources-other.md ---
@@ -0,0 +1,114 @@
+---
+layout: global
+title: Other Data Sources
+displayTitle: Other
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22746#discussion_r226208608
--- Diff: docs/sql-distributed-sql-engine.md ---
@@ -0,0 +1,85 @@
+---
+layout: global
+title: Distributed SQL Engine
+displayTitle
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22746#discussion_r226202227
--- Diff: docs/_data/menu-sql.yaml ---
@@ -0,0 +1,81 @@
+- text: Getting Started
+ url: sql-getting-started.html
+ subitems
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22746#discussion_r226202492
--- Diff: docs/sql-data-sources-other.md ---
@@ -0,0 +1,114 @@
+---
+layout: global
+title: Other Data Sources
+displayTitle: Other
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22746
My pleasure, thanks for reviewing this!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22746#discussion_r226011439
--- Diff: docs/sql-reference.md ---
@@ -0,0 +1,641 @@
+---
+layout: global
+title: Reference
+displayTitle: Reference
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22746#discussion_r226011057
--- Diff: docs/_data/menu-sql.yaml ---
@@ -0,0 +1,79 @@
+- text: Getting Started
+ url: sql-getting-started.html
+ subitems
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22575#discussion_r225998489
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/StreamTableDDLCommandSuite.scala
---
@@ -0,0 +1,43 @@
+/*
+ * Licensed
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22575#discussion_r225997731
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -631,6 +631,33 @@ object SQLConf {
.intConf
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22575#discussion_r225992780
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
---
@@ -63,7 +63,9 @@ private[kafka010
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22746#discussion_r225794532
--- Diff: docs/sql-reference.md ---
@@ -0,0 +1,641 @@
+---
+layout: global
+title: Reference
+displayTitle: Reference
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22746#discussion_r225794477
--- Diff: docs/sql-getting-started.md ---
@@ -0,0 +1,369 @@
+---
+layout: global
+title: Getting Started
+displayTitle: Getting Started
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22746#discussion_r225789933
--- Diff: docs/sql-getting-started.md ---
@@ -0,0 +1,369 @@
+---
+layout: global
+title: Getting Started
+displayTitle: Getting Started
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22746
@gatorsmile Sorry for the late on this, please have a look when you have
time.
---
-
To unsubscribe, e-mail: reviews
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/22746
[SPARK-24499][Doc] Split the page of sql-programming-guide
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22561#discussion_r225029074
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
---
@@ -39,21 +40,31 @@ private[sql
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22561#discussion_r225029122
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
---
@@ -39,21 +40,31 @@ private[sql
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22165#discussion_r224955955
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/ContextBarrierStateSuite.scala
---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22583
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22590#discussion_r223414086
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
---
@@ -194,6 +195,22 @@ class CSVSuite extends
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/22583
[SPARK-10816][SS] SessionWindow support for Structure Streaming
## What changes were proposed in this pull request?
As the discussion in
[SPARK-10816](https://issues.apache.org/jira
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22326
Thanks everyone for your review and advise.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22165#discussion_r220986263
--- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala ---
@@ -187,6 +191,12 @@ private[spark] class BarrierCoordinator
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22165#discussion_r220984492
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BarrierCoordinatorSuite.scala ---
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22165#discussion_r220984340
--- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala ---
@@ -187,6 +191,12 @@ private[spark] class BarrierCoordinator
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22165#discussion_r220983212
--- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala ---
@@ -141,7 +145,7 @@ private[spark] class BarrierCoordinator
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220777535
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/python/BatchEvalPythonExecSuite.scala
---
@@ -100,6 +105,29 @@ class
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220777383
--- Diff: python/pyspark/sql/tests.py ---
@@ -552,6 +552,96 @@ def test_udf_in_filter_on_top_of_join(self):
df = left.crossJoin(right
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220628624
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/python/BatchEvalPythonExecSuite.scala
---
@@ -100,6 +104,28 @@ class
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220576239
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,53 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220576188
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,53 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220576115
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,53 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220576062
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,53 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220575840
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,53 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22165
gental ping @jiangxb1987 @kiszk
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220568332
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,51 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220567623
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,51 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220567381
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,51 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220567301
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,51 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220567216
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,51 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220562279
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,51 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220526661
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,51 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220524866
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,51 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220524840
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,51 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220523238
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,51 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220522504
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,51 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220479127
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,56 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220478633
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,56 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220437721
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1308,6 +1312,16 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220433980
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,56 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220433916
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,56 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220433940
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,56 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220433995
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -165,6 +165,8 @@ abstract class Optimizer
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220433896
--- Diff: python/pyspark/sql/tests.py ---
@@ -552,6 +552,92 @@ def test_udf_in_filter_on_top_of_join(self):
df = left.crossJoin(right
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220433190
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,56 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220432728
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,56 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220432468
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1304,10 +1307,27 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220418201
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1304,10 +1307,27 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220267389
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1304,10 +1307,27 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220236374
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,60 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220128279
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1234,6 +1237,59 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220111919
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1234,6 +1237,59 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220109689
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1234,6 +1237,59 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220102842
--- Diff: python/pyspark/sql/tests.py ---
@@ -547,6 +547,92 @@ def test_udf_in_filter_on_top_of_join(self):
df = left.crossJoin(right
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22326
```
move this rule to optimizer, as the last batch (but before the
UpdateAttributeReferences batch). Since we apply this rule after filter
pushdown, we can simply pull out any python udf
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22326
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22524
@viirya As @shaneknapp reply in mail-list, you can try
https://hadrian.ist.berkeley.edu/jenkins/. Thanks @shaneknapp
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22524
Got it, same with me :(
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22524
```
Is jenkins down now?
```
Does this means you got a `Reason: Error during SSL Handshake with remote
server` after open the jenkins link
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22326
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22326
@cloud-fan Great thanks for your offline guidance, as our discussion, I
reimplement this by adding a new rule `HandlePythonUDFInJoinCondition` in
Analyzer, revert all changes in `Optimizer
101 - 200 of 777 matches
Mail list logo
