Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21853
Thank you very much @gatorsmile and @maropu
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21863
@gatorsmile Got it. Thank you.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21863
@gatorsmile Hi sean, isn't @mgaido91 working in the same area with the in
subq pr ?
---
-
To unsubscribe, e-mail: reviews
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204891972
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1275,6 +1276,64 @@ object
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21857
@gatorsmile I have the link to the design doc in the description ? Is there
another way ?
---
-
To unsubscribe, e-mail
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r20408
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
---
@@ -52,7 +52,7 @@ trait CheckAnalysis extends
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204777353
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
---
@@ -182,14 +182,16 @@ case class
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204775243
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1275,6 +1276,64 @@ object
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204774985
--- Diff: sql/core/src/test/resources/sql-tests/inputs/except-all.sql ---
@@ -0,0 +1,146 @@
+CREATE TEMPORARY VIEW tab1 AS SELECT * FROM VALUES
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204774728
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
---
@@ -222,6 +222,32 @@ case class Stack(children
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204773666
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
---
@@ -222,6 +222,32 @@ case class Stack(children
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204773452
--- Diff: sql/core/src/test/resources/sql-tests/inputs/except-all.sql ---
@@ -0,0 +1,146 @@
+CREATE TEMPORARY VIEW tab1 AS SELECT * FROM VALUES
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204772193
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
---
@@ -182,14 +182,16 @@ case class
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21845
@HyukjinKwon Super. Thanks a lot for fixing.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21845
@HyukjinKwon I saw the following test run for 11 minutes on jenkins for one
of my PR. Not sure if its a transient problem. Just thought, i should let you
know. On the nightly runs, should we
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21857#discussion_r204679789
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
---
@@ -222,6 +222,37 @@ case class Stack(children
GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/21857
[SPARK-21274] Implement EXCEPT ALL clause.
## What changes were proposed in this pull request?
Implements EXCEPT ALL clause through query rewrites using existing
operators in Spark
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/16677
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/21853
[SPARK-23957][SQL] Sorts in subqueries are redundant and can be removed
## What changes were proposed in this pull request?
Thanks to @henryr for the original idea at
https://github.com
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21822#discussion_r204169456
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
---
@@ -533,7 +537,8 @@ trait CheckAnalysis extends
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21822#discussion_r204166360
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
---
@@ -533,7 +537,8 @@ trait CheckAnalysis extends
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21813
Thank you very much @gatorsmile @viirya
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21049
@gatorsmile Sure.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21822#discussion_r203924609
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
---
@@ -533,7 +537,8 @@ trait CheckAnalysis extends
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21813#discussion_r203917186
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -442,17 +442,35 @@ class Analyzer
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21813#discussion_r203917039
--- Diff: sql/core/src/test/resources/sql-tests/inputs/grouping_set.sql ---
@@ -13,5 +13,39 @@ SELECT a, b, c, count(d) FROM grouping GROUP BY a, b, c
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21813#discussion_r203882365
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -442,17 +442,32 @@ class Analyzer
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21813#discussion_r203861013
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveGroupingAnalyticsSuite.scala
---
@@ -91,6 +91,40 @@ class
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21813#discussion_r203859940
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -442,17 +442,32 @@ class Analyzer
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21813
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/21813
[SPARK 24424] Support ANSI-SQL compliant syntax for GROUPING SET
## What changes were proposed in this pull request?
Enhances the parser and analyzer to support ANSI compliant syntax
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20795#discussion_r202177330
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/LookupFunctionsSuite.scala
---
@@ -0,0 +1,107 @@
+/*
+ * Licensed
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20795#discussion_r202129753
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/LookupFunctionsSuite.scala
---
@@ -0,0 +1,107 @@
+/*
+ * Licensed
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20795#discussion_r202128129
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1204,16 +1207,32 @@ class Analyzer(
* only
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20795
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20795#discussion_r201926692
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1204,16 +1207,32 @@ class Analyzer(
* only
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21590
Thank you very much @gatorsmile @maropu @viirya @HyukjinKwon @gengliangwang
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r198315709
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -109,6 +134,20 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r198022351
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,32 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r198022370
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,32 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r198022388
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,32 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r198022295
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -174,3 +209,25 @@ object JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r198022202
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -150,6 +183,7 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r198022168
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,32 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r198022141
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -174,3 +209,25 @@ object JDBCOptions
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21590
@gatorsmile @maropu I have hopefully addressed the comments. Please take a
look when you get a chance.
---
-
To unsubscribe
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21590
@gatorsmile Thanks a lot. I will process your comments and get back.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r197355161
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,38 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r196975167
--- Diff: docs/sql-programming-guide.md ---
@@ -1302,9 +1302,18 @@ the following case-insensitive options:
dbtable
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r196897983
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,38 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r196895138
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,38 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r196847962
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,38 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r196848042
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,38 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r196847882
--- Diff: docs/sql-programming-guide.md ---
@@ -1302,9 +1302,18 @@ the following case-insensitive options:
dbtable
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r196846549
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,38 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r196634511
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -1206,4 +1207,92 @@ class JDBCSuite extends SparkFunSuite
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21590
@gatorsmile Sorry to be late on this. Please look at this when you have
time.
---
-
To unsubscribe, e-mail: reviews
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r196490214
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,38 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r196489749
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -109,6 +134,20 @@ class JDBCOptions
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21590#discussion_r196487627
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -65,13 +65,38 @@ class JDBCOptions
GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/21590
[SPARK-24423][SQL] Add a new option for JDBC sources
## What changes were proposed in this pull request?
Here is the description in the JIRA -
Currently, our JDBC connector
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21240
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21240#discussion_r186279795
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
---
@@ -222,6 +222,51 @@ case class Stack(children
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21240#discussion_r186279765
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -702,6 +703,20 @@ object TypeCoercion
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21240#discussion_r186277072
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
---
@@ -222,6 +222,51 @@ case class Stack(children
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21240#discussion_r186277065
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/udtf_replicate_rows.sql ---
@@ -0,0 +1,38 @@
+CREATE TEMPORARY VIEW tab1 AS SELECT * FROM
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21240#discussion_r186277062
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -702,6 +703,20 @@ object TypeCoercion
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21240
@maropu @viirya Thanks for the comments. I have made the changes.
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/21240
[SPARK-21274][SQL] Add a new generator function replicate_rows to support
EXCEPT ALL and INTERSECT ALL
## What changes were proposed in this pull request?
Add a new UDTF replicate_rows
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21049
@henryr
I might be a bit of a hardliner on this, but I think it's correct to
eliminate the {{ORDER BY}} from common table expressions (e.g. MSSQL agrees
with me, see this link).
DB
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21049
@henryr
Is there any reason to actually use an alias at the root of a plan like
this (outside of composing with other plans, where this optimization would
apply)?
I can't think
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21174
@gatorsmile @maropu Thank you very much !!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21049
@henryr Since SubqueryAlias is used as a correlation name and used mostly
for resolving attributes, in my understanding its not safe to apply this
optimization. I will borrow @gatorsmile 's
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21174
@gatorsmile Thanks a lot. Addressed the comments.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21174#discussion_r184609062
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
---
@@ -55,7 +55,9 @@ private[sql] object
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21174#discussion_r184609073
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
---
@@ -955,4 +955,28 @@ class SubquerySuite extends QueryTest
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21174#discussion_r184609040
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -76,7 +76,10 @@ object FileSourceStrategy
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21174
@maropu So with the fix, if the query predicate contains an scalar subquery
expression, then that expression is not considered for partition pruning. For
example, if the predicate
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21174
@maropu Thanks for your response. ORC has CONVERT_METASTORE_ORC set to
false as default. So its not converted to a file based datasource. If we set
this to true then we would see the same issue
GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/21174
[SPARK-24085] Query returns UnsupportedOperationException when scalar
subquery is present in partitioning expression
## What changes were proposed in this pull request?
In this case
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/21049
@henryr @gatorsmile I agree with Sean. To the best of my knowledge, spark
does not treat "select ... from ( query)" as a subquery. It treats it as an
aliased query. Please see the t
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
Thanks a lot @cloud-fan @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20795#discussion_r176292845
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1192,11 +1195,23 @@ class Analyzer(
* @see
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20579#discussion_r176232893
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -719,4 +720,27 @@ object DataSource extends
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20579#discussion_r176187422
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -719,4 +720,27 @@ object DataSource extends
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20868
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20579#discussion_r175952404
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
---
@@ -77,7 +77,6 @@ class
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20579#discussion_r175952408
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -719,4 +720,27 @@ object DataSource extends
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
@cloud-fan ok.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
@cloud-fan Thank you. I assumed (wrongly) that we don't want to change the
behaviour for an external file based datasource and we wanted to scope our
check only to spark built in data sources
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
@gatorsmile When you get a chance, could you please see if the check for
internal datasource looks reasonable
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
@gatorsmile Thank you Sean. I will follow your suggestion.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
@cloud-fan OK.. i was thinking of adding this check in each built in
datasource like Text, CSV, Parquet, ORC, JSON etc. Just like we check it in
Parquet with this PR. Would you have any
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20795#discussion_r175156300
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1192,11 +1195,23 @@ class Analyzer(
* @see
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20579#discussion_r175154988
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -542,6 +542,11 @@ case class DataSource
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20795#discussion_r175026674
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1192,11 +1195,24 @@ class Analyzer(
* @see
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20795#discussion_r175025479
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1192,11 +1195,23 @@ class Analyzer(
* @see
401 - 500 of 1257 matches
Mail list logo
