Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r171618754
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StringFormat.scala
---
@@ -0,0 +1,98
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r171617989
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StringFormat.scala
---
@@ -0,0 +1,98
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r171347060
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StringFormat.scala
---
@@ -0,0 +1,94
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r171311785
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StringFormat.scala
---
@@ -0,0 +1,94
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20647
One more thing on matching the behavior of other nodes for the schema:
since you agree that the scan nodes should report types, can you open an issue
for it? I'm also fine adding it here instead
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20647
@cloud-fan, I found a couple new problems but otherwise this looks good. I
see you rolled back the equality changes for the scan node and streaming
relation. What is the JIRA issue to update v2
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170989638
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExec.scala
---
@@ -36,10 +37,22 @@ import
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170989246
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StringFormat.scala
---
@@ -0,0 +1,94
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20647
> I've rolled back unnecessary style-only changes, but leave the one that
clean up unused imports.
Sounds good. Thank you!
> If we think this is the right eq
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170774593
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2QueryPlan.scala
---
@@ -0,0 +1,99 @@
+/*
+ * Licensed
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170774510
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2QueryPlan.scala
---
@@ -0,0 +1,99 @@
+/*
+ * Licensed
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170774203
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2QueryPlan.scala
---
@@ -0,0 +1,99 @@
+/*
+ * Licensed
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20647
Thanks for removing the equality methods. This changes equality for the
scan and streaming relation, though. Are those significant changes?
I still think this should not be committed until
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170311026
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
---
@@ -23,11 +23,11 @@ import
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170307194
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -35,15 +35,14 @@ case class
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170303937
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -107,17 +106,24 @@ case class
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170303699
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
---
@@ -77,31 +79,32 @@ class MicroBatchExecution
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170302233
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
---
@@ -23,11 +23,11 @@ import
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20645
> A quick question: after this change, extraJavaOptions is still able to
cleanly override whatever's set in defaultJavaOptions, is that right?
No, the intent is for both sets of opti
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170097795
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
---
@@ -415,12 +418,14 @@ class MicroBatchExecution
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170092263
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala
---
@@ -17,15 +17,12 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170092091
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala ---
@@ -492,16 +492,16 @@ class StreamSuite extends StreamTest
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170091954
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
---
@@ -415,12 +418,14 @@ class MicroBatchExecution
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170090758
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
---
@@ -415,12 +418,14 @@ class MicroBatchExecution
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170090323
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
---
@@ -77,31 +79,32 @@ class MicroBatchExecution
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170089913
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownOperatorsToDataSource.scala
---
@@ -26,7 +26,7 @@ object
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170089465
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
---
@@ -23,11 +23,11 @@ import
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170089368
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -107,17 +106,24 @@ case class
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170088798
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -35,15 +35,14 @@ case class
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r170087573
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -107,17 +106,24 @@ case class
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20645
I like the `ConfigBuilder` approach. That would make this much more useful.
I'll add an implementation like that.
I think append option syntax would be confusing for users
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20535#discussion_r169708283
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -190,10 +190,15 @@ class DataFrameReader private[sql](sparkSession
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20535#discussion_r169707938
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/DataSourceOptions.java
---
@@ -97,4 +130,20 @@ public double getDouble(String key, double
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20535#discussion_r169707644
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/DataSourceOptions.java
---
@@ -27,6 +27,39 @@
/**
* An immutable string
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20603
Thanks for reviewing, @cloud-fan!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20645
@vanzin, you might be interested in this one because it makes applying
administrator settings for Spark much easier
GitHub user rdblue opened a pull request:
https://github.com/apache/spark/pull/20645
SPARK-23472: Add defaultJavaOptions for drivers and executors.
## What changes were proposed in this pull request?
This adds two new config properties: spark.driver.defaultJavaOptions
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20603
@cloud-fan, this implements SPARK-23418, which rejects user-supplied
schemas when ReadSupportWithSchema is not available
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
Thanks for all your help getting this committed, @cloud-fan!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
Thanks for the update! Enjoy your vacation, and thanks for letting me know.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r168933227
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownOperatorsToDataSource.scala
---
@@ -17,130 +17,55
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
@cloud-fan, is there anything else that needs to be updated, or is this
ready to be merged?
---
-
To unsubscribe, e-mail
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
@cloud-fan, can you have a look at this? I've made the requested changes
and tests are passing.
---
-
To unsubscribe, e-mail
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
Okay, I rebased again after SPARK-23303 was reverted.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
@cloud-fan, I've rebased and made the requested changes.
#20603 reverts the last commit that adds back support for user-supplied
schemas that are identical to the source schema
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20603
@cloud-fan, here's the PR to fix user-supplied schema behavior. Once #20387
is committed, I'll rebase and remove its commits from this PR
GitHub user rdblue opened a pull request:
https://github.com/apache/spark/pull/20603
[SPARK-23418][SQL]: Fail DataSourceV2 reads when user schema is passed, but
not supported.
## What changes were proposed in this pull request?
DataSourceV2 initially allowed user-supplied
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20490
Accidentally closed & reopened. Oops.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comm
GitHub user rdblue reopened a pull request:
https://github.com/apache/spark/pull/20490
[SPARK-23323][SQL]: Support commit coordinator for DataSourceV2 writes
## What changes were proposed in this pull request?
DataSourceV2 batch writes should use the output commit
Github user rdblue closed the pull request at:
https://github.com/apache/spark/pull/20490
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20490
@cloud-fan, I don't think we can rely on user code propagating
`InterruptedException`. How to handle that exception in particular is one of
the least understood parts of the JVM
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r167645370
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataSourceWriter.java
---
@@ -78,10 +88,11 @@ default void onDataWriterCommit
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r167644516
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataSourceWriter.java
---
@@ -78,10 +88,11 @@ default void onDataWriterCommit
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167639047
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,80 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167638689
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -37,22 +100,129 @@ case class
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167374952
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,131 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167374648
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,131 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167374579
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,131 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167369570
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,131 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167346297
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,131 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167346043
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownOperatorsToDataSource.scala
---
@@ -17,130 +17,55
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167345858
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,131 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167343364
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,131 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r167321301
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataSourceWriter.java
---
@@ -62,6 +62,16 @@
*/
DataWriterFactory
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167320953
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,130 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167295506
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,130 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167288341
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,130 @@
package
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
> See FindDataSourceTable.readDataSourceTable about how we handle the path
option.
Sorry, what do you want to cha
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r167280511
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataSourceWriter.java
---
@@ -62,6 +62,16 @@
*/
DataWriterFactory
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20490
The test failure doesn't look related to these changes to me. How can I get
on the list to ask jenkins to retest a PR
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20490
@cloud-fan, please have another look. I fixed the problems you spotted.
I haven't added support for the streaming side. It is different enough that
I think we should do it in a follow-up
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r167011220
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataSourceWriter.java
---
@@ -78,10 +78,11 @@ default void onDataWriterCommit
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r167011250
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataSourceWriter.java
---
@@ -78,10 +78,11 @@ default void onDataWriterCommit
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r167011291
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataWriterFactory.java
---
@@ -32,6 +32,16 @@
@InterfaceStability.Evolving
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r167009143
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala
---
@@ -117,20 +118,43 @@ object
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167005107
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,130 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167004846
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,130 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r167001183
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,130 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r166998163
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,130 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r166997697
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,130 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r166997241
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,130 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r166995080
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataSourceWriter.java
---
@@ -78,10 +78,11 @@ default void onDataWriterCommit
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20535#discussion_r166709990
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -171,7 +171,8 @@ class DataFrameReader private[sql](sparkSession
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20535
This should move the standard options to `DataSourceV2Relation` to avoid
needing to instantiate `DataSourceOptions` wherever the relation is created
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20387#discussion_r166691646
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -17,17 +17,139 @@
package
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
@cloud-fan: Rebased and removed path.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r166684043
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataWriterFactory.java
---
@@ -20,6 +20,7 @@
import java.io.Serializable
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r166463921
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala
---
@@ -117,20 +118,43 @@ object
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20521#discussion_r166425787
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -493,9 +510,23 @@ case class DataSource
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r166418424
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala
---
@@ -117,20 +118,43 @@ object
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20521
@cloud-fan, the tests failed but I believe it was unrelated to this change.
As you noticed, the `table` parameter that was removed wasn't actually used
anywhere
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20521#discussion_r166414290
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -493,9 +510,23 @@ case class DataSource
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
Will do.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20521#discussion_r166410690
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -493,9 +510,23 @@ case class DataSource
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
> Can you give a use case about this?
We've added a resolution rule from `UnresolvedRelation` to
`DataSourceV2Relation` that uses our implementation. `UnresolvedRelation` needs
to pass
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20521#discussion_r166407366
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -493,9 +510,23 @@ case class DataSource
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
The reader is to be created and configured by the relation, then the
relation needs to be able to set the table, path, and other properties. This
adds necessary data to the relation
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20490#discussion_r166398432
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala
---
@@ -117,20 +118,43 @@ object
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20521
Overall this looks fine to me as a fix. Thanks for noting that we want to
move away from commands-within-commands over the next few releases
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/20521#discussion_r166396780
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -493,9 +510,23 @@ case class DataSource
701 - 800 of 1330 matches
Mail list logo
