[ 
https://issues.apache.org/jira/browse/SPARK-33638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243803#comment-17243803
 ] 

Jungtaek Lim commented on SPARK-33638:
--------------------------------------

I don't agree with handling this in DataStreamWriter, hence I changed the 
title. My claim is designing DataStreamWriterV2, nothing else.

I also don't agree that we need to deal with partition columns verification in 
such way. DataFrameWriterV2 does this nicely, via branching the path between 
appending/overwriting/truncating table vs creating/replacing table and enforce 
latter whenever the configuration for creating table is provided. I think this 
is pretty much clearer for end users, rather than letting they concern about 
the impact.

For sure, even we address it with DataStreamWriterV2, we still need to deal 
with the consistency in DataStreamWriter.toTable(). Given DataStreamWriterV2 is 
taking place and recommended for table write, that would be less important.

> Full support of V2 table creation in Structured Streaming writer path
> ---------------------------------------------------------------------
>
>                 Key: SPARK-33638
>                 URL: https://issues.apache.org/jira/browse/SPARK-33638
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.1.0
>            Reporter: Yuanjian Li
>            Priority: Blocker
>
> Currently, we want to add support of creating if not exists in 
> DataStreamWriter.toTable API. Since the file format in streaming doesn't 
> support DSv2 for now, the current implementation mainly focuses on V1 
> support. We need more work to do for the full support of V2 table creation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to