bump to see anyone interested or concerned about this On Sun, Sep 20, 2020 at 1:59 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote:
> Hi devs, > > We have a capability check in DSv2 defining which operations can be done > against the data source both read and write. The concept was brought in > DSv2, so it's not weird for DSv1 to don't have a concept. > > In SS the problem arises - if I understand correctly, we would like to > couple the output mode in the query and the output table. That said, > complete mode should enforce the output table to truncate the content. > Update mode should enforce the output table to "upsert" or "delete and > append" the content. > > Nothing has been done against the DSv1 sink - Spark doesn't enforce > anything and works as append mode, though the query still respects the > output mode on stateful operations. > > I understand we don't want to make end users surprised on broken > compatibility, but shouldn't it be an "temporary" "exceptional" case > and DSv2 never does it again? I'm seeing many built-in data sources being > migrated to DSv2 with the exception of "do nothing for update/truncate", > which simply destruct the rationalization on capability. > > In addition, they don't add TRUNCATE in capability but add > SupportsTruncate in WriteBuilder, which is weird. It works as of now > because SS misses checking capability on the writer side (I guess it only > checks STREAMING_WRITE), but once we check capability in first place, > things will break. > (I'm looking into adding a writer plan in SS before analyzer, and check > capability there.) > > What would be our best fix on this issue? Would we leave the > responsibility of handling "truncate" on the data source (so do nothing is > fine if it's intended), and just add TRUNCATE to the capability? (That > should be documented in its data source description though.) Or drop the > support on truncate if the data source is unable to truncate? (Foreach and > Kafka output tables will be unable to apply complete mode afterwards.) > > Looking forward to hear everyone's thoughts. > > Thanks, > Jungtaek Lim (HeartSaVioR) >
