+1 Regards, Viquar Khan On Fri, Jul 10, 2026, 4:05 PM Uroš Bojanić <[email protected]> wrote:
> +1 (non-binding) > > This is a real pain for wide tables, thank you for working on it Huaxin! > > On 2026/07/07 22:41:28 huaxin gao wrote: > > Hi Spark community: > > > > I would like to call a vote on SPIP Write schema narrowing for > column-level > > UPDATE and MERGE in DSv2. > > > > *Motivation* > > > > Today, Spark's row-level UPDATE and MERGE operations always read and > write > > full rows, even when only a small subset of columns is modified. For wide > > tables common in AI/ML workloads, this leads to significant read and > write > > amplification and prevents connectors from implementing efficient > > column-level update strategies. > > > > This SPIP proposes an opt-in Data Source V2 API that allows connectors to > > declare the exact data columns required for UPDATE and MERGE operations. > > Spark will narrow the write schema accordingly while preserving existing > > behavior for connectors that do not opt in. > > > > *Proposal (summary)* > > > > > > - Introduce a new SupportsColumnUpdates mix-in for row-level > operations. > > - Add updatedColumns() to RowLevelOperationInfo so Spark can expose > the > > columns being updated. > > - Add updateSchema() to LogicalWriteInfo to expose the narrowed write > > schema for update rows. > > - Add default DataWriter.writeUpdate(...) methods for handling > narrowed > > update rows. > > - The design is fully additive and backward compatible. Existing > > connectors continue to receive full-row writes unless they explicitly > opt > > into the new capability. > > > > Full API details, design rationale, rejected alternatives, and > > compatibility considerations are documented in the SPIP. > > > > *Relevant links* > > > > > > - SPIP > > < > https://docs.google.com/document/d/1-Wiw9U54ESpbLakb9Cn_mO4AviM4nrk4TF7rNhI3JZg/edit?tab=t.0#heading=h.yoitjxhaitk8 > > > > - Discussion thread > > <https://www.mail-archive.com/dev%40spark.apache.org/msg35434.html> > > - JIRA <https://issues.apache.org/jira/browse/SPARK-56599> > > > > *Vote* > > > > Please vote on accepting this proposal as an official SPIP (the SPIP text > > above; implementation and follow-up JIRAs can land incrementally after > > acceptance). > > > > [ ] +1: Accept the proposal as an official SPIP > > > > [ ] +0: No opinion > > > > [ ] -1: I do not think we should adopt this SPIP (please explain why) > > > > The vote will remain open for at least 72 hours. > > > > Thanks to everyone who participated in the discussion and helped refine > the > > design. > > > > Best, > > > > Huaxin > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
