+1 (non-binding) This is a real pain for wide tables, thank you for working on it Huaxin!
On 2026/07/07 22:41:28 huaxin gao wrote: > Hi Spark community: > > I would like to call a vote on SPIP Write schema narrowing for column-level > UPDATE and MERGE in DSv2. > > *Motivation* > > Today, Spark's row-level UPDATE and MERGE operations always read and write > full rows, even when only a small subset of columns is modified. For wide > tables common in AI/ML workloads, this leads to significant read and write > amplification and prevents connectors from implementing efficient > column-level update strategies. > > This SPIP proposes an opt-in Data Source V2 API that allows connectors to > declare the exact data columns required for UPDATE and MERGE operations. > Spark will narrow the write schema accordingly while preserving existing > behavior for connectors that do not opt in. > > *Proposal (summary)* > > > - Introduce a new SupportsColumnUpdates mix-in for row-level operations. > - Add updatedColumns() to RowLevelOperationInfo so Spark can expose the > columns being updated. > - Add updateSchema() to LogicalWriteInfo to expose the narrowed write > schema for update rows. > - Add default DataWriter.writeUpdate(...) methods for handling narrowed > update rows. > - The design is fully additive and backward compatible. Existing > connectors continue to receive full-row writes unless they explicitly opt > into the new capability. > > Full API details, design rationale, rejected alternatives, and > compatibility considerations are documented in the SPIP. > > *Relevant links* > > > - SPIP > > <https://docs.google.com/document/d/1-Wiw9U54ESpbLakb9Cn_mO4AviM4nrk4TF7rNhI3JZg/edit?tab=t.0#heading=h.yoitjxhaitk8> > - Discussion thread > <https://www.mail-archive.com/dev%40spark.apache.org/msg35434.html> > - JIRA <https://issues.apache.org/jira/browse/SPARK-56599> > > *Vote* > > Please vote on accepting this proposal as an official SPIP (the SPIP text > above; implementation and follow-up JIRAs can land incrementally after > acceptance). > > [ ] +1: Accept the proposal as an official SPIP > > [ ] +0: No opinion > > [ ] -1: I do not think we should adopt this SPIP (please explain why) > > The vote will remain open for at least 72 hours. > > Thanks to everyone who participated in the discussion and helped refine the > design. > > Best, > > Huaxin > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
