I’d like to call a vote for committing to getting DataSourceV2 in a functional state for Spark 3.0.
For more context, please see the discussion thread, but here is a quick summary about what this commitment means: - We think that a “functional DSv2” is an achievable goal for the Spark 3.0 release - We will consider this a blocker for Spark 3.0, and take reasonable steps to make it happen - We will *not* delay the release without a community discussion Here’s what we’ve defined as a functional DSv2: - Add a plugin system for catalogs - Add an interface for table catalogs (see the ongoing SPIP vote) - Add an implementation of the new interface that calls SessionCatalog to load v2 tables - Add a resolution rule to load v2 tables from the v2 catalog - Add CTAS logical and physical plan nodes - Add conversions from SQL parsed plans to v2 logical plans (e.g., INSERT INTO support) Please vote in the next 3 days on whether you agree with committing to this goal. [ ] +1: Agree that we should consider a functional DSv2 implementation a blocker for Spark 3.0 [ ] +0: . . . [ ] -1: I disagree with this goal because . . . Thank you! -- Ryan Blue Software Engineer Netflix
