Thanks to everyone that attended the sync! We had some good discussions. Here are my notes for anyone that missed it or couldn’t join the live stream. If anyone wants to add to this, please send additional thoughts or corrections.
*Attendees:* - Ryan Blue - Netflix - Using v2 to integrate Iceberg with Spark. SQL, DDL/schema evolution, delete support, and hidden partitioning working in Netflix’s branch. - John Zhuge - Netflix - Working on multi-catalog support. - Felix Cheung - Uber - Interested in integrating Uber data sources. External catalog would be useful - Reynold Xin - DataBricks - Working more on SQL and sources - Arun M - HortonWorks - Interested in streaming, unified continuous and micro-batch modes - Dale Richardson - Private developer - Interested in non-FS based partitions - Dongjoon Hyun - HortonWorks - Looking at ORC support in v2, transactional processing on the write side, data lineage - Genglian Wang - DataBricks - ORC with v2 API - Hyukjin Kwon - HortonWorks - DSv2 API implementation for Hive warehouse, LLAP - Kevin Yu - IBM - Design for v2 - Matt Cheah - Palantir - Interested in integrating a custom data store - Thomas D’Silva - Salesforce - Interested in a v2 Phoenix connector - Wenchen Fan - DataBricks - Proposed DSv2 - Xiao Li - DataBricks - SparkSQL, reviews v2 patches - Yuanjian Li - Interested in continuous streaming, new catalog API *Goals for this sync:* - Build consensus around the context that affects roadmap planning - Set priorities and some reasonable milestones - In the future, we’ll open things up to more general technical discussions, but we will be more effective if we are aligned first. *Conclusions:* - Current blocker is Wenchen’s API update. Please review the refactor proposal <https://docs.google.com/document/d/1uUmKCpWLdh9vHxP7AWJ9EgbwB_U6T3EJYNjhISGmiQg/edit#heading=h.l22hv3trducc> and PR #22547 <https://github.com/apache/spark/pull/22547> - Catalog support: this is a high priority blocker for SQL and real use of DSv2 - Please review the TableCatalog SPIP proposal <https://docs.google.com/document/d/1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo/edit#heading=h.m45webtwxf2d> and the implementation, PR #21306 <https://github.com/apache/spark/pull/21306> - A proposal for incrementally introducing catalogs is in PR #21978 <https://github.com/apache/spark/pull/21978> - Generic and specific catalog support should use the same API - Replacing the current global catalog will be done in parts with more specific APIs like TableCatalog and FunctionCatalog - Behavior compatibility: - V2 will have well-defined behavior, primarily implemented by Spark to ensure consistency across sources (e.g. CTAS) - Uses of V1 sources should not see behavior changes when sources are updated to use V2. - Reconciling these concerns is difficult. File-based sources may need to implement compatibility hacks, like checking spark.sql.sources.partitionOverwriteMode - Explicit overwrite is preferred to automatic partition overwrite. This mechanism could be used to translate some behaviors of INSERT OVERWRITE ... PARTITION for V2 sources. *Context that affects roadmap planning:* This is a *summary* of the long discussion, not quotes. It may not be in the right order, but I think it captures the highlights. - The community adopted the SPIP to standardize logical plans <https://docs.google.com/document/d/1gYm5Ji2Mge3QBdOliFV5gSPTKlX4q1DCBXIkiyMv62A/edit?ts=5a987801#heading=h.m45webtwxf2d> and this requires a catalog API for sources <https://docs.google.com/document/d/1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo/edit#heading=h.m45webtwxf2d>. With multi-catalog support also coming soon, it makes sense to incorporate this in the design and planning. - Wenchen mentioned that there are two types of catalogs. First, generic catalogs that can track tables with configurable implementations (like the current catalog that can track Parquet, JDBC, JSON, etc. tables). Second, there are specific catalogs that expose a certain type of table (like a JDBC catalog that exposes all of the tables in a relational DB or a Cassandra catalog). - Ryan: we should be able to use the same catalog API for both of these use cases. - Reynold: DataBricks is interested in a catalog API and it should replace the existing API. Replacing the existing API is difficult because there are many concerns, like tracking functions. The current API is complicated and may be difficult to replace. - Ryan: Past discussions have suggested replacing the current catalog API in pieces, like the proposed TableCatalog API for named tables, a FunctionCatalog API to track UDFs, and a PathCatalog for path-based tables. I’ve also proposed a way to maintain behavior by keeping the existing code paths separate from v2 code paths using CatalogTableIdentifier. (PR #21978 <https://github.com/apache/spark/pull/21978>) - Reynold: How would path-based tables work? Most DataBricks users don’t use a table catalog - Ryan: SQL execution should be based on a Table abstraction (where table is a source of data and could be a stream). Different catalog types, like path-based or name-based, would provide table instances that are used by the rest of Spark. - Reynold: Are catalogs only needed for CTAS? - Ryan: No, v2 should support full DDL operations. Netflix has create/drop/alter implemented and has extended DDL to full schema evolution including nested types. I see no reason not to add this upstream. - Reynold: The problem is that there are a lot of difficult design challenges for a replacement catalog API. - Ryan: What needs to be passed for the table catalog API is well-defined because it is SQL DDL. - We’ve already agreed that Wenchen’s API update is the next blocker, so please review the refactor proposal <https://docs.google.com/document/d/1uUmKCpWLdh9vHxP7AWJ9EgbwB_U6T3EJYNjhISGmiQg/edit#heading=h.l22hv3trducc> and PR #22547 <https://github.com/apache/spark/pull/22547> - By standardizing behavior in V2, we get consistent and reliable behavior across sources. But because V1 behavior is not consistent, we can’t simply move everything to V2. - Example consistency problems: SaveMode.Overwrite may drop and recreate a table, may drop all rows and insert, or may overwrite portions of a table. Path-based tables don’t use an existence check, so all writes are validated using CTAS rules and not the insert rules. - Reynold: We should be able to mimic what the file-based sources do (e.g. parquet, orc) - Matt: We are interested in RTAS, file source, and metastore - Ryan: Keeping the same behavior as the file sources is not possible because that behavior differs based on Spark properties, like spark.sql.sources.partitionOverwriteMode introduced by SPARK-20236 <https://issues.apache.org/jira/browse/SPARK-20236> - Felix and Reynold: We can’t change the behavior of file-based sources - Ryan: Agreed, but we also need to deliver well-defined behavior without requiring a variety of modes that sources are required to implement for V2. - Ryan: Main problem is INSERT OVERWRITE ... PARTITIONS. This query implicitly overwrites data. Overwrites should be explicit. We have built INSERT OVERWRITE table WHERE delete-expr SELECT ... to make the data that is replaced explicit. Spark could use explicit overwrites to implement some overwrite behaviors, like static partition overwrites. *Next sync will include more specific technical topics, please send suggestions!* On Thu, Oct 25, 2018 at 1:09 PM Ryan Blue <rb...@netflix.com> wrote: Hi everyone, > > There's been some great discussion for DataSourceV2 in the last few > months, but it has been difficult to resolve some of the discussions and I > don't think that we have a very clear roadmap for getting the work done. > > To coordinate better as a community, I'd like to start a regular sync-up > over google hangouts. We use this in the Parquet community to have more > effective community discussions about thorny technical issues and to get > aligned on an overall roadmap. It is really helpful in that community and I > think it would help us get DSv2 done more quickly. > > Here's how it works: people join the hangout, we go around the list to > gather topics, have about an hour-long discussion, and then send a summary > of the discussion to the dev list for anyone that couldn't participate. > That way we can move topics along, but we keep the broader community in the > loop as well for further discussion on the mailing list. > > I'll volunteer to set up the sync and send invites to anyone that wants to > attend. If you're interested, please reply with the email address you'd > like to put on the invite list (if there's a way to do this without > specific invites, let me know). Also for the first sync, please note what > times would work for you so we can try to account for people in different > time zones. > > For the first one, I was thinking some day next week (time TBD by those > interested) and starting off with a general roadmap discussion before > diving into specific technical topics. > > Thanks, > > rb > > -- > Ryan Blue > Software Engineer > Netflix > -- Ryan Blue Software Engineer Netflix