Jayesh, The current catalog in Spark is a little weird. It uses a Hive catalog and adds metadata that only Spark understands to track tables, in addition to regular Hive tables. Some of those tables are actually just pointers to tables that exist in some other source of truth. This is what makes the default implementation “generic”.
The reason why Spark is this way is that it only supports one global catalog. I want Spark to be able to use multiple catalogs, so I can plug in a catalog implementation that talks directly to the source of truth for another system, like Cassandra or JDBC. But Cassandra tracks Cassandra tables and wouldn’t be a generic catalog. I also think that the easiest way to expose multiple catalogs is to simply let users specify which one they want to interact with: SELECT * FROM jdbc_test.db.table; USE CATALOG cassandra_prod; SELECT * FROM some_c_table; The public-facing user interaction part is still open for discussion and I think Xiao has a different opinion. The work I’ve posed solves two different problems: 1. How should Spark interact with catalog implementations? For tables, I’ve proposed the TableCatalog API and I wrote an SPIP explaining why. 2. How should Spark internally track tables in different catalogs? For this, I’ve proposed a CatalogTableIdentifier and outlined how to keep these out of paths that don’t expect tables in other catalogs. Spark catalog be the common denominator of the other catalogs (least featured) or a super-feature catalog? I think anything we want to expose in Spark would need to be in an API implemented by catalogs. I prefer an incremental approach here: as we build things that Spark can take advantage of, we can add them to the API that catalog implementations provide. On Sat, Dec 1, 2018 at 9:10 PM Thakrar, Jayesh jthak...@conversantmedia.com <http://mailto:jthak...@conversantmedia.com> wrote: Just curious on the need for a catalog within Spark. > > > > So Spark interface with other systems – many of which have a catalog of > their own – e.g. RDBMSes, HBase, Cassandra, etc. and some don’t (e.g. HDFS, > filesyststem, etc). > > So what is the purpose of having this catalog within Spark for tables > defined in Spark (which could be a front for other “catalogs”)? > > Is it trying to fulfill some void/need….. > > Also, would the Spark catalog be the common denominator of the other > catalogs (least featured) or a super-feature catalog? > > > > *From: *Xiao Li <gatorsm...@gmail.com> > *Date: *Saturday, December 1, 2018 at 10:49 PM > *To: *Ryan Blue <rb...@netflix.com> > *Cc: *"u...@spark.apache.org" <dev@spark.apache.org> > *Subject: *Re: DataSourceV2 community sync #3 > > > > Hi, Ryan, > > > > Let us first focus on answering the most fundamental problem before > discussing various related topics. What is a catalog in Spark SQL? > > > > My definition of catalog is based on the database catalog. Basically, the > catalog provides a service that manage the metadata/definitions of database > objects (e.g., database, views, tables, functions, user roles, and so on). > > > > In Spark SQL, all the external objects accessed through our data source > APIs are called "tables". I do not think we will expand the support in the > near future. That means, the metadata we need from the external data > sources are for table only. > > > > These data sources should not use the Catalog identifier to identify. That > means, in "catalog.database.table", catalog is only used to identify the > actual catalog instead of data sources. > > > > For a Spark cluster, we could mount multiple catalogs (e.g., > hive_metastore_1, hive_metastore_2 and glue_1) at the same time. We could > get the metadata of the tables, database, functions by accessing different > catalog: "hive_metastore_1.db1.tab1", "hive_metastore_2.db2.tab2", > "glue.db3.tab2". In the future, if Spark has its own catalog > implementation, we might have something like, "spark_catalog1.db3.tab2". > The catalog will be used for registering all the external data sources, > various Spark UDFs and so on. > > > > At the same time, we should NOT mix the table-level data sources with > catalog support. That means, "Cassandra1.db1.tab1", "Kafka2.db2.tab1", > "Hbase3.db1.tab2" will not appear. > > > > Do you agree on my definition of catalog in Spark SQL? > > > > Xiao > > > > > > Ryan Blue <rb...@netflix.com> 于2018年12月1日周六 下午7:25写道： > > I try to avoid discussing each specific topic about the catalog federation > before we deciding the framework of multi-catalog supports. > > I’ve tried to open discussions on this for the last 6+ months because we > need it. I understand that you’d like a comprehensive plan for supporting > more than one catalog before moving forward, but I think most of us are > okay with the incremental approach. It’s better to make progress toward the > goal. > > In general, data source API V2 and catalog API should be orthogonal > I agree with you, and they are. The API that Wenchen is working on for > reading and writing data and the TableCatalog API are orthogonal efforts. > As I said, they only overlap with the Table interface, and clearly tables > loaded from a catalog need to be able to plug into the read/write API. > > The reason these two efforts are related is that the community voted to > standardize > logical plans > <https://docs.google.com/document/d/1gYm5Ji2Mge3QBdOliFV5gSPTKlX4q1DCBXIkiyMv62A/edit?ts=5a987801#heading=h.m45webtwxf2d>. > Those standard plans have well-defined behavior for operations like CTAS, > instead of relying on the data source plugin to do … something undefined. > To implement this, we need a way for Spark to create tables, drop tables, > etc. That’s why we need a way for sources to plug in Table-related catalog > operations. (Sorry if this was already clear. I know I talked about it at > length in the first v2 sync up.) > > While the two APIs are orthogonal and serve different purposes, > implementing common operations requires that we have both. > > I would not call it a table catalog. I do not expect the data source > should/need to implement a catalog. Since you might want an atomic CTAS, we > can improve the table metadata resolution logic to support it with > different resolution priorities. For example, try to get the metadata from > the external data source, if the table metadata is not available in the > catalog. > > It sounds like your definition of a “catalog” is different. I think you’re > referring to a global catalog? Could you explain what you’re talking about > here? > > I’m talking about an API to interface with an external data source, which > I think we need for the reasons I outlined above. I don’t care what we call > it, but your comment seems to hint that there would be an API to look up > tables in external sources. That’s the thing I’m talking about. > > CatalogTableIdentifier: The PR is doing nothing but adding an interface. > > Yes. I opened this PR to discuss how Spark should track tables from > different catalogs and avoid passing those references to code paths that > don’t support them. The use of table identifiers with a catalog part was > discussed in the “Multiple catalog support” thread. I’ve also brought it up > and pointed out how I think it should be used in syncs a couple of times. > > Sorry if this discussion isn’t how you would have done it, but it’s a > fairly simple idea that I don’t think requires its own doc. > > > > On Sat, Dec 1, 2018 at 5:12 PM Xiao Li <gatorsm...@gmail.com> wrote: > > Hi, Ryan, > > > > I try to avoid discussing each specific topic about the catalog federation > before we deciding the framework of multi-catalog supports. > > > > - *CatalogTableIdentifier*: The PR > https://github.com/apache/spark/pull/21978 is doing nothing but adding an > interface. In the PR, we did not discuss how to resolve it, any restriction > on the naming and what is a catalog.This requires more doc for explaining > it. For example, > https://docs.microsoft.com/en-us/sql/t-sql/language-elements/transact-sql-syntax-conventions-transact-sql?view=sql-server-2017 > Normally, we do not merge a PR without showing how to use it. > > > > - *TableCatalog*: First, I would not call it a table catalog. I do not > expect the data source should/need to implement a catalog. Since you might > want an atomic CTAS, we can improve the table metadata resolution logic to > support it with different resolution priorities. For example, try to get > the metadata from the external data source, if the table metadata is not > available in the catalog. However, the catalog should do what the catalog > is expected to do. If we follow what our data source API V2 is doing, > basically, the data source is just a table. It is not related to database, > view, or function. Mixing catalog with data source API V2 just makes the > whole things more complex. > > > > In general, data source API V2 and catalog API should be orthogonal. I > believe the data source API V2 and catalog APIs are two separate projects. > Hopefully, you understand my concern. If we really want to mix them > together, I want to read the design of your multi-catalog support and > understand more details. > > > > Thanks, > > > > Xiao > > > > > > > > > > Ryan Blue <rb...@netflix.com> 于2018年12月1日周六 下午3:22写道： > > Xiao, > > > > I do have opinions about how multi-catalog support should work, but I > don't think we are at a point where there is consensus. That's why I've > started discussion threads and added the CatalogTableIdentifier PR instead > of a comprehensive design doc. You have opinions about how users should > interact with catalogs as well (your "federated catalog") and we should > discuss our options here. > > > > But the crucial point is that the user interaction doesn't need to be > completely decided in order to move forward. A design for multi-catalog > support isn't what we need right now; we need an API that plugins can > implement to expose table operations. > > > > I've proposed that API, TableCatalog, and a way to manage catalog plugins. > I've made an argument for why I think that API is flexible enough for the > task and still fairly simple. > > > > I think that we can add TableCatalog now and work on multi-catalog support > incrementally, and I have yet to hear your argument for why that is not the > case. > > > > rb > > > > On Sat, Dec 1, 2018 at 12:36 PM Xiao Li <gatorsm...@gmail.com> wrote: > > Hi, Ryan, > > > > Catalog is a really important component for Spark SQL or any analytics > platform, I have to emphasize. Thus, a careful design is needed to ensure > it works as expected. Based on my previous discussion with many community > members, Spark SQL needs a catalog interface so that we can mount multiple > external physical catalogs and they can be presented as a single logical > catalog [which is a so-called global federated catalog]. In the future, we > can use this interface to develop our own catalog (instead of Hive > metastore) for more efficient metadata management. We can also plug in ACL > management if needed. > > > > Based on your previous answers, it sounds like you have many ideas in your > mind about building a Catalog interface for Spark SQL, but it is not shown > in the design doc. Could you write them down in a single doc? We can try to > leave comments in the design doc, instead of discussing various issues in > PRs, emails and meetings. It can also help the whole community understand > your proposal and post their comments. > > > > Thanks, > > > > Xiao > > > > > > > > Ryan Blue <rb...@netflix.com> 于2018年11月29日周四 下午7:06写道： > > Xiao, > > For the questions in this last email about how catalogs interact and how > functions and other future features work: we discussed those last night. As > I said then, I think that the right approach is incremental. We don’t want > to design all of that in one gigantic proposal up front. To do that is to > put ourselves into analysis paralysis. > > We don’t have a design for how catalogs interact with one another, but I > think we made a strong case for two points: first, that the proposed > structure doesn’t preclude any of those future decisions (hence we should > proceed incrementally). Second, that those situations aren’t that hard to > think through if you’re concerned about them: functions that can run in > Spark can be run on any data, functions that run in external sources cannot > be run on any data. > > You’re right that I haven’t completely covered your *new* questions. But > to the questions in your first email: > > · You asked how, for example, Glue may be plugged in. That is > well covered in the PR that adds catalogs as a plugin > <https://github.com/apache/spark/pull/21306#issue-187572913>, the > response I sent to Wenchen’s questions, and the earlier discussion thread I > posted to this list with the subject “[DISCUSS] Multiple catalog support”. > The short answer is that implementations are configured with Spark config > properties and loaded with reflection. > > · You asked how users implement an external catalog without > adding new data sources. That’s also covered in the “Multiple catalog > support” proposal, the table catalog PR, and ongoing discussions on the v2 > redesign. The answer is that a catalog returns a table instance that > implements the various interfaces from Wenchen’s work. A table may > implement them directly or return other existing implementations. Here’s > how it worked in the old API > <https://github.com/apache/spark/pull/21306/files#diff-db51e7934b9ee539ad599197a935cb86R35> > . > > I hope that you don’t think I expect you to go “without seeing the design”! > > rb > > > > On Thu, Nov 29, 2018 at 3:17 PM Xiao Li <gatorsm...@gmail.com> wrote: > > Ryan, > > > > All the proposal I read is only related to Table metadata. Catalog > contains the metadata of database, functions, columns, views, and so on. > When we have multiple catalogs, how these catalogs interact with each > other? How the global catalog works? How a view, table, function, database > and column is resolved? Do we have nickname, mapping, wrapper? > > > > Or I might miss the design docs you send? Could you post the doc? > > > > Thanks, > > > > Xiao > > > > > > > > > > Ryan Blue <rb...@netflix.com> 于2018年11月29日周四 下午3:06写道： > > Xiao, > > > > Please have a look at the pull requests and documents I've posted over the > last few months. > > > > If you still have questions about how you might plug in Glue, let me know > and I can clarify. > > > > rb > > > > On Thu, Nov 29, 2018 at 2:56 PM Xiao Li <gatorsm...@gmail.com> wrote: > > Ryan, > > > > Thanks for leading the discussion and sending out the memo! > > > > Xiao suggested that there are restrictions for how tables and functions > interact. Because of this, he doesn’t think that separate TableCatalog and > FunctionCatalog APIs are feasible. > > > > Anything is possible. It depends on how we design the two interfaces. Now, > most parts are unknown to me without seeing the design. > > > > I think we need to see the user stories, and high-level design before > working on a small portion of Catalog federation. We do not need an > exhaustive design in the current stage, but we need to know how the new > proposal works. For example, how to plug in a new Hive metastore? How to > plug in a Glue? How do users implement a new external catalog without > adding any new data sources? Without knowing more details, it is hard to > say whether this TableCatalog can satisfy all the requirements. > > > > Cheers, > > > > Xiao > > > > > > Ryan Blue <rb...@netflix.com.invalid> 于2018年11月29日周四 下午2:32写道： > > Hi everyone, > > Here are my notes from last night’s sync. Some attendees that joined > during discussion may be missing, since I made the list while we were > waiting for people to join. > > If you have topic suggestions for the next sync, please start sending them > to me. Thank you! > > *Attendees:* > > Ryan Blue > John Zhuge > Jamison Bennett > Yuanjian Li > Xiao Li > stczwd > Matt Cheah > Wenchen Fan > Genglian Wang > Kevin Yu > Maryann Xue > Cody Koeninger > Bruce Robbins > Rohit Karlupia > > *Agenda:* > > · Follow-up issues or discussion on Wenchen’s PR #23086 > > · TableCatalog proposal > > · CatalogTableIdentifier > > *Notes:* > > · Discussion about PR #23086 > > o Where should the catalog API live since it needs to be accessible to > catalyst rules, but the catalyst module is private? > > o Wenchen suggested creating a sql-api module for v2 API interfaces, > making catalyst depend on it > > o Consensus was to use Wenchen’s suggestion > > · In discussion about #23086, Xiao asked how adding catalog to a > table identifier will work > > o Background from Ryan: existing code paths use TableIdentifier and > don’t expect a catalog portion. If an identifier with a catalog were passed > to existing code, that code may use the default catalog not knowing that a > different one was requested, which would be incorrect behavior. > > o Ryan: The proposal for CatalogTableIdentifier addresses this > problem. TableIdentifier is used for identifiers that have no catalog set. > By enforcing that requirement, passing a TableIdentifier to old code > ensures that no catalogs leak into that code. This is also used when the > catalog is set from context. For example, the TableCatalog API accepts only > TableIdentifier because the catalog is already determined. > > · Xiao asked whether FunctionIdentifier needs to be updated in > the same way as CatalogTableIdentifier. > > o Ryan: Yes, when a FunctionCatalog API is added > > · The remaining time was spent discussing whether the plan to > incrementally replace the current catalog API will work. [Not great notes > here, feel free to add your take in a reply] > > o Xiao suggested that there are restrictions for how tables and > functions interact. Because of this, he doesn’t think that separate > TableCatalog and FunctionCatalog APIs are feasible. > > o Wenchen and Ryan think that functions should be orthogonal to data > sources > > o Matt and Ryan think that catalog design can be done incrementally as > new interfaces (i.e. FunctionCatalog) are added and that the proposed > TableCatalog does not preclude designing for Xiao’s concerns later > > o [I forget who] pointed out that there are restrictions in some > databases for views from different sources > > o There was some discussion about when functions or views cannot be > orthogonal. For example, where the code runs is important. Functions pushed > to sources cannot necessarily be run on other sources and Spark functions > cannot necessarily be pushed down to sources. > > o Xiao would like a full catalog replacement design, including views, > databases, and functions and how they interact, before moving forward with > the proposed TableCatalog API > > o Ryan [and Matt, I think] think that TableCatalog is compatible with > future decisions and the best path forward is to build incrementally. An > exhaustive design process blocks progress on v2. > > > > On Mon, Nov 26, 2018 at 2:54 PM Ryan Blue <rb...@netflix.com> wrote: > > Hi everyone, > > I just sent out an invite for the next DSv2 community sync for Wednesday, > 28 Nov at 5PM PST. > > We have a few topics left over from last time to cover. A few people > wanted to cover catalog APIs, so I put two items on the agenda: > > · The TableCatalog proposal (and other catalog APIs) > > · Using CatalogTableIdentifier to separate v1 and v2 code paths > and avoid unintended behavior changes > > As I noted in the summary last time, please send topics ahead of time so > we can get started more quickly. > > If you would like to be added to the google hangout invite, please let me > know and I’ll add you. Thanks! > > rb > > -- > > Ryan Blue > > Software Engineer > > Netflix > > > > > -- > > Ryan Blue > > Software Engineer > > Netflix > > > > > -- > > Ryan Blue > > Software Engineer > > Netflix > > > > > -- > > Ryan Blue > > Software Engineer > > Netflix > > > > > -- > > Ryan Blue > > Software Engineer > > Netflix > > > > > -- > > Ryan Blue > > Software Engineer > > Netflix > > -- Ryan Blue Software Engineer Netflix
