Re: DataSourceV2 community sync #3

Xiao, For the questions in this last email about how catalogs interact and how functions and other future features work: we discussed those last night. As I said then, I think that the right approach is incremental. We don’t want to design all of that in one gigantic proposal up front. To do that

Re: DataSourceV2 community sync #3

Ryan, All the proposal I read is only related to Table metadata. Catalog contains the metadata of database, functions, columns, views, and so on. When we have multiple catalogs, how these catalogs interact with each other? How the global catalog works? How a view, table, function, database and col

Catalog API discussion (was: DataSourceV2 community sync #3)

Hi Wenchen, I’ll add my responses inline. The answers are based on the proposed TableCatalog API: - SPIP: Spark table metadata - PR #21306

Re: DataSourceV2 community sync #3

Xiao, Please have a look at the pull requests and documents I've posted over the last few months. If you still have questions about how you might plug in Glue, let me know and I can clarify. rb On Thu, Nov 29, 2018 at 2:56 PM Xiao Li wrote: > Ryan, > > Thanks for leading the discussion and se

Re: DataSourceV2 community sync #3

Ryan, Thanks for leading the discussion and sending out the memo! > Xiao suggested that there are restrictions for how tables and functions > interact. Because of this, he doesn’t think that separate TableCatalog and > FunctionCatalog APIs are feasible. Anything is possible. It depends on how

Re: DataSourceV2 community sync #3

Hi everyone, Here are my notes from last night’s sync. Some attendees that joined during discussion may be missing, since I made the list while we were waiting for people to join. If you have topic suggestions for the next sync, please start sending them to me. Thank you! *Attendees:* Ryan Blue