Thanks for clarification, Wenchen. I’ll create JIRA for item 2 and do some investigation on UDF catalog.
From: Wenchen Fan <cloud0...@gmail.com> Date: Monday, January 13, 2020 at 4:17 PM To: "colinma(马骏杰)" <coli...@tencent.com> Cc: "dev@spark.apache.org" <dev@spark.apache.org> Subject: Re: Question about Datasource V2(Internet mail) 1. we plan to add view support in future releases. 2. can you open a JIRA ticket? This seems like a bug to me. 3. instead of defining a lot of fields in the table, we decide to use properties to keep all the extra information. We've defined some reserved properties like "comment", "location", which is not user properties and have special meanings. We can add "Create Time" as a reserved property as well, as this should be set by Spark, not users. 4. we plan to add view catalog, UDF catalog, bucket join, but we don't have an official roadmap yet. On Fri, Jan 10, 2020 at 10:19 AM colinma(马骏杰) <coli...@tencent.com<mailto:coli...@tencent.com>> wrote: Hi devs, I did some investigation on data source v2 feature and have some questions: 1. View is not supported for now, eg, CREATE VIEW, this feature is not finished or datasource v2 won’t support view? 2. Column resolution are failed when use datasource v2, eg, select tbl.col from tbl will failed with cannot resolve '`tbl.col`' given input columns select col from tbl is ok 1. For “describe table”, when implement the datasource V2 table, we can set the user properties and get it according to the interface in Table: Map<String, String> properties(). But for some properties, eg, “Create time”, “Create by”, they are not user properties, should we also put them in properties? 2. Is there a plan for datasource v2 and I’m interested in take some related tasks. Best regards, Colin Ma
