Thanks for clarification, Wenchen.
I’ll create JIRA for item 2 and do some investigation on UDF catalog.

From: Wenchen Fan <cloud0...@gmail.com>
Date: Monday, January 13, 2020 at 4:17 PM
To: "colinma(马骏杰)" <coli...@tencent.com>
Cc: "dev@spark.apache.org" <dev@spark.apache.org>
Subject: Re: Question about Datasource V2(Internet mail)

1. we plan to add view support in future releases.
2. can you open a JIRA ticket? This seems like a bug to me.
3. instead of defining a lot of fields in the table, we decide to use 
properties to keep all the extra information. We've defined some reserved 
properties like "comment", "location", which is not user properties and have 
special meanings. We can add "Create Time" as a reserved property as well, as 
this should be set by Spark, not users.
4. we plan to add view catalog, UDF catalog, bucket join, but we don't have an 
official roadmap yet.

On Fri, Jan 10, 2020 at 10:19 AM colinma(马骏杰) 
<coli...@tencent.com<mailto:coli...@tencent.com>> wrote:
Hi devs,

I did some investigation on data source v2 feature and have some questions:

  1.  View is not supported for now, eg, CREATE VIEW,  this feature is not 
finished or datasource v2 won’t support view?
  2.  Column resolution are failed when use datasource v2, eg,

select tbl.col from tbl   will failed with cannot resolve '`tbl.col`' given 
input columns

select col from tbl  is ok

  1.  For “describe table”, when implement the datasource V2 table,  we can set 
the user properties and get it according to the interface in Table: Map<String, 
String> properties(). But for some properties, eg,  “Create time”, “Create by”, 
they are not user properties, should we also put them in properties?
  2.  Is there a plan for datasource v2 and I’m interested in take some related 
tasks.



Best regards,
Colin Ma

Reply via email to