[jira] [Commented] (CALCITE-5964) Support additional metadata attributes in GET_TABLES and GET_COLUMNS
[
https://issues.apache.org/jira/browse/CALCITE-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17778893#comment-17778893
]
Julian Hyde commented on CALCITE-5964:
--
[~oliverlee], Do you think the PR is ready? If so I will merge it. I plan to
rename the `CLASS` property to `TYPE` (and `getClazz` to `getType`).
> Support additional metadata attributes in GET_TABLES and GET_COLUMNS
>
>
> Key: CALCITE-5964
> URL: https://issues.apache.org/jira/browse/CALCITE-5964
> Project: Calcite
> Issue Type: New Feature
> Components: avatica
>Reporter: Oliver Lee
>Assignee: Oliver Lee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The goal is for Avatica to be able to handle additional columns in the
> standard JDBC getTables() and getColumns() responses.
> After testing, it appears that the Avatica client in RemoteMeta is able to
> handle Signatures of different shapes that don't necessarily conform to the
> JDBC standard
> [getTables.|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> I have opened a PR that adds a test to RemoteDriverMockTest that verifies
> that if a Signature comes in with additional columns, that Avatica is able to
> handle it fluently.
>
> This ticket is related to CALCITE-5982 , and includes some additions to
> connection properties to allow Calcite to create and send Signatures of
> different shapes.
>
>
>
>
> -The goal is to add to Avatica a mechanism such that additional metadata
> fields pertaining to tables and columns can be transmitted alongside the
> standard JDBC
> [getTables|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> and {{getColumns}} calls.-
> -The Avatica client needs the response to be extensible such that revisions
> to metadata fields send and future additions does not require a new JAR
> file.-
> -Requirements:-
> # -Avatica user does not need to download new jar files if the server
> decides to send over new metadata data in the future-
> # -If the client makes modifications to support additional columns, they
> should always be present in the call and appear with null values, as opposed
> to complete omission (Number of columns in response stays the same)-
> # -Can handle attributes of varying types i.e. {{numberOne: int}} and
> {{booleanOne: boolean}}-
> # -Allows value retrieval from the {{ResultSet}} through calling
> {{resultSet.getInt(“booleanOne”)}} or {{resultSet.getBoolean(“booleanOne”)}}-
> -Current proposal is to modify the {{MetaTable}} and {{MetaColumn}} classes
> to include a map.-
> -{{{}HashMap{}}}, such that when instantiating the
> {{CalciteMetaTable}} in the {{{}ResultSet{}}}, new entries could be added in
> the future without changes to Avatica.-
> -One we have a list of additional metadata fields to be emitted in
> {{{}CalciteMetaImpl{}}}, the {{ResultSet}} would be created with the
> appropriate values.-
> --
> -There are still some challenges identified below and I would love some
> input:-
> -Challenges:-
> * -Currently the {{MetaTable}} class that is instantiated is a
> {{{}CalciteMetaImpl{}}}. For the {{getTables()}} call, the response will be a
> list composed of schema tables of class {{CalciteMetaTable}} and database
> tables which can potentially be overloaded into 1 or more different
> subclasses. From this one heterogeneous list, we must determine the full list
> of columns to be included in the additional metadata hash. My initial plan
> was to provide a function in Calcite’s {{Table}} class such as
> {{getAdditionalColumns}} and allow it to be overloaded, but then I discovered
> the heterogeneity of the list.-
> * -Modifying the MetaTable class to include the hashmap of values could be
> easily done, but the challenge lies at {{{}RemoteMeta{}}}, to be able to
> serialize this cleanly so that requirement (4) is met and users can retrieve
> the values nicely. {{RemoteMeta}} currently serializes the response using
> reflection by looking at MetaTable.class and its attributes. The addition of
> one map is not immediately compatible with iterating over the keys of the map
> and turning each of those into fields. I’m looking into the idea of
> processing the enumerable in {{CalciteMetaImpl}} before the Frame gets
> created-
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5964) Support additional metadata attributes in GET_TABLES and GET_COLUMNS
[
https://issues.apache.org/jira/browse/CALCITE-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763972#comment-17763972
]
Oliver Lee commented on CALCITE-5964:
-
Hey [~julianhyde] ,
I have updated the Jira ticket description to reflect the current findings.
I have a PR for this ready for review:
[https://github.com/apache/calcite-avatica/pull/227]
> Support additional metadata attributes in GET_TABLES and GET_COLUMNS
>
>
> Key: CALCITE-5964
> URL: https://issues.apache.org/jira/browse/CALCITE-5964
> Project: Calcite
> Issue Type: New Feature
> Components: avatica
>Reporter: Oliver Lee
>Assignee: Oliver Lee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The goal is for Avatica to be able to handle additional columns in the
> standard JDBC getTables() and getColumns() responses.
> After testing, it appears that the Avatica client in RemoteMeta is able to
> handle Signatures of different shapes that don't necessarily conform to the
> JDBC standard
> [getTables.|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> I have opened a PR that adds a test to RemoteDriverMockTest that verifies
> that if a Signature comes in with additional columns, that Avatica is able to
> handle it fluently.
>
> This ticket is related to CALCITE-5982 , and includes some additions to
> connection properties to allow Calcite to create and send Signatures of
> different shapes.
>
>
>
>
> -The goal is to add to Avatica a mechanism such that additional metadata
> fields pertaining to tables and columns can be transmitted alongside the
> standard JDBC
> [getTables|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> and {{getColumns}} calls.-
> -The Avatica client needs the response to be extensible such that revisions
> to metadata fields send and future additions does not require a new JAR
> file.-
> -Requirements:-
> # -Avatica user does not need to download new jar files if the server
> decides to send over new metadata data in the future-
> # -If the client makes modifications to support additional columns, they
> should always be present in the call and appear with null values, as opposed
> to complete omission (Number of columns in response stays the same)-
> # -Can handle attributes of varying types i.e. {{numberOne: int}} and
> {{booleanOne: boolean}}-
> # -Allows value retrieval from the {{ResultSet}} through calling
> {{resultSet.getInt(“booleanOne”)}} or {{resultSet.getBoolean(“booleanOne”)}}-
> -Current proposal is to modify the {{MetaTable}} and {{MetaColumn}} classes
> to include a map.-
> -{{{}HashMap{}}}, such that when instantiating the
> {{CalciteMetaTable}} in the {{{}ResultSet{}}}, new entries could be added in
> the future without changes to Avatica.-
> -One we have a list of additional metadata fields to be emitted in
> {{{}CalciteMetaImpl{}}}, the {{ResultSet}} would be created with the
> appropriate values.-
> --
> -There are still some challenges identified below and I would love some
> input:-
> -Challenges:-
> * -Currently the {{MetaTable}} class that is instantiated is a
> {{{}CalciteMetaImpl{}}}. For the {{getTables()}} call, the response will be a
> list composed of schema tables of class {{CalciteMetaTable}} and database
> tables which can potentially be overloaded into 1 or more different
> subclasses. From this one heterogeneous list, we must determine the full list
> of columns to be included in the additional metadata hash. My initial plan
> was to provide a function in Calcite’s {{Table}} class such as
> {{getAdditionalColumns}} and allow it to be overloaded, but then I discovered
> the heterogeneity of the list.-
> * -Modifying the MetaTable class to include the hashmap of values could be
> easily done, but the challenge lies at {{{}RemoteMeta{}}}, to be able to
> serialize this cleanly so that requirement (4) is met and users can retrieve
> the values nicely. {{RemoteMeta}} currently serializes the response using
> reflection by looking at MetaTable.class and its attributes. The addition of
> one map is not immediately compatible with iterating over the keys of the map
> and turning each of those into fields. I’m looking into the idea of
> processing the enumerable in {{CalciteMetaImpl}} before the Frame gets
> created-
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5964) Support additional metadata attributes in GET_TABLES and GET_COLUMNS
[
https://issues.apache.org/jira/browse/CALCITE-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761498#comment-17761498
]
Oliver Lee commented on CALCITE-5964:
-
Hey [~julianhyde]
I've added a commit here (very rough, not tidied up yet) but would love to get
some feedback before I go ahead and make the parallel changes to support
getColumns as well.
[https://github.com/looker-open-source/calcite/pull/37/commits/513f3da154954bbe48f67d9fd9c0e5089c86c34b#diff-202586edbf28484a3a4c1498d8f61096e7a12c0a9c813260fb4282b5c19a3e96]
Since discovering that Avatica itself does not need to be changed to handle
responses of additional shapes, the changes would be in Calcite to support
handling subclasses. My Avatica PR will just include my tests.
My idea is to pass in a subclass of {{CalciteMetaTable}} when instantiating
{{{}CalciteMetaImpl{}}}, passed in through the {{connection.info}} (
{{CalciteConnectionProperty}} ) and use that defined subclass to create
enumerables. The subclass will contain additional attributes that hold the
additional metadata.
The subclass would have the requirements of having the same constructor shape
as CalciteMetaTable,
{quote}{{CalciteMetaTable(Table calciteTable, String tableCat, String
tableSchem, String tableName)}}
{quote}
and also have to extend CalciteMetaTable. It will also have to provide a
function override for {{getColumnNames()}} to list out all of the class'
attributes (in the correct order) which is used for reflection and creating the
list of {{ColumnMetaData}} for the Signature.
> Support additional metadata attributes in GET_TABLES and GET_COLUMNS
>
>
> Key: CALCITE-5964
> URL: https://issues.apache.org/jira/browse/CALCITE-5964
> Project: Calcite
> Issue Type: New Feature
> Components: avatica
>Reporter: Oliver Lee
>Assignee: Oliver Lee
>Priority: Major
>
> The goal is to add to Avatica a mechanism such that additional metadata
> fields pertaining to tables and columns can be transmitted alongside the
> standard JDBC
> [getTables|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> and {{getColumns}} calls.
> The Avatica client needs the response to be extensible such that revisions to
> metadata fields send and future additions does not require a new JAR file.
> Requirements:
> # Avatica user does not need to download new jar files if the server decides
> to send over new metadata data in the future
> # If the client makes modifications to support additional columns, they
> should always be present in the call and appear with null values, as opposed
> to complete omission (Number of columns in response stays the same)
> # Can handle attributes of varying types i.e. {{numberOne: int}} and
> {{booleanOne: boolean}}
> # Allows value retrieval from the {{ResultSet}} through calling
> {{resultSet.getInt(“booleanOne”)}} or {{resultSet.getBoolean(“booleanOne”)}}
> Current proposal is to modify the {{MetaTable}} and {{MetaColumn}} classes to
> include a map.
> {{{}HashMap{}}}, such that when instantiating the
> {{CalciteMetaTable}} in the {{{}ResultSet{}}}, new entries could be added in
> the future without changes to Avatica.
> One we have a list of additional metadata fields to be emitted in
> {{{}CalciteMetaImpl{}}}, the {{ResultSet}} would be created with the
> appropriate values.
>
> There are still some challenges identified below and I would love some input:
> Challenges:
> * Currently the {{MetaTable}} class that is instantiated is a
> {{{}CalciteMetaImpl{}}}. For the {{getTables()}} call, the response will be a
> list composed of schema tables of class {{CalciteMetaTable}} and database
> tables which can potentially be overloaded into 1 or more different
> subclasses. From this one heterogeneous list, we must determine the full list
> of columns to be included in the additional metadata hash. My initial plan
> was to provide a function in Calcite’s {{Table}} class such as
> {{getAdditionalColumns}} and allow it to be overloaded, but then I discovered
> the heterogeneity of the list.
> * Modifying the MetaTable class to include the hashmap of values could be
> easily done, but the challenge lies at {{{}RemoteMeta{}}}, to be able to
> serialize this cleanly so that requirement (4) is met and users can retrieve
> the values nicely. {{RemoteMeta}} currently serializes the response using
> reflection by looking at MetaTable.class and its attributes. The addition of
> one map is not immediately compatible with iterating over the keys of the map
> and turning each of those into fields. I’m looking into the idea of
> processing the enumerable in {{Calcite
[jira] [Commented] (CALCITE-5964) Support additional metadata attributes in GET_TABLES and GET_COLUMNS
[
https://issues.apache.org/jira/browse/CALCITE-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761146#comment-17761146
]
Oliver Lee commented on CALCITE-5964:
-
In terms of having this work with Calcite and making it more generic,
This is where it takes the MetaTable class hardcoded and constructs the list of
{{ColumnMetaData}} which is used to make the resulting Signature
[https://github.com/apache/calcite/blob/ada4fe2715aac8e30751cbcfb84e0b60d89db4ee/core/src/main/java/org/apache/calcite/jdbc/CalciteMetaImpl.java#L270]
To get around a forked commit, maybe I could add to {{MetaImpl}} or the
{{Meta}} interface a {{getMetaTableClass()}} that returns {{MetaTable.class}}
that would be overriden, or passed in somewhere so that
{{CalciteMetaImpl.getTables()}} has access to it.
> Support additional metadata attributes in GET_TABLES and GET_COLUMNS
>
>
> Key: CALCITE-5964
> URL: https://issues.apache.org/jira/browse/CALCITE-5964
> Project: Calcite
> Issue Type: New Feature
> Components: avatica
>Reporter: Oliver Lee
>Assignee: Oliver Lee
>Priority: Major
>
> The goal is to add to Avatica a mechanism such that additional metadata
> fields pertaining to tables and columns can be transmitted alongside the
> standard JDBC
> [getTables|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> and {{getColumns}} calls.
> The Avatica client needs the response to be extensible such that revisions to
> metadata fields send and future additions does not require a new JAR file.
> Requirements:
> # Avatica user does not need to download new jar files if the server decides
> to send over new metadata data in the future
> # If the client makes modifications to support additional columns, they
> should always be present in the call and appear with null values, as opposed
> to complete omission (Number of columns in response stays the same)
> # Can handle attributes of varying types i.e. {{numberOne: int}} and
> {{booleanOne: boolean}}
> # Allows value retrieval from the {{ResultSet}} through calling
> {{resultSet.getInt(“booleanOne”)}} or {{resultSet.getBoolean(“booleanOne”)}}
> Current proposal is to modify the {{MetaTable}} and {{MetaColumn}} classes to
> include a map.
> {{{}HashMap{}}}, such that when instantiating the
> {{CalciteMetaTable}} in the {{{}ResultSet{}}}, new entries could be added in
> the future without changes to Avatica.
> One we have a list of additional metadata fields to be emitted in
> {{{}CalciteMetaImpl{}}}, the {{ResultSet}} would be created with the
> appropriate values.
>
> There are still some challenges identified below and I would love some input:
> Challenges:
> * Currently the {{MetaTable}} class that is instantiated is a
> {{{}CalciteMetaImpl{}}}. For the {{getTables()}} call, the response will be a
> list composed of schema tables of class {{CalciteMetaTable}} and database
> tables which can potentially be overloaded into 1 or more different
> subclasses. From this one heterogeneous list, we must determine the full list
> of columns to be included in the additional metadata hash. My initial plan
> was to provide a function in Calcite’s {{Table}} class such as
> {{getAdditionalColumns}} and allow it to be overloaded, but then I discovered
> the heterogeneity of the list.
> * Modifying the MetaTable class to include the hashmap of values could be
> easily done, but the challenge lies at {{{}RemoteMeta{}}}, to be able to
> serialize this cleanly so that requirement (4) is met and users can retrieve
> the values nicely. {{RemoteMeta}} currently serializes the response using
> reflection by looking at MetaTable.class and its attributes. The addition of
> one map is not immediately compatible with iterating over the keys of the map
> and turning each of those into fields. I’m looking into the idea of
> processing the enumerable in {{CalciteMetaImpl}} before the Frame gets created
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5964) Support additional metadata attributes in GET_TABLES and GET_COLUMNS
[
https://issues.apache.org/jira/browse/CALCITE-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761141#comment-17761141
]
Oliver Lee commented on CALCITE-5964:
-
I've been testing with a mocked json response using {{RemoteDriverMockTest}}
and my finding is that as long as the {{Signature}} is set, the created
ResultSet will take shape (in terms of column ordering and number of columns)
of the supplied signature, and only resort to reflection on {{MetaTable.class}}
if the {{Signature}} is {{{}null{}}}. (Is it expected that signature will be
null?)
With this finding I believe that Avatica doesn't need to have any changes to
support ResultSets with additional columns.
My PR for this ticket will likely just be my test in {{RemoteDriverMockTest}}
> Support additional metadata attributes in GET_TABLES and GET_COLUMNS
>
>
> Key: CALCITE-5964
> URL: https://issues.apache.org/jira/browse/CALCITE-5964
> Project: Calcite
> Issue Type: New Feature
> Components: avatica
>Reporter: Oliver Lee
>Assignee: Oliver Lee
>Priority: Major
>
> The goal is to add to Avatica a mechanism such that additional metadata
> fields pertaining to tables and columns can be transmitted alongside the
> standard JDBC
> [getTables|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> and {{getColumns}} calls.
> The Avatica client needs the response to be extensible such that revisions to
> metadata fields send and future additions does not require a new JAR file.
> Requirements:
> # Avatica user does not need to download new jar files if the server decides
> to send over new metadata data in the future
> # If the client makes modifications to support additional columns, they
> should always be present in the call and appear with null values, as opposed
> to complete omission (Number of columns in response stays the same)
> # Can handle attributes of varying types i.e. {{numberOne: int}} and
> {{booleanOne: boolean}}
> # Allows value retrieval from the {{ResultSet}} through calling
> {{resultSet.getInt(“booleanOne”)}} or {{resultSet.getBoolean(“booleanOne”)}}
> Current proposal is to modify the {{MetaTable}} and {{MetaColumn}} classes to
> include a map.
> {{{}HashMap{}}}, such that when instantiating the
> {{CalciteMetaTable}} in the {{{}ResultSet{}}}, new entries could be added in
> the future without changes to Avatica.
> One we have a list of additional metadata fields to be emitted in
> {{{}CalciteMetaImpl{}}}, the {{ResultSet}} would be created with the
> appropriate values.
>
> There are still some challenges identified below and I would love some input:
> Challenges:
> * Currently the {{MetaTable}} class that is instantiated is a
> {{{}CalciteMetaImpl{}}}. For the {{getTables()}} call, the response will be a
> list composed of schema tables of class {{CalciteMetaTable}} and database
> tables which can potentially be overloaded into 1 or more different
> subclasses. From this one heterogeneous list, we must determine the full list
> of columns to be included in the additional metadata hash. My initial plan
> was to provide a function in Calcite’s {{Table}} class such as
> {{getAdditionalColumns}} and allow it to be overloaded, but then I discovered
> the heterogeneity of the list.
> * Modifying the MetaTable class to include the hashmap of values could be
> easily done, but the challenge lies at {{{}RemoteMeta{}}}, to be able to
> serialize this cleanly so that requirement (4) is met and users can retrieve
> the values nicely. {{RemoteMeta}} currently serializes the response using
> reflection by looking at MetaTable.class and its attributes. The addition of
> one map is not immediately compatible with iterating over the keys of the map
> and turning each of those into fields. I’m looking into the idea of
> processing the enumerable in {{CalciteMetaImpl}} before the Frame gets created
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5964) Support additional metadata attributes in GET_TABLES and GET_COLUMNS
[
https://issues.apache.org/jira/browse/CALCITE-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761080#comment-17761080
]
Julian Hyde commented on CALCITE-5964:
--
It's to be expected that the existing tests seem to work on a 'standard'
configuration (hsql database) whereas this task requires a non-standard
configuration. Tests for extensions have to create a test environment that is
very non-standard. In fact, allowing tests to easily create non-standard
environments is a significant part of the task. Accomplishing it without doing
major damage to the test infrastructure is a fine art.
I don't know the details, but I know for sure that when you are testing the
{{getTables}} call it will not end up calling into hsqldb.
> Support additional metadata attributes in GET_TABLES and GET_COLUMNS
>
>
> Key: CALCITE-5964
> URL: https://issues.apache.org/jira/browse/CALCITE-5964
> Project: Calcite
> Issue Type: New Feature
> Components: avatica
>Reporter: Oliver Lee
>Assignee: Oliver Lee
>Priority: Major
>
> The goal is to add to Avatica a mechanism such that additional metadata
> fields pertaining to tables and columns can be transmitted alongside the
> standard JDBC
> [getTables|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> and {{getColumns}} calls.
> The Avatica client needs the response to be extensible such that revisions to
> metadata fields send and future additions does not require a new JAR file.
> Requirements:
> # Avatica user does not need to download new jar files if the server decides
> to send over new metadata data in the future
> # If the client makes modifications to support additional columns, they
> should always be present in the call and appear with null values, as opposed
> to complete omission (Number of columns in response stays the same)
> # Can handle attributes of varying types i.e. {{numberOne: int}} and
> {{booleanOne: boolean}}
> # Allows value retrieval from the {{ResultSet}} through calling
> {{resultSet.getInt(“booleanOne”)}} or {{resultSet.getBoolean(“booleanOne”)}}
> Current proposal is to modify the {{MetaTable}} and {{MetaColumn}} classes to
> include a map.
> {{{}HashMap{}}}, such that when instantiating the
> {{CalciteMetaTable}} in the {{{}ResultSet{}}}, new entries could be added in
> the future without changes to Avatica.
> One we have a list of additional metadata fields to be emitted in
> {{{}CalciteMetaImpl{}}}, the {{ResultSet}} would be created with the
> appropriate values.
>
> There are still some challenges identified below and I would love some input:
> Challenges:
> * Currently the {{MetaTable}} class that is instantiated is a
> {{{}CalciteMetaImpl{}}}. For the {{getTables()}} call, the response will be a
> list composed of schema tables of class {{CalciteMetaTable}} and database
> tables which can potentially be overloaded into 1 or more different
> subclasses. From this one heterogeneous list, we must determine the full list
> of columns to be included in the additional metadata hash. My initial plan
> was to provide a function in Calcite’s {{Table}} class such as
> {{getAdditionalColumns}} and allow it to be overloaded, but then I discovered
> the heterogeneity of the list.
> * Modifying the MetaTable class to include the hashmap of values could be
> easily done, but the challenge lies at {{{}RemoteMeta{}}}, to be able to
> serialize this cleanly so that requirement (4) is met and users can retrieve
> the values nicely. {{RemoteMeta}} currently serializes the response using
> reflection by looking at MetaTable.class and its attributes. The addition of
> one map is not immediately compatible with iterating over the keys of the map
> and turning each of those into fields. I’m looking into the idea of
> processing the enumerable in {{CalciteMetaImpl}} before the Frame gets created
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5964) Support additional metadata attributes in GET_TABLES and GET_COLUMNS
[
https://issues.apache.org/jira/browse/CALCITE-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761044#comment-17761044
]
Oliver Lee commented on CALCITE-5964:
-
Hey [~julianhyde]
The RemoteDriverTest testGetTables() seems to read out of the hsqldb connection
and the hsqldb-scott dataset
Question: it seems like I would have to build and update the hsql jar file to
be able to end-to-end test adding a new column to see if it can deserialize
properly, this would make it difficult to merge in with that dependency change
Is my understanding of this correct?
> Support additional metadata attributes in GET_TABLES and GET_COLUMNS
>
>
> Key: CALCITE-5964
> URL: https://issues.apache.org/jira/browse/CALCITE-5964
> Project: Calcite
> Issue Type: New Feature
> Components: avatica
>Reporter: Oliver Lee
>Assignee: Oliver Lee
>Priority: Major
>
> The goal is to add to Avatica a mechanism such that additional metadata
> fields pertaining to tables and columns can be transmitted alongside the
> standard JDBC
> [getTables|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> and {{getColumns}} calls.
> The Avatica client needs the response to be extensible such that revisions to
> metadata fields send and future additions does not require a new JAR file.
> Requirements:
> # Avatica user does not need to download new jar files if the server decides
> to send over new metadata data in the future
> # If the client makes modifications to support additional columns, they
> should always be present in the call and appear with null values, as opposed
> to complete omission (Number of columns in response stays the same)
> # Can handle attributes of varying types i.e. {{numberOne: int}} and
> {{booleanOne: boolean}}
> # Allows value retrieval from the {{ResultSet}} through calling
> {{resultSet.getInt(“booleanOne”)}} or {{resultSet.getBoolean(“booleanOne”)}}
> Current proposal is to modify the {{MetaTable}} and {{MetaColumn}} classes to
> include a map.
> {{{}HashMap{}}}, such that when instantiating the
> {{CalciteMetaTable}} in the {{{}ResultSet{}}}, new entries could be added in
> the future without changes to Avatica.
> One we have a list of additional metadata fields to be emitted in
> {{{}CalciteMetaImpl{}}}, the {{ResultSet}} would be created with the
> appropriate values.
>
> There are still some challenges identified below and I would love some input:
> Challenges:
> * Currently the {{MetaTable}} class that is instantiated is a
> {{{}CalciteMetaImpl{}}}. For the {{getTables()}} call, the response will be a
> list composed of schema tables of class {{CalciteMetaTable}} and database
> tables which can potentially be overloaded into 1 or more different
> subclasses. From this one heterogeneous list, we must determine the full list
> of columns to be included in the additional metadata hash. My initial plan
> was to provide a function in Calcite’s {{Table}} class such as
> {{getAdditionalColumns}} and allow it to be overloaded, but then I discovered
> the heterogeneity of the list.
> * Modifying the MetaTable class to include the hashmap of values could be
> easily done, but the challenge lies at {{{}RemoteMeta{}}}, to be able to
> serialize this cleanly so that requirement (4) is met and users can retrieve
> the values nicely. {{RemoteMeta}} currently serializes the response using
> reflection by looking at MetaTable.class and its attributes. The addition of
> one map is not immediately compatible with iterating over the keys of the map
> and turning each of those into fields. I’m looking into the idea of
> processing the enumerable in {{CalciteMetaImpl}} before the Frame gets created
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5964) Support additional metadata attributes in GET_TABLES and GET_COLUMNS
[
https://issues.apache.org/jira/browse/CALCITE-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760560#comment-17760560
]
Oliver Lee commented on CALCITE-5964:
-
Design change:
Going to look into the idea of passing through a MetaTable subclass on avatica
server startup as a property or intrinsic server property so that
{{RemoteMeta}} on the client side has access to it, and {{CalciteMetaImpl}} on
the server has access to it. One idea is setting a
{{CalciteConnectionProperty}} on JDBC connect.
The subclass would contain the additional fields of metadata, and this allows
users to configure the subclass to their needs.
Unit tests in Avatica to be written
* Validate that using a MetaTable subclass with additional attributes will
deserialize the json properly
* A test that works end to end for a JDBC server use case, connecting to a db
(mock) and checks that the tables generate the right JSON.
* Avatica test that checks if a subclass of MetaTable is configured, that the
additional fields will be generated
* Validate that if the subclass attributes aren’t in the expected order of
JDBC standard that it behaves correctly
> Support additional metadata attributes in GET_TABLES and GET_COLUMNS
>
>
> Key: CALCITE-5964
> URL: https://issues.apache.org/jira/browse/CALCITE-5964
> Project: Calcite
> Issue Type: New Feature
> Components: avatica
>Reporter: Oliver Lee
>Assignee: Oliver Lee
>Priority: Major
>
> The goal is to add to Avatica a mechanism such that additional metadata
> fields pertaining to tables and columns can be transmitted alongside the
> standard JDBC
> [getTables|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> and {{getColumns}} calls.
> The Avatica client needs the response to be extensible such that revisions to
> metadata fields send and future additions does not require a new JAR file.
> Requirements:
> # Avatica user does not need to download new jar files if the server decides
> to send over new metadata data in the future
> # If the client makes modifications to support additional columns, they
> should always be present in the call and appear with null values, as opposed
> to complete omission (Number of columns in response stays the same)
> # Can handle attributes of varying types i.e. {{numberOne: int}} and
> {{booleanOne: boolean}}
> # Allows value retrieval from the {{ResultSet}} through calling
> {{resultSet.getInt(“booleanOne”)}} or {{resultSet.getBoolean(“booleanOne”)}}
> Current proposal is to modify the {{MetaTable}} and {{MetaColumn}} classes to
> include a map.
> {{{}HashMap{}}}, such that when instantiating the
> {{CalciteMetaTable}} in the {{{}ResultSet{}}}, new entries could be added in
> the future without changes to Avatica.
> One we have a list of additional metadata fields to be emitted in
> {{{}CalciteMetaImpl{}}}, the {{ResultSet}} would be created with the
> appropriate values.
>
> There are still some challenges identified below and I would love some input:
> Challenges:
> * Currently the {{MetaTable}} class that is instantiated is a
> {{{}CalciteMetaImpl{}}}. For the {{getTables()}} call, the response will be a
> list composed of schema tables of class {{CalciteMetaTable}} and database
> tables which can potentially be overloaded into 1 or more different
> subclasses. From this one heterogeneous list, we must determine the full list
> of columns to be included in the additional metadata hash. My initial plan
> was to provide a function in Calcite’s {{Table}} class such as
> {{getAdditionalColumns}} and allow it to be overloaded, but then I discovered
> the heterogeneity of the list.
> * Modifying the MetaTable class to include the hashmap of values could be
> easily done, but the challenge lies at {{{}RemoteMeta{}}}, to be able to
> serialize this cleanly so that requirement (4) is met and users can retrieve
> the values nicely. {{RemoteMeta}} currently serializes the response using
> reflection by looking at MetaTable.class and its attributes. The addition of
> one map is not immediately compatible with iterating over the keys of the map
> and turning each of those into fields. I’m looking into the idea of
> processing the enumerable in {{CalciteMetaImpl}} before the Frame gets created
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5964) Support additional metadata attributes in GET_TABLES and GET_COLUMNS
[
https://issues.apache.org/jira/browse/CALCITE-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759956#comment-17759956
]
Oliver Lee commented on CALCITE-5964:
-
Thanks for taking a look [~julianhyde]
I had just gotten to reading into {{CursorFactory}} before you've replied. It
looks like the proper style to use for no set class and values not being
directly read from the object's attributes is {{ARRAY}} based off of what I'm
seeing at {{{}deduce(){}}}.
My understanding so far of how CursorFactory styles works is that {{Style.MAP}}
is for the case when the enumerable is just a map object (not applicable here),
and {{Style.ARRAY}} is for scenarios when we're not reading right off of a
class's attributes.
It doesn't seem like there exists a composite style allowing for arrays + map.
I'm looking at the example of Struct usage in
{{AvaticaResultSetConversionsTest.java}} for understanding.
To me that sounds like it would be best if the server supplies the Iterable
object with the map keys already flattened into attributes.
That ties into the 3rd point you brought up in that they don't necessarily have
to come from a list of {{MetaTable}} instances if I'm understanding you
correctly.
I assumed that was a restriction because the current implementation in
{{CalciteMetaImpl}} calls {{tables()}} which returns {{Iterable}}
> Support additional metadata attributes in GET_TABLES and GET_COLUMNS
>
>
> Key: CALCITE-5964
> URL: https://issues.apache.org/jira/browse/CALCITE-5964
> Project: Calcite
> Issue Type: New Feature
> Components: avatica
>Reporter: Oliver Lee
>Assignee: Oliver Lee
>Priority: Major
>
> The goal is to add to Avatica a mechanism such that additional metadata
> fields pertaining to tables and columns can be transmitted alongside the
> standard JDBC
> [getTables|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> and {{getColumns}} calls.
> The Avatica client needs the response to be extensible such that revisions to
> metadata fields send and future additions does not require a new JAR file.
> Requirements:
> # Avatica user does not need to download new jar files if the server decides
> to send over new metadata data in the future
> # If the client makes modifications to support additional columns, they
> should always be present in the call and appear with null values, as opposed
> to complete omission (Number of columns in response stays the same)
> # Can handle attributes of varying types i.e. {{numberOne: int}} and
> {{booleanOne: boolean}}
> # Allows value retrieval from the {{ResultSet}} through calling
> {{resultSet.getInt(“booleanOne”)}} or {{resultSet.getBoolean(“booleanOne”)}}
> Current proposal is to modify the {{MetaTable}} and {{MetaColumn}} classes to
> include a map.
> {{{}HashMap{}}}, such that when instantiating the
> {{CalciteMetaTable}} in the {{{}ResultSet{}}}, new entries could be added in
> the future without changes to Avatica.
> One we have a list of additional metadata fields to be emitted in
> {{{}CalciteMetaImpl{}}}, the {{ResultSet}} would be created with the
> appropriate values.
>
> There are still some challenges identified below and I would love some input:
> Challenges:
> * Currently the {{MetaTable}} class that is instantiated is a
> {{{}CalciteMetaImpl{}}}. For the {{getTables()}} call, the response will be a
> list composed of schema tables of class {{CalciteMetaTable}} and database
> tables which can potentially be overloaded into 1 or more different
> subclasses. From this one heterogeneous list, we must determine the full list
> of columns to be included in the additional metadata hash. My initial plan
> was to provide a function in Calcite’s {{Table}} class such as
> {{getAdditionalColumns}} and allow it to be overloaded, but then I discovered
> the heterogeneity of the list.
> * Modifying the MetaTable class to include the hashmap of values could be
> easily done, but the challenge lies at {{{}RemoteMeta{}}}, to be able to
> serialize this cleanly so that requirement (4) is met and users can retrieve
> the values nicely. {{RemoteMeta}} currently serializes the response using
> reflection by looking at MetaTable.class and its attributes. The addition of
> one map is not immediately compatible with iterating over the keys of the map
> and turning each of those into fields. I’m looking into the idea of
> processing the enumerable in {{CalciteMetaImpl}} before the Frame gets created
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5964) Support additional metadata attributes in GET_TABLES and GET_COLUMNS
[
https://issues.apache.org/jira/browse/CALCITE-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759769#comment-17759769
]
Julian Hyde commented on CALCITE-5964:
--
Let's look at changes required on server, RPC protocol, client.
*1. RPC protocol*
The client needs to know what the names and types of the additional columns. I
believe that
[TablesRequest|https://calcite.apache.org/avatica/docs/json_reference.html#tablesrequest]
yields a ResultSetResponse, and that response includes a list of column names
and types, so we're good.
The values of those columns need to come back. I believe we're also good.
*2. Client*
In the client, {{DatabaseMetaData.getTables}} will end up calling
{{RemoteMeta.getTables}}, and that calls {{toResultSet(MetaTable.class,
response)}}. Nowhere does the client create an instance of {{class MetaTable}}.
It just uses its fields to populate a {{ColumnMetaData.StructType}} that
contains a list of {{ColumnMetaData}}, one for each column. You can create some
additional {{ColumnMetaData}} instances, one for each extension column.
The values are returned in batches. Each batch is an instance of {{class
Frame}}, and contains {{List rows}}. You should investigate how those
row objects are created from the JSON result. If you're lucky they are
{{LinkedHashMap}} objects.
You'll need to use the right implementation of {{CursorFactory}} and that may
mean choosing the right style (ARRAY, LIST, MAP, ...).
*3. Server*
The server needs to generate the responses with the extra fields. It does not
necessarily do that from a list of {{MetaTable}} instances. (The server might
be implemented in C or Prolog, for all we know.)
{quote}Modifying the MetaTable class to include the hashmap of values{quote}
Minor quibble: the values will be in a {{java.util.Map}} but this will not
necessarily be a {{HashMap}}
> Support additional metadata attributes in GET_TABLES and GET_COLUMNS
>
>
> Key: CALCITE-5964
> URL: https://issues.apache.org/jira/browse/CALCITE-5964
> Project: Calcite
> Issue Type: New Feature
> Components: avatica
>Reporter: Oliver Lee
>Assignee: Oliver Lee
>Priority: Major
>
> The goal is to add to Avatica a mechanism such that additional metadata
> fields pertaining to tables and columns can be transmitted alongside the
> standard JDBC
> [getTables|https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getTables-java.lang.String-java.lang.String-java.lang.String-java.lang.String:A-)]
> and {{getColumns}} calls.
> The Avatica client needs the response to be extensible such that revisions to
> metadata fields send and future additions does not require a new JAR file.
> Requirements:
> # Avatica user does not need to download new jar files if the server decides
> to send over new metadata data in the future
> # If the client makes modifications to support additional columns, they
> should always be present in the call and appear with null values, as opposed
> to complete omission (Number of columns in response stays the same)
> # Can handle attributes of varying types i.e. {{numberOne: int}} and
> {{booleanOne: boolean}}
> # Allows value retrieval from the {{ResultSet}} through calling
> {{resultSet.getInt(“booleanOne”)}} or {{resultSet.getBoolean(“booleanOne”)}}
> Current proposal is to modify the {{MetaTable}} and {{MetaColumn}} classes to
> include a map.
> {{{}HashMap{}}}, such that when instantiating the
> {{CalciteMetaTable}} in the {{{}ResultSet{}}}, new entries could be added in
> the future without changes to Avatica.
> One we have a list of additional metadata fields to be emitted in
> {{{}CalciteMetaImpl{}}}, the {{ResultSet}} would be created with the
> appropriate values.
>
> There are still some challenges identified below and I would love some input:
> Challenges:
> * Currently the {{MetaTable}} class that is instantiated is a
> {{{}CalciteMetaImpl{}}}. For the {{getTables()}} call, the response will be a
> list composed of schema tables of class {{CalciteMetaTable}} and database
> tables which can potentially be overloaded into 1 or more different
> subclasses. From this one heterogeneous list, we must determine the full list
> of columns to be included in the additional metadata hash. My initial plan
> was to provide a function in Calcite’s {{Table}} class such as
> {{getAdditionalColumns}} and allow it to be overloaded, but then I discovered
> the heterogeneity of the list.
> * Modifying the MetaTable class to include the hashmap of values could be
> easily done, but the challenge lies at {{{}RemoteMeta{}}}, to be able to
> serialize this cleanly so that requirement (4) is met and users can retrieve
> the values nicely. {{RemoteMeta}} currently serializes t
