Hello Naveen,
1. If I understand correctly, then yes you can extract nested fields from
struct type. The syntax depends on StructKind value for your data type,
for example for FULLY_QUALIFIED struct you should first
make alias for your table and then request nested field like,
table_alias.struct_column.nested_field. In rel tree such expressions are
presented as RexCall with SqlItemOperator operator.
2. Yes, this ability was implemented in CALCITE-3138 [1]. It builds call to
ROW type constructor function on top of flattened tree for necessary
columns.
3. Yes, examples of such functions are ROW(...), ANY_VALUE(...) etc.
In current implementation of flattener invocation of ROW constructor
function is done despite of null handling same issue exists for some
aggregate function flattening, like COUNT(struct_column).
Proper null handling is real pain for flattener, original idea was to
handle special null indicator for each flattened struct, but in practice I
recognized that it's really hard to deal with flattened fields indices when
related methods are called from very different points, so for now the
problem remains unsolved.
If you can't avoid dealing with null values in your struct columns you
could try to avoid invocation to SqlToRelConverter.flattenTypes(...) and
check whether final plan acceptable for you. As far as I know
there is no reading material for given topic, you can investigate source
code by debugging RelStructuredTypeFlattener and reading some related plans
in SqlToRelConverterTest.java and SqlToRelConverterTest.xml.
[1] https://issues.apache.org/jira/browse/CALCITE-3138
Thanks,
Igor
On Thu, Oct 24, 2019 at 12:57 PM Naveen Kumar
<naveenkuma...@flipkart.com.invalid> wrote:
> Hi,
>
> I work at Flipkart, we are using Calcite in our streaming platform. In most
> of our use cases, input data is nested. I understand Calcite flattens
> structs in scan and references fields positionally.
>
> I had a few questions on handling nested data -
>
> 1. Can RelNode DAG work with nested data (instead of flattened fields)
> by referencing fields through their nested structure eg,
> data.order.orderId
> 2. In the current flattened behavior, can output of a query be a struct.
> Eg if *orderId, orderData.timestamp, orderData.category* are output of
> select query, can I declaratively organise output to below json
> structure -
> 1.
>
>
>
>
>
> *{ "orderId": "order1", "orderData": { "timestamp": 1571904384814,
> "category": "shoes" } }*
> 3. Can output of a UDF be struct type
>
> Please point me to any reading material or example that would help with
> these questions.
>
> Regards,
> Naveen
>
> --
>
>
>
>
> *-----------------------------------------------------------------------------------------*
>
>
> *This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they are
> addressed. If you have received this email in error, please notify the
> system manager. This message contains confidential information and is
> intended only for the individual named. If you are not the named
> addressee,
> you should not disseminate, distribute or copy this email. Please notify
> the sender immediately by email if you have received this email by mistake
> and delete this email from your system. If you are not the intended
> recipient, you are notified that disclosing, copying, distributing or
> taking any action in reliance on the contents of this information is
> strictly prohibited.*****
>
> ****
>
> *Any views or opinions presented in this
> email are solely those of the author and do not necessarily represent
> those
> of the organization. Any information on shares, debentures or similar
> instruments, recommended product pricing, valuations and the like are for
> information purposes only. It is not meant to be an instruction or
> recommendation, as the case may be, to buy or to sell securities,
> products,
> services nor an offer to buy or sell securities, products or services
> unless specifically stated to be so on behalf of the Flipkart group.
> Employees of the Flipkart group of companies are expressly required not to
> make defamatory statements and not to infringe or authorise any
> infringement of copyright or any other legal right by email
> communications.
> Any such communication is contrary to organizational policy and outside
> the
> scope of the employment of the individual concerned. The organization will
> not accept any liability in respect of such communication, and the
> employee
> responsible will be personally liable for any damages or other liability
> arising.*****
>
> ****
>
> *Our organization accepts no liability for the
> content of this email, or for the consequences of any actions taken on the
> basis of the information *provided,* unless that information is
> subsequently confirmed in writing. If you are not the intended recipient,
> you are notified that disclosing, copying, distributing or taking any
> action in reliance on the contents of this information is strictly
> prohibited.*
>
>
>
> _-----------------------------------------------------------------------------------------_
>
>
