[
https://issues.apache.org/jira/browse/TAJO-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14059250#comment-14059250
]
David Chen commented on TAJO-710:
---------------------------------
Hi Hyunsik,
Sorry for the delay in my response.
Using either Protocol Buffers or Parquet's schema representation would be
viable options. After thinking about this some more, I think it may be better
to use Protocol Buffers for both the schema and the record representation since
Protocol Buffers provides a generic implementation of both.
I am hesitant to reuse Parquet's representation since MessageType contains some
implementation details that are more specific to Parquet and that Parquet, in
fact, does not have a standalone record format; the different Parquet readers
and writers materialize rows into various other record formats but there is no
"official" Parquet record representation. Parquet's MessageType has some
support for annotating fields, but these annotations are more or less specific
to Parquet and if we need to add additional annotations, we would either have
to fork or contribute those changes back to Parquet.
On the other hand, Protocol Buffers provides a generic nested schema
representation and record representation as well as support for custom
annotations (https://developers.google.com/protocol-buffers/docs/proto).
According to Google's papers, both F1 and Dremel appear to be using Protocol
Buffers for their data model with F1 also using Protocol Buffers for storing
data (http://research.google.com/pubs/pub41344.html).
As a result, I think that using Protocol Buffers is a good approach to start
with for implementing nested schemas and non-scalar types.
Thanks,
David
> Add support for nested schemas and non-scalar types
> ---------------------------------------------------
>
> Key: TAJO-710
> URL: https://issues.apache.org/jira/browse/TAJO-710
> Project: Tajo
> Issue Type: New Feature
> Components: data type
> Reporter: David Chen
> Assignee: David Chen
>
> Add support for nested schemas and non-scalar types (maps, arrays, enums, and
> unions). Here are some ways other systems handle nested schemas:
> * Pig and Hive uses complex data types, such as bags, structs, arrays, etc.
> * Impala doesn't support nested schemas or non-scalar data types
> (http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_langref_unsupported.html)
> and disallows complex types in their Parquet support
> (http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_parquet.html).
> * Presto also does not support non-scalar types
> (http://prestodb.io/docs/current/language/types.html)
> From the discussion in TAJO-30:
> {quote}
> I have a plan for nested schema. Currently, Tajo only supports a flat schema
> like relational DBMS. So, even though Tajo is extended to nested data mode,
> it will not break the compatibility.
> I'm thinking that Tajo takes Parquet data model (= protobuf or BigQuery).
> When I consider nested data model, I thought two main points. Parquet data
> model satisfies with these points. The first point that I've thought is the
> processing model on nested data. Parquet data model is the same to that of
> BigQuery, and BigQuery already concreted the processing model including
> flattening, cross production on repeated fields, and aggregation on repeated
> fields [1][2]. The second point is file format. Parquet is a native file
> format for this model. Parquet already includes the efficient record assembly
> method. Besides, Parquet is already mature and is widely used in many systems.
> [1] http://research.google.com/pubs/pub36632.html
> [2] https://developers.google.com/bigquery/docs/data
> I'm thinking that we need three stages for this work. Firstly, we can start
> with a small change to improve our schema system. Then, we will add some
> physical operator to just flatten one nested row into a number of flattened
> rows. Finally, we will solve some query optimization issues like
> projection/filter push down on nested schema and will add some physical
> operators to directly process nested rows.
> If you have any idea, feel free to share with us.
> Thanks,
> Hyunsik
> {quote}
> This ticket may need to be broken up into multiple sub-tasks. Each sub-task
> will involve defining an extension to the query language to support the data
> type, implementing the new data type, then adding support for the data type
> in each of the storage types. I have opened tickets for each of these four
> tasks but not as subtasks because it is very likely that each of these tasks
> will have subtasks of their own:
> * TAJO-721: Adding support for nested records
> * TAJO-722: Adding support for maps
> * TAJO-723: Adding support for array
> * TAJO-724: Adding support for unions
> Adding support for the enum type can be a consideration, but is lower
> priority than the other four complex types. Neither Hive nor Pig currently
> have an enum type (even though storage formats such as Avro and Parquet do)
> and, I believe, simply convert enum values to strings.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
