[
https://issues.apache.org/jira/browse/PARQUET-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933846#comment-14933846
]
Ryan Blue commented on PARQUET-379:
-----------------------------------
I think this is part of a larger issue of handling schema evolution. The main
use case I know of for union is merging file schemas into a metadata summary
file. Those are no longer really needed because each schema is resolved against
the requested schema individually on the reader, which eliminates the
bottle-neck that the metadata file was intended to avoid. And as you note,
union doesn't really create a union as one might expect: a schema that can be
used to read both of the input schemas.
> PrimitiveType.union erases original type
> ----------------------------------------
>
> Key: PARQUET-379
> URL: https://issues.apache.org/jira/browse/PARQUET-379
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0
> Reporter: Cheng Lian
>
> The following ScalaTest test case
> {code}
> test("merge primitive types") {
> val expected =
> Types.buildMessage()
> .addField(
> Types
> .required(INT32)
> .as(DECIMAL)
> .precision(7)
> .scale(2)
> .named("f"))
> .named("root")
> assert(expected.union(expected) === expected)
> }
> {code}
> produces the following assertion error
> {noformat}
> message root {
> required int32 f;
> }
> did not equal message root {
> required int32 f (DECIMAL(9,0));
> }
> {noformat}
> This is because {{PrimitiveType.union}} doesn't handle original type
> properly. An open question is that, can two primitive types with the same
> primitive type name but different original types be unioned?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
