[
https://issues.apache.org/jira/browse/PIG-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated PIG-4326:
----------------------------
Attachment: PIG-4326-0.patch
I tried and seems testLoadRecordsWithMapOfRecords pass. I attached my change in
AvroStorageSchemaConversionUtilities.
If there is inconsistency between avro schema and Pig, we shall fill this gap.
For example, if avro map always contain a record, we shall remove it when
translating to Pig. Pig always has a tuple inside bag, when translating to
avro, we shall remove the tuple. If we are not doing that, then it is a bug we
shall fix.
> AvroStorageSchemaConversionUtilities does not properly convert schema for
> maps of arrays of records
> ---------------------------------------------------------------------------------------------------
>
> Key: PIG-4326
> URL: https://issues.apache.org/jira/browse/PIG-4326
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.12.0, 0.13.0
> Reporter: Michael Prim
> Assignee: Michael Prim
> Fix For: 0.15.0
>
> Attachments: PIG-4326-0.patch, mapsOfArraysOfRecords.patch
>
>
> I tried to convert the avro schema of a map of arrays of records into the
> proper pig schema and got always empty map schemas in pig.
> The reason is that the AvroStorageSchemaConversionUtilities does only assume
> records or primitive types as content of the map. However, a map of arrays,
> or a map of map, could have a schema itself and requires recursive calling to
> derive the full schema.
> I wrote a unit test to test for maps of arrays of records which fails with
> every pig release since the AvroStorage was rewritten (I think this was in
> 0.12), and there have been no changes since then in the trunk.
> Further the attached patch contains the (rather simple) fix that makes the
> schema conversion utils succeed.
> Would appreciate further comments and if this can be included upstream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
