[
https://issues.apache.org/jira/browse/CRUNCH-480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Wills resolved CRUNCH-480.
-------------------------------
Resolution: Fixed
Fix Version/s: 0.12.0
Committed. Thanks everyone!
> AvroParquetFileSource doesn't properly configure user-supplied read schema
> --------------------------------------------------------------------------
>
> Key: CRUNCH-480
> URL: https://issues.apache.org/jira/browse/CRUNCH-480
> Project: Crunch
> Issue Type: Bug
> Components: IO
> Affects Versions: 0.10.0
> Reporter: E. Sammer
> Assignee: Gabriel Reid
> Priority: Blocker
> Fix For: 0.12.0
>
> Attachments: CRUNCH-480.1.patch, CRUNCH-480.2.patch,
> CRUNCH-480.3.patch, CRUNCH-480.patch
>
>
> It seems like AvroParquetFileSource doesn't properly set the configuration
> param required to use a user-supplied read schema that differs from the
> schema in the file.
> Deep in the guts of Parquet (InternalParquetReader#initialize()), I found
> this:
> {code}
> this.recordConverter = readSupport.prepareForRead(
> configuration, extraMetadata, fileSchema,
> new ReadSupport.ReadContext(requestedSchema, readSupportMetadata));
> {code}
> Later, in Parquet's AvroReadSupport#prepareForRead(), it appears to ignore
> the supplied requestedSchema and, instead, looks for the key avro.read.schema
> in the readSupportMetadata map. This is seriously kookie code in Parquet
> (i.e. wrong), but because Crunch doesn't supply readSupportMetadata, we can
> never properly supply a read schema. Boooo hisssss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
