[
https://issues.apache.org/jira/browse/CRUNCH-329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889660#comment-13889660
]
Gabriel Reid commented on CRUNCH-329:
-------------------------------------
After a look at the code I see what you mean. I was thinking (incorrectly) that
the inner types would only be needed by the MapFns used by the PType, and
seeing as those get initialized then they would have the information that they
need.
I guess this means that Writables.WRITABLE_CODES probably needs to get updated
somehow based on the Configuration (so that its state is published to remote
JVMs). I guess this could be as simple as a static
Writables.initialize(Configuration) function that would merge in the codes
contained in the Configuration, which feels really ugly to me but I don't see a
lot of other easily-implementable options. What do you think?
> Re-add type info to TupleWritable to make fields sort correctly
> ---------------------------------------------------------------
>
> Key: CRUNCH-329
> URL: https://issues.apache.org/jira/browse/CRUNCH-329
> Project: Crunch
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.10.0, 0.8.3
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.10.0, 0.8.3
>
> Attachments: CRUNCH-329.patch, fix-ss-writables.patch
>
>
> Secondary sorts aren't currently working correctly for Writable types after
> we hacked the TupleWritable impl to make all of the fields BytesWritables
> (e.g., secondary IntWritable values will no longer be sorted correctly, even
> though everything is still grouped correctly.)
> The least-bad way that I came up with to fix this is to use integer codes for
> each possible WritableComparable type in a pipeline that we can use to decode
> what Writable type each tuple field corresponds to. This allows us to keep
> the various fields sortable while still doing a reasonable job of minimizing
> the serialization required to pass the type information along.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
