[
https://issues.apache.org/jira/browse/HIVE-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162709#comment-13162709
]
Krishna Kumar commented on HIVE-2604:
-------------------------------------
The current implementation works as follows:
- Adds a serde UberCompressorSerde, which is used to convert the cell values
to bytes
- Adds a codec UberCompressionCodec which uses user-specific config to
compress each block of column values through one of three possible mechanisms
- Config for the column: "codec:<codecname>" - Apply a CompressionCodec on
the UberCompressorSerde serialized bytestream
- Config for the column: "codec:<codecname>,<serdename>" Re-serialize the
bytestream through serdename and then apply codecname on it
- Config for the column: "compressor:<compressorname>" compress the cell
values by sending them through the type-specific compressor
- [As a future enhancement, the config, say if is "dynamic", can let the
codec decide the mechanism on the current block stats/previous seen blocks]
The idea is to maintain the ability to use a serde/codec combination (as we do
now) for any columns which are not 'interesting' and use type-specific
compressors only for special columns.
Type-specific compressor is also an extension point only; no implementation
attached to this jira. Have attached one sample compressor to HIVE-2623, while
many others are possible.
> Add UberCompressor Serde/Codec to contrib which allows per-column compression
> strategies
> ----------------------------------------------------------------------------------------
>
> Key: HIVE-2604
> URL: https://issues.apache.org/jira/browse/HIVE-2604
> Project: Hive
> Issue Type: Sub-task
> Components: Query Processor, Serializers/Deserializers
> Reporter: Krishna Kumar
> Assignee: Krishna Kumar
> Attachments: HIVE-2604.v0.patch
>
>
> The strategies supported are
> 1. using a specified codec on the column
> 2. using a specific codec on the column which is serialized via a specific
> serde
> 3. using a specific "TypeSpecificCompressor" instance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
