[
https://issues.apache.org/jira/browse/PHOENIX-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15010075#comment-15010075
]
James Taylor commented on PHOENIX-2417:
---------------------------------------
Thanks to [~julienledem] for pointing me to a couple of simple classes that
implement a simple prefix encoder that we could use:
- On the write side:
https://github.com/Parquet/parquet-mr/blob/master/parquet-column/src/main/java/parquet/column/values/deltastrings/DeltaByteArrayWriter.java
- On the read side (materializing a single value at a time as you iterate):
https://github.com/Parquet/parquet-mr/blob/fa8957d7939b59e8d391fa17000b34e865de015d/parquet-column/src/main/java/parquet/column/values/deltastrings/DeltaByteArrayReader.java#L61
We can either copy/paste or use directly (especially if there's already a
Parquet dependency).
> Compress memory used by row key byte[] of guideposts
> ----------------------------------------------------
>
> Key: PHOENIX-2417
> URL: https://issues.apache.org/jira/browse/PHOENIX-2417
> Project: Phoenix
> Issue Type: Bug
> Reporter: James Taylor
>
> We've found that smaller guideposts are better in terms of minimizing any
> increase in latency for point scans. However, this increases the amount of
> memory significantly when caching the guideposts on the client. Guidepost are
> equidistant row keys in the form of raw byte[] which are likely to have a
> large percentage of their leading bytes in common (as they're stored in
> sorted order. We should use a simple compression technique to mitigate this.
> I noticed that Apache Parquet has a run length encoding - perhaps we can use
> that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
