Hi Jason,
Thank you for the explanation.
I have several *low* cardinality fields that contain semi-long values and
they are, I think, a perfect candidate for dictionary encoding.
I assumed that the choose to use dictionary encoding was a bit smarter than
this and would rely on Strings type column where x% repeated values were a
clear signal for it's use.
If you can outline what needs to be done and where then I will gladly take
a stab at it :).
Several questions along those lines:
- Does the Parquet library that Drill uses allow for programmatic
section?
- What metadata, regarding the column content, is available when the
choice is made?
- Where in the Parquet part of Drill is this logic?
- Is there no ongoing effort in parquet-mr to make the automatic
handling smarter?
- Are all Parquet encoding options being used by drill?
- Like the encoding of longs where delta between semi-subsequent numbers
is stored. (As I understand it)
thanks again.
Regards,
-Stefan
On Thu, Feb 4, 2016 at 3:36 PM, Jason Altekruse <altekruseja...@gmail.com>
wrote:
> Hi Stefan,
>
> There is a reason that dictionary is disabled by default. The parquet-mr
> library we leverage for writing parquet files currently has the behavior to
> write nearly all columns as dictionary encoded for all types when
> dictionary encoding is enabled. This includes columns with integers,
> doubles, dates and timestamps.
>
> Do you have some data that you believe is well suited for dictionary
> encoding in the dataset? I think there are good uses for it, such as data
> coming from systems that support enumerations, that might be represented as
> strings when exported from a database for use with Big Data tools like
> Drill. Unfortunately we do not currently provide a mechanism for requesting
> dictionary encoding on only some columns, and we don't do anything like
> buffer values to determine if a given column is well-suited for dictionary
> encoding before starting to write them.
>
> In many cases it obviously is not a good choice, and so we actually take a
> performance hit re-materializing the data out of the dictionary upon read.
>
> If you would be interested in trying to contribute such an enhancement I
> would be willing to help you get started with it.
>
> - Jason
>
> On Wed, Feb 3, 2016 at 5:15 AM, Stefán Baxter <ste...@activitystream.com>
> wrote:
>
> > Hi,
> >
> > I'm converting Avro to parquest and I'm getting this log entry back for a
> > timestamp field:
> >
> > Written 1,008,842B for [occurred_at] INT64: 591,435 values, 2,169,557B
> raw,
> > 1,008,606B comp, 5 pages, encodings: [BIT_PACKED, PLAIN,
> PLAIN_DICTIONARY,
> > RLE], dic { 123,832 entries, 990,656B raw, 123,832B comp}
> >
> > Can someone please tell me if this is the expected encoding for a
> timestamp
> > field.
> >
> > I'm a bit surprised that it seems to be dictionary based. (Yes, I have
> > enabled dictionary encoding for Parquet files).
> >
> > Regards,
> > -Stefán
> >
>
