[
https://issues.apache.org/jira/browse/ARROW-81?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430047#comment-15430047
]
Wes McKinney commented on ARROW-81:
-----------------------------------
You could have a case of redundancy if the span of dictionary indices and
dictionary values is the same. In practice, dictionary encoding integers with a
wide span (for example -- if max(values) - min(values) is some large number)
can have significant performance benefits as you can do things normally
requiring a hash table (e.g. computing a frequency table) with much less effort.
> [Format] Add a Category logical type (distinct from dictionary-encoding)
> ------------------------------------------------------------------------
>
> Key: ARROW-81
> URL: https://issues.apache.org/jira/browse/ARROW-81
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Assignee: Wes McKinney
>
> A Category (or "factor") is a dictionary-encoded array whose dictionary has
> semantic meaning. The data consists of
> - An array of integer "codes"
> - A child array of some other type, known as the "categories" or "levels" of
> the array. Typically there is an "ordered" boolean flag indicating whether
> the order of the categories is meaningful.
> Category/factor types are used in a number of common statistical analyses.
> See, for example,
> http://www.voteview.com/R_Ordered_Logistic_or_Probit_Regression.htm. It is a
> basic requirement for Python and R, at least, as Arrow C++ consumers, to have
> this type. Separately, we should consider what is necessary to be able to
> transmit category data in IPCs -- possible an expansion of the Arrow format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
