Jefffrey commented on issue #10160: URL: https://github.com/apache/arrow-rs/issues/10160#issuecomment-4769309225
the arrow spec explicitly does state that dictionaries can have duplicate values: > Note that a dictionary is permitted to contain duplicate values or nulls: > ``` > data VarBinary (dictionary-encoded) > index_type: Int32 > values: [0, 1, 3, 1, 4, 2] > > dictionary > type: VarBinary > values: ['foo', 'bar', 'baz', 'foo', null] > ``` - https://arrow.apache.org/docs/format/Columnar.html#dictionary-encoded-layout so this would be more a feature request, in wanting to deduplicate a dictionary for downstream performance reasons -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
