etseidl commented on PR #10777: URL: https://github.com/apache/arrow-rs/pull/10777#issuecomment-5376304272
I need some time to mull this over. One issue as I see it is that the fallback here always re-encodes the buffered values. But I see many instances where the dictionary is effective, but too many keys accrete within a large column chunk, so fallback occurs anyway. The current page would still benefit from the dictionary, but with this change that partial page will be re-encoded with PLAIN. Perhaps this re-encode should only occur if it's the first page in the chunk (i.e. `retain_dictionary` is `false`). I'll also admit I find the presence of two `Option<DictEncoder>`s a bit confusing. Maybe a boolean to indicate if fallback has occurred would be better? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
