dbtsai commented on PR #50327: URL: https://github.com/apache/arrow/pull/50327#issuecomment-4920283598
**Follow-up idea: reuse decoded strings for dictionary-encoded arrays** The new `StringArray._getitem_py` calls `PyUnicode_DecodeUTF8` per element, so a repeated string value produces a fresh `str` object every time (N allocations + N UTF-8 decodes for N elements). Two places where value reuse could help: 1. **`DictionaryArray` (highest value, no correctness risk):** the dictionary already stores each distinct value once. A specialized path could decode the D dictionary entries once and then return interned references (`decoded[index[i]]` + `Py_INCREF`) — turning N decodes into D decodes + N pointer reuses. This PR currently doesn't specialize `DictionaryArray`, so it falls back to the Scalar path and this win is unclaimed. Since `str` is immutable, sharing references is semantically identical to copying. 2. **Plain `StringArray` with repeated values:** an explicit dedup cache (or `PyUnicode_InternInPlace`) would cut allocations on low-cardinality data, but adds a hash/lookup per element and regresses high-cardinality (mostly-unique) data. Probably only worth it behind a cardinality heuristic / opt-in, not by default. Not blocking for this PR — just flagging `DictionaryArray` as a natural, safe follow-up where value reuse is a large win. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
