granthamtaylor commented on PR #45001: URL: https://github.com/apache/arrow/pull/45001#issuecomment-5622465473
I have a use case for the feature I would love to see resolved. In [relflow](https://relflow.ai), I am hashing scalar values in nested arrays to safely represent arbitrary values (phone numbers, emails, credit cards numbers, etc) within n-dimensional contexts. The hash function only needs to be consistent within a batch such that the model cannot overfit to any actual values - only the equalities among values within a context (IE: "for what other financial transactions / clickstream events has this customer authenticated with this arbitrary phone number?") I am currently moving the arrow data over to a polars series (which does support such scalar hashing). This is reasonably cheap but I would love an arrow-native solution ! Happy to help / contribute. Thank you for the work here ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
