cooldome commented on issue #49646: URL: https://github.com/apache/arrow/issues/49646#issuecomment-6285648363
Hi, I missed your request for comment. Here is our use case. We are storing trades in arrow ipc files. Main assets like currency code, equity names, interest rate curves, bond cusip codes are dictionarised. In this file currency code column appear more than 180 times. Main offender is the column with trade economics, it is dense union column with trade types like IR swap, Equity Future, Bond Future, FX Option and etc. Dense union has around 70 alternatives. For these 70 alternatives: currency code appears more than 100 times, bond cusips 5 times, equity name 35 times, interest rate 60 times and etc. Bond dictionary has 200k+ elements repeated 5 times. Equity dictionary has 100k elements repeated 35 times. We don't use plain string because everything gets loaded into pytorch tensors in the end and pytorch doesn't work with strings. We estimated that we would save 2.2 gb every day if would stop duplicating dictionaries -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
