tobixdev commented on issue #12644: URL: https://github.com/apache/datafusion/issues/12644#issuecomment-2592666978
I second @paleolimbot here, as I am also interested in this topic. First of all, thank you for all your efforts in this area! To provide some context, we are currently creating a prototype that handles data that is encoded as a union with 10-ish different variants. Logically, this union represents the encoding of a single type. While we make ends meet by creating logical plans manually with specialized UDFs (e.g., for equality), we are in a very early phase. Only working with physical types will add lots of complexity to our code that can be eliminated with [logical types](#12622) and its benefits (e.g., #12635). So what our (dream) scenario is that we could define this extension type (and its encoding) together with implementations for equality, orderings, etc, and DataFusion would automatically make use of these when joining or sorting (AFAIK, this should be supported according to #7923). I'd like to support you in these efforts. However, while I read a few discussions and proposals, I am still a bit lost on where I can/should help out as this is (from what I see) a huge ongoing project across multiple issues and epics. Do you have pointers on where I could start helping out? For example, I found #13301, but I am unsure if the efforts on simple functions #12635 make these changes somewhat premature. Thank you for helping me to navigate this project! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
