Hi all, I wanted to share some work one of my (now former) students, Karshit Shah, has done with integrating Apache Arrow into Calcite. Karshit has written an Arrow adapter that's able to perform filtering and projections natively on Arrow data using Gandiva so these expressions can be JITed using LLVM. The pull request[0] needs some cleanup, but the code is in relatively good shape.
Right now, the adapter only reads from files, but I think there are a number of exciting extensions to this that are possible. For example, Arrow has a client-server framework Flight which could be connected with Calcite, perhaps via Avatica. (Andy Grove was doing some work on this last year[1] although I'm not sure of the progress.) The biggest blocker on this is actually not the Calcite code, but the availability of a suitably built Arrow dependency with Gandiva along with the appropriate CI configuration. I opened a JIRA on the Arrow project with some more details[2]. I'd love some thoughts on the approach and some help in pushing this over the finish line. [0] https://github.com/apache/calcite/pull/2133 [1] https://mail-archives.apache.org/mod_mbox/calcite-dev/202002.mbox/%3cCAJEf=X5xvXLQpJkX_VjJk=TnNRwT52v0=p28sczmid1tyce...@mail.gmail.com%3e [2] https://issues.apache.org/jira/browse/ARROW-11135 -- Michael Mior mm...@apache.org
