Hi Michael, Thanks for sharing the great work. I believe it is important work for both communities.
Best, Liya Fan On Thu, Apr 8, 2021 at 3:30 AM Michael Mior <mm...@apache.org> wrote: > Hi all, > > I wanted to share some work one of my (now former) students, Karshit > Shah, has done with integrating Apache Arrow into Calcite. Karshit has > written an Arrow adapter that's able to perform filtering and > projections natively on Arrow data using Gandiva so these expressions > can be JITed using LLVM. The pull request[0] needs some cleanup, but > the code is in relatively good shape. > > Right now, the adapter only reads from files, but I think there are a > number of exciting extensions to this that are possible. For example, > Arrow has a client-server framework Flight which could be connected > with Calcite, perhaps via Avatica. (Andy Grove was doing some work on > this last year[1] although I'm not sure of the progress.) > > The biggest blocker on this is actually not the Calcite code, but the > availability of a suitably built Arrow dependency with Gandiva along > with the appropriate CI configuration. I opened a JIRA on the Arrow > project with some more details[2]. > > I'd love some thoughts on the approach and some help in pushing this > over the finish line. > > [0] https://github.com/apache/calcite/pull/2133 > [1] > https://mail-archives.apache.org/mod_mbox/calcite-dev/202002.mbox/%3cCAJEf=X5xvXLQpJkX_VjJk=TnNRwT52v0=p28sczmid1tyce...@mail.gmail.com%3e > [2] https://issues.apache.org/jira/browse/ARROW-11135 > -- > Michael Mior > mm...@apache.org >
