Re: Looking for advice on integrating with a custom data source

Hi Andy, Congratulations on making such fast progress! The code to do filter pushdowns is rather complex and, it seems, most plugins copy/paste the same wad of code (with the same bugs). PR 1914 provides a layer that converts the messy Drill logical plan into a nice, simple set of predicates.

Re: Looking for advice on integrating with a custom data source

With some extra debugging I can see that the getNewWithChildren call is made to an earlier instance of GroupScan and not the instance created by the filter push-down rule. I'm wondering if this is some kind of hashCode/equals/toString/getDigest issue? On Tue, Jan 14, 2020 at 7:52 PM Andy Grove wr

Re: Looking for advice on integrating with a custom data source

I'm now working on predicate push down ... I have a filter rule that is correctly extracting the predicates that the backend database supports and I am creating a new GroupScan containing these predicates, using the Kafka plugin as a reference. I see the GroupScan constructor being called after thi

Re: Apache Drill documentation updates

Hi Bridget, Thanks much for the update and for all your efforts on Drill documentation over the last several years. Thanks for squeezing in time to handle the release updates. Also, thanks for documenting how to update the documentation: the team should be able to keep information updated withi

Re: querying json from multiple subdirectories

Hi Prabhakar, I would think that the following query would work: SELECT FROM dfs..`transactions/` That should merge everything into one table and you should get a dir0 column with the directory names. --C > On Jan 14, 2020, at 4:56 AM, Prabhakar Bhosaale wrote: > > Hi All, > > I am new to

Re: querying json from multiple subdirectories

Hi, Drill can easily query directories including subdirectories and then join data with other directories, tables etc. Please refer to Drill documentation for more details. For example, you can start from this article: https://drill.apache.org/docs/querying-directories/

querying json from multiple subdirectories

Hi All, I am new to apache drill and trying to retrieve data from json files by querying the directories. The directory structure is |-->Year2012--->trans.json | | transactions-->| |