Hi Rahul,
Thanks for sharing your code. I was trying to get plugin for solr engine. But I
thought of using solr's rest api to do the queries ,get schema metadata info
etc.
The goal for me is to expose a solr engine to tools like Tableau or MS Excel
and user can do stuff there.
I am still very new to this and there is a learning curve. It would be great if
you can comment/review whatever I've done so far.
https://github.com/sudipmukherjee/drill/tree/master/contrib/storage-solr
Thanks,
Sudip
-Original Message-
From: rahul challapalli [mailto:challapallira...@gmail.com]
Sent: 10 August 2015 AM 05:21
To: dev@drill.apache.org
Subject: Re: Lucene Format Plugin
Below is the link to my branch which contains the changes related to the format
plugin.
https://github.com/rchallapalli/drill/tree/lucene/contrib/format-lucene
Any thoughts on how to handle contributions like this which still have some
work to be done?
- Rahul
On Mon, Aug 3, 2015 at 12:21 PM, rahul challapalli
challapallira...@gmail.com wrote:
Thanks Jason.
I want to look at the solr plugin and see where we can collaborate or
if we already duplicated part of the effort.
I still need to push a few commits. I will share the code once I get
these changes pushed.
- Rahul
On Mon, Aug 3, 2015 at 11:31 AM, Jason Altekruse
altekruseja...@gmail.com
wrote:
Hey Rahul,
This is really cool! Thanks for all of the time you put into writing
this, I think we have a lot of available opportunities to reach new
communities with efforts like this.
I noticed last week another contributor opened a JIRA for a solr
plugin, there might be a good opportunity for the two of you to join
efforts, as I believe he likely stated working on a lucene reader as
part of his solr work.
Would you like to post a link to your work on Github or another
public host of your code?
https://issues.apache.org/jira/browse/DRILL-3585
On Mon, Aug 3, 2015 at 2:29 AM, Stefán Baxter
ste...@activitystream.com
wrote:
Hi,
I'm pretty new around here but I just wanted to tell you how much
your
work
can benefit us. This is great!.
Look forward to trying it out.
Regards,
-Stefán
On Mon, Aug 3, 2015 at 8:38 AM, rahul challapalli
challapallira...@gmail.com wrote:
Hello Drillers,
I have been working on a lucene format plugin. In its current
state,
the
below sample query successfully searches a lucene index and
returns
the
results.
select path from dfs_test.`/search-index` where
contents='maxItemsPerBlock'
and contents = 'BlockTreeTermsIndex'
*High Level Overview of Current Implementation:*
*Parallelization:* A lucene segment is the lowest level of
parrallelization.
*Filter Pushdown:* Currently the format plugin is designed to
push the complete filter into the scan.
*Filter Evaluation:* Each condition in the filter is treated as a
lucene
TermQuery
http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/search/Ter
mQuery.html
and multiple conditions are joined using a BooleanQuery
http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/search/Boo
leanQuery.html
.
If we *do not* use a TermQuery, then we have to know the exact
type of Analyzer
https://lucene.apache.org/core/5_2_1/core/org/apache/lucene/analysis/
Analyzer.html
to use with each field in the query.
Ex: 'contents' field might have been analyzed using a
StandardAnalyzer
https://lucene.apache.org/core/5_2_1/analyzers-common/org/apache/luce
ne/analysis/standard/StandardAnalyzer.html
and the 'path' field might not have been analyzed at all.
If desired, support for raw lucene queries with a reserved word
should be
easy to add.
Ex: select * from dfs.`search-index` where searchQuery =
+contents:maxItemsPerBlock
+path:/home/file.txt;
*Converting SqlFilter to Lucene Query:* Currently only = and !=
operators are handled while converting a sql filter into a lucene
query.
For indexed fields this might be sufficient to handle a good
number of cases. For non-indexed fields operators like ,, like
etc need to
be
handled.
*FileSystems:* Currently the format plugin only works on a local
filesystem.
Though far from complete, I want to work with the community to
get
some
feedback and avoid any chance of duplication of work. Kindly let
me
know
your thoughts
- Rahul
***Legal Disclaimer***
This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you.
**
