[jira] [Updated] (LUCENE-7481) SpanPayloadCheckQuery is missing rewrite method
[ https://issues.apache.org/jira/browse/LUCENE-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Chyla updated LUCENE-7481: Description: If used with a wildcard query, the result is a failure saying: "Rewrite query first" The SpanNearQuery has the rewrite method; however the SpanPayloadCheckQuery just returns the query itself. this works: ``` spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), SpanMultiTermQueryWrapper(vectrfield:m*), SpanMultiTermQueryWrapper(vectrfield:f*)], 0, true) ``` code to generate the query: ``` private Query getSpanQuery(String[] parts, int howMany, boolean truncate) throws UnsupportedEncodingException { SpanQuery[] clauses = new SpanQuery[howMany+1]; clauses[0] = new SpanTermQuery(new Term("vectrfield", parts[0])); // surname for (int i = 0; i < howMany; i++) { if (truncate) { SpanMultiTermQueryWrapper q = new SpanMultiTermQueryWrapper(new WildcardQuery(new Term("vectrfield", parts[i+1].substring(0, 1) + "*"))); clauses[i+1] = q; } else { clauses[i+1] = new SpanTermQuery(new Term("vectrfield", parts[i+1])); } } SpanNearQuery sq = new SpanNearQuery(clauses, 0, true); // match in order return sq; } ``` and this fails: ``` spanPayCheck(spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), SpanMultiTermQueryWrapper(vectrfield:m*), SpanMultiTermQueryWrapper(vectrfield:f*)], 1, true), payloadRef: 0;1;2;3;) ``` each clause is made of: ``` new SpanMultiTermQueryWrapper(new WildcardQuery(new Term("vectrfield", parts[i+1].substring(0, 1) + "*"))); ``` It is a regression; the code was working well in SOLR4.x was: If used with a wildcard query, the result is a failure saying: "Rewrite query first" The SpanNearQuery has the rewrite method; however the SpanPayloadCheckQuery just returns the query itself. this works: ``` spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), SpanMultiTermQueryWrapper(vectrfield:m*), SpanMultiTermQueryWrapper(vectrfield:f*)], 0, true) ``` code to generate the query: ``` private Query getSpanQuery(String[] parts, int howMany, boolean truncate) throws UnsupportedEncodingException { SpanQuery[] clauses = new SpanQuery[howMany+1]; clauses[0] = new SpanTermQuery(new Term("vectrfield", parts[0])); // surname for (int i = 0; i < howMany; i++) { if (truncate) { SpanMultiTermQueryWrapper q = new SpanMultiTermQueryWrapper(new WildcardQuery(new Term("vectrfield", parts[i+1].substring(0, 1) + "*"))); clauses[i+1] = q; } else { clauses[i+1] = new SpanTermQuery(new Term("vectrfield", parts[i+1])); } } SpanNearQuery sq = new SpanNearQuery(clauses, 0, true); // match in order return sq; } ``` and this fails: {code:java} spanPayCheck(spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), SpanMultiTermQueryWrapper(vectrfield:m*), SpanMultiTermQueryWrapper(vectrfield:f*)], 1, true), payloadRef: 0;1;2;3;) {/code} each clause is made of: ``` new SpanMultiTermQueryWrapper(new WildcardQuery(new Term("vectrfield", parts[i+1].substring(0, 1) + "*"))); ``` It is a regression; the code was working well in SOLR4.x > SpanPayloadCheckQuery is missing rewrite method > --- > > Key: LUCENE-7481 > URL: https://issues.apache.org/jira/browse/LUCENE-7481 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: 6.x >Reporter: Roman Chyla > > If used with a wildcard query, the result is a failure saying: "Rewrite query > first" > The SpanNearQuery has the rewrite method; however the SpanPayloadCheckQuery > just returns the query itself. > this works: > ``` > spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), > SpanMultiTermQueryWrapper(vectrfield:m*), > SpanMultiTermQueryWrapper(vectrfield:f*)], 0, true) > ``` > code to generate the query: > ``` > private Query getSpanQuery(String[] parts, int howMany, boolean truncate) > throws UnsupportedEncodingException { > SpanQuery[] clauses = new SpanQuery[howMany+1]; > clauses[0] = new SpanTermQuery(new Term("vectrfield", > parts[0])); // surname > for (int i = 0; i < howMany; i++) { > if (truncate) { > SpanMultiTermQueryWrapper q = new > SpanMultiTermQueryW
[jira] [Updated] (LUCENE-7481) SpanPayloadCheckQuery is missing rewrite method
[ https://issues.apache.org/jira/browse/LUCENE-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Chyla updated LUCENE-7481: Description: If used with a wildcard query, the result is a failure saying: "Rewrite query first" The SpanNearQuery has the rewrite method; however the SpanPayloadCheckQuery just returns the query itself. this works: ``` spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), SpanMultiTermQueryWrapper(vectrfield:m*), SpanMultiTermQueryWrapper(vectrfield:f*)], 0, true) ``` code to generate the query: ``` private Query getSpanQuery(String[] parts, int howMany, boolean truncate) throws UnsupportedEncodingException { SpanQuery[] clauses = new SpanQuery[howMany+1]; clauses[0] = new SpanTermQuery(new Term("vectrfield", parts[0])); // surname for (int i = 0; i < howMany; i++) { if (truncate) { SpanMultiTermQueryWrapper q = new SpanMultiTermQueryWrapper(new WildcardQuery(new Term("vectrfield", parts[i+1].substring(0, 1) + "*"))); clauses[i+1] = q; } else { clauses[i+1] = new SpanTermQuery(new Term("vectrfield", parts[i+1])); } } SpanNearQuery sq = new SpanNearQuery(clauses, 0, true); // match in order return sq; } ``` and this fails: {code:java} spanPayCheck(spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), SpanMultiTermQueryWrapper(vectrfield:m*), SpanMultiTermQueryWrapper(vectrfield:f*)], 1, true), payloadRef: 0;1;2;3;) {/code} each clause is made of: ``` new SpanMultiTermQueryWrapper(new WildcardQuery(new Term("vectrfield", parts[i+1].substring(0, 1) + "*"))); ``` It is a regression; the code was working well in SOLR4.x was: If used with a wildcard query, the result is a failure saying: "Rewrite query first" The SpanNearQuery has the rewrite method; however the SpanPayloadCheckQuery just returns the query itself. this works: ``` spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), SpanMultiTermQueryWrapper(vectrfield:m*), SpanMultiTermQueryWrapper(vectrfield:f*)], 0, true) ``` code to generate the query: ``` private Query getSpanQuery(String[] parts, int howMany, boolean truncate) throws UnsupportedEncodingException { SpanQuery[] clauses = new SpanQuery[howMany+1]; clauses[0] = new SpanTermQuery(new Term("vectrfield", parts[0])); // surname for (int i = 0; i < howMany; i++) { if (truncate) { SpanMultiTermQueryWrapper q = new SpanMultiTermQueryWrapper(new WildcardQuery(new Term("vectrfield", parts[i+1].substring(0, 1) + "*"))); clauses[i+1] = q; } else { clauses[i+1] = new SpanTermQuery(new Term("vectrfield", parts[i+1])); } } SpanNearQuery sq = new SpanNearQuery(clauses, 0, true); // match in order return sq; } ``` and this fails: ``` spanPayCheck(spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), SpanMultiTermQueryWrapper(vectrfield:m*), SpanMultiTermQueryWrapper(vectrfield:f*)], 1, true), payloadRef: 0;1;2;3;) ``` each clause is made of: ``` new SpanMultiTermQueryWrapper(new WildcardQuery(new Term("vectrfield", parts[i+1].substring(0, 1) + "*"))); ``` It is a regression; the code was working well in SOLR4.x > SpanPayloadCheckQuery is missing rewrite method > --- > > Key: LUCENE-7481 > URL: https://issues.apache.org/jira/browse/LUCENE-7481 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: 6.x >Reporter: Roman Chyla > > If used with a wildcard query, the result is a failure saying: "Rewrite query > first" > The SpanNearQuery has the rewrite method; however the SpanPayloadCheckQuery > just returns the query itself. > this works: > ``` > spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), > SpanMultiTermQueryWrapper(vectrfield:m*), > SpanMultiTermQueryWrapper(vectrfield:f*)], 0, true) > ``` > code to generate the query: > ``` > private Query getSpanQuery(String[] parts, int howMany, boolean truncate) > throws UnsupportedEncodingException { > SpanQuery[] clauses = new SpanQuery[howMany+1]; > clauses[0] = new SpanTermQuery(new Term("vectrfield", > parts[0])); // surname > for (int i = 0; i < howMany; i++) { > if (truncate) { > SpanMultiTermQueryWrapper q = new > SpanMultiTermQueryW
[jira] [Created] (LUCENE-7481) SpanPayloadCheckQuery is missing rewrite method
Roman Chyla created LUCENE-7481: --- Summary: SpanPayloadCheckQuery is missing rewrite method Key: LUCENE-7481 URL: https://issues.apache.org/jira/browse/LUCENE-7481 Project: Lucene - Core Issue Type: Bug Affects Versions: 6.x Reporter: Roman Chyla If used with a wildcard query, the result is a failure saying: "Rewrite query first" The SpanNearQuery has the rewrite method; however the SpanPayloadCheckQuery just returns the query itself. this works: ``` spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), SpanMultiTermQueryWrapper(vectrfield:m*), SpanMultiTermQueryWrapper(vectrfield:f*)], 0, true) ``` code to generate the query: ``` private Query getSpanQuery(String[] parts, int howMany, boolean truncate) throws UnsupportedEncodingException { SpanQuery[] clauses = new SpanQuery[howMany+1]; clauses[0] = new SpanTermQuery(new Term("vectrfield", parts[0])); // surname for (int i = 0; i < howMany; i++) { if (truncate) { SpanMultiTermQueryWrapper q = new SpanMultiTermQueryWrapper(new WildcardQuery(new Term("vectrfield", parts[i+1].substring(0, 1) + "*"))); clauses[i+1] = q; } else { clauses[i+1] = new SpanTermQuery(new Term("vectrfield", parts[i+1])); } } SpanNearQuery sq = new SpanNearQuery(clauses, 0, true); // match in order return sq; } ``` and this fails: ``` spanPayCheck(spanNear([vectrfield:ebyuugz, SpanMultiTermQueryWrapper(vectrfield:e*), SpanMultiTermQueryWrapper(vectrfield:m*), SpanMultiTermQueryWrapper(vectrfield:f*)], 1, true), payloadRef: 0;1;2;3;) ``` each clause is made of: ``` new SpanMultiTermQueryWrapper(new WildcardQuery(new Term("vectrfield", parts[i+1].substring(0, 1) + "*"))); ``` It is a regression; the code was working well in SOLR4.x -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6468) Regression: StopFilterFactory doesn't work properly without enablePositionIncrements="false"
[ https://issues.apache.org/jira/browse/SOLR-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514785#comment-15514785 ] Roman Chyla commented on SOLR-6468: --- Ha! :-) I've found my own comment above, 2 years later I'm facing this situation again, I completely forgot (and truth be told: preferred running old solr 4x). This is how the new solr sees things: A 350-MHz GBT Survey of 50 Faint Fermi γ ray Sources for Radio Millisecond Pulsars is indexed as ``` null_1 1 :350|350mhz 2 :mhz|syn::mhz 3 :acr::gbt|gbt|syn::gbt|syn::green bank telescope 4 :survey|syn::survey null_1 6 :50 ``` the 1st and 5th position is a gap - so the search for "350-MHz GBT Survey of 50 Faint" will fail - because 'of' is a stopword and the stop-filter will always increment the position (what's the purpose of a stopfilter; if it is leaving gaps?) anyways, the solution with CharFilterFactory cannot work for me, I have to do this: 1. search for synonyms (they can contain stopwords) 2. remove stopwords 3. search for other synonyms (that don't have stopwords) I'm afraid the real life is little bit more complex than what it seems; but there is a logic to your choices, SOLR devs, I'm afraid I can agree with you. People who understand the *why* will make it work again as it *should*. Others will happily keep using the 'simplified' version. > Regression: StopFilterFactory doesn't work properly without > enablePositionIncrements="false" > > > Key: SOLR-6468 > URL: https://issues.apache.org/jira/browse/SOLR-6468 > Project: Solr > Issue Type: Bug >Affects Versions: 4.8.1, 4.9 >Reporter: Alexander S. > > Setup: > * Schema version is 1.5 > * Field config: > {code} > autoGeneratePhraseQueries="true"> > > > ignoreCase="true" /> > > > > {code} > * Stop words: > {code} > http > https > ftp > www > {code} > So very simple. In the index I have: > * twitter.com/testuser > All these queries do match: > * twitter.com/testuser > * com/testuser > * testuser > But none of these does: > * https://twitter.com/testuser > * https://www.twitter.com/testuser > * www.twitter.com/testuser > Debug output shows: > "parsedquery_toString": "+(url_words_ngram:\"? twitter com testuser\")" > But we need: > "parsedquery_toString": "+(url_words_ngram:\"twitter com testuser\")" > Complete debug outputs: > * a valid search: > http://pastie.org/pastes/9500661/text?key=rgqj5ivlgsbk1jxsudx9za > * an invalid search: > http://pastie.org/pastes/9500662/text?key=b4zlh2oaxtikd8jvo5xaww > The complete discussion and explanation of the problem is here: > http://lucene.472066.n3.nabble.com/Help-with-StopFilterFactory-td4153839.html > I didn't find a clear explanation how can we upgrade Solr, there's no any > replacement or a workarround to this, so this is not just a major change but > a major disrespect to all existing Solr users who are using this feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6468) Regression: StopFilterFactory doesn't work properly without enablePositionIncrements="false"
[ https://issues.apache.org/jira/browse/SOLR-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225186#comment-14225186 ] Roman Chyla commented on SOLR-6468: --- I also find this change to be unfortunate. If this is just a developers making decisions for users (then it causes problems to users who really know why they do need that feature: for phrase search that should ignore stopwords). But if the underlying issue is something serious with the indexer not being able to work with the position, than it would be even weirder - and actually very bad for many users. I don't really understand benefits of this change. Any chance to return to the original? > Regression: StopFilterFactory doesn't work properly without > enablePositionIncrements="false" > > > Key: SOLR-6468 > URL: https://issues.apache.org/jira/browse/SOLR-6468 > Project: Solr > Issue Type: Bug >Affects Versions: 4.8.1, 4.9 >Reporter: Alexander S. > > Setup: > * Schema version is 1.5 > * Field config: > {code} > autoGeneratePhraseQueries="true"> > > > ignoreCase="true" /> > > > > {code} > * Stop words: > {code} > http > https > ftp > www > {code} > So very simple. In the index I have: > * twitter.com/testuser > All these queries do match: > * twitter.com/testuser > * com/testuser > * testuser > But none of these does: > * https://twitter.com/testuser > * https://www.twitter.com/testuser > * www.twitter.com/testuser > Debug output shows: > "parsedquery_toString": "+(url_words_ngram:\"? twitter com testuser\")" > But we need: > "parsedquery_toString": "+(url_words_ngram:\"twitter com testuser\")" > Complete debug outputs: > * a valid search: > http://pastie.org/pastes/9500661/text?key=rgqj5ivlgsbk1jxsudx9za > * an invalid search: > http://pastie.org/pastes/9500662/text?key=b4zlh2oaxtikd8jvo5xaww > The complete discussion and explanation of the problem is here: > http://lucene.472066.n3.nabble.com/Help-with-StopFilterFactory-td4153839.html > I didn't find a clear explanation how can we upgrade Solr, there's no any > replacement or a workarround to this, so this is not just a major change but > a major disrespect to all existing Solr users who are using this feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5014) ANTLR Lucene query parser
[ https://issues.apache.org/jira/browse/LUCENE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Chyla updated LUCENE-5014: Attachment: LUCENE-5014.txt Added solr qparserplugin > ANTLR Lucene query parser > - > > Key: LUCENE-5014 > URL: https://issues.apache.org/jira/browse/LUCENE-5014 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser, modules/queryparser >Affects Versions: 4.3 > Environment: all >Reporter: Roman Chyla > Labels: antlr, query, queryparser > Attachments: LUCENE-5014.txt, LUCENE-5014.txt, LUCENE-5014.txt, > LUCENE-5014.txt, LUCENE-5014.txt > > > I would like to propose a new way of building query parsers for Lucene. > Currently, most Lucene parsers are hard to extend because they are either > written in Java (ie. the SOLR query parser, or edismax) or the parsing logic > is 'married' with the query building logic (i.e. the standard lucene parser, > generated by JavaCC) - which makes any extension really hard. > Few years back, Lucene got the contrib/modern query parser (later renamed to > 'flexible'), yet that parser didn't become a star (it must be very confusing > for many users). However, that parsing framework is very powerful! And it is > a real pity that there aren't more parsers already using it - because it > allows us to add/extend/change almost any aspect of the query parsing. > So, if we combine ANTLR + queryparser.flexible, we can get very powerful > framework for building almost any query language one can think of. And I hope > this extension can become useful. > The details: > - every new query syntax is written in EBNF, it lives in separate files (and > can be tested/developed independently - using 'gunit') > - ANTLR parser generates parsing code (and it can generate parsers in > several languages, the main target is Java, but it can also do Python - which > may be interesting for pylucene) > - the parser generates AST (abstract syntax tree) which is consumed by a > 'pipeline' of processors, users can easily modify this pipeline to add a > desired functionality > - the new parser contains a few (very important) debugging functions; it can > print results of every stage of the build, generate AST's as graphical > charts; ant targets help to build/test/debug grammars > - I've tried to reuse the existing queryparser.flexible components as much > as possible, only adding new processors when necessary > Assumptions about the grammar: > - every grammar must have one top parse rule called 'mainQ' > - parsers must generate AST (Abstract Syntax Tree) > The structure of the AST is left open, there are components which make > assumptions about the shape of the AST (ie. that MODIFIER is parent of a a > FIELD) however users are free to choose/write different processors with > different assumptions about the AST shape. > More documentation on how to use the parser can be seen here: > http://29min.wordpress.com/category/antlrqueryparser/ > The parser has been created more than one year back and is used in production > (http://labs.adsabs.harvard.edu/adsabs/). A different dialects of query > languages (with proximity operatos, functions, special logic etc) - can be > seen here: > https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs > https://github.com/romanchyla/montysolr/tree/master/contrib/invenio -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5014) ANTLR Lucene query parser
[ https://issues.apache.org/jira/browse/LUCENE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13699417#comment-13699417 ] Roman Chyla commented on LUCENE-5014: - New addition: solr qparser plugin. It is unfortunately not as easy as one may think, because of various defaults - e.g. user may want to specify different defaultField, whether wildcards are allowed at the beginning, what is the maximum range for proximity values... some of which should be only in solrconfig.xml, and some also in query params. So here is a stab at it, it works, but may require more config options - there is also a new unittest. Only that Ivy mirrors decided to not work now (ughhh) so I could not test solr unittests - ihope it works. Lucene's 'ant test' went fine. If sb wants to try in solr, please make sure you have antlr-runtime.jar in your solr libs and this should go inside solrconfig.xml {code} text {code} > ANTLR Lucene query parser > - > > Key: LUCENE-5014 > URL: https://issues.apache.org/jira/browse/LUCENE-5014 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser, modules/queryparser >Affects Versions: 4.3 > Environment: all >Reporter: Roman Chyla > Labels: antlr, query, queryparser > Attachments: LUCENE-5014.txt, LUCENE-5014.txt, LUCENE-5014.txt, > LUCENE-5014.txt, LUCENE-5014.txt > > > I would like to propose a new way of building query parsers for Lucene. > Currently, most Lucene parsers are hard to extend because they are either > written in Java (ie. the SOLR query parser, or edismax) or the parsing logic > is 'married' with the query building logic (i.e. the standard lucene parser, > generated by JavaCC) - which makes any extension really hard. > Few years back, Lucene got the contrib/modern query parser (later renamed to > 'flexible'), yet that parser didn't become a star (it must be very confusing > for many users). However, that parsing framework is very powerful! And it is > a real pity that there aren't more parsers already using it - because it > allows us to add/extend/change almost any aspect of the query parsing. > So, if we combine ANTLR + queryparser.flexible, we can get very powerful > framework for building almost any query language one can think of. And I hope > this extension can become useful. > The details: > - every new query syntax is written in EBNF, it lives in separate files (and > can be tested/developed independently - using 'gunit') > - ANTLR parser generates parsing code (and it can generate parsers in > several languages, the main target is Java, but it can also do Python - which > may be interesting for pylucene) > - the parser generates AST (abstract syntax tree) which is consumed by a > 'pipeline' of processors, users can easily modify this pipeline to add a > desired functionality > - the new parser contains a few (very important) debugging functions; it can > print results of every stage of the build, generate AST's as graphical > charts; ant targets help to build/test/debug grammars > - I've tried to reuse the existing queryparser.flexible components as much > as possible, only adding new processors when necessary > Assumptions about the grammar: > - every grammar must have one top parse rule called 'mainQ' > - parsers must generate AST (Abstract Syntax Tree) > The structure of the AST is left open, there are components which make > assumptions about the shape of the AST (ie. that MODIFIER is parent of a a > FIELD) however users are free to choose/write different processors with > different assumptions about the AST shape. > More documentation on how to use the parser can be seen here: > http://29min.wordpress.com/category/antlrqueryparser/ > The parser has been created more than one year back and is used in production > (http://labs.adsabs.harvard.edu/adsabs/). A different dialects of query > languages (with proximity operatos, functions, special logic etc) - can be > seen here: > https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs > https://github.com/romanchyla/montysolr/tree/master/contrib/invenio -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5014) ANTLR Lucene query parser
[ https://issues.apache.org/jira/browse/LUCENE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698985#comment-13698985 ] Roman Chyla commented on LUCENE-5014: - will it be OK to include the solr parts in this ticket? besides the jira name, that seems s aa best option to me. > ANTLR Lucene query parser > - > > Key: LUCENE-5014 > URL: https://issues.apache.org/jira/browse/LUCENE-5014 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser, modules/queryparser >Affects Versions: 4.3 > Environment: all >Reporter: Roman Chyla > Labels: antlr, query, queryparser > Attachments: LUCENE-5014.txt, LUCENE-5014.txt, LUCENE-5014.txt, > LUCENE-5014.txt > > > I would like to propose a new way of building query parsers for Lucene. > Currently, most Lucene parsers are hard to extend because they are either > written in Java (ie. the SOLR query parser, or edismax) or the parsing logic > is 'married' with the query building logic (i.e. the standard lucene parser, > generated by JavaCC) - which makes any extension really hard. > Few years back, Lucene got the contrib/modern query parser (later renamed to > 'flexible'), yet that parser didn't become a star (it must be very confusing > for many users). However, that parsing framework is very powerful! And it is > a real pity that there aren't more parsers already using it - because it > allows us to add/extend/change almost any aspect of the query parsing. > So, if we combine ANTLR + queryparser.flexible, we can get very powerful > framework for building almost any query language one can think of. And I hope > this extension can become useful. > The details: > - every new query syntax is written in EBNF, it lives in separate files (and > can be tested/developed independently - using 'gunit') > - ANTLR parser generates parsing code (and it can generate parsers in > several languages, the main target is Java, but it can also do Python - which > may be interesting for pylucene) > - the parser generates AST (abstract syntax tree) which is consumed by a > 'pipeline' of processors, users can easily modify this pipeline to add a > desired functionality > - the new parser contains a few (very important) debugging functions; it can > print results of every stage of the build, generate AST's as graphical > charts; ant targets help to build/test/debug grammars > - I've tried to reuse the existing queryparser.flexible components as much > as possible, only adding new processors when necessary > Assumptions about the grammar: > - every grammar must have one top parse rule called 'mainQ' > - parsers must generate AST (Abstract Syntax Tree) > The structure of the AST is left open, there are components which make > assumptions about the shape of the AST (ie. that MODIFIER is parent of a a > FIELD) however users are free to choose/write different processors with > different assumptions about the AST shape. > More documentation on how to use the parser can be seen here: > http://29min.wordpress.com/category/antlrqueryparser/ > The parser has been created more than one year back and is used in production > (http://labs.adsabs.harvard.edu/adsabs/). A different dialects of query > languages (with proximity operatos, functions, special logic etc) - can be > seen here: > https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs > https://github.com/romanchyla/montysolr/tree/master/contrib/invenio -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5014) ANTLR Lucene query parser
[ https://issues.apache.org/jira/browse/LUCENE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698981#comment-13698981 ] Roman Chyla commented on LUCENE-5014: - HiErik, i'll add a solr qparser plugin too. thanks for reminding me. > ANTLR Lucene query parser > - > > Key: LUCENE-5014 > URL: https://issues.apache.org/jira/browse/LUCENE-5014 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser, modules/queryparser >Affects Versions: 4.3 > Environment: all >Reporter: Roman Chyla > Labels: antlr, query, queryparser > Attachments: LUCENE-5014.txt, LUCENE-5014.txt, LUCENE-5014.txt, > LUCENE-5014.txt > > > I would like to propose a new way of building query parsers for Lucene. > Currently, most Lucene parsers are hard to extend because they are either > written in Java (ie. the SOLR query parser, or edismax) or the parsing logic > is 'married' with the query building logic (i.e. the standard lucene parser, > generated by JavaCC) - which makes any extension really hard. > Few years back, Lucene got the contrib/modern query parser (later renamed to > 'flexible'), yet that parser didn't become a star (it must be very confusing > for many users). However, that parsing framework is very powerful! And it is > a real pity that there aren't more parsers already using it - because it > allows us to add/extend/change almost any aspect of the query parsing. > So, if we combine ANTLR + queryparser.flexible, we can get very powerful > framework for building almost any query language one can think of. And I hope > this extension can become useful. > The details: > - every new query syntax is written in EBNF, it lives in separate files (and > can be tested/developed independently - using 'gunit') > - ANTLR parser generates parsing code (and it can generate parsers in > several languages, the main target is Java, but it can also do Python - which > may be interesting for pylucene) > - the parser generates AST (abstract syntax tree) which is consumed by a > 'pipeline' of processors, users can easily modify this pipeline to add a > desired functionality > - the new parser contains a few (very important) debugging functions; it can > print results of every stage of the build, generate AST's as graphical > charts; ant targets help to build/test/debug grammars > - I've tried to reuse the existing queryparser.flexible components as much > as possible, only adding new processors when necessary > Assumptions about the grammar: > - every grammar must have one top parse rule called 'mainQ' > - parsers must generate AST (Abstract Syntax Tree) > The structure of the AST is left open, there are components which make > assumptions about the shape of the AST (ie. that MODIFIER is parent of a a > FIELD) however users are free to choose/write different processors with > different assumptions about the AST shape. > More documentation on how to use the parser can be seen here: > http://29min.wordpress.com/category/antlrqueryparser/ > The parser has been created more than one year back and is used in production > (http://labs.adsabs.harvard.edu/adsabs/). A different dialects of query > languages (with proximity operatos, functions, special logic etc) - can be > seen here: > https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs > https://github.com/romanchyla/montysolr/tree/master/contrib/invenio -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5014) ANTLR Lucene query parser
[ https://issues.apache.org/jira/browse/LUCENE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Chyla updated LUCENE-5014: Attachment: LUCENE-5014.txt The patch that *actually* contains the extended parser with NEAR operator support > ANTLR Lucene query parser > - > > Key: LUCENE-5014 > URL: https://issues.apache.org/jira/browse/LUCENE-5014 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser, modules/queryparser >Affects Versions: 4.3 > Environment: all >Reporter: Roman Chyla > Labels: antlr, query, queryparser > Attachments: LUCENE-5014.txt, LUCENE-5014.txt, LUCENE-5014.txt, > LUCENE-5014.txt > > > I would like to propose a new way of building query parsers for Lucene. > Currently, most Lucene parsers are hard to extend because they are either > written in Java (ie. the SOLR query parser, or edismax) or the parsing logic > is 'married' with the query building logic (i.e. the standard lucene parser, > generated by JavaCC) - which makes any extension really hard. > Few years back, Lucene got the contrib/modern query parser (later renamed to > 'flexible'), yet that parser didn't become a star (it must be very confusing > for many users). However, that parsing framework is very powerful! And it is > a real pity that there aren't more parsers already using it - because it > allows us to add/extend/change almost any aspect of the query parsing. > So, if we combine ANTLR + queryparser.flexible, we can get very powerful > framework for building almost any query language one can think of. And I hope > this extension can become useful. > The details: > - every new query syntax is written in EBNF, it lives in separate files (and > can be tested/developed independently - using 'gunit') > - ANTLR parser generates parsing code (and it can generate parsers in > several languages, the main target is Java, but it can also do Python - which > may be interesting for pylucene) > - the parser generates AST (abstract syntax tree) which is consumed by a > 'pipeline' of processors, users can easily modify this pipeline to add a > desired functionality > - the new parser contains a few (very important) debugging functions; it can > print results of every stage of the build, generate AST's as graphical > charts; ant targets help to build/test/debug grammars > - I've tried to reuse the existing queryparser.flexible components as much > as possible, only adding new processors when necessary > Assumptions about the grammar: > - every grammar must have one top parse rule called 'mainQ' > - parsers must generate AST (Abstract Syntax Tree) > The structure of the AST is left open, there are components which make > assumptions about the shape of the AST (ie. that MODIFIER is parent of a a > FIELD) however users are free to choose/write different processors with > different assumptions about the AST shape. > More documentation on how to use the parser can be seen here: > http://29min.wordpress.com/category/antlrqueryparser/ > The parser has been created more than one year back and is used in production > (http://labs.adsabs.harvard.edu/adsabs/). A different dialects of query > languages (with proximity operatos, functions, special logic etc) - can be > seen here: > https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs > https://github.com/romanchyla/montysolr/tree/master/contrib/invenio -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5014) ANTLR Lucene query parser
[ https://issues.apache.org/jira/browse/LUCENE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Chyla updated LUCENE-5014: Attachment: LUCENE-5014.txt The same patch + lucene grammar extended with NEARx operator > ANTLR Lucene query parser > - > > Key: LUCENE-5014 > URL: https://issues.apache.org/jira/browse/LUCENE-5014 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser, modules/queryparser >Affects Versions: 4.3 > Environment: all >Reporter: Roman Chyla > Labels: antlr, query, queryparser > Attachments: LUCENE-5014.txt, LUCENE-5014.txt, LUCENE-5014.txt > > > I would like to propose a new way of building query parsers for Lucene. > Currently, most Lucene parsers are hard to extend because they are either > written in Java (ie. the SOLR query parser, or edismax) or the parsing logic > is 'married' with the query building logic (i.e. the standard lucene parser, > generated by JavaCC) - which makes any extension really hard. > Few years back, Lucene got the contrib/modern query parser (later renamed to > 'flexible'), yet that parser didn't become a star (it must be very confusing > for many users). However, that parsing framework is very powerful! And it is > a real pity that there aren't more parsers already using it - because it > allows us to add/extend/change almost any aspect of the query parsing. > So, if we combine ANTLR + queryparser.flexible, we can get very powerful > framework for building almost any query language one can think of. And I hope > this extension can become useful. > The details: > - every new query syntax is written in EBNF, it lives in separate files (and > can be tested/developed independently - using 'gunit') > - ANTLR parser generates parsing code (and it can generate parsers in > several languages, the main target is Java, but it can also do Python - which > may be interesting for pylucene) > - the parser generates AST (abstract syntax tree) which is consumed by a > 'pipeline' of processors, users can easily modify this pipeline to add a > desired functionality > - the new parser contains a few (very important) debugging functions; it can > print results of every stage of the build, generate AST's as graphical > charts; ant targets help to build/test/debug grammars > - I've tried to reuse the existing queryparser.flexible components as much > as possible, only adding new processors when necessary > Assumptions about the grammar: > - every grammar must have one top parse rule called 'mainQ' > - parsers must generate AST (Abstract Syntax Tree) > The structure of the AST is left open, there are components which make > assumptions about the shape of the AST (ie. that MODIFIER is parent of a a > FIELD) however users are free to choose/write different processors with > different assumptions about the AST shape. > More documentation on how to use the parser can be seen here: > http://29min.wordpress.com/category/antlrqueryparser/ > The parser has been created more than one year back and is used in production > (http://labs.adsabs.harvard.edu/adsabs/). A different dialects of query > languages (with proximity operatos, functions, special logic etc) - can be > seen here: > https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs > https://github.com/romanchyla/montysolr/tree/master/contrib/invenio -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5014) ANTLR Lucene query parser
[ https://issues.apache.org/jira/browse/LUCENE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13695149#comment-13695149 ] Roman Chyla commented on LUCENE-5014: - Adding an example, standard lucene grammar extended with NEAR operators (as discussed above) This should illustrate how easy it is to extend/modify/add a new query dialect. Handling of NEAR operators is not at all trivial, so I hope you will have some fun realizing it can be done in two lines ;) {code} setGrammarName("ExtendedLuceneGrammar"); ((AqpQueryTreeBuilder) qp.getQueryBuilder()).setBuilder(AqpNearQueryNode.class, new AqpNearQueryNodeBuilder()); {code} Have a look at TestAqpExtendedLGSimple > ANTLR Lucene query parser > - > > Key: LUCENE-5014 > URL: https://issues.apache.org/jira/browse/LUCENE-5014 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser, modules/queryparser >Affects Versions: 4.3 > Environment: all >Reporter: Roman Chyla > Labels: antlr, query, queryparser > Attachments: LUCENE-5014.txt, LUCENE-5014.txt > > > I would like to propose a new way of building query parsers for Lucene. > Currently, most Lucene parsers are hard to extend because they are either > written in Java (ie. the SOLR query parser, or edismax) or the parsing logic > is 'married' with the query building logic (i.e. the standard lucene parser, > generated by JavaCC) - which makes any extension really hard. > Few years back, Lucene got the contrib/modern query parser (later renamed to > 'flexible'), yet that parser didn't become a star (it must be very confusing > for many users). However, that parsing framework is very powerful! And it is > a real pity that there aren't more parsers already using it - because it > allows us to add/extend/change almost any aspect of the query parsing. > So, if we combine ANTLR + queryparser.flexible, we can get very powerful > framework for building almost any query language one can think of. And I hope > this extension can become useful. > The details: > - every new query syntax is written in EBNF, it lives in separate files (and > can be tested/developed independently - using 'gunit') > - ANTLR parser generates parsing code (and it can generate parsers in > several languages, the main target is Java, but it can also do Python - which > may be interesting for pylucene) > - the parser generates AST (abstract syntax tree) which is consumed by a > 'pipeline' of processors, users can easily modify this pipeline to add a > desired functionality > - the new parser contains a few (very important) debugging functions; it can > print results of every stage of the build, generate AST's as graphical > charts; ant targets help to build/test/debug grammars > - I've tried to reuse the existing queryparser.flexible components as much > as possible, only adding new processors when necessary > Assumptions about the grammar: > - every grammar must have one top parse rule called 'mainQ' > - parsers must generate AST (Abstract Syntax Tree) > The structure of the AST is left open, there are components which make > assumptions about the shape of the AST (ie. that MODIFIER is parent of a a > FIELD) however users are free to choose/write different processors with > different assumptions about the AST shape. > More documentation on how to use the parser can be seen here: > http://29min.wordpress.com/category/antlrqueryparser/ > The parser has been created more than one year back and is used in production > (http://labs.adsabs.harvard.edu/adsabs/). A different dialects of query > languages (with proximity operatos, functions, special logic etc) - can be > seen here: > https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs > https://github.com/romanchyla/montysolr/tree/master/contrib/invenio -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-5014) ANTLR Lucene query parser
[ https://issues.apache.org/jira/browse/LUCENE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13667908#comment-13667908 ] Roman Chyla edited comment on LUCENE-5014 at 5/27/13 7:04 PM: -- Hi David, In practical terms ANTLR can do exactly the same thing as PEG (ie lookahead, backtracking,memoization) - see this http://stackoverflow.com/questions/8816759/ll-versus-peg-parsers-what-is-the-difference But it is also capable of doing more things than PEG (ie. better error recovery - PEG parser needs to parse the whole tree before it discovers an error; then the error recovery is not the same thing) PEG's can be easier *especially* because of the first-choice operator; in fact at times I wished that ANTLR just chose the first available option (well, it does, but it reports and error and I didn't want to have grammar with errors). So, in CFG&ANTLR world, ambiguity is solved using syntactic predicates (lookahead) -- so far, this has been a theoretical, here are few more points: Grammar vs code === I looked at the presentation and the parser contains the operator precedence, however there it is spread across several screens of java code, i find the following much more readable {code} mainQ : clauseOr+ EOF ; clauseOr : clauseAnd (or clauseAnd )* ; clauseAnd : clauseNot (and clauseNot)* ; {code} It is essentially the same thing, but it is independent of the Java and I can see it on few lines - and extend it adding few more lines. The patch I wrote makes the handling of separate grammar and generated code seamless. So the 2/3 advantages of PEG over ANTLR disappear. Syntax vs semantics (business logic) The example from the presentation needs to be much more involved if it is to be used in the real life. Consider this query: {noformat} dog NEAR cat {noformat} This is going to work only in the simplest case, where each term is a single TermQuery. Yet if there was a synonym expansion (where would it go inside the PEG parser, is one question) - the parser needs to *rewrite* the query something like: {noformat} (dog|canin) NEAR cat --> (dog NEAR cat) OR (canin NEAR cat) {noformat} So, there you get the 'spaghetti problem' - in the example presented, the logic that rewrites the query must reside in the same place as the query parsing. That is not an improvement IMO, it is the same thing as the old Lucene parsers written in JavaCC which are very difficult to extend or debug I think I'll add a new grammar with the proximity operators so that you can see how easy it is to solve the same situation with ANTLR (but you will need to read the patch this time ;)) btw. the patch is big because i included the html with SVG charts of the generated parse trees and one Excel file (that one helps in writing unittest for the grammar) Developer vs user experience I think PEG definitely looks simpler to developers (in the presented example) and its main advantage is the first-choice operator. But since ANTLR can do the same and it has programming language independent grammar, it can do the same job. The difference may be in maturity of the project, tools available (ie debuggers) - and of course implementation (see the link above for details) I can imagine that for PEG you can use your IDE of choice, while with ANTLR there is this 'pesky' level of abstraction - but there are tools that make life bearable, such as ANTLRWorks or Eclipse ANTLR debugger (though I have not liked that one); grammar unittest and I added ways to debug/view the grammar. If you apply the patch, you can try: {code} ant -f aqp-build.xml gunit # edit StandardLuceneGrammar and save as 'mytestgrammar' ant -f aqp-build.xml try-view -Dquery="foo NEAR bar" -Dgrammar=mytestgrammar {code} There may be of course more things to consider, but I believe the 3 issues above present some interesting vantage points. was (Author: rchyla): Hi David, In practical terms ANTLR can do exactly the same thing as PEG (ie lookahead, backtracking,memoization) - see this http://stackoverflow.com/questions/8816759/ll-versus-peg-parsers-what-is-the-difference But it is also capable of doing more things than PEG (ie. better error recovery - PEG parser needs to parse the whole tree before it discovers an error; then the error recovery is not the same thing) PEG's can be easier *especially* because of the first-choice operator; in fact at times I wished that ANTLR just chose the first available option (well, it does, but it reports and error and I didn't want to have grammar with errors). So, in CFG&ANTLR world, ambiguity is solved using syntactic predicated (lookahead) -- so far, this has been a theoretical, here are few more points: Clarity === I looked at the presentation and the parser contai
[jira] [Commented] (LUCENE-5014) ANTLR Lucene query parser
[ https://issues.apache.org/jira/browse/LUCENE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13667908#comment-13667908 ] Roman Chyla commented on LUCENE-5014: - Hi David, In practical terms ANTLR can do exactly the same thing as PEG (ie lookahead, backtracking,memoization) - see this http://stackoverflow.com/questions/8816759/ll-versus-peg-parsers-what-is-the-difference But it is also capable of doing more things than PEG (ie. better error recovery - PEG parser needs to parse the whole tree before it discovers an error; then the error recovery is not the same thing) PEG's can be easier *especially* because of the first-choice operator; in fact at times I wished that ANTLR just chose the first available option (well, it does, but it reports and error and I didn't want to have grammar with errors). So, in CFG&ANTLR world, ambiguity is solved using syntactic predicated (lookahead) -- so far, this has been a theoretical, here are few more points: Clarity === I looked at the presentation and the parser contains the operator precedence, however there it is spread across several screens of java code, i find the following much more readable {code} mainQ : clauseOr+ EOF ; clauseOr : clauseAnd (or clauseAnd )* ; clauseAnd : clauseNot (and clauseNot)* ; {code} It is essentially the same thing, but it is independent of the Java and I can see it on few lines - and extend it adding few more lines. The patch I wrote makes the handling of separate grammar and generated code seamless. So the 2/3 advantages of PEG over ANTLR disappear. Syntax vs semantics (business logic) The example from the presentation needs to be much more involved if it is to be used in the real life. Consider this query: {noformat} dog NEAR cat {noformat} This is going to work only in the simplest case, where each term is a single TermQuery. Yet if there was a synonym expansion (where would it go inside the PEG parser, is one question) - the parser needs to *rewrite* the query something like: {noformat} (dog|canin) NEAR cat --> (dog NEAR cat) OR (canin NEAR cat) {noformat} So, there you get the 'spaghetti problem' - in the example presented, the logic that rewrites the query must reside in the same place as the query parsing. That is not an improvement IMO, it is the same thing as the old Lucene parsers written in JavaCC which are very difficult to extend or debug I think I'll add a new grammar with the proximity operators so that you can see how easy it is to solve the same situation with ANTLR (but you will need to read the patch this time ;)) btw. the patch is big because i included the html with SVG charts of the generated parse trees and one Excel file (that one helps in writing unittest for the grammar) Developer vs user experience I think PEG definitely looks simpler (in the presented example) and its main advantage is the first-choice operator. But since ANTLR can do the same and it has programming language independent grammar, it can do the same job. The difference may be in maturity of the project, tools available (ie debuggers) - and of course implementation (see the link above for details) I can imagine that for PEG you can use your IDE of choice, while with ANTLR there is this 'pesky' level of abstraction - but there are tools that make life bearable, such as ANTLRWorks or Eclipse ANTLR debugger (though I have not liked that one); grammar unittest and I added ways to debug/view the grammar. Again, I recommend trying it, e.g. {code} ant -f aqp-build.xml gunit # edit StandardLuceneGrammar and save as 'mytestgrammar' ant -f aqp-build.xml try-view -Dquery="foo NEAR bar" -Dgrammar=mytestgrammar {code} There may be of course more things to consider, but I believe the 3 issues above present some interesting vantage points. > ANTLR Lucene query parser > - > > Key: LUCENE-5014 > URL: https://issues.apache.org/jira/browse/LUCENE-5014 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser, modules/queryparser >Affects Versions: 4.3 > Environment: all >Reporter: Roman Chyla > Labels: antlr, query, queryparser > Attachments: LUCENE-5014.txt, LUCENE-5014.txt > > > I would like to propose a new way of building query parsers for Lucene. > Currently, most Lucene parsers are hard to extend because they are either > written in Java (ie. the SOLR query parser, or edismax) or the parsing logic > is 'married' with the query building logic (i.e. the standard lucene parser, > generated by JavaCC) - which makes any extension really hard. > Few years back, Lucene got the contrib/modern query parser (later renamed to > 'flexible'), yet that parser di
[jira] [Updated] (LUCENE-5014) ANTLR Lucene query parser
[ https://issues.apache.org/jira/browse/LUCENE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Chyla updated LUCENE-5014: Attachment: LUCENE-5014.txt Includes binary files (ie. one jar and xls) svn diff --force --diff-cmd /usr/bin/diff -x -au > LUCENE-5014.txt > ANTLR Lucene query parser > - > > Key: LUCENE-5014 > URL: https://issues.apache.org/jira/browse/LUCENE-5014 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser, modules/queryparser >Affects Versions: 4.3 > Environment: all >Reporter: Roman Chyla > Labels: antlr, query, queryparser > Attachments: LUCENE-5014.txt, LUCENE-5014.txt > > > I would like to propose a new way of building query parsers for Lucene. > Currently, most Lucene parsers are hard to extend because they are either > written in Java (ie. the SOLR query parser, or edismax) or the parsing logic > is 'married' with the query building logic (i.e. the standard lucene parser, > generated by JavaCC) - which makes any extension really hard. > Few years back, Lucene got the contrib/modern query parser (later renamed to > 'flexible'), yet that parser didn't become a star (it must be very confusing > for many users). However, that parsing framework is very powerful! And it is > a real pity that there aren't more parsers already using it - because it > allows us to add/extend/change almost any aspect of the query parsing. > So, if we combine ANTLR + queryparser.flexible, we can get very powerful > framework for building almost any query language one can think of. And I hope > this extension can become useful. > The details: > - every new query syntax is written in EBNF, it lives in separate files (and > can be tested/developed independently - using 'gunit') > - ANTLR parser generates parsing code (and it can generate parsers in > several languages, the main target is Java, but it can also do Python - which > may be interesting for pylucene) > - the parser generates AST (abstract syntax tree) which is consumed by a > 'pipeline' of processors, users can easily modify this pipeline to add a > desired functionality > - the new parser contains a few (very important) debugging functions; it can > print results of every stage of the build, generate AST's as graphical > charts; ant targets help to build/test/debug grammars > - I've tried to reuse the existing queryparser.flexible components as much > as possible, only adding new processors when necessary > Assumptions about the grammar: > - every grammar must have one top parse rule called 'mainQ' > - parsers must generate AST (Abstract Syntax Tree) > The structure of the AST is left open, there are components which make > assumptions about the shape of the AST (ie. that MODIFIER is parent of a a > FIELD) however users are free to choose/write different processors with > different assumptions about the AST shape. > More documentation on how to use the parser can be seen here: > http://29min.wordpress.com/category/antlrqueryparser/ > The parser has been created more than one year back and is used in production > (http://labs.adsabs.harvard.edu/adsabs/). A different dialects of query > languages (with proximity operatos, functions, special logic etc) - can be > seen here: > https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs > https://github.com/romanchyla/montysolr/tree/master/contrib/invenio -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5014) ANTLR Lucene query parser
[ https://issues.apache.org/jira/browse/LUCENE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Chyla updated LUCENE-5014: Attachment: LUCENE-5014.txt Patch without binary files (if possible, use the other patch) > ANTLR Lucene query parser > - > > Key: LUCENE-5014 > URL: https://issues.apache.org/jira/browse/LUCENE-5014 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser, modules/queryparser >Affects Versions: 4.3 > Environment: all >Reporter: Roman Chyla > Labels: antlr, query, queryparser > Attachments: LUCENE-5014.txt > > > I would like to propose a new way of building query parsers for Lucene. > Currently, most Lucene parsers are hard to extend because they are either > written in Java (ie. the SOLR query parser, or edismax) or the parsing logic > is 'married' with the query building logic (i.e. the standard lucene parser, > generated by JavaCC) - which makes any extension really hard. > Few years back, Lucene got the contrib/modern query parser (later renamed to > 'flexible'), yet that parser didn't become a star (it must be very confusing > for many users). However, that parsing framework is very powerful! And it is > a real pity that there aren't more parsers already using it - because it > allows us to add/extend/change almost any aspect of the query parsing. > So, if we combine ANTLR + queryparser.flexible, we can get very powerful > framework for building almost any query language one can think of. And I hope > this extension can become useful. > The details: > - every new query syntax is written in EBNF, it lives in separate files (and > can be tested/developed independently - using 'gunit') > - ANTLR parser generates parsing code (and it can generate parsers in > several languages, the main target is Java, but it can also do Python - which > may be interesting for pylucene) > - the parser generates AST (abstract syntax tree) which is consumed by a > 'pipeline' of processors, users can easily modify this pipeline to add a > desired functionality > - the new parser contains a few (very important) debugging functions; it can > print results of every stage of the build, generate AST's as graphical > charts; ant targets help to build/test/debug grammars > - I've tried to reuse the existing queryparser.flexible components as much > as possible, only adding new processors when necessary > Assumptions about the grammar: > - every grammar must have one top parse rule called 'mainQ' > - parsers must generate AST (Abstract Syntax Tree) > The structure of the AST is left open, there are components which make > assumptions about the shape of the AST (ie. that MODIFIER is parent of a a > FIELD) however users are free to choose/write different processors with > different assumptions about the AST shape. > More documentation on how to use the parser can be seen here: > http://29min.wordpress.com/category/antlrqueryparser/ > The parser has been created more than one year back and is used in production > (http://labs.adsabs.harvard.edu/adsabs/). A different dialects of query > languages (with proximity operatos, functions, special logic etc) - can be > seen here: > https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs > https://github.com/romanchyla/montysolr/tree/master/contrib/invenio -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-5014) ANTLR Lucene query parser
Roman Chyla created LUCENE-5014: --- Summary: ANTLR Lucene query parser Key: LUCENE-5014 URL: https://issues.apache.org/jira/browse/LUCENE-5014 Project: Lucene - Core Issue Type: Improvement Components: core/queryparser, modules/queryparser Affects Versions: 4.3 Environment: all Reporter: Roman Chyla I would like to propose a new way of building query parsers for Lucene. Currently, most Lucene parsers are hard to extend because they are either written in Java (ie. the SOLR query parser, or edismax) or the parsing logic is 'married' with the query building logic (i.e. the standard lucene parser, generated by JavaCC) - which makes any extension really hard. Few years back, Lucene got the contrib/modern query parser (later renamed to 'flexible'), yet that parser didn't become a star (it must be very confusing for many users). However, that parsing framework is very powerful! And it is a real pity that there aren't more parsers already using it - because it allows us to add/extend/change almost any aspect of the query parsing. So, if we combine ANTLR + queryparser.flexible, we can get very powerful framework for building almost any query language one can think of. And I hope this extension can become useful. The details: - every new query syntax is written in EBNF, it lives in separate files (and can be tested/developed independently - using 'gunit') - ANTLR parser generates parsing code (and it can generate parsers in several languages, the main target is Java, but it can also do Python - which may be interesting for pylucene) - the parser generates AST (abstract syntax tree) which is consumed by a 'pipeline' of processors, users can easily modify this pipeline to add a desired functionality - the new parser contains a few (very important) debugging functions; it can print results of every stage of the build, generate AST's as graphical charts; ant targets help to build/test/debug grammars - I've tried to reuse the existing queryparser.flexible components as much as possible, only adding new processors when necessary Assumptions about the grammar: - every grammar must have one top parse rule called 'mainQ' - parsers must generate AST (Abstract Syntax Tree) The structure of the AST is left open, there are components which make assumptions about the shape of the AST (ie. that MODIFIER is parent of a a FIELD) however users are free to choose/write different processors with different assumptions about the AST shape. More documentation on how to use the parser can be seen here: http://29min.wordpress.com/category/antlrqueryparser/ The parser has been created more than one year back and is used in production (http://labs.adsabs.harvard.edu/adsabs/). A different dialects of query languages (with proximity operatos, functions, special logic etc) - can be seen here: https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs https://github.com/romanchyla/montysolr/tree/master/contrib/invenio -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4679) LowercaseExpandedTermsQueryNodeProcessor changes regex queries
[ https://issues.apache.org/jira/browse/LUCENE-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Chyla updated LUCENE-4679: Description: This is really a very silly request, but could the lowercase processor 'abstain' from changing regex queries? For example, W should stay uppercase, but it is lowercased. was: This is really a very silly request, but could the lowercase processor 'abstain' from changing regex queries? For example, W should stay uppercase, but it will be lowercased. > LowercaseExpandedTermsQueryNodeProcessor changes regex queries > -- > > Key: LUCENE-4679 > URL: https://issues.apache.org/jira/browse/LUCENE-4679 > Project: Lucene - Core > Issue Type: Wish >Reporter: Roman Chyla >Priority: Trivial > Attachments: LUCENE-4679.patch > > > This is really a very silly request, but could the lowercase processor > 'abstain' from changing regex queries? For example, W should stay > uppercase, but it is lowercased. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4679) LowercaseExpandedTermsQueryNodeProcessor changes regex queries
[ https://issues.apache.org/jira/browse/LUCENE-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Chyla updated LUCENE-4679: Description: This is really a very silly request, but could the lowercase processor 'abstain' from changing regex queries? For example, W should stay uppercase, but it will be lowercased. was: This is really a very silly request, but could the lowercase processor 'abstain' from changing regex queries? For example, \\W should stay uppercase, but it will be lowercased. > LowercaseExpandedTermsQueryNodeProcessor changes regex queries > -- > > Key: LUCENE-4679 > URL: https://issues.apache.org/jira/browse/LUCENE-4679 > Project: Lucene - Core > Issue Type: Wish >Reporter: Roman Chyla >Priority: Trivial > Attachments: LUCENE-4679.patch > > > This is really a very silly request, but could the lowercase processor > 'abstain' from changing regex queries? For example, W should stay > uppercase, but it will be lowercased. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4679) LowercaseExpandedTermsQueryNodeProcessor changes regex queries
[ https://issues.apache.org/jira/browse/LUCENE-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Chyla updated LUCENE-4679: Attachment: LUCENE-4679.patch > LowercaseExpandedTermsQueryNodeProcessor changes regex queries > -- > > Key: LUCENE-4679 > URL: https://issues.apache.org/jira/browse/LUCENE-4679 > Project: Lucene - Core > Issue Type: Wish >Reporter: Roman Chyla >Priority: Trivial > Attachments: LUCENE-4679.patch > > > This is really a very silly request, but could the lowercase processor > 'abstain' from changing regex queries? For example, \\W should stay > uppercase, but it will be lowercased. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-4679) LowercaseExpandedTermsQueryNodeProcessor changes regex queries
Roman Chyla created LUCENE-4679: --- Summary: LowercaseExpandedTermsQueryNodeProcessor changes regex queries Key: LUCENE-4679 URL: https://issues.apache.org/jira/browse/LUCENE-4679 Project: Lucene - Core Issue Type: Wish Reporter: Roman Chyla Priority: Trivial This is really a very silly request, but could the lowercase processor 'abstain' from changing regex queries? For example, \\W should stay uppercase, but it will be lowercased. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4499) Multi-word synonym filter (synonym expansion)
[ https://issues.apache.org/jira/browse/LUCENE-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Chyla updated LUCENE-4499: Attachment: LUCENE-4499.patch A new patch, as the old version was extending wrong class (which cause web tests to fail) > Multi-word synonym filter (synonym expansion) > - > > Key: LUCENE-4499 > URL: https://issues.apache.org/jira/browse/LUCENE-4499 > Project: Lucene - Core > Issue Type: Improvement > Components: core/other >Affects Versions: 4.1, 5.0 >Reporter: Roman Chyla >Priority: Minor > Labels: analysis, multi-word, synonyms > Fix For: 5.0 > > Attachments: LUCENE-4499.patch, LUCENE-4499.patch > > > I apologize for bringing the multi-token synonym expansion up again. There is > an old, unresolved issue at LUCENE-1622 [1] > While solving the problem for our needs [2], I discovered that the current > SolrSynonym parser (and the wonderful FTS) have almost everything to > satisfactorily handle both the query and index time synonym expansion. It > seems that people often need to use the synonym filter *slightly* differently > at indexing and query time. > In our case, we must do different things during indexing and querying. > Example sentence: Mirrors of the Hubble space telescope pointed at XA5 > This is what we need (comma marks position bump): > indexing: mirrors,hubble|hubble space > telescope|hst,space,telescope,pointed,xa5|astroobject#5 > querying: +mirrors +(hubble space telescope | hst) +pointed > +(xa5|astroboject#5) > This translated to following needs: > indexing time: > single-token synonyms => return only synonyms > multi-token synonyms => return original tokens *AND* the synonyms > query time: > single-token: return only synonyms (but preserve case) > multi-token: return only synonyms > > We need the original tokens for the proximity queries, if we indexed 'hubble > space telescope' > as one token, we cannot search for 'hubble NEAR telescope' > You may (not) be surprised, but Lucene already supports ALL of these > requirements. The patch is an attempt to state the problem differently. I am > not sure if it is the best option, however it works perfectly for our needs > and it seems it could work for general public too. Especially if the > SynonymFilterFactory had a preconfigured sets of SynonymMapBuilders - and > people would just choose what situation they use. Please look at the unittest. > links: > [1] https://issues.apache.org/jira/browse/LUCENE-1622 > [2] http://labs.adsabs.harvard.edu/trac/ads-invenio/ticket/158 > [3] seems to have similar request: > http://lucene.472066.n3.nabble.com/Proposal-Full-support-for-multi-word-synonyms-at-query-time-td4000522.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4499) Multi-word synonym filter (synonym expansion)
[ https://issues.apache.org/jira/browse/LUCENE-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507440#comment-13507440 ] Roman Chyla commented on LUCENE-4499: - Hi Nolan, your case seems to confirm a need for some solution. You have decided to make a seaprate query parser, I have put the expanding logic into a query parser as well. See this for the working example: https://github.com/romanchyla/montysolr/blob/master/contrib/adsabs/src/test/org/apache/solr/analysis/TestAdsabsTypeFulltextParsing.java And its config https://github.com/romanchyla/montysolr/blob/master/contrib/examples/adsabs/solr/collection1/conf/schema.xml#L325 I see two added benefits (besides not needing a query parser plugin - in our case, it must be plugged into our qparser): 1. you can use the filter at index/query time inside a standard query parser 2. special configuration for synonym expansion (for example, we have found it very useful to be able to search for multi-tokens in case-insensitive manner, but recognize single tokens only case-sensitively; or expand with multi-token synonyms only for multi-word originals and output also the original words, otherwise eat them (replace them)) Nice blog post, I wish I could write as instructively as well :) > Multi-word synonym filter (synonym expansion) > - > > Key: LUCENE-4499 > URL: https://issues.apache.org/jira/browse/LUCENE-4499 > Project: Lucene - Core > Issue Type: Improvement > Components: core/other >Affects Versions: 4.1, 5.0 >Reporter: Roman Chyla >Priority: Minor > Labels: analysis, multi-word, synonyms > Fix For: 5.0 > > Attachments: LUCENE-4499.patch > > > I apologize for bringing the multi-token synonym expansion up again. There is > an old, unresolved issue at LUCENE-1622 [1] > While solving the problem for our needs [2], I discovered that the current > SolrSynonym parser (and the wonderful FTS) have almost everything to > satisfactorily handle both the query and index time synonym expansion. It > seems that people often need to use the synonym filter *slightly* differently > at indexing and query time. > In our case, we must do different things during indexing and querying. > Example sentence: Mirrors of the Hubble space telescope pointed at XA5 > This is what we need (comma marks position bump): > indexing: mirrors,hubble|hubble space > telescope|hst,space,telescope,pointed,xa5|astroobject#5 > querying: +mirrors +(hubble space telescope | hst) +pointed > +(xa5|astroboject#5) > This translated to following needs: > indexing time: > single-token synonyms => return only synonyms > multi-token synonyms => return original tokens *AND* the synonyms > query time: > single-token: return only synonyms (but preserve case) > multi-token: return only synonyms > > We need the original tokens for the proximity queries, if we indexed 'hubble > space telescope' > as one token, we cannot search for 'hubble NEAR telescope' > You may (not) be surprised, but Lucene already supports ALL of these > requirements. The patch is an attempt to state the problem differently. I am > not sure if it is the best option, however it works perfectly for our needs > and it seems it could work for general public too. Especially if the > SynonymFilterFactory had a preconfigured sets of SynonymMapBuilders - and > people would just choose what situation they use. Please look at the unittest. > links: > [1] https://issues.apache.org/jira/browse/LUCENE-1622 > [2] http://labs.adsabs.harvard.edu/trac/ads-invenio/ticket/158 > [3] seems to have similar request: > http://lucene.472066.n3.nabble.com/Proposal-Full-support-for-multi-word-synonyms-at-query-time-td4000522.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org