[ https://issues.apache.org/jira/browse/LUCENE-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113281#comment-13113281 ]
Karl Wright commented on LUCENE-3450: ------------------------------------- Apologies for not finishing this ticket promptly, but minutes after creating it I lost internet and phone service for seven hours. Some notes: (1) The filter-based plugin passes the tests, the query-based one fails two out of the four tests. I've been concentrating initially on the failure that comes from ManifoldCFSecurityFilterTest.java line 100. Here's the error: {code} [junit] ------------- Standard Error ----------------- [junit] +((-allow_token_share:* -deny_token_share:*) allow_token_share:token1 -deny_token_share:token1) +((-allow_token_document:* -deny_token_document:*) allow_token_document:token1 -deny_token_document:token1) [junit] 22/09/2011 08:26:50 ? org.apache.solr.SolrTestCaseJ4 assertQ [junit] SEVERE: REQUEST FAILED: xpath=//*[@numFound='3'] [junit] xml response was: <?xml version="1.0" encoding="UTF-8"?> [junit] <response> [junit] <lst name="responseHeader"><int name="status">0</int><int name="QTime">116</int><lst name="params"><str name="echoParams">all</str><str name="fl">id</str><str name="q">*:*</str><str name="qt">/mcf</str><str name="UserTokens">token1</str><str name="mcf">true</str></lst></lst><result name="response" numFound="0" start="0"></result> [junit] </response> {code} This output includes the actual query I am generating for this test, which looks correct to me. (2) The test indexes 5 documents into four access token fields. These documents are summarized below. {code} // | share | document // |--------------|-------------- // | allow | deny | allow | deny // ------------+-------+------+-------+------ // da12 | | | 1, 2 | // ------------+-------+------+-------+------ // da13-dd3 | | | 1,3 | 3 // ------------+-------+------+-------+------ // sa123-sd13 | 1,2,3 | 1, 3 | | // ------------+-------+------+-------+------ // sa3-sd1-da23| 3 | 1 | 2,3 | // ------------+-------+------+-------+------ // notoken | | | | // ------------+-------+------+-------+------ {code} (2) Initially the query-based plugin directly corresponded to the filter based one. Still failed in exactly the same way. Here's the diff between the filter-based one and the initial query-based one: {code} Index: src/java/org/apache/solr/mcf/ManifoldCFSecurityFilter.java =================================================================== --- src/java/org/apache/solr/mcf/ManifoldCFSecurityFilter.java (revision 1173895) +++ src/java/org/apache/solr/mcf/ManifoldCFSecurityFilter.java (working copy) @@ -150,7 +150,8 @@ userAccessTokens = getAccessTokens(authenticatedUserName); } - BooleanFilter bf = new BooleanFilter(); + BooleanQuery bq = new BooleanQuery(); + //bf.setMaxClauseCount(100000); if (userAccessTokens.size() == 0) { @@ -159,28 +160,26 @@ // (fieldAllowShare is empty AND fieldDenyShare is empty AND fieldAllowDocument is empty AND fieldDenyDocument is empty) // We're trying to map to: -(fieldAllowShare:*) , which should be pretty efficient in Solr because it is negated. If this turns out not to be so, then we should // have the SolrConnector inject a special token into these fields when they otherwise would be empty, and we can trivially match on that token. - bf.add(new FilterClause(new QueryWrapperFilter(new WildcardQuery(new Term(fieldAllowShare,"*"))),BooleanClause.Occur.MUST_NOT)); - bf.add(new FilterClause(new QueryWrapperFilter(new WildcardQuery(new Term(fieldDenyShare,"*"))),BooleanClause.Occur.MUST_NOT)); - bf.add(new FilterClause(new QueryWrapperFilter(new WildcardQuery(new Term(fieldAllowDocument,"*"))),BooleanClause.Occur.MUST_NOT)); - bf.add(new FilterClause(new QueryWrapperFilter(new WildcardQuery(new Term(fieldDenyDocument,"*"))),BooleanClause.Occur.MUST_NOT)); + bq.add(new WildcardQuery(new Term(fieldAllowShare,"*")),BooleanClause.Occur.MUST_NOT); + bq.add(new WildcardQuery(new Term(fieldDenyShare,"*")),BooleanClause.Occur.MUST_NOT); + bq.add(new WildcardQuery(new Term(fieldAllowDocument,"*")),BooleanClause.Occur.MUST_NOT); + bq.add(new WildcardQuery(new Term(fieldDenyDocument,"*")),BooleanClause.Occur.MUST_NOT); } else { // Extend the query appropriately for each user access token. - bf.add(new FilterClause(calculateCompleteSubfilter(fieldAllowShare,fieldDenyShare,userAccessTokens),BooleanClause.Occur.MUST)); - bf.add(new FilterClause(calculateCompleteSubfilter(fieldAllowDocument,fieldDenyDocument,userAccessTokens),BooleanClause.Occur.MUST)); + bq.add(calculateCompleteSubquery(fieldAllowShare,fieldDenyShare,userAccessTokens),BooleanClause.Occur.MUST); + bq.add(calculateCompleteSubquery(fieldAllowDocument,fieldDenyDocument,userAccessTokens),BooleanClause.Occur.MUST); } // Concatenate with the user's original query. - //FilteredQuery query = new FilteredQuery(rb.getQuery(),bf); - //rb.setQuery(query); List<Query> list = rb.getFilters(); if (list == null) { list = new ArrayList<Query>(); rb.setFilters(list); } - list.add(new ConstantScoreQuery(bf)); + list.add(new ConstantScoreQuery(bq)); } @Override @@ -193,28 +192,27 @@ * ((fieldAllowShare is empty AND fieldDenyShare is empty) OR fieldAllowShare HAS token1 OR fieldAllowShare HAS token2 ...) * AND fieldDenyShare DOESN'T_HAVE token1 AND fieldDenyShare DOESN'T_HAVE token2 ... */ - protected Filter calculateCompleteSubfilter(String allowField, String denyField, List<String> userAccessTokens) + protected Query calculateCompleteSubquery(String allowField, String denyField, List<String> userAccessTokens) { - BooleanFilter bf = new BooleanFilter(); + BooleanQuery bq = new BooleanQuery(); + bq.setMaxClauseCount(1000000); // Add a clause for each token. This will be added directly to the main filter (as a deny test), as well as to an OR's subclause (as an allow test). - BooleanFilter orFilter = new BooleanFilter(); + BooleanQuery orQuery = new BooleanQuery(); + orQuery.setMaxClauseCount(1000000); + // Add the empty-acl case - BooleanFilter subUnprotectedClause = new BooleanFilter(); - subUnprotectedClause.add(new FilterClause(new QueryWrapperFilter(new WildcardQuery(new Term(allowField,"*"))),BooleanClause.Occur.MUST_NOT)); - subUnprotectedClause.add(new FilterClause(new QueryWrapperFilter(new WildcardQuery(new Term(denyField,"*"))),BooleanClause.Occur.MUST_NOT)); - orFilter.add(new FilterClause(subUnprotectedClause,BooleanClause.Occur.SHOULD)); + BooleanQuery subUnprotectedClause = new BooleanQuery(); + subUnprotectedClause.add(new WildcardQuery(new Term(allowField,"*")),BooleanClause.Occur.MUST_NOT); + subUnprotectedClause.add(new WildcardQuery(new Term(denyField,"*")),BooleanClause.Occur.MUST_NOT); + orQuery.add(subUnprotectedClause,BooleanClause.Occur.SHOULD); for (String accessToken : userAccessTokens) { - TermsFilter tf = new TermsFilter(); - tf.addTerm(new Term(allowField,accessToken)); - orFilter.add(new FilterClause(tf,BooleanClause.Occur.SHOULD)); - tf = new TermsFilter(); - tf.addTerm(new Term(denyField,accessToken)); - bf.add(new FilterClause(tf,BooleanClause.Occur.MUST_NOT)); + orQuery.add(new TermQuery(new Term(allowField,accessToken)),BooleanClause.Occur.SHOULD); + bq.add(new TermQuery(new Term(denyField,accessToken)),BooleanClause.Occur.MUST_NOT); } - bf.add(new FilterClause(orFilter,BooleanClause.Occur.MUST)); - return bf; + bq.add(orQuery,BooleanClause.Occur.MUST); + return bq; } {code} After this I experimented for a bit, flattening out one level of BooleanQuery, commenting out the ConstantScoreQuery wrapper, and adding output to stderr of the generated query. None of this fixed things, although the test behavior did change somewhat, which was another worrisome sign. The failure at ManifoldCFSecurityFilterTest.java line 100 still occurred just the same, however. This is as far as I've gotten so far. If you need a more condensed test case, I'm happy to try and produce one, but I am unfamiliar with the Lucene-side testing infrastructure so I'd need to be pointed at an example to emulate. > BooleanQuery seems broken on trunk > ---------------------------------- > > Key: LUCENE-3450 > URL: https://issues.apache.org/jira/browse/LUCENE-3450 > Project: Lucene - Java > Issue Type: Bug > Reporter: Karl Wright > Attachments: filter.diff, query.diff > > > Converting a Solr SearchComponent from using filters to using equivalent > queries causes its test cases to fail. The reasons are unclear, and I've > tried several things to isolate the problem, but with no luck so far. > The failure manifests itself by the constructed query returning no documents > whatsoever, at least within the Solr context it's being used in. The > equivalent filter version of the component works properly (as you can see by > the passed test). > To run the test: > cd solr/contrib/mcf > ant test > I will attach both filter and query versions of the code, as patches. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org