Re: is SearchComponent the correct way?
@Override
public void collect(int doc) throws IOException {
docNumsByIds.put(context.reader().doc(doc).getField("id"), doc);
// don't call delegate
}
@Override
protected void doSetNextReader(LeafReaderContext context) throws
IOException {
flush(); // strictly before the next segment
super.doSetNextReader(context)
}
void flush(){
Set allowedIDs = callRedisToRetainAllowed(docNumsByIds.keys())
docNumsByIds.keys().retainAll(allowedIDs )
for(int docNum:docNumsByIds.values()) {
delegate.collect(docNum);
}
}
public void finish() throws IOException {
flush();
super.finish()
}
On Thu, Nov 29, 2018 at 2:36 PM John Thorhauer
wrote:
> So my understanding is that the DelegatingCollector.collect() method has
> access to a single doc. At that point I must choose to either call
> super.collect() or not. So this is the point at which I have to check
> redis for security data for a single doc and determine if this doc should
> be allowed as part of the result set or not. So it seems that I have to
> check my redis cache one doc at a time since I am only provided one doc in
> the collect() method and I must determine at this point if I should call
> the super.collect() or not.
>
> I would like to find an option where I can get all the docs in the
> postfilter and run a single query to redis with all of the docs at once to
> get a single answer back from redis and then determine, based on the redis
> response, which of the docs should be allowed to pass thru my postfilter.
>
>
>
>
> On Fri, Nov 16, 2018 at 4:30 PM Mikhail Khludnev wrote:
>
> > On Tue, Nov 13, 2018 at 6:36 AM John Thorhauer
> > wrote:
> >
> > > Mikhail,
> > >
> > > Where do I implement the buffering? I can not do it in then collect()
> > > method.
> >
> > Please clarify why exactly? Notice my statement about one segment only.
> >
> >
> > > I can not see how I can get access to what I need in the finish()
> > > method.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Nov 6, 2018 at 12:44 PM Mikhail Khludnev
> > wrote:
> > >
> > > > Not really. It expect to work segment by segment. So it can buffer
> all
> > > doc
> > > > from one segment, hit redis and push all results into delegating
> > > collector.
> > > >
> > > > On Tue, Nov 6, 2018 at 8:29 PM John Thorhauer <
> jthorha...@yakabod.com>
> > > > wrote:
> > > >
> > > > > Mikhail,
> > > > >
> > > > > Thanks for the suggestion. After looking over the PostFilter
> > interface
> > > > and
> > > > > the DelegatingCollector, it appears that this would require me to
> > query
> > > > my
> > > > > outside datastore (redis) for security information once for each
> > > > document.
> > > > > This would be a big performance issue. I would like to be able to
> > > > iterate
> > > > > through the documents, gathering all the critical ID's and then
> send
> > a
> > > > > single query to redis, getting back my security related data, and
> > then
> > > > > iterate through the documents, pulling out the ones that the user
> > > should
> > > > > not see.
> > > > >
> > > > > Is this possible?
> > > > >
> > > > > Thanks again for your help!
> > > > > John
> > > > >
> > > > >
> > > > > On Tue, Nov 6, 2018 at 6:24 AM John Thorhauer <
> > jthorha...@yakabod.com>
> > > > > wrote:
> > > > >
> > > > > > We have a need to check the results of a search against a set of
> > > > security
> > > > > > lists that are maintained in a redis cache. I need to be able to
> > > take
> > > > > each
> > > > > > document that is returned for a search and check the redis cache
> to
> > > see
> > > > > if
> > > > > > the document should be displayed or not.
> > > > > >
> > > > > > I am attempting to do this by creating a SearchComponent. I am
> > able
> > > to
> > > > > > iterate thru the results and identify the items I want to remove
> > from
> > > > the
> > > > > > results but I am not sure how to proceed in removing them.
> > > > > >
> > > > > > Is SearchComponent the best way to do this? If so, any thoughts
> on
> > > how
> > > > > to
> > > > > > proceed?
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > John Thorhauer
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > John Thorhauer
> > > > > Vice President, Software Development
> > > > > Yakabod, Inc.
> > > > > Cell: 240-818-9050
> > > > > Office: 301-662-4554 x2105
> > > > >
> > > >
> > > >
> > > > --
> > > > Sincerely yours
> > > > Mikhail Khludnev
> > > >
> > >
> > >
> > > --
> > > John Thorhauer
> > > Vice President, Software Development
> > > Yakabod, Inc.
> > > Cell: 240-818-9050
> > > Office: 301-662-4554 x2105
> > >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>
--
Sincerely yours
Mikhail Khludnev
Re: is SearchComponent the correct way?
So my understanding is that the DelegatingCollector.collect() method has access to a single doc. At that point I must choose to either call super.collect() or not. So this is the point at which I have to check redis for security data for a single doc and determine if this doc should be allowed as part of the result set or not. So it seems that I have to check my redis cache one doc at a time since I am only provided one doc in the collect() method and I must determine at this point if I should call the super.collect() or not. I would like to find an option where I can get all the docs in the postfilter and run a single query to redis with all of the docs at once to get a single answer back from redis and then determine, based on the redis response, which of the docs should be allowed to pass thru my postfilter. On Fri, Nov 16, 2018 at 4:30 PM Mikhail Khludnev wrote: > On Tue, Nov 13, 2018 at 6:36 AM John Thorhauer > wrote: > > > Mikhail, > > > > Where do I implement the buffering? I can not do it in then collect() > > method. > > Please clarify why exactly? Notice my statement about one segment only. > > > > I can not see how I can get access to what I need in the finish() > > method. > > > > Thanks, > > John > > > > On Tue, Nov 6, 2018 at 12:44 PM Mikhail Khludnev > wrote: > > > > > Not really. It expect to work segment by segment. So it can buffer all > > doc > > > from one segment, hit redis and push all results into delegating > > collector. > > > > > > On Tue, Nov 6, 2018 at 8:29 PM John Thorhauer > > > wrote: > > > > > > > Mikhail, > > > > > > > > Thanks for the suggestion. After looking over the PostFilter > interface > > > and > > > > the DelegatingCollector, it appears that this would require me to > query > > > my > > > > outside datastore (redis) for security information once for each > > > document. > > > > This would be a big performance issue. I would like to be able to > > > iterate > > > > through the documents, gathering all the critical ID's and then send > a > > > > single query to redis, getting back my security related data, and > then > > > > iterate through the documents, pulling out the ones that the user > > should > > > > not see. > > > > > > > > Is this possible? > > > > > > > > Thanks again for your help! > > > > John > > > > > > > > > > > > On Tue, Nov 6, 2018 at 6:24 AM John Thorhauer < > jthorha...@yakabod.com> > > > > wrote: > > > > > > > > > We have a need to check the results of a search against a set of > > > security > > > > > lists that are maintained in a redis cache. I need to be able to > > take > > > > each > > > > > document that is returned for a search and check the redis cache to > > see > > > > if > > > > > the document should be displayed or not. > > > > > > > > > > I am attempting to do this by creating a SearchComponent. I am > able > > to > > > > > iterate thru the results and identify the items I want to remove > from > > > the > > > > > results but I am not sure how to proceed in removing them. > > > > > > > > > > Is SearchComponent the best way to do this? If so, any thoughts on > > how > > > > to > > > > > proceed? > > > > > > > > > > > > > > > Thanks, > > > > > John Thorhauer > > > > > > > > > > > > > > > > > > -- > > > > John Thorhauer > > > > Vice President, Software Development > > > > Yakabod, Inc. > > > > Cell: 240-818-9050 > > > > Office: 301-662-4554 x2105 > > > > > > > > > > > > > -- > > > Sincerely yours > > > Mikhail Khludnev > > > > > > > > > -- > > John Thorhauer > > Vice President, Software Development > > Yakabod, Inc. > > Cell: 240-818-9050 > > Office: 301-662-4554 x2105 > > > > > -- > Sincerely yours > Mikhail Khludnev >
Re: is SearchComponent the correct way?
On Tue, Nov 13, 2018 at 6:36 AM John Thorhauer wrote: > Mikhail, > > Where do I implement the buffering? I can not do it in then collect() > method. Please clarify why exactly? Notice my statement about one segment only. > I can not see how I can get access to what I need in the finish() > method. > > Thanks, > John > > On Tue, Nov 6, 2018 at 12:44 PM Mikhail Khludnev wrote: > > > Not really. It expect to work segment by segment. So it can buffer all > doc > > from one segment, hit redis and push all results into delegating > collector. > > > > On Tue, Nov 6, 2018 at 8:29 PM John Thorhauer > > wrote: > > > > > Mikhail, > > > > > > Thanks for the suggestion. After looking over the PostFilter interface > > and > > > the DelegatingCollector, it appears that this would require me to query > > my > > > outside datastore (redis) for security information once for each > > document. > > > This would be a big performance issue. I would like to be able to > > iterate > > > through the documents, gathering all the critical ID's and then send a > > > single query to redis, getting back my security related data, and then > > > iterate through the documents, pulling out the ones that the user > should > > > not see. > > > > > > Is this possible? > > > > > > Thanks again for your help! > > > John > > > > > > > > > On Tue, Nov 6, 2018 at 6:24 AM John Thorhauer > > > wrote: > > > > > > > We have a need to check the results of a search against a set of > > security > > > > lists that are maintained in a redis cache. I need to be able to > take > > > each > > > > document that is returned for a search and check the redis cache to > see > > > if > > > > the document should be displayed or not. > > > > > > > > I am attempting to do this by creating a SearchComponent. I am able > to > > > > iterate thru the results and identify the items I want to remove from > > the > > > > results but I am not sure how to proceed in removing them. > > > > > > > > Is SearchComponent the best way to do this? If so, any thoughts on > how > > > to > > > > proceed? > > > > > > > > > > > > Thanks, > > > > John Thorhauer > > > > > > > > > > > > > > -- > > > John Thorhauer > > > Vice President, Software Development > > > Yakabod, Inc. > > > Cell: 240-818-9050 > > > Office: 301-662-4554 x2105 > > > > > > > > > -- > > Sincerely yours > > Mikhail Khludnev > > > > > -- > John Thorhauer > Vice President, Software Development > Yakabod, Inc. > Cell: 240-818-9050 > Office: 301-662-4554 x2105 > -- Sincerely yours Mikhail Khludnev
Re: is SearchComponent the correct way?
Mikhail, Where do I implement the buffering? I can not do it in then collect() method. I can not see how I can get access to what I need in the finish() method. Thanks, John On Tue, Nov 6, 2018 at 12:44 PM Mikhail Khludnev wrote: > Not really. It expect to work segment by segment. So it can buffer all doc > from one segment, hit redis and push all results into delegating collector. > > On Tue, Nov 6, 2018 at 8:29 PM John Thorhauer > wrote: > > > Mikhail, > > > > Thanks for the suggestion. After looking over the PostFilter interface > and > > the DelegatingCollector, it appears that this would require me to query > my > > outside datastore (redis) for security information once for each > document. > > This would be a big performance issue. I would like to be able to > iterate > > through the documents, gathering all the critical ID's and then send a > > single query to redis, getting back my security related data, and then > > iterate through the documents, pulling out the ones that the user should > > not see. > > > > Is this possible? > > > > Thanks again for your help! > > John > > > > > > On Tue, Nov 6, 2018 at 6:24 AM John Thorhauer > > wrote: > > > > > We have a need to check the results of a search against a set of > security > > > lists that are maintained in a redis cache. I need to be able to take > > each > > > document that is returned for a search and check the redis cache to see > > if > > > the document should be displayed or not. > > > > > > I am attempting to do this by creating a SearchComponent. I am able to > > > iterate thru the results and identify the items I want to remove from > the > > > results but I am not sure how to proceed in removing them. > > > > > > Is SearchComponent the best way to do this? If so, any thoughts on how > > to > > > proceed? > > > > > > > > > Thanks, > > > John Thorhauer > > > > > > > > > > -- > > John Thorhauer > > Vice President, Software Development > > Yakabod, Inc. > > Cell: 240-818-9050 > > Office: 301-662-4554 x2105 > > > > > -- > Sincerely yours > Mikhail Khludnev > -- John Thorhauer Vice President, Software Development Yakabod, Inc. Cell: 240-818-9050 Office: 301-662-4554 x2105
Re: is SearchComponent the correct way?
Not really. It expect to work segment by segment. So it can buffer all doc from one segment, hit redis and push all results into delegating collector. On Tue, Nov 6, 2018 at 8:29 PM John Thorhauer wrote: > Mikhail, > > Thanks for the suggestion. After looking over the PostFilter interface and > the DelegatingCollector, it appears that this would require me to query my > outside datastore (redis) for security information once for each document. > This would be a big performance issue. I would like to be able to iterate > through the documents, gathering all the critical ID's and then send a > single query to redis, getting back my security related data, and then > iterate through the documents, pulling out the ones that the user should > not see. > > Is this possible? > > Thanks again for your help! > John > > > On Tue, Nov 6, 2018 at 6:24 AM John Thorhauer > wrote: > > > We have a need to check the results of a search against a set of security > > lists that are maintained in a redis cache. I need to be able to take > each > > document that is returned for a search and check the redis cache to see > if > > the document should be displayed or not. > > > > I am attempting to do this by creating a SearchComponent. I am able to > > iterate thru the results and identify the items I want to remove from the > > results but I am not sure how to proceed in removing them. > > > > Is SearchComponent the best way to do this? If so, any thoughts on how > to > > proceed? > > > > > > Thanks, > > John Thorhauer > > > > > > -- > John Thorhauer > Vice President, Software Development > Yakabod, Inc. > Cell: 240-818-9050 > Office: 301-662-4554 x2105 > -- Sincerely yours Mikhail Khludnev
Re: is SearchComponent the correct way?
Mikhail, Thanks for the suggestion. After looking over the PostFilter interface and the DelegatingCollector, it appears that this would require me to query my outside datastore (redis) for security information once for each document. This would be a big performance issue. I would like to be able to iterate through the documents, gathering all the critical ID's and then send a single query to redis, getting back my security related data, and then iterate through the documents, pulling out the ones that the user should not see. Is this possible? Thanks again for your help! John On Tue, Nov 6, 2018 at 6:24 AM John Thorhauer wrote: > We have a need to check the results of a search against a set of security > lists that are maintained in a redis cache. I need to be able to take each > document that is returned for a search and check the redis cache to see if > the document should be displayed or not. > > I am attempting to do this by creating a SearchComponent. I am able to > iterate thru the results and identify the items I want to remove from the > results but I am not sure how to proceed in removing them. > > Is SearchComponent the best way to do this? If so, any thoughts on how to > proceed? > > > Thanks, > John Thorhauer > > -- John Thorhauer Vice President, Software Development Yakabod, Inc. Cell: 240-818-9050 Office: 301-662-4554 x2105
Re: is SearchComponent the correct way?
It should be postfilter https://lucidworks.com/2017/11/27/caching-and-filters-and-post-filters/, I believe. On Tue, Nov 6, 2018 at 2:24 PM John Thorhauer wrote: > We have a need to check the results of a search against a set of security > lists that are maintained in a redis cache. I need to be able to take each > document that is returned for a search and check the redis cache to see if > the document should be displayed or not. > > I am attempting to do this by creating a SearchComponent. I am able to > iterate thru the results and identify the items I want to remove from the > results but I am not sure how to proceed in removing them. > > Is SearchComponent the best way to do this? If so, any thoughts on how to > proceed? > > > Thanks, > John Thorhauer > -- Sincerely yours Mikhail Khludnev
