Hello! The problem here is that Solr is a multi-year effort by a lot of people. We can't match that.
Maybe we could integrate with Solr/Solr Cloud instead, by feeding our cache information into their storage for indexing and relying on their own mechanisms for distributed IR sorting? Regards, -- Ilya Kasnacheev вт, 26 нояб. 2019 г. в 13:59, Zhenya Stanilovsky <arzamas...@mail.ru.invalid >: > > Ilya Kasnacheev, what a problem in Solr with Ignite functionality ? > > thanks ! > > >Вторник, 26 ноября 2019, 13:50 +03:00 от Ilya Kasnacheev < > ilya.kasnach...@gmail.com>: > > > >Hello! > > > >I have a hunch that we are trying to build Apache Solr (or Solr Cloud) > into > >Apache Ignite. I think that's a lot of effort that is not very justified. > > > >I don't think we should try to implement sorting in Apache Ignite, because > >it is a lot of work, and a lot of code in our code base which we don't > >really want. > > > >Regards, > >-- > >Ilya Kasnacheev > > > > > >пт, 22 нояб. 2019 г. в 20:59, Yuriy Shuliga < shul...@gmail.com >: > > > >> Dear Igniters, > >> > >> The first part of TextQuery improvement - a result limit - was developed > >> and merged. > >> Now we have to develop most important functionality here - proper > sorting > >> of Lucene index response and correct reducing of them for distributed > >> queries. > >> > >> *There are two Lucene based aspects* > >> > >> 1. In case of using no sorting fields, the documents in response are > still > >> ordered by relevance. > >> Actually this is ScoreDoc.score value. > >> In order to reduce the distributed results correctly, the score should > be > >> passed with response. > >> > >> 2. When sorting by conventional fields, then Lucene should have these > >> fields properly indexed and > >> corresponding Sort object should be applied to Lucene's search call. > >> In order to mark those fields a new annotation like '@SortField' may be > >> introduced. > >> > >> *Reducing on Ignite * > >> > >> The obvious point of distributed response reduction is class > >> GridCacheDistributedQueryFuture. > >> Though, @Ivan Pavlukhin mentioned class with similar functionality: > >> ReduceIndexSorted > >> What I see here, that it is tangled with H2 related classes ( > >> org.h2.result.Row) and might not be unified with TextQuery reduction. > >> > >> Still need a support here. > >> > >> Overall, the goal of this letter is to initiate discussion on TextQuery > >> Sorting implementation and come closer to ticket creation. > >> > >> BR, > >> Yuriy Shuliha > >> > >> вт, 22 жовт. 2019 о 13:31 Andrey Mashenkov < andrey.mashen...@gmail.com > > > >> пише: > >> > >> > Hi Dmitry, Yuriy. > >> > > >> > I've found GridCacheQueryFutureAdapter has newly added AtomicInteger > >> > 'total' field and 'limit; field as primitive int. > >> > > >> > Both fields are used inside synchronized block only. > >> > So, we can make both private and downgrade AtomicInteger to primitive > >> int. > >> > > >> > Most likely, these fields can be replaced with one field. > >> > > >> > > >> > > >> > On Mon, Oct 21, 2019 at 10:01 PM Dmitriy Pavlov < dpav...@apache.org > > > >> > wrote: > >> > > >> > > Hi Andrey, > >> > > > >> > > I've checked this ticket comments, and there is a TC Bot visa (with > no > >> > > blockers). > >> > > > >> > > Do you have any concerns related to this patch? > >> > > > >> > > Sincerely, > >> > > Dmitriy Pavlov > >> > > > >> > > чт, 17 окт. 2019 г. в 16:43, Yuriy Shuliga < shul...@gmail.com >: > >> > > > >> > >> Andrey, > >> > >> > >> > >> Per you request, I created ticket > >> > >> https://issues.apache.org/jira/browse/IGNITE-12291 linked to > >> > >> > https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-12189 > >> > >> > >> > >> Could you please proceed with PR merge ? > >> > >> > >> > >> BR, > >> > >> Yuriy Shuliha > >> > >> > >> > >> ср, 9 жовт. 2019 о 12:52 Andrey Mashenkov < > andrey.mashen...@gmail.com > >> > > >> > >> пише: > >> > >> > >> > >> > Hi Yuri, > >> > >> > > >> > >> > To get access to TC Bot you should register as TeamCity user > [1], if > >> > you > >> > >> > didn't do this already. > >> > >> > Then you will be able to authorize on Ignite TC Bot page with > same > >> > >> > credentials. > >> > >> > > >> > >> > [1] https://ci.ignite.apache.org/registerUser.html > >> > >> > > >> > >> > On Fri, Oct 4, 2019 at 3:10 PM Yuriy Shuliga < shul...@gmail.com > > > >> > wrote: > >> > >> > > >> > >> >> Andrew, > >> > >> >> > >> > >> >> I have corrected PR according to your notes. Please review. > >> > >> >> What will be the next steps in order to merge in? > >> > >> >> > >> > >> >> Y. > >> > >> >> > >> > >> >> чт, 3 жовт. 2019 о 17:47 Andrey Mashenkov < > >> > andrey.mashen...@gmail.com > > >> > >> >> пише: > >> > >> >> > >> > >> >> > Yuri, > >> > >> >> > > >> > >> >> > I've done with review. > >> > >> >> > No crime found, but trivial compatibility bug. > >> > >> >> > > >> > >> >> > On Thu, Oct 3, 2019 at 3:54 PM Yuriy Shuliga < > shul...@gmail.com > > >> > >> wrote: > >> > >> >> > > >> > >> >> > > Denis, > >> > >> >> > > > >> > >> >> > > Thank you for your attention to this. > >> > >> >> > > as for now, the > >> > https://issues.apache.org/jira/browse/IGNITE-12189 > >> > >> >> > ticket > >> > >> >> > > is still pending review. > >> > >> >> > > Do we have a chance to move it forward somehow? > >> > >> >> > > > >> > >> >> > > BR, > >> > >> >> > > Yuriy Shuliha > >> > >> >> > > > >> > >> >> > > пн, 30 вер. 2019 о 23:35 Denis Magda < dma...@apache.org > > пише: > >> > >> >> > > > >> > >> >> > > > Yuriy, > >> > >> >> > > > > >> > >> >> > > > I've seen you opening a pull-request with the first > changes: > >> > >> >> > > > https://issues.apache.org/jira/browse/IGNITE-12189 > >> > >> >> > > > > >> > >> >> > > > Alex Scherbakov and Ivan are you the right guys to do the > >> > review? > >> > >> >> > > > > >> > >> >> > > > - > >> > >> >> > > > Denis > >> > >> >> > > > > >> > >> >> > > > > >> > >> >> > > > On Fri, Sep 27, 2019 at 8:48 AM Павлухин Иван < > >> > >> vololo...@gmail.com > > >> > >> >> > > wrote: > >> > >> >> > > > > >> > >> >> > > > > Yuriy, > >> > >> >> > > > > > >> > >> >> > > > > Thank you for providing details! Quite interesting. > >> > >> >> > > > > > >> > >> >> > > > > Yes, we already have support of distributed limit and > >> merging > >> > >> >> sorted > >> > >> >> > > > > subresults for SQL queries. E.g. ReduceIndexSorted and > >> > >> >> > > > > MergeStreamIterator are used for merging sorted streams. > >> > >> >> > > > > > >> > >> >> > > > > Could you please also clarify about score/relevance? Is > it > >> > >> >> provided > >> > >> >> > by > >> > >> >> > > > > Lucene engine for each query result? I am thinking how > to > >> do > >> > >> >> sorted > >> > >> >> > > > > merge properly in this case. > >> > >> >> > > > > > >> > >> >> > > > > ср, 25 сент. 2019 г. в 18:56, Yuriy Shuliga < > >> > shul...@gmail.com > >> > >> >: > >> > >> >> > > > > > > >> > >> >> > > > > > Ivan, > >> > >> >> > > > > > > >> > >> >> > > > > > Thank you for interesting question! > >> > >> >> > > > > > > >> > >> >> > > > > > Text searches (or full text searches) are mostly > >> > >> human-oriented. > >> > >> >> > And > >> > >> >> > > > the > >> > >> >> > > > > > point of user's interest is topmost part of response. > >> > >> >> > > > > > Then user can read it, evaluate and use the given > records > >> > for > >> > >> >> > further > >> > >> >> > > > > > purposes. > >> > >> >> > > > > > > >> > >> >> > > > > > Particularly in our case, we use Ignite for operations > >> with > >> > >> >> > financial > >> > >> >> > > > > data, > >> > >> >> > > > > > and there lots of text stuff like assets names, fin. > >> > >> >> instruments, > >> > >> >> > > > > companies > >> > >> >> > > > > > etc. > >> > >> >> > > > > > In order to operate with this quickly and reliably, > users > >> > >> used > >> > >> >> to > >> > >> >> > > work > >> > >> >> > > > > with > >> > >> >> > > > > > text search, type-ahead completions, suggestions. > >> > >> >> > > > > > > >> > >> >> > > > > > For this purposes we are indexing particular string > data > >> in > >> > >> >> > separate > >> > >> >> > > > > caches. > >> > >> >> > > > > > > >> > >> >> > > > > > Sorting capabilities and response size limitations are > >> very > >> > >> >> > important > >> > >> >> > > > > > there. As our API have to provide most relevant > >> information > >> > >> in > >> > >> >> view > >> > >> >> > > of > >> > >> >> > > > > > limited size. > >> > >> >> > > > > > > >> > >> >> > > > > > Now let me comment some Ignite/Lucene perspective. > >> > >> >> > > > > > Actually Ignite queries and Lucene returns > >> > >> *TopDocs.scoresDocs > >> > >> >> > > *already > >> > >> >> > > > > > sorted by *score *(relevance). So most relevant > documents > >> > >> are on > >> > >> >> > the > >> > >> >> > > > top. > >> > >> >> > > > > > And currently distributed queries responses from > >> different > >> > >> nodes > >> > >> >> > are > >> > >> >> > > > > merged > >> > >> >> > > > > > into final query cursor queue in arbitrary way. > >> > >> >> > > > > > So in fact we already have the score order ruined > here. > >> > Also > >> > >> >> Ignite > >> > >> >> > > > > > requests all possible documents from Lucene that is > >> > redundant > >> > >> >> and > >> > >> >> > not > >> > >> >> > > > > good > >> > >> >> > > > > > for performance. > >> > >> >> > > > > > > >> > >> >> > > > > > I'm implementing *limit* parameter to be part of > >> *TextQuery > >> > >> *and > >> > >> >> > have > >> > >> >> > > > to > >> > >> >> > > > > > notice that we still have to add sorting for text > queries > >> > >> >> > processing > >> > >> >> > > in > >> > >> >> > > > > > order to have applicable results. > >> > >> >> > > > > > > >> > >> >> > > > > > *Limit* parameter itself should improve the part of > >> issues > >> > >> from > >> > >> >> > > above, > >> > >> >> > > > > but > >> > >> >> > > > > > definitely, sorting by document score at least should > be > >> > >> >> > implemented > >> > >> >> > > > > along > >> > >> >> > > > > > with limit. > >> > >> >> > > > > > > >> > >> >> > > > > > This is a pretty short commentary if you still have > any > >> > >> >> questions, > >> > >> >> > > > please > >> > >> >> > > > > > ask, do not hesitate) > >> > >> >> > > > > > > >> > >> >> > > > > > BR, > >> > >> >> > > > > > Yuriy Shuliha > >> > >> >> > > > > > > >> > >> >> > > > > > чт, 19 вер. 2019 о 11:38 Павлухин Иван < > >> > vololo...@gmail.com > > >> > >> >> пише: > >> > >> >> > > > > > > >> > >> >> > > > > > > Yuriy, > >> > >> >> > > > > > > > >> > >> >> > > > > > > Greatly appreciate your interest. > >> > >> >> > > > > > > > >> > >> >> > > > > > > Could you please elaborate a little bit about > sorting? > >> > What > >> > >> >> tasks > >> > >> >> > > > does > >> > >> >> > > > > > > it help to solve and how? It would be great to > provide > >> an > >> > >> >> > example. > >> > >> >> > > > > > > > >> > >> >> > > > > > > ср, 18 сент. 2019 г. в 09:39, Alexei Scherbakov < > >> > >> >> > > > > > > alexey.scherbak...@gmail.com >: > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > Denis, > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > I like the idea of throwing an exception for > enabled > >> > text > >> > >> >> > queries > >> > >> >> > > > on > >> > >> >> > > > > > > > persistent caches. > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > Also I'm fine with proposed limit for unsorted > >> > searches. > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > Yury, please proceed with ticket creation. > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > вт, 17 сент. 2019 г., 22:06 Denis Magda < > >> > >> dma...@apache.org > >> > >> >> >: > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > > Igniters, > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > I see nothing wrong with Yury's proposal in > regards > >> > >> >> full-text > >> > >> >> > > > > search > >> > >> >> > > > > > > API > >> > >> >> > > > > > > > > evolution as long as Yury is ready to push it > >> > forward. > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > As for the in-memory mode only, it makes total > >> sense > >> > >> for > >> > >> >> > > > in-memory > >> > >> >> > > > > data > >> > >> >> > > > > > > > > grid deployments when Ignite caches data of an > >> > >> underlying > >> > >> >> DB > >> > >> >> > > like > >> > >> >> > > > > > > Postgres. > >> > >> >> > > > > > > > > As part of the changes, I would simply throw an > >> > >> exception > >> > >> >> (by > >> > >> >> > > > > default) > >> > >> >> > > > > > > if > >> > >> >> > > > > > > > > the one attempts to use text indices with the > >> native > >> > >> >> > > persistence > >> > >> >> > > > > > > enabled. > >> > >> >> > > > > > > > > If the person is ready to live with that > limitation > >> > >> that > >> > >> >> an > >> > >> >> > > > > explicit > >> > >> >> > > > > > > > > configuration change is needed to come around > the > >> > >> >> exception. > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > Thoughts? > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > - > >> > >> >> > > > > > > > > Denis > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > On Tue, Sep 17, 2019 at 7:44 AM Yuriy Shuliga < > >> > >> >> > > shul...@gmail.com > >> > >> >> > > > > > >> > >> >> > > > > > > wrote: > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > > Hello to all again, > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > Thank you for important comments and notes > given > >> > >> below! > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > Let me answer and continue the discussion. > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > (I) Overall needs in Lucene indexing > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > Alexei has referenced to > >> > >> >> > > > > > > > > > > >> https://issues.apache.org/jira/browse/IGNITE-5371 > >> > >> where > >> > >> >> > > > > > > > > > absence of index persistence was declared as > an > >> > >> >> obstacle to > >> > >> >> > > > > further > >> > >> >> > > > > > > > > > development. > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > a) This ticket is already closed as not > valid.b) > >> > >> There > >> > >> >> are > >> > >> >> > > > > definite > >> > >> >> > > > > > > needs > >> > >> >> > > > > > > > > > (and in our project as well) in just in-memory > >> > >> indexing > >> > >> >> of > >> > >> >> > > > > selected > >> > >> >> > > > > > > data. > >> > >> >> > > > > > > > > > We intend to use search capabilities for > fetching > >> > >> >> limited > >> > >> >> > > > amount > >> > >> >> > > > > of > >> > >> >> > > > > > > > > records > >> > >> >> > > > > > > > > > that should be used in type-ahead search / > >> > >> suggestions. > >> > >> >> > > > > > > > > > Not all of the data will be indexed and the > are > >> no > >> > >> need > >> > >> >> in > >> > >> >> > > > Lucene > >> > >> >> > > > > > > index > >> > >> >> > > > > > > > > to > >> > >> >> > > > > > > > > > be persistence. Hope this is a wide pattern of > >> > >> >> text-search > >> > >> >> > > > usage. > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > (II) Necessary fixes in current > implementation. > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > a) Implementation of correct *limit *(*offset* > >> > seems > >> > >> to > >> > >> >> be > >> > >> >> > > not > >> > >> >> > > > > > > required > >> > >> >> > > > > > > > > in > >> > >> >> > > > > > > > > > text-search tasks for now) > >> > >> >> > > > > > > > > > I have investigated the data flow for > distributed > >> > >> text > >> > >> >> > > queries. > >> > >> >> > > > > it > >> > >> >> > > > > > > was > >> > >> >> > > > > > > > > > simple test prefix query, like 'name'*='ene*'* > >> > >> >> > > > > > > > > > For now each server-node returns all response > >> > >> records to > >> > >> >> > the > >> > >> >> > > > > > > client-node > >> > >> >> > > > > > > > > > and it may contain ~thousands, ~hundred > thousands > >> > >> >> records. > >> > >> >> > > > > > > > > > Event if we need only first 10-100. Again, all > >> the > >> > >> >> results > >> > >> >> > > are > >> > >> >> > > > > added > >> > >> >> > > > > > > to > >> > >> >> > > > > > > > > > queue in GridCacheQueryFutureAdapter in > arbitrary > >> > >> order > >> > >> >> by > >> > >> >> > > > pages. > >> > >> >> > > > > > > > > > I did not find here any means to deliver > >> > >> deterministic > >> > >> >> > > result. > >> > >> >> > > > > > > > > > So implementing limit as part of query and > >> > >> >> > > > > (GridCacheQueryRequest) > >> > >> >> > > > > > > will > >> > >> >> > > > > > > > > not > >> > >> >> > > > > > > > > > change the nature of response but will limit > load > >> > on > >> > >> >> nodes > >> > >> >> > > and > >> > >> >> > > > > > > > > networking. > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > Can we consider to open a ticket for this? > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > (III) Further extension of Lucene API > exposition > >> to > >> > >> >> Ignite > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > a) Sorting > >> > >> >> > > > > > > > > > The solution for this could be: > >> > >> >> > > > > > > > > > - Make entities comparable > >> > >> >> > > > > > > > > > - Add custom comparator to entity > >> > >> >> > > > > > > > > > - Add annotations to mark sorted fields for > >> Lucene > >> > >> >> indexing > >> > >> >> > > > > > > > > > - Use comparators when merging responses or > >> > reducing > >> > >> to > >> > >> >> > > desired > >> > >> >> > > > > > > limit on > >> > >> >> > > > > > > > > > client node. > >> > >> >> > > > > > > > > > Will require full result set to be loaded into > >> > >> memory. > >> > >> >> > Though > >> > >> >> > > > > can be > >> > >> >> > > > > > > used > >> > >> >> > > > > > > > > > for relatively small limits. > >> > >> >> > > > > > > > > > BR, > >> > >> >> > > > > > > > > > Yuriy Shuliha > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > пт, 30 серп. 2019 о 10:37 Alexei Scherbakov < > >> > >> >> > > > > > > > > alexey.scherbak...@gmail.com > > >> > >> >> > > > > > > > > > пише: > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > > Yuriy, > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > Note what one of major blockers for text > >> queries > >> > is > >> > >> >> [1] > >> > >> >> > > which > >> > >> >> > > > > makes > >> > >> >> > > > > > > > > > lucene > >> > >> >> > > > > > > > > > > indexes unusable with persistence and main > >> reason > >> > >> for > >> > >> >> > > > > > > discontinuation. > >> > >> >> > > > > > > > > > > Probably it's should be addressed first to > make > >> > >> text > >> > >> >> > > queries > >> > >> >> > > > a > >> > >> >> > > > > > > valid > >> > >> >> > > > > > > > > > > product feature. > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > Distributed sorting and advanved querying is > >> > indeed > >> > >> >> not a > >> > >> >> > > > > trivial > >> > >> >> > > > > > > task. > >> > >> >> > > > > > > > > > > Some kind of merging must be implemented on > >> query > >> > >> >> > > originating > >> > >> >> > > > > node. > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > [1] > >> > >> https://issues.apache.org/jira/browse/IGNITE-5371 > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > чт, 29 авг. 2019 г. в 23:38, Denis Magda < > >> > >> >> > > dma...@apache.org > >> > >> >> > > > >: > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Yuriy, > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > If you are ready to take over the > full-text > >> > >> search > >> > >> >> > > indexes > >> > >> >> > > > > then > >> > >> >> > > > > > > > > please > >> > >> >> > > > > > > > > > go > >> > >> >> > > > > > > > > > > > ahead. The primary reason why the > community > >> > >> wants to > >> > >> >> > > > > discontinue > >> > >> >> > > > > > > them > >> > >> >> > > > > > > > > > > first > >> > >> >> > > > > > > > > > > > (and, probable, resurrect later) are the > >> > >> limitations > >> > >> >> > > listed > >> > >> >> > > > > by > >> > >> >> > > > > > > Andrey > >> > >> >> > > > > > > > > > and > >> > >> >> > > > > > > > > > > > minimal support from the community end. > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > - > >> > >> >> > > > > > > > > > > > Denis > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > On Thu, Aug 29, 2019 at 1:29 PM Andrey > >> > Mashenkov > >> > >> < > >> > >> >> > > > > > > > > > > > andrey.mashen...@gmail.com > > >> > >> >> > > > > > > > > > > > wrote: > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Hi Yuriy, > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Unfortunatelly, there is a plan to > >> > discontinue > >> > >> >> > > > TextQueries > >> > >> >> > > > > in > >> > >> >> > > > > > > > > Ignite > >> > >> >> > > > > > > > > > > [1]. > >> > >> >> > > > > > > > > > > > > Motivation here is text indexes are not > >> > >> >> persistent, > >> > >> >> > not > >> > >> >> > > > > > > > > transactional > >> > >> >> > > > > > > > > > > and > >> > >> >> > > > > > > > > > > > > can't be user together with SQL or > inside > >> > SQL. > >> > >> >> > > > > > > > > > > > > and there is a lack of interest from > >> > community > >> > >> >> side. > >> > >> >> > > > > > > > > > > > > You are weclome to take on these issues > and > >> > >> make > >> > >> >> > > > > TextQueries > >> > >> >> > > > > > > great. > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > 1, PageSize can't be used to limit > >> > resultset. > >> > >> >> > > > > > > > > > > > > Query results return from data node to > >> > >> client-side > >> > >> >> > > cursor > >> > >> >> > > > > in > >> > >> >> > > > > > > > > > > page-by-page > >> > >> >> > > > > > > > > > > > > manner and > >> > >> >> > > > > > > > > > > > > this parameter is designed control page > >> size. > >> > >> It > >> > >> >> is > >> > >> >> > > > > supposed > >> > >> >> > > > > > > query > >> > >> >> > > > > > > > > > > > executes > >> > >> >> > > > > > > > > > > > > lazily on server side and > >> > >> >> > > > > > > > > > > > > it is not excepted full resultset be > loaded > >> > to > >> > >> >> memory > >> > >> >> > > on > >> > >> >> > > > > server > >> > >> >> > > > > > > > > side > >> > >> >> > > > > > > > > > at > >> > >> >> > > > > > > > > > > > > once, but by pages. > >> > >> >> > > > > > > > > > > > > Do you mean you found Lucene load entire > >> > >> resultset > >> > >> >> > into > >> > >> >> > > > > memory > >> > >> >> > > > > > > > > before > >> > >> >> > > > > > > > > > > > first > >> > >> >> > > > > > > > > > > > > page is sent to client? > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > I'd think a new parameter should be > added > >> to > >> > >> limit > >> > >> >> > > > result. > >> > >> >> > > > > The > >> > >> >> > > > > > > best > >> > >> >> > > > > > > > > > > > > solution is to use query language > commands > >> > for > >> > >> >> this, > >> > >> >> > > e.g. > >> > >> >> > > > > > > > > > > "LIMIT/OFFSET" > >> > >> >> > > > > > > > > > > > in > >> > >> >> > > > > > > > > > > > > SQL. > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > This task doesn't look trivial. Query is > >> > >> >> distributed > >> > >> >> > > > > operation > >> > >> >> > > > > > > and > >> > >> >> > > > > > > > > > same > >> > >> >> > > > > > > > > > > > > user query will be executed on data > nodes > >> > >> >> > > > > > > > > > > > > and then results from all nodes should > be > >> > >> correcly > >> > >> >> > > merged > >> > >> >> > > > > > > before > >> > >> >> > > > > > > > > > being > >> > >> >> > > > > > > > > > > > > returned via client-cursor. > >> > >> >> > > > > > > > > > > > > So, LIMIT should be applied on every > node > >> and > >> > >> >> then on > >> > >> >> > > > merge > >> > >> >> > > > > > > phase. > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Also, this may be non-obviuos, limiting > >> > results > >> > >> >> make > >> > >> >> > no > >> > >> >> > > > > sence > >> > >> >> > > > > > > > > without > >> > >> >> > > > > > > > > > > > > sorting, > >> > >> >> > > > > > > > > > > > > as there is no guarantee every next > query > >> run > >> > >> will > >> > >> >> > > return > >> > >> >> > > > > same > >> > >> >> > > > > > > data > >> > >> >> > > > > > > > > > > > because > >> > >> >> > > > > > > > > > > > > of page reordeing. > >> > >> >> > > > > > > > > > > > > Basically, merge phase receive results > from > >> > >> data > >> > >> >> > nodes > >> > >> >> > > > > > > > > asynchronously > >> > >> >> > > > > > > > > > > and > >> > >> >> > > > > > > > > > > > > messages from different nodes can't be > >> > ordered. > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > 2. > >> > >> >> > > > > > > > > > > > > a. "tokenize" param name (for > >> > @QueryTextFiled) > >> > >> >> looks > >> > >> >> > > more > >> > >> >> > > > > > > verbose, > >> > >> >> > > > > > > > > > > isn't > >> > >> >> > > > > > > > > > > > > it. > >> > >> >> > > > > > > > > > > > > b,c. What about distributed query? How > >> > partial > >> > >> >> > results > >> > >> >> > > > from > >> > >> >> > > > > > > nodes > >> > >> >> > > > > > > > > > will > >> > >> >> > > > > > > > > > > be > >> > >> >> > > > > > > > > > > > > merged? > >> > >> >> > > > > > > > > > > > > Does Lucene allows to configure > comparator > >> > for > >> > >> >> data > >> > >> >> > > > > sorting? > >> > >> >> > > > > > > > > > > > > What comparator Ignite should choose to > >> sort > >> > >> >> result > >> > >> >> > on > >> > >> >> > > > > merge > >> > >> >> > > > > > > phase? > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > 3. For now Lucene engine is not > >> configurable > >> > at > >> > >> >> all. > >> > >> >> > > E.g. > >> > >> >> > > > > it is > >> > >> >> > > > > > > > > > > > impossible > >> > >> >> > > > > > > > > > > > > to configure Tokenizer. > >> > >> >> > > > > > > > > > > > > I'd think about possible ways to > configure > >> > >> engine > >> > >> >> at > >> > >> >> > > > first > >> > >> >> > > > > and > >> > >> >> > > > > > > only > >> > >> >> > > > > > > > > > > then > >> > >> >> > > > > > > > > > > > go > >> > >> >> > > > > > > > > > > > > further to discuss\implement complex > >> > features, > >> > >> >> > > > > > > > > > > > > that may depends on engine config. > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > On Thu, Aug 29, 2019 at 8:17 PM Yuriy > >> > Shuliga < > >> > >> >> > > > > > > shul...@gmail.com > > >> > >> >> > > > > > > > > > > wrote: > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Dear community, > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > By starting this chain I'd like to > open > >> > >> >> discussion > >> > >> >> > > that > >> > >> >> > > > > would > >> > >> >> > > > > > > > > come > >> > >> >> > > > > > > > > > to > >> > >> >> > > > > > > > > > > > > > contribution results in subj. area. > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Ignite has indexing capabilities, > backed > >> up > >> > >> by > >> > >> >> > > > different > >> > >> >> > > > > > > > > > mechanisms, > >> > >> >> > > > > > > > > > > > > > including Lucene. > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Currently, Lucene 7.5.0 is used (past > >> year > >> > >> >> > release). > >> > >> >> > > > > > > > > > > > > > This is a wide spread and mature > >> technology > >> > >> that > >> > >> >> > > covers > >> > >> >> > > > > text > >> > >> >> > > > > > > > > search > >> > >> >> > > > > > > > > > > > area > >> > >> >> > > > > > > > > > > > > > and beyond (e.g. spacial data > indexing). > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > My goal is to *expose more Lucene > >> > >> functionality > >> > >> >> to > >> > >> >> > > > Ignite > >> > >> >> > > > > > > > > indexing > >> > >> >> > > > > > > > > > > and > >> > >> >> > > > > > > > > > > > > > query mechanisms for text data*. > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > It's quite simple request at current > >> stage. > >> > >> It > >> > >> >> is > >> > >> >> > > > coming > >> > >> >> > > > > > > from our > >> > >> >> > > > > > > > > > > > > project's > >> > >> >> > > > > > > > > > > > > > needs, but i believe, will be useful > for > >> a > >> > >> lot > >> > >> >> more > >> > >> >> > > > > people. > >> > >> >> > > > > > > > > > > > > > Let's walk through and vote or discuss > >> > about > >> > >> >> Jira > >> > >> >> > > > > tickets for > >> > >> >> > > > > > > > > them. > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > 1.[trivial] Use > dataQuery.getPageSize() > >> > to > >> > >> >> limit > >> > >> >> > > > search > >> > >> >> > > > > > > > > response > >> > >> >> > > > > > > > > > > > items > >> > >> >> > > > > > > > > > > > > > inside GridLuceneIndex.query(). > Currently > >> > it > >> > >> is > >> > >> >> > > calling > >> > >> >> > > > > > > > > > > > > > IndexSearcher.search(query, > >> > >> >> *Integer.MAX_VALUE*) - > >> > >> >> > so > >> > >> >> > > > > > > basically > >> > >> >> > > > > > > > > all > >> > >> >> > > > > > > > > > > > > scored > >> > >> >> > > > > > > > > > > > > > matches will me returned, what we do > not > >> > >> need in > >> > >> >> > most > >> > >> >> > > > > cases. > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > 2.[simple] Add sorting. Then more > >> capable > >> > >> >> search > >> > >> >> > > call > >> > >> >> > > > > can be > >> > >> >> > > > > > > > > > > > > > executed: *IndexSearcher.search(query, > >> > count, > >> > >> >> > > > > > > > > > > > > > sort) * > >> > >> >> > > > > > > > > > > > > > Implementation steps: > >> > >> >> > > > > > > > > > > > > > a) Introduce boolean *sortField* > >> parameter > >> > in > >> > >> >> > > > > > > *@QueryTextFiled * > >> > >> >> > > > > > > > > > > > > > annotation. If > >> > >> >> > > > > > > > > > > > > > *true *the filed will be indexed but > not > >> > >> >> tokenized. > >> > >> >> > > > > Number > >> > >> >> > > > > > > types > >> > >> >> > > > > > > > > > are > >> > >> >> > > > > > > > > > > > > > preferred here. > >> > >> >> > > > > > > > > > > > > > b) Add *sort* collection to > *TextQuery* > >> > >> >> > constructor. > >> > >> >> > > It > >> > >> >> > > > > > > should > >> > >> >> > > > > > > > > > define > >> > >> >> > > > > > > > > > > > > > desired sort fields used for querying. > >> > >> >> > > > > > > > > > > > > > c) Implement Lucene sort usage in > >> > >> >> > > > > GridLuceneIndex.query(). > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > 3.[moderate] Build complex queries > with > >> > >> >> > *TextQuery*, > >> > >> >> > > > > > > including > >> > >> >> > > > > > > > > > > > > > terms/queries boosting. > >> > >> >> > > > > > > > > > > > > > *This section for voting only, as > >> requires > >> > >> more > >> > >> >> > > > detailed > >> > >> >> > > > > > > work. > >> > >> >> > > > > > > > > > Should > >> > >> >> > > > > > > > > > > > be > >> > >> >> > > > > > > > > > > > > > extended if community is interested in > >> it.* > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Looking forward to your comments! > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > BR, > >> > >> >> > > > > > > > > > > > > > Yuriy Shuliha > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > -- > >> > >> >> > > > > > > > > > > > > Best regards, > >> > >> >> > > > > > > > > > > > > Andrey V. Mashenkov > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > -- > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > Best regards, > >> > >> >> > > > > > > > > > > Alexei Scherbakov > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > >> > >> >> > > > > > > > >> > >> >> > > > > > > > >> > >> >> > > > > > > -- > >> > >> >> > > > > > > Best regards, > >> > >> >> > > > > > > Ivan Pavlukhin > >> > >> >> > > > > > > > >> > >> >> > > > > > >> > >> >> > > > > > >> > >> >> > > > > > >> > >> >> > > > > -- > >> > >> >> > > > > Best regards, > >> > >> >> > > > > Ivan Pavlukhin > >> > >> >> > > > > > >> > >> >> > > > > >> > >> >> > > > >> > >> >> > > >> > >> >> > > >> > >> >> > -- > >> > >> >> > Best regards, > >> > >> >> > Andrey V. Mashenkov > >> > >> >> > > >> > >> >> > >> > >> > > >> > >> > > >> > >> > -- > >> > >> > Best regards, > >> > >> > Andrey V. Mashenkov > >> > >> > > >> > >> > >> > > > >> > > >> > -- > >> > Best regards, > >> > Andrey V. Mashenkov > >> > > >> > > > >
