Ilya Kasnacheev, what a problem in Solr with Ignite functionality ?
thanks !
>Вторник, 26 ноября 2019, 13:50 +03:00 от Ilya Kasnacheev
><ilya.kasnach...@gmail.com>:
>
>Hello!
>
>I have a hunch that we are trying to build Apache Solr (or Solr Cloud) into
>Apache Ignite. I think that's a lot of effort that is not very justified.
>
>I don't think we should try to implement sorting in Apache Ignite, because
>it is a lot of work, and a lot of code in our code base which we don't
>really want.
>
>Regards,
>--
>Ilya Kasnacheev
>
>
>пт, 22 нояб. 2019 г. в 20:59, Yuriy Shuliga < shul...@gmail.com >:
>
>> Dear Igniters,
>>
>> The first part of TextQuery improvement - a result limit - was developed
>> and merged.
>> Now we have to develop most important functionality here - proper sorting
>> of Lucene index response and correct reducing of them for distributed
>> queries.
>>
>> *There are two Lucene based aspects*
>>
>> 1. In case of using no sorting fields, the documents in response are still
>> ordered by relevance.
>> Actually this is ScoreDoc.score value.
>> In order to reduce the distributed results correctly, the score should be
>> passed with response.
>>
>> 2. When sorting by conventional fields, then Lucene should have these
>> fields properly indexed and
>> corresponding Sort object should be applied to Lucene's search call.
>> In order to mark those fields a new annotation like '@SortField' may be
>> introduced.
>>
>> *Reducing on Ignite *
>>
>> The obvious point of distributed response reduction is class
>> GridCacheDistributedQueryFuture.
>> Though, @Ivan Pavlukhin mentioned class with similar functionality:
>> ReduceIndexSorted
>> What I see here, that it is tangled with H2 related classes (
>> org.h2.result.Row) and might not be unified with TextQuery reduction.
>>
>> Still need a support here.
>>
>> Overall, the goal of this letter is to initiate discussion on TextQuery
>> Sorting implementation and come closer to ticket creation.
>>
>> BR,
>> Yuriy Shuliha
>>
>> вт, 22 жовт. 2019 о 13:31 Andrey Mashenkov < andrey.mashen...@gmail.com >
>> пише:
>>
>> > Hi Dmitry, Yuriy.
>> >
>> > I've found GridCacheQueryFutureAdapter has newly added AtomicInteger
>> > 'total' field and 'limit; field as primitive int.
>> >
>> > Both fields are used inside synchronized block only.
>> > So, we can make both private and downgrade AtomicInteger to primitive
>> int.
>> >
>> > Most likely, these fields can be replaced with one field.
>> >
>> >
>> >
>> > On Mon, Oct 21, 2019 at 10:01 PM Dmitriy Pavlov < dpav...@apache.org >
>> > wrote:
>> >
>> > > Hi Andrey,
>> > >
>> > > I've checked this ticket comments, and there is a TC Bot visa (with no
>> > > blockers).
>> > >
>> > > Do you have any concerns related to this patch?
>> > >
>> > > Sincerely,
>> > > Dmitriy Pavlov
>> > >
>> > > чт, 17 окт. 2019 г. в 16:43, Yuriy Shuliga < shul...@gmail.com >:
>> > >
>> > >> Andrey,
>> > >>
>> > >> Per you request, I created ticket
>> > >> https://issues.apache.org/jira/browse/IGNITE-12291 linked to
>> > >> https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-12189
>> > >>
>> > >> Could you please proceed with PR merge ?
>> > >>
>> > >> BR,
>> > >> Yuriy Shuliha
>> > >>
>> > >> ср, 9 жовт. 2019 о 12:52 Andrey Mashenkov < andrey.mashen...@gmail.com
>> >
>> > >> пише:
>> > >>
>> > >> > Hi Yuri,
>> > >> >
>> > >> > To get access to TC Bot you should register as TeamCity user [1], if
>> > you
>> > >> > didn't do this already.
>> > >> > Then you will be able to authorize on Ignite TC Bot page with same
>> > >> > credentials.
>> > >> >
>> > >> > [1] https://ci.ignite.apache.org/registerUser.html
>> > >> >
>> > >> > On Fri, Oct 4, 2019 at 3:10 PM Yuriy Shuliga < shul...@gmail.com >
>> > wrote:
>> > >> >
>> > >> >> Andrew,
>> > >> >>
>> > >> >> I have corrected PR according to your notes. Please review.
>> > >> >> What will be the next steps in order to merge in?
>> > >> >>
>> > >> >> Y.
>> > >> >>
>> > >> >> чт, 3 жовт. 2019 о 17:47 Andrey Mashenkov <
>> > andrey.mashen...@gmail.com >
>> > >> >> пише:
>> > >> >>
>> > >> >> > Yuri,
>> > >> >> >
>> > >> >> > I've done with review.
>> > >> >> > No crime found, but trivial compatibility bug.
>> > >> >> >
>> > >> >> > On Thu, Oct 3, 2019 at 3:54 PM Yuriy Shuliga < shul...@gmail.com >
>> > >> wrote:
>> > >> >> >
>> > >> >> > > Denis,
>> > >> >> > >
>> > >> >> > > Thank you for your attention to this.
>> > >> >> > > as for now, the
>> > https://issues.apache.org/jira/browse/IGNITE-12189
>> > >> >> > ticket
>> > >> >> > > is still pending review.
>> > >> >> > > Do we have a chance to move it forward somehow?
>> > >> >> > >
>> > >> >> > > BR,
>> > >> >> > > Yuriy Shuliha
>> > >> >> > >
>> > >> >> > > пн, 30 вер. 2019 о 23:35 Denis Magda < dma...@apache.org > пише:
>> > >> >> > >
>> > >> >> > > > Yuriy,
>> > >> >> > > >
>> > >> >> > > > I've seen you opening a pull-request with the first changes:
>> > >> >> > > > https://issues.apache.org/jira/browse/IGNITE-12189
>> > >> >> > > >
>> > >> >> > > > Alex Scherbakov and Ivan are you the right guys to do the
>> > review?
>> > >> >> > > >
>> > >> >> > > > -
>> > >> >> > > > Denis
>> > >> >> > > >
>> > >> >> > > >
>> > >> >> > > > On Fri, Sep 27, 2019 at 8:48 AM Павлухин Иван <
>> > >> vololo...@gmail.com >
>> > >> >> > > wrote:
>> > >> >> > > >
>> > >> >> > > > > Yuriy,
>> > >> >> > > > >
>> > >> >> > > > > Thank you for providing details! Quite interesting.
>> > >> >> > > > >
>> > >> >> > > > > Yes, we already have support of distributed limit and
>> merging
>> > >> >> sorted
>> > >> >> > > > > subresults for SQL queries. E.g. ReduceIndexSorted and
>> > >> >> > > > > MergeStreamIterator are used for merging sorted streams.
>> > >> >> > > > >
>> > >> >> > > > > Could you please also clarify about score/relevance? Is it
>> > >> >> provided
>> > >> >> > by
>> > >> >> > > > > Lucene engine for each query result? I am thinking how to
>> do
>> > >> >> sorted
>> > >> >> > > > > merge properly in this case.
>> > >> >> > > > >
>> > >> >> > > > > ср, 25 сент. 2019 г. в 18:56, Yuriy Shuliga <
>> > shul...@gmail.com
>> > >> >:
>> > >> >> > > > > >
>> > >> >> > > > > > Ivan,
>> > >> >> > > > > >
>> > >> >> > > > > > Thank you for interesting question!
>> > >> >> > > > > >
>> > >> >> > > > > > Text searches (or full text searches) are mostly
>> > >> human-oriented.
>> > >> >> > And
>> > >> >> > > > the
>> > >> >> > > > > > point of user's interest is topmost part of response.
>> > >> >> > > > > > Then user can read it, evaluate and use the given records
>> > for
>> > >> >> > further
>> > >> >> > > > > > purposes.
>> > >> >> > > > > >
>> > >> >> > > > > > Particularly in our case, we use Ignite for operations
>> with
>> > >> >> > financial
>> > >> >> > > > > data,
>> > >> >> > > > > > and there lots of text stuff like assets names, fin.
>> > >> >> instruments,
>> > >> >> > > > > companies
>> > >> >> > > > > > etc.
>> > >> >> > > > > > In order to operate with this quickly and reliably, users
>> > >> used
>> > >> >> to
>> > >> >> > > work
>> > >> >> > > > > with
>> > >> >> > > > > > text search, type-ahead completions, suggestions.
>> > >> >> > > > > >
>> > >> >> > > > > > For this purposes we are indexing particular string data
>> in
>> > >> >> > separate
>> > >> >> > > > > caches.
>> > >> >> > > > > >
>> > >> >> > > > > > Sorting capabilities and response size limitations are
>> very
>> > >> >> > important
>> > >> >> > > > > > there. As our API have to provide most relevant
>> information
>> > >> in
>> > >> >> view
>> > >> >> > > of
>> > >> >> > > > > > limited size.
>> > >> >> > > > > >
>> > >> >> > > > > > Now let me comment some Ignite/Lucene perspective.
>> > >> >> > > > > > Actually Ignite queries and Lucene returns
>> > >> *TopDocs.scoresDocs
>> > >> >> > > *already
>> > >> >> > > > > > sorted by *score *(relevance). So most relevant documents
>> > >> are on
>> > >> >> > the
>> > >> >> > > > top.
>> > >> >> > > > > > And currently distributed queries responses from
>> different
>> > >> nodes
>> > >> >> > are
>> > >> >> > > > > merged
>> > >> >> > > > > > into final query cursor queue in arbitrary way.
>> > >> >> > > > > > So in fact we already have the score order ruined here.
>> > Also
>> > >> >> Ignite
>> > >> >> > > > > > requests all possible documents from Lucene that is
>> > redundant
>> > >> >> and
>> > >> >> > not
>> > >> >> > > > > good
>> > >> >> > > > > > for performance.
>> > >> >> > > > > >
>> > >> >> > > > > > I'm implementing *limit* parameter to be part of
>> *TextQuery
>> > >> *and
>> > >> >> > have
>> > >> >> > > > to
>> > >> >> > > > > > notice that we still have to add sorting for text queries
>> > >> >> > processing
>> > >> >> > > in
>> > >> >> > > > > > order to have applicable results.
>> > >> >> > > > > >
>> > >> >> > > > > > *Limit* parameter itself should improve the part of
>> issues
>> > >> from
>> > >> >> > > above,
>> > >> >> > > > > but
>> > >> >> > > > > > definitely, sorting by document score at least should be
>> > >> >> > implemented
>> > >> >> > > > > along
>> > >> >> > > > > > with limit.
>> > >> >> > > > > >
>> > >> >> > > > > > This is a pretty short commentary if you still have any
>> > >> >> questions,
>> > >> >> > > > please
>> > >> >> > > > > > ask, do not hesitate)
>> > >> >> > > > > >
>> > >> >> > > > > > BR,
>> > >> >> > > > > > Yuriy Shuliha
>> > >> >> > > > > >
>> > >> >> > > > > > чт, 19 вер. 2019 о 11:38 Павлухин Иван <
>> > vololo...@gmail.com >
>> > >> >> пише:
>> > >> >> > > > > >
>> > >> >> > > > > > > Yuriy,
>> > >> >> > > > > > >
>> > >> >> > > > > > > Greatly appreciate your interest.
>> > >> >> > > > > > >
>> > >> >> > > > > > > Could you please elaborate a little bit about sorting?
>> > What
>> > >> >> tasks
>> > >> >> > > > does
>> > >> >> > > > > > > it help to solve and how? It would be great to provide
>> an
>> > >> >> > example.
>> > >> >> > > > > > >
>> > >> >> > > > > > > ср, 18 сент. 2019 г. в 09:39, Alexei Scherbakov <
>> > >> >> > > > > > > alexey.scherbak...@gmail.com >:
>> > >> >> > > > > > > >
>> > >> >> > > > > > > > Denis,
>> > >> >> > > > > > > >
>> > >> >> > > > > > > > I like the idea of throwing an exception for enabled
>> > text
>> > >> >> > queries
>> > >> >> > > > on
>> > >> >> > > > > > > > persistent caches.
>> > >> >> > > > > > > >
>> > >> >> > > > > > > > Also I'm fine with proposed limit for unsorted
>> > searches.
>> > >> >> > > > > > > >
>> > >> >> > > > > > > > Yury, please proceed with ticket creation.
>> > >> >> > > > > > > >
>> > >> >> > > > > > > > вт, 17 сент. 2019 г., 22:06 Denis Magda <
>> > >> dma...@apache.org
>> > >> >> >:
>> > >> >> > > > > > > >
>> > >> >> > > > > > > > > Igniters,
>> > >> >> > > > > > > > >
>> > >> >> > > > > > > > > I see nothing wrong with Yury's proposal in regards
>> > >> >> full-text
>> > >> >> > > > > search
>> > >> >> > > > > > > API
>> > >> >> > > > > > > > > evolution as long as Yury is ready to push it
>> > forward.
>> > >> >> > > > > > > > >
>> > >> >> > > > > > > > > As for the in-memory mode only, it makes total
>> sense
>> > >> for
>> > >> >> > > > in-memory
>> > >> >> > > > > data
>> > >> >> > > > > > > > > grid deployments when Ignite caches data of an
>> > >> underlying
>> > >> >> DB
>> > >> >> > > like
>> > >> >> > > > > > > Postgres.
>> > >> >> > > > > > > > > As part of the changes, I would simply throw an
>> > >> exception
>> > >> >> (by
>> > >> >> > > > > default)
>> > >> >> > > > > > > if
>> > >> >> > > > > > > > > the one attempts to use text indices with the
>> native
>> > >> >> > > persistence
>> > >> >> > > > > > > enabled.
>> > >> >> > > > > > > > > If the person is ready to live with that limitation
>> > >> that
>> > >> >> an
>> > >> >> > > > > explicit
>> > >> >> > > > > > > > > configuration change is needed to come around the
>> > >> >> exception.
>> > >> >> > > > > > > > >
>> > >> >> > > > > > > > > Thoughts?
>> > >> >> > > > > > > > >
>> > >> >> > > > > > > > >
>> > >> >> > > > > > > > > -
>> > >> >> > > > > > > > > Denis
>> > >> >> > > > > > > > >
>> > >> >> > > > > > > > >
>> > >> >> > > > > > > > > On Tue, Sep 17, 2019 at 7:44 AM Yuriy Shuliga <
>> > >> >> > > shul...@gmail.com
>> > >> >> > > > >
>> > >> >> > > > > > > wrote:
>> > >> >> > > > > > > > >
>> > >> >> > > > > > > > > > Hello to all again,
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > Thank you for important comments and notes given
>> > >> below!
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > Let me answer and continue the discussion.
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > (I) Overall needs in Lucene indexing
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > Alexei has referenced to
>> > >> >> > > > > > > > > >
>> https://issues.apache.org/jira/browse/IGNITE-5371
>> > >> where
>> > >> >> > > > > > > > > > absence of index persistence was declared as an
>> > >> >> obstacle to
>> > >> >> > > > > further
>> > >> >> > > > > > > > > > development.
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > a) This ticket is already closed as not valid.b)
>> > >> There
>> > >> >> are
>> > >> >> > > > > definite
>> > >> >> > > > > > > needs
>> > >> >> > > > > > > > > > (and in our project as well) in just in-memory
>> > >> indexing
>> > >> >> of
>> > >> >> > > > > selected
>> > >> >> > > > > > > data.
>> > >> >> > > > > > > > > > We intend to use search capabilities for fetching
>> > >> >> limited
>> > >> >> > > > amount
>> > >> >> > > > > of
>> > >> >> > > > > > > > > records
>> > >> >> > > > > > > > > > that should be used in type-ahead search /
>> > >> suggestions.
>> > >> >> > > > > > > > > > Not all of the data will be indexed and the are
>> no
>> > >> need
>> > >> >> in
>> > >> >> > > > Lucene
>> > >> >> > > > > > > index
>> > >> >> > > > > > > > > to
>> > >> >> > > > > > > > > > be persistence. Hope this is a wide pattern of
>> > >> >> text-search
>> > >> >> > > > usage.
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > (II) Necessary fixes in current implementation.
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > a) Implementation of correct *limit *(*offset*
>> > seems
>> > >> to
>> > >> >> be
>> > >> >> > > not
>> > >> >> > > > > > > required
>> > >> >> > > > > > > > > in
>> > >> >> > > > > > > > > > text-search tasks for now)
>> > >> >> > > > > > > > > > I have investigated the data flow for distributed
>> > >> text
>> > >> >> > > queries.
>> > >> >> > > > > it
>> > >> >> > > > > > > was
>> > >> >> > > > > > > > > > simple test prefix query, like 'name'*='ene*'*
>> > >> >> > > > > > > > > > For now each server-node returns all response
>> > >> records to
>> > >> >> > the
>> > >> >> > > > > > > client-node
>> > >> >> > > > > > > > > > and it may contain ~thousands, ~hundred thousands
>> > >> >> records.
>> > >> >> > > > > > > > > > Event if we need only first 10-100. Again, all
>> the
>> > >> >> results
>> > >> >> > > are
>> > >> >> > > > > added
>> > >> >> > > > > > > to
>> > >> >> > > > > > > > > > queue in GridCacheQueryFutureAdapter in arbitrary
>> > >> order
>> > >> >> by
>> > >> >> > > > pages.
>> > >> >> > > > > > > > > > I did not find here any means to deliver
>> > >> deterministic
>> > >> >> > > result.
>> > >> >> > > > > > > > > > So implementing limit as part of query and
>> > >> >> > > > > (GridCacheQueryRequest)
>> > >> >> > > > > > > will
>> > >> >> > > > > > > > > not
>> > >> >> > > > > > > > > > change the nature of response but will limit load
>> > on
>> > >> >> nodes
>> > >> >> > > and
>> > >> >> > > > > > > > > networking.
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > Can we consider to open a ticket for this?
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > (III) Further extension of Lucene API exposition
>> to
>> > >> >> Ignite
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > a) Sorting
>> > >> >> > > > > > > > > > The solution for this could be:
>> > >> >> > > > > > > > > > - Make entities comparable
>> > >> >> > > > > > > > > > - Add custom comparator to entity
>> > >> >> > > > > > > > > > - Add annotations to mark sorted fields for
>> Lucene
>> > >> >> indexing
>> > >> >> > > > > > > > > > - Use comparators when merging responses or
>> > reducing
>> > >> to
>> > >> >> > > desired
>> > >> >> > > > > > > limit on
>> > >> >> > > > > > > > > > client node.
>> > >> >> > > > > > > > > > Will require full result set to be loaded into
>> > >> memory.
>> > >> >> > Though
>> > >> >> > > > > can be
>> > >> >> > > > > > > used
>> > >> >> > > > > > > > > > for relatively small limits.
>> > >> >> > > > > > > > > > BR,
>> > >> >> > > > > > > > > > Yuriy Shuliha
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > пт, 30 серп. 2019 о 10:37 Alexei Scherbakov <
>> > >> >> > > > > > > > > alexey.scherbak...@gmail.com >
>> > >> >> > > > > > > > > > пише:
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > > > > Yuriy,
>> > >> >> > > > > > > > > > >
>> > >> >> > > > > > > > > > > Note what one of major blockers for text
>> queries
>> > is
>> > >> >> [1]
>> > >> >> > > which
>> > >> >> > > > > makes
>> > >> >> > > > > > > > > > lucene
>> > >> >> > > > > > > > > > > indexes unusable with persistence and main
>> reason
>> > >> for
>> > >> >> > > > > > > discontinuation.
>> > >> >> > > > > > > > > > > Probably it's should be addressed first to make
>> > >> text
>> > >> >> > > queries
>> > >> >> > > > a
>> > >> >> > > > > > > valid
>> > >> >> > > > > > > > > > > product feature.
>> > >> >> > > > > > > > > > >
>> > >> >> > > > > > > > > > > Distributed sorting and advanved querying is
>> > indeed
>> > >> >> not a
>> > >> >> > > > > trivial
>> > >> >> > > > > > > task.
>> > >> >> > > > > > > > > > > Some kind of merging must be implemented on
>> query
>> > >> >> > > originating
>> > >> >> > > > > node.
>> > >> >> > > > > > > > > > >
>> > >> >> > > > > > > > > > > [1]
>> > >> https://issues.apache.org/jira/browse/IGNITE-5371
>> > >> >> > > > > > > > > > >
>> > >> >> > > > > > > > > > > чт, 29 авг. 2019 г. в 23:38, Denis Magda <
>> > >> >> > > dma...@apache.org
>> > >> >> > > > >:
>> > >> >> > > > > > > > > > >
>> > >> >> > > > > > > > > > > > Yuriy,
>> > >> >> > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > If you are ready to take over the full-text
>> > >> search
>> > >> >> > > indexes
>> > >> >> > > > > then
>> > >> >> > > > > > > > > please
>> > >> >> > > > > > > > > > go
>> > >> >> > > > > > > > > > > > ahead. The primary reason why the community
>> > >> wants to
>> > >> >> > > > > discontinue
>> > >> >> > > > > > > them
>> > >> >> > > > > > > > > > > first
>> > >> >> > > > > > > > > > > > (and, probable, resurrect later) are the
>> > >> limitations
>> > >> >> > > listed
>> > >> >> > > > > by
>> > >> >> > > > > > > Andrey
>> > >> >> > > > > > > > > > and
>> > >> >> > > > > > > > > > > > minimal support from the community end.
>> > >> >> > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > -
>> > >> >> > > > > > > > > > > > Denis
>> > >> >> > > > > > > > > > > >
>> > >> >> > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > On Thu, Aug 29, 2019 at 1:29 PM Andrey
>> > Mashenkov
>> > >> <
>> > >> >> > > > > > > > > > > > andrey.mashen...@gmail.com >
>> > >> >> > > > > > > > > > > > wrote:
>> > >> >> > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > Hi Yuriy,
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > Unfortunatelly, there is a plan to
>> > discontinue
>> > >> >> > > > TextQueries
>> > >> >> > > > > in
>> > >> >> > > > > > > > > Ignite
>> > >> >> > > > > > > > > > > [1].
>> > >> >> > > > > > > > > > > > > Motivation here is text indexes are not
>> > >> >> persistent,
>> > >> >> > not
>> > >> >> > > > > > > > > transactional
>> > >> >> > > > > > > > > > > and
>> > >> >> > > > > > > > > > > > > can't be user together with SQL or inside
>> > SQL.
>> > >> >> > > > > > > > > > > > > and there is a lack of interest from
>> > community
>> > >> >> side.
>> > >> >> > > > > > > > > > > > > You are weclome to take on these issues and
>> > >> make
>> > >> >> > > > > TextQueries
>> > >> >> > > > > > > great.
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > 1, PageSize can't be used to limit
>> > resultset.
>> > >> >> > > > > > > > > > > > > Query results return from data node to
>> > >> client-side
>> > >> >> > > cursor
>> > >> >> > > > > in
>> > >> >> > > > > > > > > > > page-by-page
>> > >> >> > > > > > > > > > > > > manner and
>> > >> >> > > > > > > > > > > > > this parameter is designed control page
>> size.
>> > >> It
>> > >> >> is
>> > >> >> > > > > supposed
>> > >> >> > > > > > > query
>> > >> >> > > > > > > > > > > > executes
>> > >> >> > > > > > > > > > > > > lazily on server side and
>> > >> >> > > > > > > > > > > > > it is not excepted full resultset be loaded
>> > to
>> > >> >> memory
>> > >> >> > > on
>> > >> >> > > > > server
>> > >> >> > > > > > > > > side
>> > >> >> > > > > > > > > > at
>> > >> >> > > > > > > > > > > > > once, but by pages.
>> > >> >> > > > > > > > > > > > > Do you mean you found Lucene load entire
>> > >> resultset
>> > >> >> > into
>> > >> >> > > > > memory
>> > >> >> > > > > > > > > before
>> > >> >> > > > > > > > > > > > first
>> > >> >> > > > > > > > > > > > > page is sent to client?
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > I'd think a new parameter should be added
>> to
>> > >> limit
>> > >> >> > > > result.
>> > >> >> > > > > The
>> > >> >> > > > > > > best
>> > >> >> > > > > > > > > > > > > solution is to use query language commands
>> > for
>> > >> >> this,
>> > >> >> > > e.g.
>> > >> >> > > > > > > > > > > "LIMIT/OFFSET"
>> > >> >> > > > > > > > > > > > in
>> > >> >> > > > > > > > > > > > > SQL.
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > This task doesn't look trivial. Query is
>> > >> >> distributed
>> > >> >> > > > > operation
>> > >> >> > > > > > > and
>> > >> >> > > > > > > > > > same
>> > >> >> > > > > > > > > > > > > user query will be executed on data nodes
>> > >> >> > > > > > > > > > > > > and then results from all nodes should be
>> > >> correcly
>> > >> >> > > merged
>> > >> >> > > > > > > before
>> > >> >> > > > > > > > > > being
>> > >> >> > > > > > > > > > > > > returned via client-cursor.
>> > >> >> > > > > > > > > > > > > So, LIMIT should be applied on every node
>> and
>> > >> >> then on
>> > >> >> > > > merge
>> > >> >> > > > > > > phase.
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > Also, this may be non-obviuos, limiting
>> > results
>> > >> >> make
>> > >> >> > no
>> > >> >> > > > > sence
>> > >> >> > > > > > > > > without
>> > >> >> > > > > > > > > > > > > sorting,
>> > >> >> > > > > > > > > > > > > as there is no guarantee every next query
>> run
>> > >> will
>> > >> >> > > return
>> > >> >> > > > > same
>> > >> >> > > > > > > data
>> > >> >> > > > > > > > > > > > because
>> > >> >> > > > > > > > > > > > > of page reordeing.
>> > >> >> > > > > > > > > > > > > Basically, merge phase receive results from
>> > >> data
>> > >> >> > nodes
>> > >> >> > > > > > > > > asynchronously
>> > >> >> > > > > > > > > > > and
>> > >> >> > > > > > > > > > > > > messages from different nodes can't be
>> > ordered.
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > 2.
>> > >> >> > > > > > > > > > > > > a. "tokenize" param name (for
>> > @QueryTextFiled)
>> > >> >> looks
>> > >> >> > > more
>> > >> >> > > > > > > verbose,
>> > >> >> > > > > > > > > > > isn't
>> > >> >> > > > > > > > > > > > > it.
>> > >> >> > > > > > > > > > > > > b,c. What about distributed query? How
>> > partial
>> > >> >> > results
>> > >> >> > > > from
>> > >> >> > > > > > > nodes
>> > >> >> > > > > > > > > > will
>> > >> >> > > > > > > > > > > be
>> > >> >> > > > > > > > > > > > > merged?
>> > >> >> > > > > > > > > > > > > Does Lucene allows to configure comparator
>> > for
>> > >> >> data
>> > >> >> > > > > sorting?
>> > >> >> > > > > > > > > > > > > What comparator Ignite should choose to
>> sort
>> > >> >> result
>> > >> >> > on
>> > >> >> > > > > merge
>> > >> >> > > > > > > phase?
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > 3. For now Lucene engine is not
>> configurable
>> > at
>> > >> >> all.
>> > >> >> > > E.g.
>> > >> >> > > > > it is
>> > >> >> > > > > > > > > > > > impossible
>> > >> >> > > > > > > > > > > > > to configure Tokenizer.
>> > >> >> > > > > > > > > > > > > I'd think about possible ways to configure
>> > >> engine
>> > >> >> at
>> > >> >> > > > first
>> > >> >> > > > > and
>> > >> >> > > > > > > only
>> > >> >> > > > > > > > > > > then
>> > >> >> > > > > > > > > > > > go
>> > >> >> > > > > > > > > > > > > further to discuss\implement complex
>> > features,
>> > >> >> > > > > > > > > > > > > that may depends on engine config.
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > On Thu, Aug 29, 2019 at 8:17 PM Yuriy
>> > Shuliga <
>> > >> >> > > > > > > shul...@gmail.com >
>> > >> >> > > > > > > > > > > wrote:
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > > Dear community,
>> > >> >> > > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > > By starting this chain I'd like to open
>> > >> >> discussion
>> > >> >> > > that
>> > >> >> > > > > would
>> > >> >> > > > > > > > > come
>> > >> >> > > > > > > > > > to
>> > >> >> > > > > > > > > > > > > > contribution results in subj. area.
>> > >> >> > > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > > Ignite has indexing capabilities, backed
>> up
>> > >> by
>> > >> >> > > > different
>> > >> >> > > > > > > > > > mechanisms,
>> > >> >> > > > > > > > > > > > > > including Lucene.
>> > >> >> > > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > > Currently, Lucene 7.5.0 is used (past
>> year
>> > >> >> > release).
>> > >> >> > > > > > > > > > > > > > This is a wide spread and mature
>> technology
>> > >> that
>> > >> >> > > covers
>> > >> >> > > > > text
>> > >> >> > > > > > > > > search
>> > >> >> > > > > > > > > > > > area
>> > >> >> > > > > > > > > > > > > > and beyond (e.g. spacial data indexing).
>> > >> >> > > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > > My goal is to *expose more Lucene
>> > >> functionality
>> > >> >> to
>> > >> >> > > > Ignite
>> > >> >> > > > > > > > > indexing
>> > >> >> > > > > > > > > > > and
>> > >> >> > > > > > > > > > > > > > query mechanisms for text data*.
>> > >> >> > > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > > It's quite simple request at current
>> stage.
>> > >> It
>> > >> >> is
>> > >> >> > > > coming
>> > >> >> > > > > > > from our
>> > >> >> > > > > > > > > > > > > project's
>> > >> >> > > > > > > > > > > > > > needs, but i believe, will be useful for
>> a
>> > >> lot
>> > >> >> more
>> > >> >> > > > > people.
>> > >> >> > > > > > > > > > > > > > Let's walk through and vote or discuss
>> > about
>> > >> >> Jira
>> > >> >> > > > > tickets for
>> > >> >> > > > > > > > > them.
>> > >> >> > > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > > 1.[trivial] Use dataQuery.getPageSize()
>> > to
>> > >> >> limit
>> > >> >> > > > search
>> > >> >> > > > > > > > > response
>> > >> >> > > > > > > > > > > > items
>> > >> >> > > > > > > > > > > > > > inside GridLuceneIndex.query(). Currently
>> > it
>> > >> is
>> > >> >> > > calling
>> > >> >> > > > > > > > > > > > > > IndexSearcher.search(query,
>> > >> >> *Integer.MAX_VALUE*) -
>> > >> >> > so
>> > >> >> > > > > > > basically
>> > >> >> > > > > > > > > all
>> > >> >> > > > > > > > > > > > > scored
>> > >> >> > > > > > > > > > > > > > matches will me returned, what we do not
>> > >> need in
>> > >> >> > most
>> > >> >> > > > > cases.
>> > >> >> > > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > > 2.[simple] Add sorting. Then more
>> capable
>> > >> >> search
>> > >> >> > > call
>> > >> >> > > > > can be
>> > >> >> > > > > > > > > > > > > > executed: *IndexSearcher.search(query,
>> > count,
>> > >> >> > > > > > > > > > > > > > sort) *
>> > >> >> > > > > > > > > > > > > > Implementation steps:
>> > >> >> > > > > > > > > > > > > > a) Introduce boolean *sortField*
>> parameter
>> > in
>> > >> >> > > > > > > *@QueryTextFiled *
>> > >> >> > > > > > > > > > > > > > annotation. If
>> > >> >> > > > > > > > > > > > > > *true *the filed will be indexed but not
>> > >> >> tokenized.
>> > >> >> > > > > Number
>> > >> >> > > > > > > types
>> > >> >> > > > > > > > > > are
>> > >> >> > > > > > > > > > > > > > preferred here.
>> > >> >> > > > > > > > > > > > > > b) Add *sort* collection to *TextQuery*
>> > >> >> > constructor.
>> > >> >> > > It
>> > >> >> > > > > > > should
>> > >> >> > > > > > > > > > define
>> > >> >> > > > > > > > > > > > > > desired sort fields used for querying.
>> > >> >> > > > > > > > > > > > > > c) Implement Lucene sort usage in
>> > >> >> > > > > GridLuceneIndex.query().
>> > >> >> > > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > > 3.[moderate] Build complex queries with
>> > >> >> > *TextQuery*,
>> > >> >> > > > > > > including
>> > >> >> > > > > > > > > > > > > > terms/queries boosting.
>> > >> >> > > > > > > > > > > > > > *This section for voting only, as
>> requires
>> > >> more
>> > >> >> > > > detailed
>> > >> >> > > > > > > work.
>> > >> >> > > > > > > > > > Should
>> > >> >> > > > > > > > > > > > be
>> > >> >> > > > > > > > > > > > > > extended if community is interested in
>> it.*
>> > >> >> > > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > > Looking forward to your comments!
>> > >> >> > > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > > BR,
>> > >> >> > > > > > > > > > > > > > Yuriy Shuliha
>> > >> >> > > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > > > --
>> > >> >> > > > > > > > > > > > > Best regards,
>> > >> >> > > > > > > > > > > > > Andrey V. Mashenkov
>> > >> >> > > > > > > > > > > > >
>> > >> >> > > > > > > > > > > >
>> > >> >> > > > > > > > > > >
>> > >> >> > > > > > > > > > >
>> > >> >> > > > > > > > > > > --
>> > >> >> > > > > > > > > > >
>> > >> >> > > > > > > > > > > Best regards,
>> > >> >> > > > > > > > > > > Alexei Scherbakov
>> > >> >> > > > > > > > > > >
>> > >> >> > > > > > > > > >
>> > >> >> > > > > > > > >
>> > >> >> > > > > > >
>> > >> >> > > > > > >
>> > >> >> > > > > > >
>> > >> >> > > > > > > --
>> > >> >> > > > > > > Best regards,
>> > >> >> > > > > > > Ivan Pavlukhin
>> > >> >> > > > > > >
>> > >> >> > > > >
>> > >> >> > > > >
>> > >> >> > > > >
>> > >> >> > > > > --
>> > >> >> > > > > Best regards,
>> > >> >> > > > > Ivan Pavlukhin
>> > >> >> > > > >
>> > >> >> > > >
>> > >> >> > >
>> > >> >> >
>> > >> >> >
>> > >> >> > --
>> > >> >> > Best regards,
>> > >> >> > Andrey V. Mashenkov
>> > >> >> >
>> > >> >>
>> > >> >
>> > >> >
>> > >> > --
>> > >> > Best regards,
>> > >> > Andrey V. Mashenkov
>> > >> >
>> > >>
>> > >
>> >
>> > --
>> > Best regards,
>> > Andrey V. Mashenkov
>> >
>>
