Hi, Andrey! Am I right, that you mean this ticket [1] *IGNITE-12291 Create controllable paged query requests / responses for TextQuery similar to current SQL result processing*, when talked about incorrect limit work for TextQueries?
[1] https://issues.apache.org/jira/browse/IGNITE-12291 On Thu, Apr 8, 2021 at 4:32 PM Maksim Timonin <timonin.ma...@gmail.com> wrote: > Hi, Andrey! > > >> ScanQuery, TextQuery and partially SQL query share the same > infrastructure > I think I understand what you mean. I debug query processing and now agree > that it's a nice idea to try to reuse the infrastructure of scan and text > queries. Also as I can see there already Reducer functionality exists, so I > hope we can use that. I'm not absolutely confident now that it will work > fine, but I'm going to start there. Thanks for pointing me this direction! > > >> I don't like the idea a user code will be executed inside BTree > operation > On the confluence page I've shown that a predicate passes as > TreeRowClosure. In this case you're right, any exception in a predicate > will lead to a CorruptedTreeException. But I see another legal way to > implement the predicate operation. BPlusTree.find accepts the X param that > passed to IO.getRow(). As I understand this param helps to control how much > returned row is filled. Then we can use it to return an object that > contains only basic info - link, pageAddr, offset. Then predicate operation > will be applied on the higher level on a cursor returned by a tree (like > H2TreeIndex does). It's safe to run user code there, we can handle > exceptions there. > > > > On Wed, Apr 7, 2021 at 4:46 PM Andrey Mashenkov < > andrey.mashen...@gmail.com> wrote: > >> Maksim, >> >> The ScanQuery API provides a filter as >> > param that for case of index query should be splitted on such >> conditions. >> > It looks like a non-trivial task. >> > >> ScanQuery, TextQuery and partially SQL query share the same >> infrastructure. >> I've thought we could extend, improve and reuse some ScanQuery code that >> already works fine: map query on topology, IO, batching. >> Add IndexCondition alongside the Filter, and abstract query executor from >> source (primary and secondary Indexes). >> Add a sorted merge algorithm to the query merge stage. It can be very >> useful also for TextQueries that suffers from the absence of sorted merge >> and a "limit' condition work incorrectly. >> >> If you think it will be too hard than creating from scratch, I'm ok. >> >> 3. Ignite creates a proxy object that is filled with objects that are >> > inlined. If a user tries to access a field that isn't inlined or not >> > indexed, then deserialization will start and Ignite will log.warn() >> about >> > that. >> > >> Agree, this can be faster. >> I don't like the idea a user code will be executed inside BTree operation, >> any exception can cause FailureHandler triggering and stop the node. >> >> There is one more thing that could be improved. >> ScanQuery now iterates over per-partition PK Hash index trees and has >> performance issues on a small grid with a large number of partitions. >> So, there are many partitions on every node and many trees should be >> scanned. >> In this case scan over a secondary index gives significant boots even if >> every row is materialized, because we need to traverse over a single tree >> per-node. >> Having the ability to run a ScanQuery over a secondary index (if one >> exists) instead of PK Hash will be great. >> >> >> On Wed, Apr 7, 2021 at 11:18 AM Maksim Timonin <timonin.ma...@gmail.com> >> wrote: >> >> > Hi, Andrey! >> > >> > Thanks for the review and your comments! >> > >> > >> Is it possible to extend ScanQuery functionality to pass index >> condition >> > I investigated this way and see some issues: >> > 1. Querying of indexes is not a scan actually. It's >> > a tree traverse (predicate operation is an exclusion, other operations >> like >> > gt, lt, min, max have explicit boundaries). An index query consists of >> > conditions that match an index structure. In general for a multi-key >> index >> > there can be multiple conditions. The ScanQuery API provides a filter as >> > param that for case of index query should be splitted on such >> conditions. >> > It looks like a non-trivial task. >> > 2. Querying of an index requires a sorted result, while The ScanQuery >> > doesn't matter about that. So there will be a different behavior of the >> > iterator for scanning a cache and querying indexes. It's not much to >> > implement I think, but it can make ScanQuery unclear for a user. >> > >> > Maybe it's a point to separate traverse (gt, lt, in, etc...) and scan >> > (predicate) index operations to different API. So there still will be a >> new >> > query type for the traversing. >> > >> > But we will introduce some inheritors for ScanQuery, like TableScanQuery >> > and IndexScanQuery, for scan and filter. Then the question is about >> > ordering, Cache and Table scans aren't ordered, but Index is. Then we >> can >> > introduce an optional param "order" for ScanQuery too. >> > >> > WDYT? >> > >> > >> Functional indices >> > >> This task looks like a huge one because the lifecycle of such classes >> > should be described first >> > I agree with you. That this part should be investigated deeper than I >> did. >> > So let's postpone discussion about functional indexes for a while. >> IEP-71 >> > declares some phases, functional indexes are part of the 2nd phase, but >> > users will get new functionality already from the 1st phase. Then I'll >> dig >> > into things you mentioned. Thanks for pointing them out. >> > >> > >> IndexScan by the predicate is questionable >> > Also in comments to the IEP on the Confluence you mentioned about >> > deserialization that is required to get an object for predicate >> function. >> > Now I see it like that: >> > 1. The predicate should operate only with indexed fields; >> > 2. User win from predicate only if index is inlined properly (even a >> part >> > of rows aren't inlined due to varlen - it still can be faster then make >> a >> > ScanQuery); >> > 3. Ignite creates a proxy object that is filled with objects that are >> > inlined. If a user tries to access a field that isn't inlined or not >> > indexed, then deserialization will start and Ignite will log.warn() >> about >> > that. >> > >> > So, I think it's a valid use case. Is there smth I'm missing? >> > >> > >> > >> > >> > >> > On Tue, Apr 6, 2021 at 6:21 PM Andrey Mashenkov < >> > andrey.mashen...@gmail.com> >> > wrote: >> > >> > > Hi Maksim, >> > > >> > > Nice idea, I'd like to see this feature in Ignite. >> > > The motivation is clear to me, it would be nice to have fast scans and >> > omit >> > > SQL overhead on planning, parsing and etc in some simple use-cases. >> > > >> > > I've left few minor comments to the IEP, but I have the next questions >> > > which answer I failed to find in IEP. >> > > 1. Is it possible to extend ScanQuery functionality to pass index >> > condition >> > > as a hint/parameter rather than create a separate query type? >> > > This allows a user to run a query over the particular table (for >> > > multi-table per cache case) and use an index for some type of >> conditions. >> > > >> > > 2. Functional indices, as you wrote, should use Functions distributed >> via >> > > peerClassLoading mechanics. >> > > This means there will no class with function on server sides and such >> > > classes are not persistent. Seems, they can survive grid restart. >> > > This task looks like a huge one because the lifecycle of such classes >> > > should be described first. >> > > Possible pitfalls are: >> > > * Durability. Function code MUST be persistent, to survive node >> restart >> > as >> > > there can be no guaranteed classes available on the server-side. >> > > * Consistency. Server (and maybe clients) nodes MUST have the same >> class >> > > code at a time. >> > > * Code ownership. Would class code be shared or per-cache? If first, >> you >> > > can't just change class code by loading a new one, because other >> caches >> > may >> > > use this function. >> > > If second, different caches may have different code/behavior, that >> may be >> > > non-obvious to end-user. >> > > >> > > 3. IndexScan by the predicate is questionable. >> > > Maybe it will can faster if there are multiple tables in a cache, but >> > looks >> > > similar to ScanQuery with a filter. >> > > >> > > Also, I believe we can have a common API (configuring, creating, >> using) >> > for >> > > all types of Indices, but >> > > some types (e.g. functional) will be ignored in SQL due to limited >> > support >> > > on H2 side, >> > > and other types will be shared and could be used by ScanQuery engine >> as >> > > well as by SQL engine. >> > > >> > > On Tue, Apr 6, 2021 at 4:14 PM Maksim Timonin < >> timonin.ma...@gmail.com> >> > > wrote: >> > > >> > > > Hi, Igniters! >> > > > >> > > > I'd like to propose a new feature - opportunity to query and create >> > > indexes >> > > > from public API. >> > > > >> > > > It will help in some cases, where: >> > > > 1. SQL is not applicable by design of user application; >> > > > 2. Where IndexScan is preferable than ScanQuery for performance >> > reasons; >> > > > 3. Functional indexes are required. >> > > > >> > > > Also it'll be great to have a transactional support for such >> queries, >> > > like >> > > > the "select for update" query provides. But I don't dig there much. >> It >> > > will >> > > > be a next step if this API will be implemented. >> > > > >> > > > I've prepared an IEP-71 for that [1] with more details. Please share >> > your >> > > > thoughts. >> > > > >> > > > >> > > > [1] >> > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-71+Public+API+for+secondary+index+search >> > > > >> > > >> > > >> > > -- >> > > Best regards, >> > > Andrey V. Mashenkov >> > > >> > >> >> >> -- >> Best regards, >> Andrey V. Mashenkov >> >
