Re: [VOTE] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
According to KIP-968, we added ResultOrder to this KIP and support withAscendingKeys() to the RangeQuery. Sincerely, Hanyu On Tue, Oct 17, 2023 at 11:25 AM Guozhang Wang wrote: > Seems my previous msg was sent to the wrong recipient, just resending.. > > On Fri, Oct 13, 2023 at 7:06 PM Guozhang Wang > wrote: > > > > Thanks Hanyu. I made a pass on the KIP and read through the DISCUSS > > thread. Do not have any comments. +1 > > > > On Fri, Oct 13, 2023 at 9:29 AM Hanyu (Peter) Zheng > > wrote: > > > > > > Hello everyone, > > > > > > I would like to start a vote for KIP-985 that Add reverseRange and > > > reverseAll query over kv-store in IQv2. > > > > > > Sincerely, > > > Hanyu > > > > > > On Fri, Oct 13, 2023 at 9:15 AM Hanyu (Peter) Zheng < > pzh...@confluent.io> > > > wrote: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-985:+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2 > > > > > > > > -- > > > > > > > > [image: Confluent] <https://www.confluent.io> > > > > Hanyu (Peter) Zheng he/him/his > > > > Software Engineer Intern > > > > +1 (213) 431-7193 <+1+(213)+431-7193> > > > > Follow us: [image: Blog] > > > > < > https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog > >[image: > > > > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > > > > <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > > > > <https://slackpass.io/confluentcommunity>[image: YouTube] > > > > <https://youtube.com/confluent> > > > > > > > > [image: Try Confluent Cloud for Free] > > > > < > https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic > > > > > > > > > > > > > > > -- > > > > > > [image: Confluent] <https://www.confluent.io> > > > Hanyu (Peter) Zheng he/him/his > > > Software Engineer Intern > > > +1 (213) 431-7193 <+1+(213)+431-7193> > > > Follow us: [image: Blog] > > > < > https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog > >[image: > > > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > > > <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > > > <https://slackpass.io/confluentcommunity>[image: YouTube] > > > <https://youtube.com/confluent> > > > > > > [image: Try Confluent Cloud for Free] > > > < > https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic > > > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Re: [VOTE] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
According to KIP-968, we added ResultOrder to this KIP and support withAscendingKeys() to the TimestampedRangeQuery. Sincerely, Hanyu On Wed, Nov 8, 2023 at 11:04 AM Hanyu (Peter) Zheng wrote: > Hi all, > > This voting thread has been open for over 72 hours and has received enough > votes. Therefore, the vote will be closed now. > > +3 binding votes > +1 (non-binding) > > KIP-992 has PASSED. > > > Thanks all for your votes > Hanyu > > On Fri, Nov 3, 2023 at 5:10 PM Matthias J. Sax wrote: > >> Thanks for the KIP. >> >> +1 (binding) >> >> >> -Matthias >> >> On 11/3/23 6:08 AM, Lucas Brutschy wrote: >> > Hi Hanyu, >> > >> > Thanks for the KIP! >> > +1 (binding) >> > >> > Cheers >> > Lucas >> > >> > On Thu, Nov 2, 2023 at 10:19 PM Hao Li >> wrote: >> >> >> >> Hi Hanyu, >> >> >> >> Thanks for the KIP! >> >> +1 (non-binding) >> >> >> >> Hao >> >> >> >> On Thu, Nov 2, 2023 at 1:29 PM Bill Bejeck wrote: >> >> >> >>> Hi Hanyu, >> >>> >> >>> Thanks for the KIP this LGTM. >> >>> +1 (binding) >> >>> >> >>> Thanks, >> >>> Bill >> >>> >> >>> >> >>> >> >>> On Wed, Nov 1, 2023 at 1:07 PM Hanyu (Peter) Zheng >> >>> wrote: >> >>> >> >>>> Hello everyone, >> >>>> >> >>>> I would like to start a vote for KIP-992: Proposal to introduce IQv2 >> >>> Query >> >>>> Types: TimestampedKeyQuery and TimestampedRangeQuery. >> >>>> >> >>>> Sincerely, >> >>>> Hanyu >> >>>> >> >>>> On Wed, Nov 1, 2023 at 10:00 AM Hanyu (Peter) Zheng < >> pzh...@confluent.io >> >>>> >> >>>> wrote: >> >>>> >> >>>>> >> >>>>> >> >>>> >> >>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery >> >>>>> >> >>>>> -- >> >>>>> >> >>>>> [image: Confluent] <https://www.confluent.io> >> >>>>> Hanyu (Peter) Zheng he/him/his >> >>>>> Software Engineer Intern >> >>>>> +1 (213) 431-7193 <+1+(213)+431-7193> >> >>>>> Follow us: [image: Blog] >> >>>>> < >> >>>> >> >>> >> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog >> >>>>> [image: >> >>>>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >> >>>>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >> >>>>> <https://slackpass.io/confluentcommunity>[image: YouTube] >> >>>>> <https://youtube.com/confluent> >> >>>>> >> >>>>> [image: Try Confluent Cloud for Free] >> >>>>> < >> >>>> >> >>> >> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic >> >>>>> >> >>>>> >> >>>> >> >>>> >> >>>> -- >> >>>> >> >>>> [image: Confluent] <https://www.confluent.io> >> >>>> Hanyu (Peter) Zheng he/him/his >> >>>> Software Engineer Intern >> >>>> +1 (213) 431-7193 <+1+(213)+431-7193> >> >>>> Follow us: [image: Blog] >> >>>> < >> >>>> >> >>> >> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog >> >>>>> [image: >> >>>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >> >>>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >> >>>> <https://slackpass.io/confluentcommunity>[image: YouTube] >> >>>> <https://youtube.com/confluent> >> >>>> >> >>>> [image: Try Confluent Cloud for Free] >> >>>> < >> >>>> >> >>> >> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic >> >>>>> >> >>>> >> >>> >> > > > -- > > [image: Confluent] <https://www.confluent.io> > Hanyu (Peter) Zheng he/him/his > Software Engineer Intern > +1 (213) 431-7193 <+1+(213)+431-7193> > Follow us: [image: Blog] > <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > <https://slackpass.io/confluentcommunity>[image: YouTube] > <https://youtube.com/confluent> > > [image: Try Confluent Cloud for Free] > <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic> > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Re: [DISCUSS] KIP-997 Support fetch(fromKey, toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery
Thank you Alieh, After discussion with Matthias, we decide use oldTimeFrom and oldTimeTo(these two will Deprecated soom) to implement withWindowStartRange() and use timeFrom and timeTo to implement withAllKey() and withKeyRange(), I will update the KIP. Sincerely, Hanyu On Sun, Dec 3, 2023 at 3:11 PM Alieh Saeedi wrote: > Thanks, Hanyu, for the KIP and all the updates. > I just do not understand the purpose of defining new time ranges > (`newTimeFrom`, `newTimeTo`). Why don't we simply re-use the existing time > range variables? > > Bests, > Alieh > > On Thu, Nov 30, 2023 at 8:34 PM Hanyu (Peter) Zheng > wrote: > > > new KIP link: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-997%3A++update+WindowRangeQuery+and+unify+WindowKeyQuery+and+WindowRangeQuery > > > > On Wed, Nov 29, 2023 at 10:12 PM Hanyu (Peter) Zheng < > pzh...@confluent.io> > > wrote: > > > > > Thank you Bruno, > > > 1. Thank you for the notification. I have updated the ticket link > > > accordingly. > > > 2. Certainly, I'll update the KIP name. Should I initiate a new > > discussion > > > for it, because if I change the name, the link will change. > > > 3. Understood, I will add that to the KIP. > > > 4. I propose we accept both > > > `WindowRangeQuery.withAllKeys().fromTime(time1).toTime(time2)` and > > > `WindowRangeQuery.withKeyRange(key1, > > key2).fromTime(time1).toTime(time2)`, > > > while also reusing the existing `withKey` method. > > > 5. Following a discussion with Matthias, we've decided to defer the > > > implementation of order guarantees to a future KIP. > > > > > > Sincerely, > > > Hanyu > > > > > > On Wed, Nov 29, 2023 at 6:22 AM Bruno Cadonna > > wrote: > > > > > >> Hi, > > >> > > >> Thanks for the updates! > > >> > > >> > > >> 1. > > >> Could you please link the correct ticket in the KIP? > > >> > > >> 2. > > >> Could you please adapt the motivation section and the title to the > > >> updated goal of the KIP? There is no fetch() or fetchAll() method in > the > > >> query class. > > >> > > >> 3. > > >> Could you please add the "// newly added" comment to all parts that > were > > >> newly added? That is methods lowerKeyBound() and upperKeyBound(). > > >> > > >> 4. > > >> We should use a more fluent API as I proposed in my last e-mail: > > >> > > >> Here again > > >> > > >> WindowRangeQuery.withAllKeys().fromTime(time1).toTime(time2); > > >> WindowRangeQuery.withKey(key1).fromTime(time1).toTime(time2); > > >> WindowRangeQuery.withKeyRange(key1, > key2).fromTime(time1).toTime(time2); > > >> > > >> 5. > > >> We should also consider the order of the results similar as we did in > > >> KIP-968. Alternatively, we do not guarantee any order and postpone the > > >> order guarantees to a future KIP. > > >> > > >> > > >> Best, > > >> Bruno > > >> > > >> > > >> > > >> On 11/17/23 3:11 AM, Matthias J. Sax wrote: > > >> > Thanks for the KIP. > > >> > > > >> > Given how `WindowRangeQuery` works right now, it's really time to > > >> > improve it. > > >> > > > >> > > > >> > 1) Agree. It's not clear what will be added right now. I think we > > >> should > > >> > deprecate existing `getKey()` w/o an actually replacement? For > > >> > `getFromKey` and `getToKey` we should actually be `lowerKeyBound()` > > and > > >> > `upperKeyBound()` to align to KIP-969? > > >> > > > >> > Also wondering if we should deprecate existing `withKey()` and > > >> > `withWindowStartRange`? `withKey` only works for SessionStores and > > >> > implements a single-key full-time-range query. Similarly, > > >> > `withWindowStartRange` only works for WindowedStore and implements > an > > >> > all-key time-range query. Thus, both are rather special and it seems > > >> the > > >> > aim of this KIP is to generalize `WindowRangeQuery` to arbitrary > > >> > key-range/time-range queries? > > >> > > > >> > What raises one question about time-range semantics, given that we > > >> query &g
Re: [DISCUSS] KIP-997 Support fetch(fromKey, toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery
new KIP link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-997%3A++update+WindowRangeQuery+and+unify+WindowKeyQuery+and+WindowRangeQuery On Wed, Nov 29, 2023 at 10:12 PM Hanyu (Peter) Zheng wrote: > Thank you Bruno, > 1. Thank you for the notification. I have updated the ticket link > accordingly. > 2. Certainly, I'll update the KIP name. Should I initiate a new discussion > for it, because if I change the name, the link will change. > 3. Understood, I will add that to the KIP. > 4. I propose we accept both > `WindowRangeQuery.withAllKeys().fromTime(time1).toTime(time2)` and > `WindowRangeQuery.withKeyRange(key1, key2).fromTime(time1).toTime(time2)`, > while also reusing the existing `withKey` method. > 5. Following a discussion with Matthias, we've decided to defer the > implementation of order guarantees to a future KIP. > > Sincerely, > Hanyu > > On Wed, Nov 29, 2023 at 6:22 AM Bruno Cadonna wrote: > >> Hi, >> >> Thanks for the updates! >> >> >> 1. >> Could you please link the correct ticket in the KIP? >> >> 2. >> Could you please adapt the motivation section and the title to the >> updated goal of the KIP? There is no fetch() or fetchAll() method in the >> query class. >> >> 3. >> Could you please add the "// newly added" comment to all parts that were >> newly added? That is methods lowerKeyBound() and upperKeyBound(). >> >> 4. >> We should use a more fluent API as I proposed in my last e-mail: >> >> Here again >> >> WindowRangeQuery.withAllKeys().fromTime(time1).toTime(time2); >> WindowRangeQuery.withKey(key1).fromTime(time1).toTime(time2); >> WindowRangeQuery.withKeyRange(key1, key2).fromTime(time1).toTime(time2); >> >> 5. >> We should also consider the order of the results similar as we did in >> KIP-968. Alternatively, we do not guarantee any order and postpone the >> order guarantees to a future KIP. >> >> >> Best, >> Bruno >> >> >> >> On 11/17/23 3:11 AM, Matthias J. Sax wrote: >> > Thanks for the KIP. >> > >> > Given how `WindowRangeQuery` works right now, it's really time to >> > improve it. >> > >> > >> > 1) Agree. It's not clear what will be added right now. I think we >> should >> > deprecate existing `getKey()` w/o an actually replacement? For >> > `getFromKey` and `getToKey` we should actually be `lowerKeyBound()` and >> > `upperKeyBound()` to align to KIP-969? >> > >> > Also wondering if we should deprecate existing `withKey()` and >> > `withWindowStartRange`? `withKey` only works for SessionStores and >> > implements a single-key full-time-range query. Similarly, >> > `withWindowStartRange` only works for WindowedStore and implements an >> > all-key time-range query. Thus, both are rather special and it seems >> the >> > aim of this KIP is to generalize `WindowRangeQuery` to arbitrary >> > key-range/time-range queries? >> > >> > What raises one question about time-range semantics, given that we >> query >> > windows with different semantics. >> > >> > - The current `WindowStore` semantics used for >> > `WindowRangeQuery#withWindowStartRange` is considering only the window >> > start time, ie, the window-start time must fall into the query >> > time-range to be returned. >> > >> > - In contrast, `SessionStore` time ranges base on `findSession` use >> > earliest-session-end-time and latest-session-end-time and thus >> implement >> > an "window-bounds / search-time-range overlap query". >> > >> > Is there any concern about semantic differences? I would also be >> > possible to use the same semantics for both query types, and maybe even >> > let the user pick with semantics they want (let users chose might >> > actually be the best thing to do)? -- We can also do this >> incrementally, >> > and limit the scope of this KIP (or keep the full KIP scope but >> > implement it incrementally only) >> > >> > Btw: I think we should not add any ordering at this point, and >> > explicitly state that no ordering is guarantee whatsoever at this point. >> > >> > >> > >> > 2) Agreed. We should deprecate `getFromTime` and `getToTime` and add >> new >> > method `fromTime` and `toTime`. >> > >> > >> > >> > 3) About the API. If we move forward with general key-range/time-range >> I >> > agree that a more modular
Re: [DISCUSS] KIP-997 Support fetch(fromKey, toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery
; > > > > > > > -Matthias > > > > On 11/16/23 1:19 AM, Bruno Cadonna wrote: > >> Hi Hanyu, > >> > >> Thanks for the KIP! > >> > >> 1) > >> Could you please mark the pieces that you want to add to the API in > >> the code listing in the KIP? You can add a comment like "// newly > >> added" or similar. That would make reading the KIP a bit easier > >> because one does not need to compare your code with the code in the > >> current codebase. > >> > >> 2) > >> Could you -- as a side cleanup -- also change the getters to not use > >> the get-prefix anymore, please? That was apparently an oversight when > >> those methods were added. Although the API is marked as Evolving, I > >> think we should still deprecate the getX() methods, since it does not > >> cost us anything. > >> > >> 3) > >> I propose to make the API a bit more fluent. For example, something like > >> > >> WindowRangeQuery.withKey(key).fromTime(t1).toTime(t2) > >> > >> and > >> > >> WindowRangeQuery.withAllKeys().fromTime(t1).toTime(t2) > >> > >> and > >> > >> WindowRangeQuery.withKeyRange(key1, key2).fromTime(t1).toTime(t2) > >> > >> and maybe even in addition to the above add also the option to start > >> with the time range > >> > >> WindowRangeQuery.withWindowStartRange(t1, t2).fromKey(key1).toKey(key2) > >> > >> > >> 4) > >> Could you also add some usage examples? Alieh did quite a nice job > >> regarding usage examples in KIP-986. > >> > >> > >> Best, > >> Bruno > >> > >> On 11/8/23 8:02 PM, Hanyu (Peter) Zheng wrote: > >>> Hello everyone, > >>> > >>> I would like to start the discussion for KIP-997: Support > fetch(fromKey, > >>> toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and > >>> WindowRangeQuery > >>> The KIP can be found here: > >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-997%3A++Support+fetch%28fromKey%2C+toKey%2C+from%2C+to%29+to+WindowRangeQuery+and+unify+WindowKeyQuery+and+WindowRangeQuery > >>> > >>> Any suggestions are more than welcome. > >>> > >>> Many thanks, > >>> Hanyu > >>> > >>> On Wed, Nov 8, 2023 at 10:38 AM Hanyu (Peter) Zheng > >>> > >>> wrote: > >>> > >>>> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-997%3A++Support+fetch%28fromKey%2C+toKey%2C+from%2C+to%29+to+WindowRangeQuery+and+unify+WindowKeyQuery+and+WindowRangeQuery > >>>> > >>>> -- > >>>> > >>>> [image: Confluent] <https://www.confluent.io> > >>>> Hanyu (Peter) Zheng he/him/his > >>>> Software Engineer Intern > >>>> +1 (213) 431-7193 <+1+(213)+431-7193> > >>>> Follow us: [image: Blog] > >>>> < > https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog > >[image: > >>>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > >>>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > >>>> <https://slackpass.io/confluentcommunity>[image: YouTube] > >>>> <https://youtube.com/confluent> > >>>> > >>>> [image: Try Confluent Cloud for Free] > >>>> < > https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic > > > >>>> > >>> > >>> > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
[VOTE] KIP-997 Support fetch(fromKey, toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery
https://cwiki.apache.org/confluence/display/KAFKA/KIP-997%3A++Support+fetch%28fromKey%2C+toKey%2C+from%2C+to%29+to+WindowRangeQuery+and+unify+WindowKeyQuery+and+WindowRangeQuery -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Re: [VOTE] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
Hi all, This voting thread has been open for over 72 hours and has received enough votes. Therefore, the vote will be closed now. +3 binding votes +1 (non-binding) KIP-992 has PASSED. Thanks all for your votes Hanyu On Fri, Nov 3, 2023 at 5:10 PM Matthias J. Sax wrote: > Thanks for the KIP. > > +1 (binding) > > > -Matthias > > On 11/3/23 6:08 AM, Lucas Brutschy wrote: > > Hi Hanyu, > > > > Thanks for the KIP! > > +1 (binding) > > > > Cheers > > Lucas > > > > On Thu, Nov 2, 2023 at 10:19 PM Hao Li wrote: > >> > >> Hi Hanyu, > >> > >> Thanks for the KIP! > >> +1 (non-binding) > >> > >> Hao > >> > >> On Thu, Nov 2, 2023 at 1:29 PM Bill Bejeck wrote: > >> > >>> Hi Hanyu, > >>> > >>> Thanks for the KIP this LGTM. > >>> +1 (binding) > >>> > >>> Thanks, > >>> Bill > >>> > >>> > >>> > >>> On Wed, Nov 1, 2023 at 1:07 PM Hanyu (Peter) Zheng > >>> wrote: > >>> > >>>> Hello everyone, > >>>> > >>>> I would like to start a vote for KIP-992: Proposal to introduce IQv2 > >>> Query > >>>> Types: TimestampedKeyQuery and TimestampedRangeQuery. > >>>> > >>>> Sincerely, > >>>> Hanyu > >>>> > >>>> On Wed, Nov 1, 2023 at 10:00 AM Hanyu (Peter) Zheng < > pzh...@confluent.io > >>>> > >>>> wrote: > >>>> > >>>>> > >>>>> > >>>> > >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery > >>>>> > >>>>> -- > >>>>> > >>>>> [image: Confluent] <https://www.confluent.io> > >>>>> Hanyu (Peter) Zheng he/him/his > >>>>> Software Engineer Intern > >>>>> +1 (213) 431-7193 <+1+(213)+431-7193> > >>>>> Follow us: [image: Blog] > >>>>> < > >>>> > >>> > https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog > >>>>> [image: > >>>>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > >>>>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > >>>>> <https://slackpass.io/confluentcommunity>[image: YouTube] > >>>>> <https://youtube.com/confluent> > >>>>> > >>>>> [image: Try Confluent Cloud for Free] > >>>>> < > >>>> > >>> > https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic > >>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> > >>>> [image: Confluent] <https://www.confluent.io> > >>>> Hanyu (Peter) Zheng he/him/his > >>>> Software Engineer Intern > >>>> +1 (213) 431-7193 <+1+(213)+431-7193> > >>>> Follow us: [image: Blog] > >>>> < > >>>> > >>> > https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog > >>>>> [image: > >>>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > >>>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > >>>> <https://slackpass.io/confluentcommunity>[image: YouTube] > >>>> <https://youtube.com/confluent> > >>>> > >>>> [image: Try Confluent Cloud for Free] > >>>> < > >>>> > >>> > https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic > >>>>> > >>>> > >>> > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Re: [DISCUSS] KIP-997 Support fetch(fromKey, toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery
Hello everyone, I would like to start the discussion for KIP-997: Support fetch(fromKey, toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-997%3A++Support+fetch%28fromKey%2C+toKey%2C+from%2C+to%29+to+WindowRangeQuery+and+unify+WindowKeyQuery+and+WindowRangeQuery Any suggestions are more than welcome. Many thanks, Hanyu On Wed, Nov 8, 2023 at 10:38 AM Hanyu (Peter) Zheng wrote: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-997%3A++Support+fetch%28fromKey%2C+toKey%2C+from%2C+to%29+to+WindowRangeQuery+and+unify+WindowKeyQuery+and+WindowRangeQuery > > -- > > [image: Confluent] <https://www.confluent.io> > Hanyu (Peter) Zheng he/him/his > Software Engineer Intern > +1 (213) 431-7193 <+1+(213)+431-7193> > Follow us: [image: Blog] > <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > <https://slackpass.io/confluentcommunity>[image: YouTube] > <https://youtube.com/confluent> > > [image: Try Confluent Cloud for Free] > <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic> > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
[DISCUSS] KIP-997 Support fetch(fromKey, toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery
https://cwiki.apache.org/confluence/display/KAFKA/KIP-997%3A++Support+fetch%28fromKey%2C+toKey%2C+from%2C+to%29+to+WindowRangeQuery+and+unify+WindowKeyQuery+and+WindowRangeQuery -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Re: [VOTE] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
Hello everyone, I would like to start a vote for KIP-992: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery. Sincerely, Hanyu On Wed, Nov 1, 2023 at 10:00 AM Hanyu (Peter) Zheng wrote: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery > > -- > > [image: Confluent] <https://www.confluent.io> > Hanyu (Peter) Zheng he/him/his > Software Engineer Intern > +1 (213) 431-7193 <+1+(213)+431-7193> > Follow us: [image: Blog] > <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > <https://slackpass.io/confluentcommunity>[image: YouTube] > <https://youtube.com/confluent> > > [image: Try Confluent Cloud for Free] > <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic> > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
[VOTE] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Re: [DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
Hi, Hao, For TimestampedKeyQuery, it only returns the value of the key, and the value should be ValueAndTimestamp. If you want to get an iterator of `ValueAndTimestamp`, you can use TimestampedRangeQuery. Sincerely, Hanyu On Mon, Oct 30, 2023 at 1:42 PM Hanyu (Peter) Zheng wrote: > Hi, Matthias, > Now, if we use TimestampedKeyQuery to query kv-store, it will throw a > exception, > the exception like this: > java.lang.IllegalArgumentException: Cannot get result for failed query. > > Sincerely, > Hanyu > > On Thu, Oct 26, 2023 at 3:45 PM Hao Li wrote: > >> Thanks for the KIP Hanyu! One question: why not return an iterator of >> `ValueAndTimestamp` for `TimestampedKeyQuery`? I suppose for a >> ts-kv-store, there could be multiple timestamps associated with the same >> key? >> >> Hao >> >> On Thu, Oct 26, 2023 at 10:23 AM Matthias J. Sax >> wrote: >> >> > Would we really get a ClassCastException? >> > >> > From my understanding, the store would reject the query as unsupported >> > and thus the returned `QueryResult` object would have it's internal flag >> > set to indicate the failure, but no exception would be thrown directly? >> > >> > (Of course, there might be an exception thrown to the user if they don't >> > check `isSuccess()` flag but call `getResult()` directly.) >> > >> > >> > -Matthias >> > >> > On 10/25/23 8:55 AM, Hanyu (Peter) Zheng wrote: >> > > Hi, Bill, >> > > Thank you for your reply. Yes, now, if a user executes a timestamped >> > query >> > > against a non-timestamped store, It will throw ClassCastException. >> > > If a user uses KeyQuery to query kv-store or ts-kv-store, it always >> > return >> > > V. If a user uses TimestampedKeyQuery to query kv-store, it will >> throw a >> > > exception, so TimestampedKeyQuery query can only query ts-kv-store and >> > > return ValueAndTimestamp object in the end. >> > > >> > > Sincerely, >> > > Hanyu >> > > >> > > On Wed, Oct 25, 2023 at 8:51 AM Hanyu (Peter) Zheng < >> pzh...@confluent.io >> > > >> > > wrote: >> > > >> > >> Thank you Lucas, >> > >> >> > >> I will fix the capitalization. >> > >> When a user executes a timestamped query against a non-timestamped >> > store, >> > >> It will throw ClassCastException. >> > >> >> > >> Sincerely, >> > >> Hanyu >> > >> >> > >> On Tue, Oct 24, 2023 at 1:36 AM Lucas Brutschy >> > >> wrote: >> > >> >> > >>> Hi Hanyu, >> > >>> >> > >>> reading the KIP, I was wondering the same thing as Bill. >> > >>> >> > >>> Other than that, this looks good to me. Thanks for KIP. >> > >>> >> > >>> nit: you have method names `LowerBound` and `UpperBound`, where you >> > >>> probably want to fix the capitalization. >> > >>> >> > >>> Cheers, >> > >>> Lucas >> > >>> >> > >>> On Mon, Oct 23, 2023 at 5:46 PM Bill Bejeck >> wrote: >> > >>>> >> > >>>> Hey Hanyu, >> > >>>> >> > >>>> Thanks for the KIP, it's a welcomed addition. >> > >>>> Overall, the KIP looks good to me, I just have one comment. >> > >>>> >> > >>>> Can you discuss the expected behavior when a user executes a >> > timestamped >> > >>>> query against a non-timestamped store? I think it should throw an >> > >>>> exception vs. using some default value. >> > >>>> If it's the case that Kafka Stream wraps all stores in a >> > >>>> `TimestampAndValue` store and returning a plain `V` or a >> > >>>> `TimestampAndValue` object depends on the query type, then it >> would >> > >>> be >> > >>>> good to add those details to the KIP. >> > >>>> >> > >>>> Thanks, >> > >>>> Bill >> > >>>> >> > >>>> >> > >>>> >> > >>>> On Fri, Oct 20, 2023 at 5:07 PM Hanyu (Peter) Zheng >> > >>>> wrote: >> > >>>> >> > >>>>> Thank you Matthias,
Re: [DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
Hi, Matthias, Now, if we use TimestampedKeyQuery to query kv-store, it will throw a exception, the exception like this: java.lang.IllegalArgumentException: Cannot get result for failed query. Sincerely, Hanyu On Thu, Oct 26, 2023 at 3:45 PM Hao Li wrote: > Thanks for the KIP Hanyu! One question: why not return an iterator of > `ValueAndTimestamp` for `TimestampedKeyQuery`? I suppose for a > ts-kv-store, there could be multiple timestamps associated with the same > key? > > Hao > > On Thu, Oct 26, 2023 at 10:23 AM Matthias J. Sax wrote: > > > Would we really get a ClassCastException? > > > > From my understanding, the store would reject the query as unsupported > > and thus the returned `QueryResult` object would have it's internal flag > > set to indicate the failure, but no exception would be thrown directly? > > > > (Of course, there might be an exception thrown to the user if they don't > > check `isSuccess()` flag but call `getResult()` directly.) > > > > > > -Matthias > > > > On 10/25/23 8:55 AM, Hanyu (Peter) Zheng wrote: > > > Hi, Bill, > > > Thank you for your reply. Yes, now, if a user executes a timestamped > > query > > > against a non-timestamped store, It will throw ClassCastException. > > > If a user uses KeyQuery to query kv-store or ts-kv-store, it always > > return > > > V. If a user uses TimestampedKeyQuery to query kv-store, it will > throw a > > > exception, so TimestampedKeyQuery query can only query ts-kv-store and > > > return ValueAndTimestamp object in the end. > > > > > > Sincerely, > > > Hanyu > > > > > > On Wed, Oct 25, 2023 at 8:51 AM Hanyu (Peter) Zheng < > pzh...@confluent.io > > > > > > wrote: > > > > > >> Thank you Lucas, > > >> > > >> I will fix the capitalization. > > >> When a user executes a timestamped query against a non-timestamped > > store, > > >> It will throw ClassCastException. > > >> > > >> Sincerely, > > >> Hanyu > > >> > > >> On Tue, Oct 24, 2023 at 1:36 AM Lucas Brutschy > > >> wrote: > > >> > > >>> Hi Hanyu, > > >>> > > >>> reading the KIP, I was wondering the same thing as Bill. > > >>> > > >>> Other than that, this looks good to me. Thanks for KIP. > > >>> > > >>> nit: you have method names `LowerBound` and `UpperBound`, where you > > >>> probably want to fix the capitalization. > > >>> > > >>> Cheers, > > >>> Lucas > > >>> > > >>> On Mon, Oct 23, 2023 at 5:46 PM Bill Bejeck > wrote: > > >>>> > > >>>> Hey Hanyu, > > >>>> > > >>>> Thanks for the KIP, it's a welcomed addition. > > >>>> Overall, the KIP looks good to me, I just have one comment. > > >>>> > > >>>> Can you discuss the expected behavior when a user executes a > > timestamped > > >>>> query against a non-timestamped store? I think it should throw an > > >>>> exception vs. using some default value. > > >>>> If it's the case that Kafka Stream wraps all stores in a > > >>>> `TimestampAndValue` store and returning a plain `V` or a > > >>>> `TimestampAndValue` object depends on the query type, then it > would > > >>> be > > >>>> good to add those details to the KIP. > > >>>> > > >>>> Thanks, > > >>>> Bill > > >>>> > > >>>> > > >>>> > > >>>> On Fri, Oct 20, 2023 at 5:07 PM Hanyu (Peter) Zheng > > >>>> wrote: > > >>>> > > >>>>> Thank you Matthias, > > >>>>> > > >>>>> I will modify the KIP to eliminate this restriction. > > >>>>> > > >>>>> Sincerely, > > >>>>> Hanyu > > >>>>> > > >>>>> On Fri, Oct 20, 2023 at 2:04 PM Hanyu (Peter) Zheng < > > >>> pzh...@confluent.io> > > >>>>> wrote: > > >>>>> > > >>>>>> Thank you Alieh, > > >>>>>> > > >>>>>> In these two new query types, I will remove 'get' from all getter > > >>> method > > >>>>>> names. >
Re: [DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
Hi, Bill, Thank you for your reply. Yes, now, if a user executes a timestamped query against a non-timestamped store, It will throw ClassCastException. If a user uses KeyQuery to query kv-store or ts-kv-store, it always return V. If a user uses TimestampedKeyQuery to query kv-store, it will throw a exception, so TimestampedKeyQuery query can only query ts-kv-store and return ValueAndTimestamp object in the end. Sincerely, Hanyu On Wed, Oct 25, 2023 at 8:51 AM Hanyu (Peter) Zheng wrote: > Thank you Lucas, > > I will fix the capitalization. > When a user executes a timestamped query against a non-timestamped store, > It will throw ClassCastException. > > Sincerely, > Hanyu > > On Tue, Oct 24, 2023 at 1:36 AM Lucas Brutschy > wrote: > >> Hi Hanyu, >> >> reading the KIP, I was wondering the same thing as Bill. >> >> Other than that, this looks good to me. Thanks for KIP. >> >> nit: you have method names `LowerBound` and `UpperBound`, where you >> probably want to fix the capitalization. >> >> Cheers, >> Lucas >> >> On Mon, Oct 23, 2023 at 5:46 PM Bill Bejeck wrote: >> > >> > Hey Hanyu, >> > >> > Thanks for the KIP, it's a welcomed addition. >> > Overall, the KIP looks good to me, I just have one comment. >> > >> > Can you discuss the expected behavior when a user executes a timestamped >> > query against a non-timestamped store? I think it should throw an >> > exception vs. using some default value. >> > If it's the case that Kafka Stream wraps all stores in a >> > `TimestampAndValue` store and returning a plain `V` or a >> > `TimestampAndValue` object depends on the query type, then it would >> be >> > good to add those details to the KIP. >> > >> > Thanks, >> > Bill >> > >> > >> > >> > On Fri, Oct 20, 2023 at 5:07 PM Hanyu (Peter) Zheng >> > wrote: >> > >> > > Thank you Matthias, >> > > >> > > I will modify the KIP to eliminate this restriction. >> > > >> > > Sincerely, >> > > Hanyu >> > > >> > > On Fri, Oct 20, 2023 at 2:04 PM Hanyu (Peter) Zheng < >> pzh...@confluent.io> >> > > wrote: >> > > >> > > > Thank you Alieh, >> > > > >> > > > In these two new query types, I will remove 'get' from all getter >> method >> > > > names. >> > > > >> > > > Sincerely, >> > > > Hanyu >> > > > >> > > > On Fri, Oct 20, 2023 at 10:40 AM Matthias J. Sax >> > > wrote: >> > > > >> > > >> Thanks for the KIP Hanyu, >> > > >> >> > > >> One questions: >> > > >> >> > > >> > To address this inconsistency, we propose that KeyQuery should >> be >> > > >> restricted to querying kv-stores only, ensuring that it always >> returns >> > > a >> > > >> plain V type, making the behavior of the aforementioned code more >> > > >> predictable. Similarly, RangeQuery should be dedicated to querying >> > > >> kv-stores , consistently returning only the plain V . >> > > >> >> > > >> Why do you want to restrict `KeyQuery` and `RangeQuery` to >> kv-stores? I >> > > >> think it would be possible to still allow both queries for >> ts-kv-stores, >> > > >> but change the implementation to return "plain V" instead of >> > > >> `ValueAndTimestamp`, ie, the implementation would automatically >> > > >> unwrap the value. >> > > >> >> > > >> >> > > >> >> > > >> -Matthias >> > > >> >> > > >> On 10/20/23 2:32 AM, Alieh Saeedi wrote: >> > > >> > Hey Hanyu, >> > > >> > >> > > >> > Thanks for the KIP. It seems good to me. >> > > >> > Just one point: AFAIK, we are going to remove "get" from the >> name of >> > > all >> > > >> > getter methods. >> > > >> > >> > > >> > Cheers, >> > > >> > Alieh >> > > >> > >> > > >> > On Thu, Oct 19, 2023 at 5:44 PM Hanyu (Peter) Zheng >> > > >> > wrote: >> > > >> > >> > > >> >> Hello everyone, >
Re: [DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
Thank you Lucas, I will fix the capitalization. When a user executes a timestamped query against a non-timestamped store, It will throw ClassCastException. Sincerely, Hanyu On Tue, Oct 24, 2023 at 1:36 AM Lucas Brutschy wrote: > Hi Hanyu, > > reading the KIP, I was wondering the same thing as Bill. > > Other than that, this looks good to me. Thanks for KIP. > > nit: you have method names `LowerBound` and `UpperBound`, where you > probably want to fix the capitalization. > > Cheers, > Lucas > > On Mon, Oct 23, 2023 at 5:46 PM Bill Bejeck wrote: > > > > Hey Hanyu, > > > > Thanks for the KIP, it's a welcomed addition. > > Overall, the KIP looks good to me, I just have one comment. > > > > Can you discuss the expected behavior when a user executes a timestamped > > query against a non-timestamped store? I think it should throw an > > exception vs. using some default value. > > If it's the case that Kafka Stream wraps all stores in a > > `TimestampAndValue` store and returning a plain `V` or a > > `TimestampAndValue` object depends on the query type, then it would be > > good to add those details to the KIP. > > > > Thanks, > > Bill > > > > > > > > On Fri, Oct 20, 2023 at 5:07 PM Hanyu (Peter) Zheng > > wrote: > > > > > Thank you Matthias, > > > > > > I will modify the KIP to eliminate this restriction. > > > > > > Sincerely, > > > Hanyu > > > > > > On Fri, Oct 20, 2023 at 2:04 PM Hanyu (Peter) Zheng < > pzh...@confluent.io> > > > wrote: > > > > > > > Thank you Alieh, > > > > > > > > In these two new query types, I will remove 'get' from all getter > method > > > > names. > > > > > > > > Sincerely, > > > > Hanyu > > > > > > > > On Fri, Oct 20, 2023 at 10:40 AM Matthias J. Sax > > > wrote: > > > > > > > >> Thanks for the KIP Hanyu, > > > >> > > > >> One questions: > > > >> > > > >> > To address this inconsistency, we propose that KeyQuery should be > > > >> restricted to querying kv-stores only, ensuring that it always > returns > > > a > > > >> plain V type, making the behavior of the aforementioned code more > > > >> predictable. Similarly, RangeQuery should be dedicated to querying > > > >> kv-stores , consistently returning only the plain V . > > > >> > > > >> Why do you want to restrict `KeyQuery` and `RangeQuery` to > kv-stores? I > > > >> think it would be possible to still allow both queries for > ts-kv-stores, > > > >> but change the implementation to return "plain V" instead of > > > >> `ValueAndTimestamp`, ie, the implementation would automatically > > > >> unwrap the value. > > > >> > > > >> > > > >> > > > >> -Matthias > > > >> > > > >> On 10/20/23 2:32 AM, Alieh Saeedi wrote: > > > >> > Hey Hanyu, > > > >> > > > > >> > Thanks for the KIP. It seems good to me. > > > >> > Just one point: AFAIK, we are going to remove "get" from the name > of > > > all > > > >> > getter methods. > > > >> > > > > >> > Cheers, > > > >> > Alieh > > > >> > > > > >> > On Thu, Oct 19, 2023 at 5:44 PM Hanyu (Peter) Zheng > > > >> > wrote: > > > >> > > > > >> >> Hello everyone, > > > >> >> > > > >> >> I would like to start the discussion for KIP-992: Proposal to > > > introduce > > > >> >> IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery > > > >> >> > > > >> >> The KIP can be found here: > > > >> >> > > > >> >> > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery > > > >> >> > > > >> >> Any suggestions are more than welcome. > > > >> >> > > > >> >> Many thanks, > > > >> >> Hanyu > > > >> >> > > > >> >> On Thu, Oct 19, 2023 at 8:17 AM Hanyu (Peter) Zheng < > > &g
Re: [DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
Thank you Matthias, I will modify the KIP to eliminate this restriction. Sincerely, Hanyu On Fri, Oct 20, 2023 at 2:04 PM Hanyu (Peter) Zheng wrote: > Thank you Alieh, > > In these two new query types, I will remove 'get' from all getter method > names. > > Sincerely, > Hanyu > > On Fri, Oct 20, 2023 at 10:40 AM Matthias J. Sax wrote: > >> Thanks for the KIP Hanyu, >> >> One questions: >> >> > To address this inconsistency, we propose that KeyQuery should be >> restricted to querying kv-stores only, ensuring that it always returns a >> plain V type, making the behavior of the aforementioned code more >> predictable. Similarly, RangeQuery should be dedicated to querying >> kv-stores , consistently returning only the plain V . >> >> Why do you want to restrict `KeyQuery` and `RangeQuery` to kv-stores? I >> think it would be possible to still allow both queries for ts-kv-stores, >> but change the implementation to return "plain V" instead of >> `ValueAndTimestamp`, ie, the implementation would automatically >> unwrap the value. >> >> >> >> -Matthias >> >> On 10/20/23 2:32 AM, Alieh Saeedi wrote: >> > Hey Hanyu, >> > >> > Thanks for the KIP. It seems good to me. >> > Just one point: AFAIK, we are going to remove "get" from the name of all >> > getter methods. >> > >> > Cheers, >> > Alieh >> > >> > On Thu, Oct 19, 2023 at 5:44 PM Hanyu (Peter) Zheng >> > wrote: >> > >> >> Hello everyone, >> >> >> >> I would like to start the discussion for KIP-992: Proposal to introduce >> >> IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery >> >> >> >> The KIP can be found here: >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery >> >> >> >> Any suggestions are more than welcome. >> >> >> >> Many thanks, >> >> Hanyu >> >> >> >> On Thu, Oct 19, 2023 at 8:17 AM Hanyu (Peter) Zheng < >> pzh...@confluent.io> >> >> wrote: >> >> >> >>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery >> >>> >> >>> -- >> >>> >> >>> [image: Confluent] <https://www.confluent.io> >> >>> Hanyu (Peter) Zheng he/him/his >> >>> Software Engineer Intern >> >>> +1 (213) 431-7193 <+1+(213)+431-7193> >> >>> Follow us: [image: Blog] >> >>> < >> >> >> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog >> >>> [image: >> >>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >> >>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >> >>> <https://slackpass.io/confluentcommunity>[image: YouTube] >> >>> <https://youtube.com/confluent> >> >>> >> >>> [image: Try Confluent Cloud for Free] >> >>> < >> >> >> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic >> >>> >> >>> >> >> >> >> >> >> -- >> >> >> >> [image: Confluent] <https://www.confluent.io> >> >> Hanyu (Peter) Zheng he/him/his >> >> Software Engineer Intern >> >> +1 (213) 431-7193 <+1+(213)+431-7193> >> >> Follow us: [image: Blog] >> >> < >> >> >> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog >> >>> [image: >> >> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >> >> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >> >> <https://slackpass.io/confluentcommunity>[image: YouTube] >> >> <https://youtube.com/confluent> >> >> >> >> [image: Try Confluent Cloud for Free] >> >> < >> >> >> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic >> >>> >> >> >&
Re: [DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
Thank you Alieh, In these two new query types, I will remove 'get' from all getter method names. Sincerely, Hanyu On Fri, Oct 20, 2023 at 10:40 AM Matthias J. Sax wrote: > Thanks for the KIP Hanyu, > > One questions: > > > To address this inconsistency, we propose that KeyQuery should be > restricted to querying kv-stores only, ensuring that it always returns a > plain V type, making the behavior of the aforementioned code more > predictable. Similarly, RangeQuery should be dedicated to querying > kv-stores , consistently returning only the plain V . > > Why do you want to restrict `KeyQuery` and `RangeQuery` to kv-stores? I > think it would be possible to still allow both queries for ts-kv-stores, > but change the implementation to return "plain V" instead of > `ValueAndTimestamp`, ie, the implementation would automatically > unwrap the value. > > > > -Matthias > > On 10/20/23 2:32 AM, Alieh Saeedi wrote: > > Hey Hanyu, > > > > Thanks for the KIP. It seems good to me. > > Just one point: AFAIK, we are going to remove "get" from the name of all > > getter methods. > > > > Cheers, > > Alieh > > > > On Thu, Oct 19, 2023 at 5:44 PM Hanyu (Peter) Zheng > > wrote: > > > >> Hello everyone, > >> > >> I would like to start the discussion for KIP-992: Proposal to introduce > >> IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery > >> > >> The KIP can be found here: > >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery > >> > >> Any suggestions are more than welcome. > >> > >> Many thanks, > >> Hanyu > >> > >> On Thu, Oct 19, 2023 at 8:17 AM Hanyu (Peter) Zheng < > pzh...@confluent.io> > >> wrote: > >> > >>> > >>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery > >>> > >>> -- > >>> > >>> [image: Confluent] <https://www.confluent.io> > >>> Hanyu (Peter) Zheng he/him/his > >>> Software Engineer Intern > >>> +1 (213) 431-7193 <+1+(213)+431-7193> > >>> Follow us: [image: Blog] > >>> < > >> > https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog > >>> [image: > >>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > >>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > >>> <https://slackpass.io/confluentcommunity>[image: YouTube] > >>> <https://youtube.com/confluent> > >>> > >>> [image: Try Confluent Cloud for Free] > >>> < > >> > https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic > >>> > >>> > >> > >> > >> -- > >> > >> [image: Confluent] <https://www.confluent.io> > >> Hanyu (Peter) Zheng he/him/his > >> Software Engineer Intern > >> +1 (213) 431-7193 <+1+(213)+431-7193> > >> Follow us: [image: Blog] > >> < > >> > https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog > >>> [image: > >> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > >> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > >> <https://slackpass.io/confluentcommunity>[image: YouTube] > >> <https://youtube.com/confluent> > >> > >> [image: Try Confluent Cloud for Free] > >> < > >> > https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic > >>> > >> > > > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Re: [DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
Hello everyone, I would like to start the discussion for KIP-992: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery Any suggestions are more than welcome. Many thanks, Hanyu On Thu, Oct 19, 2023 at 8:17 AM Hanyu (Peter) Zheng wrote: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery > > -- > > [image: Confluent] <https://www.confluent.io> > Hanyu (Peter) Zheng he/him/his > Software Engineer Intern > +1 (213) 431-7193 <+1+(213)+431-7193> > Follow us: [image: Blog] > <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > <https://slackpass.io/confluentcommunity>[image: YouTube] > <https://youtube.com/confluent> > > [image: Try Confluent Cloud for Free] > <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic> > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
[DISCUSS] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Re: [VOTE] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Hi all, This voting thread has been open for over 72 hours and has received enough votes. Therefore, the vote will be closed now. +3 binding votes KIP-985 has PASSED. Thanks all for your votes Hanyu On Tue, Oct 17, 2023 at 8:25 AM Walker Carlson wrote: > +1 (binding_ > > Thanks! > > On Tue, Oct 17, 2023 at 3:22 AM Lucas Brutschy > wrote: > > > +1 (binding) > > > > Thanks for the KIP! > > > > On Tue, Oct 17, 2023 at 2:31 AM Matthias J. Sax > wrote: > > > > > > +1 (binding) > > > > > > > > > On 10/13/23 9:24 AM, Hanyu (Peter) Zheng wrote: > > > > Hello everyone, > > > > > > > > I would like to start a vote for KIP-985 that Add reverseRange and > > > > reverseAll query over kv-store in IQv2. > > > > > > > > Sincerely, > > > > Hanyu > > > > > > > > On Fri, Oct 13, 2023 at 9:15 AM Hanyu (Peter) Zheng < > > pzh...@confluent.io> > > > > wrote: > > > > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-985:+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2 > > > >> > > > >> -- > > > >> > > > >> [image: Confluent] <https://www.confluent.io> > > > >> Hanyu (Peter) Zheng he/him/his > > > >> Software Engineer Intern > > > >> +1 (213) 431-7193 <+1+(213)+431-7193> > > > >> Follow us: [image: Blog] > > > >> < > > > https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog > > >[image: > > > >> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > > > >> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > > > >> <https://slackpass.io/confluentcommunity>[image: YouTube] > > > >> <https://youtube.com/confluent> > > > >> > > > >> [image: Try Confluent Cloud for Free] > > > >> < > > > https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic > > > > > > >> > > > > > > > > > > > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
If no, I would also propose to make it explicit in the JavaDocs. > In > >>> addition, it raises the question if a method `forwardRange()` (for the > >>> lack of a better idea about a name right now) is actually missing to > >>> provide such a contract? > >>> > >>> > >>> Of course, we always depend on the serialization format for order, and > >>> if users need "logical order" they need to ensure to use a > serialization > >>> format that align byte[]-lexicographical order to logical order. But > for > >>> the scope of this work, I would not even try to open this can of > worms... > >>> > >>> > >>> > >>> > >>> Looking into `RangeQuery` the JavaDocs don't say anything about order. > >>> Thus, `RangeQuery#range()` could actually also be implemented by > calling > >>> `reverseRange()` without violating the contract as it seems. A > hash-base > >>> store could also implement it, without the need to explicitly sort... > >>> > >>> What brings be back to my original though about having three types of > >>> results for `Range` > >>>- no ordering guarantee > >>>- ascending (we would only give byte[]-lexicographical order) > >>>- descending (we would only give byte[]-lexicographical order) > >>> > >>> Again, I actually believe that the original intent of RangeQuery was to > >>> inherit the ascending order of `ReadOnlyKeyValueStore#range()`... > Please > >>> keep me honest about it. On the other hand, both APIs seems to be > >>> independent enough to not couple them... -- this could actually be a > >>> step into the right direction and would follow the underlying idea of > >>> IQv2 to begin with: decouple semantics for the store interfaces from > the > >>> query types and semantics... > >>> > >>> > >>> OR: we actually say that `RangeQuery#range` did implicitly inherit the > >>> (non explicit) "ascending byte[]-lexicographical" order of the > >>> underlying `ReadOnlyKeyValueStore`, and we just need to update the > >>> (Java)Docs to make it explicit. -- But it might go against the idea of > >>> IQv2 as stated above. > >>> > >>> > >>> Furthermore, the consequence would be, that a potential custom > >>> hash-based store, would need to do extra work to `range()` to do the > >>> sorting (or of course might reject the query as "not supported"). -- Of > >>> course, a hash-based store would still need to do extract work to > >>> implement `ReadOnlyKeyValueStore#(reverse)Range()` correctly (or also > >>> throw an `UnsupportedOperationException`... -- However, if we keep > store > >>> interface and query interface independent as intended by IQv2, we would > >>> allow a hash-based store to implement `RangeQuery#range()` even if the > >>> store does not support `ReadOnlyKeyValueStore#range()` (or only with > >>> additional sorting cost); such a decoupling sounds like an improvement > >>> to me. > >>> > >>> > >>> > >>> Sorry for the (too?) long email, but I wanted to be very explicit so we > >>> can hopefully settle this. Curious to hear your thought about it. > >>> > >>> > >>> > >>> -Matthias > >>> > >>> > >>> On 10/9/23 8:34 PM, Hanyu (Peter) Zheng wrote: > >>>> Thank you Colt, > >>>> > >>>> At first, we misinterpreted the JavaDoc. Upon further discussion, we > >>>> realized that after the key is converted to bytes, queries are based > >>>> on the > >>>> key's byte order, not its intrinsic order. > >>>> > >>>> Sincerely, > >>>> Hanyu > >>>> > >>>> On Mon, Oct 9, 2023 at 6:55 PM Colt McNealy > >> wrote: > >>>> > >>>>> Hanyu, > >>>>> > >>>>> I like the attention to detail! > >>>>> > >>>>> It is correct that the JavaDoc does not "guarantee" order. However, > the > >>>>> public API contract specified in the javadoc does mention > >>>>> lexicographical > >>>>> ordering of the bytes, which is a useful API contract. In our Streams > >>>>> app > >>>>> w
Re: [VOTE] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Hello everyone, I would like to start a vote for KIP-985 that Add reverseRange and reverseAll query over kv-store in IQv2. Sincerely, Hanyu On Fri, Oct 13, 2023 at 9:15 AM Hanyu (Peter) Zheng wrote: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-985:+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2 > > -- > > [image: Confluent] <https://www.confluent.io> > Hanyu (Peter) Zheng he/him/his > Software Engineer Intern > +1 (213) 431-7193 <+1+(213)+431-7193> > Follow us: [image: Blog] > <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > <https://slackpass.io/confluentcommunity>[image: YouTube] > <https://youtube.com/confluent> > > [image: Try Confluent Cloud for Free] > <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic> > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
[VOTE] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
https://cwiki.apache.org/confluence/display/KAFKA/KIP-985:+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2 -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Thank you Colt,
At first, we misinterpreted the JavaDoc. Upon further discussion, we
realized that after the key is converted to bytes, queries are based on the
key's byte order, not its intrinsic order.
Sincerely,
Hanyu
On Mon, Oct 9, 2023 at 6:55 PM Colt McNealy wrote:
> Hanyu,
>
> I like the attention to detail!
>
> It is correct that the JavaDoc does not "guarantee" order. However, the
> public API contract specified in the javadoc does mention lexicographical
> ordering of the bytes, which is a useful API contract. In our Streams app
> we make use of that contract during interactive queries (specifically, to
> guarantee correctness when doing a paginated range scan. If the order
> changes, then the "bookmark" we use for pagination would be meaningless).
>
> As such, I still think the KIP as you proposed is a highly useful feature.
> I would just make a note of the semantics in the JavaDoc and also in the
> KIP.
>
> Thanks,
> Colt McNealy
>
> *Founder, LittleHorse.dev*
>
>
> On Mon, Oct 9, 2023 at 2:22 PM Hanyu (Peter) Zheng
> wrote:
>
> > After our discussion, we discovered something intriguing. The definitions
> > for the range and reverseRange methods in the ReadOnlyKeyValueStore are
> as
> > follows:
> > /**
> > * Get an iterator over a given range of keys. This iterator must be
> > closed after use.
> > * The returned iterator must be safe from {@link
> > java.util.ConcurrentModificationException}s
> > * and must not return null values.
> > ** Order is not guaranteed as bytes lexicographical ordering might
> not
> > represent key order.*
> > *
> > * @param from The first key that could be in the range, where
> > iteration starts from.
> > * A null value indicates that the range starts with the
> > first element in the store.
> > * @param to The last key that could be in the range, where
> iteration
> > ends.
> > * A null value indicates that the range ends with the
> last
> > element in the store.
> > * @return The iterator for this range, from smallest to largest
> bytes.
> > * @throws InvalidStateStoreException if the store is not initialized
> > */
> > KeyValueIterator range(K from, K to);
> >
> > /**
> > * Get a reverse iterator over a given range of keys. This iterator
> > must be closed after use.
> > * The returned iterator must be safe from {@link
> > java.util.ConcurrentModificationException}s
> > * and must not return null values.
> > * *Order is not guaranteed as bytes lexicographical ordering might
> not
> > represent key order.*
> > *
> > * @param from The first key that could be in the range, where
> > iteration ends.
> > * A null value indicates that the range starts with the
> > first element in the store.
> > * @param to The last key that could be in the range, where
> iteration
> > starts from.
> > * A null value indicates that the range ends with the
> last
> > element in the store.
> > * @return The reverse iterator for this range, from largest to
> > smallest key bytes.
> > * @throws InvalidStateStoreException if the store is not initialized
> > */
> > default KeyValueIterator reverseRange(K from, K to) {
> > throw new UnsupportedOperationException();
> > }
> >
> > The query methods of RangeQuery ultimately invoke either the range method
> > or the reverseRange method. However, as per the JavaDoc: the order is not
> > guaranteed, since byte lexicographical ordering may not correspond to the
> > actual key order.
> >
> > Sincerely,
> > Hanyu
> >
> > On Fri, Oct 6, 2023 at 10:00 AM Hanyu (Peter) Zheng >
> > wrote:
> >
> > > Thank you, Matthias, for the detailed implementation and explanation.
> As
> > > of now, our capability is limited to executing interactive queries on
> > > individual partitions. To illustrate:
> > >
> > > Consider the IQv2StoreIntegrationTest:
> > >
> > > We have two partitions:
> > > Partition0 contains key-value pairs: <0,0> and <2,2>.
> > > Partition1 contains key-value pairs: <1,1> and <3,3>.
> > > When executing RangeQuery.withRange(1,3), the results are:
> > >
> > > Partition0: [2]
> > > Partition1: [1, 3]
> > > To support functionalities like reverseRange and reverseAll, we can
> > > introduce the wi
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
After our discussion, we discovered something intriguing. The definitions
for the range and reverseRange methods in the ReadOnlyKeyValueStore are as
follows:
/**
* Get an iterator over a given range of keys. This iterator must be
closed after use.
* The returned iterator must be safe from {@link
java.util.ConcurrentModificationException}s
* and must not return null values.
** Order is not guaranteed as bytes lexicographical ordering might not
represent key order.*
*
* @param from The first key that could be in the range, where
iteration starts from.
* A null value indicates that the range starts with the
first element in the store.
* @param to The last key that could be in the range, where iteration
ends.
* A null value indicates that the range ends with the last
element in the store.
* @return The iterator for this range, from smallest to largest bytes.
* @throws InvalidStateStoreException if the store is not initialized
*/
KeyValueIterator range(K from, K to);
/**
* Get a reverse iterator over a given range of keys. This iterator
must be closed after use.
* The returned iterator must be safe from {@link
java.util.ConcurrentModificationException}s
* and must not return null values.
* *Order is not guaranteed as bytes lexicographical ordering might not
represent key order.*
*
* @param from The first key that could be in the range, where
iteration ends.
* A null value indicates that the range starts with the
first element in the store.
* @param to The last key that could be in the range, where iteration
starts from.
* A null value indicates that the range ends with the last
element in the store.
* @return The reverse iterator for this range, from largest to
smallest key bytes.
* @throws InvalidStateStoreException if the store is not initialized
*/
default KeyValueIterator reverseRange(K from, K to) {
throw new UnsupportedOperationException();
}
The query methods of RangeQuery ultimately invoke either the range method
or the reverseRange method. However, as per the JavaDoc: the order is not
guaranteed, since byte lexicographical ordering may not correspond to the
actual key order.
Sincerely,
Hanyu
On Fri, Oct 6, 2023 at 10:00 AM Hanyu (Peter) Zheng
wrote:
> Thank you, Matthias, for the detailed implementation and explanation. As
> of now, our capability is limited to executing interactive queries on
> individual partitions. To illustrate:
>
> Consider the IQv2StoreIntegrationTest:
>
> We have two partitions:
> Partition0 contains key-value pairs: <0,0> and <2,2>.
> Partition1 contains key-value pairs: <1,1> and <3,3>.
> When executing RangeQuery.withRange(1,3), the results are:
>
> Partition0: [2]
> Partition1: [1, 3]
> To support functionalities like reverseRange and reverseAll, we can
> introduce the withDescendingKeys() method. For instance, using
> RangeQuery.withRange(1,3).withDescendingKeys(), the anticipated results are:
>
> Partition0: [2]
> Partition1: [3, 1]
>
> In response to Hao's inquiry about the boundary issue, please refer to the
> StoreQueryUtils class. The code snippet:
>
> iterator = kvStore.range(lowerRange.orElse(null), upperRange.orElse(null));
> indicates that when implementing range in each store, it's structured like:
>
> @Override
> public KeyValueIterator range(final Bytes from, final Bytes
> to) {
> if (from != null && to != null && from.compareTo(to) > 0) {
> This section performs the necessary checks.
>
> Sincerely,
> Hanyu
>
> On Thu, Oct 5, 2023 at 9:52 AM Hanyu (Peter) Zheng
> wrote:
>
>> Hi, Hao,
>>
>> In this case, it will return an empty set or list in the end.
>>
>> Sincerely,
>> Hanyu
>>
>> On Wed, Oct 4, 2023 at 10:29 PM Matthias J. Sax wrote:
>>
>>> Great discussion!
>>>
>>> It seems the only open question might be about ordering guarantees?
>>> IIRC, we had a discussion about this in the past.
>>>
>>>
>>> Technically (at least from my POV), existing `RangeQuery` does not have
>>> a guarantee that data is return in any specific order (not even on a per
>>> partitions bases). It just happens that RocksDB (and as pointed out by
>>> Hanyu already, also the built-in in-memory store that is base on a
>>> tree-map) allows us to return data ordered by key; as mentioned already,
>>> this guarantee is limited on a per partition basis.
>>>
>>> If there would be custom store base on a hashed key-value store, this
>>> store could implement RangeQuery and return data (even for a single
>>> partition) with no ordering,
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Thank you, Matthias, for the detailed implementation and explanation. As of
now, our capability is limited to executing interactive queries on
individual partitions. To illustrate:
Consider the IQv2StoreIntegrationTest:
We have two partitions:
Partition0 contains key-value pairs: <0,0> and <2,2>.
Partition1 contains key-value pairs: <1,1> and <3,3>.
When executing RangeQuery.withRange(1,3), the results are:
Partition0: [2]
Partition1: [1, 3]
To support functionalities like reverseRange and reverseAll, we can
introduce the withDescendingKeys() method. For instance, using
RangeQuery.withRange(1,3).withDescendingKeys(), the anticipated results are:
Partition0: [2]
Partition1: [3, 1]
In response to Hao's inquiry about the boundary issue, please refer to the
StoreQueryUtils class. The code snippet:
iterator = kvStore.range(lowerRange.orElse(null), upperRange.orElse(null));
indicates that when implementing range in each store, it's structured like:
@Override
public KeyValueIterator range(final Bytes from, final Bytes
to) {
if (from != null && to != null && from.compareTo(to) > 0) {
This section performs the necessary checks.
Sincerely,
Hanyu
On Thu, Oct 5, 2023 at 9:52 AM Hanyu (Peter) Zheng
wrote:
> Hi, Hao,
>
> In this case, it will return an empty set or list in the end.
>
> Sincerely,
> Hanyu
>
> On Wed, Oct 4, 2023 at 10:29 PM Matthias J. Sax wrote:
>
>> Great discussion!
>>
>> It seems the only open question might be about ordering guarantees?
>> IIRC, we had a discussion about this in the past.
>>
>>
>> Technically (at least from my POV), existing `RangeQuery` does not have
>> a guarantee that data is return in any specific order (not even on a per
>> partitions bases). It just happens that RocksDB (and as pointed out by
>> Hanyu already, also the built-in in-memory store that is base on a
>> tree-map) allows us to return data ordered by key; as mentioned already,
>> this guarantee is limited on a per partition basis.
>>
>> If there would be custom store base on a hashed key-value store, this
>> store could implement RangeQuery and return data (even for a single
>> partition) with no ordering, without violating the contract.
>>
>>
>>
>> Thus, it could actually make sense, to extend `RangeQuery` and allow
>> three options: no-order, ascending, descending. For our existing
>> Rocks/InMemory implementations, no-order could be equal to ascending and
>> nothing changes effectively, but it might be a better API contract? --
>> If we assume that there might be a custom hash-based store, such a store
>> could reject a query if "ascending" is required, or might need to do
>> more work to implement it (up to the store maintainer). This is actually
>> the beauty of IQv2 that different stores can pick what queries they want
>> to support.
>>
>> From an API contract point of view, it seems confusing to say:
>> specifying nothing means no guarantee (or ascending if the store can
>> offer it), but descending can we explicitly request. Thus, a hash-based
>> store, might be able to accept "order not specified query", but would
>> reject "descending". This seems to be somewhat unbalanced?
>>
>> Thus, I am wondering if we should actually add `withAscendingKeys()`,
>> too, even if it won't impact our current RocksDB/In-Memory
>> implementations?
>>
>>
>> The second question is about per-partition or across-partition ordering:
>> it's not possible right now to actually offer across-partition ordering
>> the way IQv2 is setup. The reason is, that the store that implements a
>> query type, is always a single shard. Thus, the implementation does not
>> have access to other shards. It's hard-coded inside Kafka Streams, to
>> query each shared, and to "accumulate" partial results, and return the
>> back to the user. Note that the API is:
>>
>>
>> > StateQueryResult result = KafkaStreams.query(...);
>> > Map> resultPerPartitions =
>> result.getPartitionResults();
>>
>>
>> Thus, if we would want to offer across-partition ordering, we cannot do
>> it right now, because Kafka Streams does not know anything about the
>> semantics of the query it distributes... -- the result is an unknown
>> type . We would need to extend IQv2 with an additional mechanism,
>> that allows users to plug in more custom code to "merge" multiple
>> partitions result into a "global result". This is clearly out-of-scope
>> for this KIP and would require a new KIP by itself.
>>
>> I seems that this contract, which is independent of th
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Hi, Hao,
In this case, it will return an empty set or list in the end.
Sincerely,
Hanyu
On Wed, Oct 4, 2023 at 10:29 PM Matthias J. Sax wrote:
> Great discussion!
>
> It seems the only open question might be about ordering guarantees?
> IIRC, we had a discussion about this in the past.
>
>
> Technically (at least from my POV), existing `RangeQuery` does not have
> a guarantee that data is return in any specific order (not even on a per
> partitions bases). It just happens that RocksDB (and as pointed out by
> Hanyu already, also the built-in in-memory store that is base on a
> tree-map) allows us to return data ordered by key; as mentioned already,
> this guarantee is limited on a per partition basis.
>
> If there would be custom store base on a hashed key-value store, this
> store could implement RangeQuery and return data (even for a single
> partition) with no ordering, without violating the contract.
>
>
>
> Thus, it could actually make sense, to extend `RangeQuery` and allow
> three options: no-order, ascending, descending. For our existing
> Rocks/InMemory implementations, no-order could be equal to ascending and
> nothing changes effectively, but it might be a better API contract? --
> If we assume that there might be a custom hash-based store, such a store
> could reject a query if "ascending" is required, or might need to do
> more work to implement it (up to the store maintainer). This is actually
> the beauty of IQv2 that different stores can pick what queries they want
> to support.
>
> From an API contract point of view, it seems confusing to say:
> specifying nothing means no guarantee (or ascending if the store can
> offer it), but descending can we explicitly request. Thus, a hash-based
> store, might be able to accept "order not specified query", but would
> reject "descending". This seems to be somewhat unbalanced?
>
> Thus, I am wondering if we should actually add `withAscendingKeys()`,
> too, even if it won't impact our current RocksDB/In-Memory implementations?
>
>
> The second question is about per-partition or across-partition ordering:
> it's not possible right now to actually offer across-partition ordering
> the way IQv2 is setup. The reason is, that the store that implements a
> query type, is always a single shard. Thus, the implementation does not
> have access to other shards. It's hard-coded inside Kafka Streams, to
> query each shared, and to "accumulate" partial results, and return the
> back to the user. Note that the API is:
>
>
> > StateQueryResult result = KafkaStreams.query(...);
> > Map> resultPerPartitions =
> result.getPartitionResults();
>
>
> Thus, if we would want to offer across-partition ordering, we cannot do
> it right now, because Kafka Streams does not know anything about the
> semantics of the query it distributes... -- the result is an unknown
> type . We would need to extend IQv2 with an additional mechanism,
> that allows users to plug in more custom code to "merge" multiple
> partitions result into a "global result". This is clearly out-of-scope
> for this KIP and would require a new KIP by itself.
>
> I seems that this contract, which is independent of the query type is
> not well understood, and thus a big +1 to fix the documentation. I don't
> think that this KIP must "define" anything, but it might of course be
> worth to add the explanation why the KIP cannot even offer
> global-ordering, as it's defined/limited by the IQv2 "framework" itself,
> not the individual queries.
>
>
>
> -Matthias
>
>
>
>
> On 10/4/23 4:38 PM, Hao Li wrote:
> > Hi Hanyu,
> >
> > Thanks for the KIP! Seems there are already a lot of good discussions. I
> > only have two comments:
> >
> > 1. Please make it clear in
> > ```
> > /**
> > * Interactive range query using a lower and upper bound to filter
> the
> > keys returned.
> > * @param lower The key that specifies the lower bound of the range
> > * @param upper The key that specifies the upper bound of the range
> > * @param The key type
> > * @param The value type
> > */
> > public static RangeQuery withRange(final K lower,
> final K
> > upper) {
> > return new RangeQuery<>(Optional.ofNullable(lower),
> > Optional.ofNullable(upper), true);
> > }
> > ```
> > that a `null` in lower or upper parameter means it's unbounded.
> > 2. What's the behavior if lower is 3 and upper is 1? Is it
> IllegalArgument
> > or will this return an empty result? Maybe also clarify this in the
> > do
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
For testing purposes, we previously used a Set to record the results in IQv2StoreIntegrationTest. Let's take an example where we now have two partitions and four key-value pairs: <0,0> in p0, <1,1> in p1, <2,2> in p0, and <3,3> in p1. If we execute withRange(1,3), it will return a Set of <1, 2, 3>. However, if we run withRange(1,3).withDescendingKeys(), and still use a Set, the result will again be a Set of <1,2,3>. This means we won't be able to determine whether the results have been reversed. To resolve this ambiguity, I've switched to using a List to record the results, ensuring the order of retrieval from partitions p0 and p1. So, withRange(1,3) would yield a List of [2, 1, 3], whereas withRange(1,3).withDescendingKeys() would produce a List of [2,3,1]. This ordering makes sense since RocksDB sorts its keys, and InMemoryStore uses a TreeMap structure, which means the keys are already sorted. Sincerely, Hanyu On Wed, Oct 4, 2023 at 9:25 AM Hanyu (Peter) Zheng wrote: > Hi, Bruno > > Thank you for your suggestions, I will update them soon. > Sincerely, > > Hanyu > > On Wed, Oct 4, 2023 at 9:25 AM Hanyu (Peter) Zheng > wrote: > >> Hi, Lucas, >> >> Thank you for your suggestions. >> I will update the KIP and code together. >> >> Sincerely, >> Hanyu >> >> On Tue, Oct 3, 2023 at 8:16 PM Hanyu (Peter) Zheng >> wrote: >> >>> If we use WithDescendingKeys() to generate a RangeQuery to do the >>> reveseQuery, how do we achieve the methods like withRange, withUpperBound, >>> and withLowerBound only in this method? >>> >>> On Tue, Oct 3, 2023 at 8:01 PM Hanyu (Peter) Zheng >>> wrote: >>> >>>> I believe there's no need to introduce a method like >>>> WithDescendingKeys(). Instead, we can simply add a reverse flag to >>>> RangeQuery. Each method within RangeQuery would then accept an additional >>>> parameter. If the reverse is set to true, it would indicate the results >>>> should be reversed. >>>> >>>> Initially, I introduced a reverse variable. When set to false, the >>>> RangeQuery class behaves normally. However, when reverse is set to true, >>>> the RangeQuery essentially takes on the functionality of ReverseRangeQuery. >>>> Further details can be found in the "Rejected Alternatives" section. >>>> >>>> In my perspective, RangeQuery is a class responsible for creating a >>>> series of RangeQuery objects. It offers methods such as withRange, >>>> withUpperBound, and withLowerBound, allowing us to generate objects >>>> representing different queries. I'm unsure how adding a >>>> withDescendingOrder() method would be compatible with the other methods, >>>> especially considering that, based on KIP 969, WithDescendingKeys() doesn't >>>> appear to take any input variables. And if withDescendingOrder() doesn't >>>> accept any input, how does it return a RangeQuery? >>>> >>>> On Tue, Oct 3, 2023 at 4:37 PM Hanyu (Peter) Zheng >>>> wrote: >>>> >>>>> Hi, Colt, >>>>> The underlying structure of inMemoryKeyValueStore is treeMap. >>>>> Sincerely, >>>>> Hanyu >>>>> >>>>> On Tue, Oct 3, 2023 at 4:34 PM Hanyu (Peter) Zheng < >>>>> pzh...@confluent.io> wrote: >>>>> >>>>>> Hi Bill, >>>>>> 1. I will update the KIP in accordance with the PR and synchronize >>>>>> their future updates. >>>>>> 2. I will use that name. >>>>>> 3. you mean add something about ordering at the motivation section? >>>>>> >>>>>> Sincerely, >>>>>> Hanyu >>>>>> >>>>>> >>>>>> On Tue, Oct 3, 2023 at 4:29 PM Hanyu (Peter) Zheng < >>>>>> pzh...@confluent.io> wrote: >>>>>> >>>>>>> Hi, Walker, >>>>>>> >>>>>>> 1. I will update the KIP in accordance with the PR and synchronize >>>>>>> their future updates. >>>>>>> 2. I will use that name. >>>>>>> 3. I'll provide additional details in that section. >>>>>>> 4. I intend to utilize rangeQuery to achieve what we're referring to >>>>>>> as reverseQuery. In essence, reverseQuery is merely a term. To clear up >>>>>>> any >>>>>>> ambiguity, I'll make necessary adjus
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Hi, Bruno Thank you for your suggestions, I will update them soon. Sincerely, Hanyu On Wed, Oct 4, 2023 at 9:25 AM Hanyu (Peter) Zheng wrote: > Hi, Lucas, > > Thank you for your suggestions. > I will update the KIP and code together. > > Sincerely, > Hanyu > > On Tue, Oct 3, 2023 at 8:16 PM Hanyu (Peter) Zheng > wrote: > >> If we use WithDescendingKeys() to generate a RangeQuery to do the >> reveseQuery, how do we achieve the methods like withRange, withUpperBound, >> and withLowerBound only in this method? >> >> On Tue, Oct 3, 2023 at 8:01 PM Hanyu (Peter) Zheng >> wrote: >> >>> I believe there's no need to introduce a method like >>> WithDescendingKeys(). Instead, we can simply add a reverse flag to >>> RangeQuery. Each method within RangeQuery would then accept an additional >>> parameter. If the reverse is set to true, it would indicate the results >>> should be reversed. >>> >>> Initially, I introduced a reverse variable. When set to false, the >>> RangeQuery class behaves normally. However, when reverse is set to true, >>> the RangeQuery essentially takes on the functionality of ReverseRangeQuery. >>> Further details can be found in the "Rejected Alternatives" section. >>> >>> In my perspective, RangeQuery is a class responsible for creating a >>> series of RangeQuery objects. It offers methods such as withRange, >>> withUpperBound, and withLowerBound, allowing us to generate objects >>> representing different queries. I'm unsure how adding a >>> withDescendingOrder() method would be compatible with the other methods, >>> especially considering that, based on KIP 969, WithDescendingKeys() doesn't >>> appear to take any input variables. And if withDescendingOrder() doesn't >>> accept any input, how does it return a RangeQuery? >>> >>> On Tue, Oct 3, 2023 at 4:37 PM Hanyu (Peter) Zheng >>> wrote: >>> >>>> Hi, Colt, >>>> The underlying structure of inMemoryKeyValueStore is treeMap. >>>> Sincerely, >>>> Hanyu >>>> >>>> On Tue, Oct 3, 2023 at 4:34 PM Hanyu (Peter) Zheng >>>> wrote: >>>> >>>>> Hi Bill, >>>>> 1. I will update the KIP in accordance with the PR and synchronize >>>>> their future updates. >>>>> 2. I will use that name. >>>>> 3. you mean add something about ordering at the motivation section? >>>>> >>>>> Sincerely, >>>>> Hanyu >>>>> >>>>> >>>>> On Tue, Oct 3, 2023 at 4:29 PM Hanyu (Peter) Zheng < >>>>> pzh...@confluent.io> wrote: >>>>> >>>>>> Hi, Walker, >>>>>> >>>>>> 1. I will update the KIP in accordance with the PR and synchronize >>>>>> their future updates. >>>>>> 2. I will use that name. >>>>>> 3. I'll provide additional details in that section. >>>>>> 4. I intend to utilize rangeQuery to achieve what we're referring to >>>>>> as reverseQuery. In essence, reverseQuery is merely a term. To clear up >>>>>> any >>>>>> ambiguity, I'll make necessary adjustments to the KIP. >>>>>> >>>>>> Sincerely, >>>>>> Hanyu >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Oct 3, 2023 at 4:09 PM Hanyu (Peter) Zheng < >>>>>> pzh...@confluent.io> wrote: >>>>>> >>>>>>> Ok, I will change it back to following the code, and update them >>>>>>> together. >>>>>>> >>>>>>> On Tue, Oct 3, 2023 at 2:27 PM Walker Carlson >>>>>>> wrote: >>>>>>> >>>>>>>> Hello Hanyu, >>>>>>>> >>>>>>>> Looking over your kip things mostly make sense but I have a couple >>>>>>>> of >>>>>>>> comments. >>>>>>>> >>>>>>>> >>>>>>>>1. You have "withDescandingOrder()". I think you mean >>>>>>>> "descending" :) >>>>>>>>Also there are still a few places in the do where its called >>>>>>>> "setReverse" >>>>>>>>2. Also I like "WithDescendingKeys()" better >>>>>
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Hi, Lucas, Thank you for your suggestions. I will update the KIP and code together. Sincerely, Hanyu On Tue, Oct 3, 2023 at 8:16 PM Hanyu (Peter) Zheng wrote: > If we use WithDescendingKeys() to generate a RangeQuery to do the > reveseQuery, how do we achieve the methods like withRange, withUpperBound, > and withLowerBound only in this method? > > On Tue, Oct 3, 2023 at 8:01 PM Hanyu (Peter) Zheng > wrote: > >> I believe there's no need to introduce a method like >> WithDescendingKeys(). Instead, we can simply add a reverse flag to >> RangeQuery. Each method within RangeQuery would then accept an additional >> parameter. If the reverse is set to true, it would indicate the results >> should be reversed. >> >> Initially, I introduced a reverse variable. When set to false, the >> RangeQuery class behaves normally. However, when reverse is set to true, >> the RangeQuery essentially takes on the functionality of ReverseRangeQuery. >> Further details can be found in the "Rejected Alternatives" section. >> >> In my perspective, RangeQuery is a class responsible for creating a >> series of RangeQuery objects. It offers methods such as withRange, >> withUpperBound, and withLowerBound, allowing us to generate objects >> representing different queries. I'm unsure how adding a >> withDescendingOrder() method would be compatible with the other methods, >> especially considering that, based on KIP 969, WithDescendingKeys() doesn't >> appear to take any input variables. And if withDescendingOrder() doesn't >> accept any input, how does it return a RangeQuery? >> >> On Tue, Oct 3, 2023 at 4:37 PM Hanyu (Peter) Zheng >> wrote: >> >>> Hi, Colt, >>> The underlying structure of inMemoryKeyValueStore is treeMap. >>> Sincerely, >>> Hanyu >>> >>> On Tue, Oct 3, 2023 at 4:34 PM Hanyu (Peter) Zheng >>> wrote: >>> >>>> Hi Bill, >>>> 1. I will update the KIP in accordance with the PR and synchronize >>>> their future updates. >>>> 2. I will use that name. >>>> 3. you mean add something about ordering at the motivation section? >>>> >>>> Sincerely, >>>> Hanyu >>>> >>>> >>>> On Tue, Oct 3, 2023 at 4:29 PM Hanyu (Peter) Zheng >>>> wrote: >>>> >>>>> Hi, Walker, >>>>> >>>>> 1. I will update the KIP in accordance with the PR and synchronize >>>>> their future updates. >>>>> 2. I will use that name. >>>>> 3. I'll provide additional details in that section. >>>>> 4. I intend to utilize rangeQuery to achieve what we're referring to >>>>> as reverseQuery. In essence, reverseQuery is merely a term. To clear up >>>>> any >>>>> ambiguity, I'll make necessary adjustments to the KIP. >>>>> >>>>> Sincerely, >>>>> Hanyu >>>>> >>>>> >>>>> >>>>> On Tue, Oct 3, 2023 at 4:09 PM Hanyu (Peter) Zheng < >>>>> pzh...@confluent.io> wrote: >>>>> >>>>>> Ok, I will change it back to following the code, and update them >>>>>> together. >>>>>> >>>>>> On Tue, Oct 3, 2023 at 2:27 PM Walker Carlson >>>>>> wrote: >>>>>> >>>>>>> Hello Hanyu, >>>>>>> >>>>>>> Looking over your kip things mostly make sense but I have a couple of >>>>>>> comments. >>>>>>> >>>>>>> >>>>>>>1. You have "withDescandingOrder()". I think you mean >>>>>>> "descending" :) >>>>>>>Also there are still a few places in the do where its called >>>>>>> "setReverse" >>>>>>>2. Also I like "WithDescendingKeys()" better >>>>>>>3. I'm not sure of what ordering guarantees we are offering. >>>>>>> Perhaps we >>>>>>>can add a section to the motivation clearly spelling out the >>>>>>> current >>>>>>>ordering and the new offering? >>>>>>>4. When you say "use unbounded reverseQuery to achieve >>>>>>> reverseAll" do >>>>>>>you mean "use unbounded RangeQuery to achieve reverseAll"? as far >>>>>>> as I can >>>
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
If we use WithDescendingKeys() to generate a RangeQuery to do the reveseQuery, how do we achieve the methods like withRange, withUpperBound, and withLowerBound only in this method? On Tue, Oct 3, 2023 at 8:01 PM Hanyu (Peter) Zheng wrote: > I believe there's no need to introduce a method like WithDescendingKeys(). > Instead, we can simply add a reverse flag to RangeQuery. Each method within > RangeQuery would then accept an additional parameter. If the reverse is set > to true, it would indicate the results should be reversed. > > Initially, I introduced a reverse variable. When set to false, the > RangeQuery class behaves normally. However, when reverse is set to true, > the RangeQuery essentially takes on the functionality of ReverseRangeQuery. > Further details can be found in the "Rejected Alternatives" section. > > In my perspective, RangeQuery is a class responsible for creating a series > of RangeQuery objects. It offers methods such as withRange, withUpperBound, > and withLowerBound, allowing us to generate objects representing different > queries. I'm unsure how adding a withDescendingOrder() method would be > compatible with the other methods, especially considering that, based on > KIP 969, WithDescendingKeys() doesn't appear to take any input variables. > And if withDescendingOrder() doesn't accept any input, how does it return a > RangeQuery? > > On Tue, Oct 3, 2023 at 4:37 PM Hanyu (Peter) Zheng > wrote: > >> Hi, Colt, >> The underlying structure of inMemoryKeyValueStore is treeMap. >> Sincerely, >> Hanyu >> >> On Tue, Oct 3, 2023 at 4:34 PM Hanyu (Peter) Zheng >> wrote: >> >>> Hi Bill, >>> 1. I will update the KIP in accordance with the PR and synchronize their >>> future updates. >>> 2. I will use that name. >>> 3. you mean add something about ordering at the motivation section? >>> >>> Sincerely, >>> Hanyu >>> >>> >>> On Tue, Oct 3, 2023 at 4:29 PM Hanyu (Peter) Zheng >>> wrote: >>> >>>> Hi, Walker, >>>> >>>> 1. I will update the KIP in accordance with the PR and synchronize >>>> their future updates. >>>> 2. I will use that name. >>>> 3. I'll provide additional details in that section. >>>> 4. I intend to utilize rangeQuery to achieve what we're referring to as >>>> reverseQuery. In essence, reverseQuery is merely a term. To clear up any >>>> ambiguity, I'll make necessary adjustments to the KIP. >>>> >>>> Sincerely, >>>> Hanyu >>>> >>>> >>>> >>>> On Tue, Oct 3, 2023 at 4:09 PM Hanyu (Peter) Zheng >>>> wrote: >>>> >>>>> Ok, I will change it back to following the code, and update them >>>>> together. >>>>> >>>>> On Tue, Oct 3, 2023 at 2:27 PM Walker Carlson >>>>> wrote: >>>>> >>>>>> Hello Hanyu, >>>>>> >>>>>> Looking over your kip things mostly make sense but I have a couple of >>>>>> comments. >>>>>> >>>>>> >>>>>>1. You have "withDescandingOrder()". I think you mean "descending" >>>>>> :) >>>>>>Also there are still a few places in the do where its called >>>>>> "setReverse" >>>>>>2. Also I like "WithDescendingKeys()" better >>>>>>3. I'm not sure of what ordering guarantees we are offering. >>>>>> Perhaps we >>>>>>can add a section to the motivation clearly spelling out the >>>>>> current >>>>>>ordering and the new offering? >>>>>>4. When you say "use unbounded reverseQuery to achieve reverseAll" >>>>>> do >>>>>>you mean "use unbounded RangeQuery to achieve reverseAll"? as far >>>>>> as I can >>>>>>tell we don't have a reverseQuery as a named object? >>>>>> >>>>>> >>>>>> Looking good so far >>>>>> >>>>>> best, >>>>>> Walker >>>>>> >>>>>> On Tue, Oct 3, 2023 at 2:13 PM Colt McNealy >>>>>> wrote: >>>>>> >>>>>> > Hello Hanyu, >>>>>> > >>>>>> > Thank you for the KIP. I agree with Matthias' proposal to keep the >>>>>> naming
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
I believe there's no need to introduce a method like WithDescendingKeys(). Instead, we can simply add a reverse flag to RangeQuery. Each method within RangeQuery would then accept an additional parameter. If the reverse is set to true, it would indicate the results should be reversed. Initially, I introduced a reverse variable. When set to false, the RangeQuery class behaves normally. However, when reverse is set to true, the RangeQuery essentially takes on the functionality of ReverseRangeQuery. Further details can be found in the "Rejected Alternatives" section. In my perspective, RangeQuery is a class responsible for creating a series of RangeQuery objects. It offers methods such as withRange, withUpperBound, and withLowerBound, allowing us to generate objects representing different queries. I'm unsure how adding a withDescendingOrder() method would be compatible with the other methods, especially considering that, based on KIP 969, WithDescendingKeys() doesn't appear to take any input variables. And if withDescendingOrder() doesn't accept any input, how does it return a RangeQuery? On Tue, Oct 3, 2023 at 4:37 PM Hanyu (Peter) Zheng wrote: > Hi, Colt, > The underlying structure of inMemoryKeyValueStore is treeMap. > Sincerely, > Hanyu > > On Tue, Oct 3, 2023 at 4:34 PM Hanyu (Peter) Zheng > wrote: > >> Hi Bill, >> 1. I will update the KIP in accordance with the PR and synchronize their >> future updates. >> 2. I will use that name. >> 3. you mean add something about ordering at the motivation section? >> >> Sincerely, >> Hanyu >> >> >> On Tue, Oct 3, 2023 at 4:29 PM Hanyu (Peter) Zheng >> wrote: >> >>> Hi, Walker, >>> >>> 1. I will update the KIP in accordance with the PR and synchronize their >>> future updates. >>> 2. I will use that name. >>> 3. I'll provide additional details in that section. >>> 4. I intend to utilize rangeQuery to achieve what we're referring to as >>> reverseQuery. In essence, reverseQuery is merely a term. To clear up any >>> ambiguity, I'll make necessary adjustments to the KIP. >>> >>> Sincerely, >>> Hanyu >>> >>> >>> >>> On Tue, Oct 3, 2023 at 4:09 PM Hanyu (Peter) Zheng >>> wrote: >>> >>>> Ok, I will change it back to following the code, and update them >>>> together. >>>> >>>> On Tue, Oct 3, 2023 at 2:27 PM Walker Carlson >>>> wrote: >>>> >>>>> Hello Hanyu, >>>>> >>>>> Looking over your kip things mostly make sense but I have a couple of >>>>> comments. >>>>> >>>>> >>>>>1. You have "withDescandingOrder()". I think you mean "descending" >>>>> :) >>>>>Also there are still a few places in the do where its called >>>>> "setReverse" >>>>>2. Also I like "WithDescendingKeys()" better >>>>>3. I'm not sure of what ordering guarantees we are offering. >>>>> Perhaps we >>>>>can add a section to the motivation clearly spelling out the current >>>>>ordering and the new offering? >>>>>4. When you say "use unbounded reverseQuery to achieve reverseAll" >>>>> do >>>>>you mean "use unbounded RangeQuery to achieve reverseAll"? as far >>>>> as I can >>>>>tell we don't have a reverseQuery as a named object? >>>>> >>>>> >>>>> Looking good so far >>>>> >>>>> best, >>>>> Walker >>>>> >>>>> On Tue, Oct 3, 2023 at 2:13 PM Colt McNealy >>>>> wrote: >>>>> >>>>> > Hello Hanyu, >>>>> > >>>>> > Thank you for the KIP. I agree with Matthias' proposal to keep the >>>>> naming >>>>> > convention consistent with KIP-969. I favor the >>>>> `.withDescendingKeys()` >>>>> > name. >>>>> > >>>>> > I am curious about one thing. RocksDB guarantees that records >>>>> returned >>>>> > during a range scan are lexicographically ordered by the bytes of >>>>> the keys >>>>> > (either ascending or descending order, as specified in the query). >>>>> This >>>>> > means that results within a single partition are indeed ordered.** My >>>>> > reading of KIP-805 s
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Hi, Colt, The underlying structure of inMemoryKeyValueStore is treeMap. Sincerely, Hanyu On Tue, Oct 3, 2023 at 4:34 PM Hanyu (Peter) Zheng wrote: > Hi Bill, > 1. I will update the KIP in accordance with the PR and synchronize their > future updates. > 2. I will use that name. > 3. you mean add something about ordering at the motivation section? > > Sincerely, > Hanyu > > > On Tue, Oct 3, 2023 at 4:29 PM Hanyu (Peter) Zheng > wrote: > >> Hi, Walker, >> >> 1. I will update the KIP in accordance with the PR and synchronize their >> future updates. >> 2. I will use that name. >> 3. I'll provide additional details in that section. >> 4. I intend to utilize rangeQuery to achieve what we're referring to as >> reverseQuery. In essence, reverseQuery is merely a term. To clear up any >> ambiguity, I'll make necessary adjustments to the KIP. >> >> Sincerely, >> Hanyu >> >> >> >> On Tue, Oct 3, 2023 at 4:09 PM Hanyu (Peter) Zheng >> wrote: >> >>> Ok, I will change it back to following the code, and update them >>> together. >>> >>> On Tue, Oct 3, 2023 at 2:27 PM Walker Carlson >>> wrote: >>> >>>> Hello Hanyu, >>>> >>>> Looking over your kip things mostly make sense but I have a couple of >>>> comments. >>>> >>>> >>>>1. You have "withDescandingOrder()". I think you mean "descending" :) >>>>Also there are still a few places in the do where its called >>>> "setReverse" >>>>2. Also I like "WithDescendingKeys()" better >>>>3. I'm not sure of what ordering guarantees we are offering. Perhaps >>>> we >>>>can add a section to the motivation clearly spelling out the current >>>>ordering and the new offering? >>>>4. When you say "use unbounded reverseQuery to achieve reverseAll" do >>>>you mean "use unbounded RangeQuery to achieve reverseAll"? as far as >>>> I can >>>>tell we don't have a reverseQuery as a named object? >>>> >>>> >>>> Looking good so far >>>> >>>> best, >>>> Walker >>>> >>>> On Tue, Oct 3, 2023 at 2:13 PM Colt McNealy >>>> wrote: >>>> >>>> > Hello Hanyu, >>>> > >>>> > Thank you for the KIP. I agree with Matthias' proposal to keep the >>>> naming >>>> > convention consistent with KIP-969. I favor the >>>> `.withDescendingKeys()` >>>> > name. >>>> > >>>> > I am curious about one thing. RocksDB guarantees that records returned >>>> > during a range scan are lexicographically ordered by the bytes of the >>>> keys >>>> > (either ascending or descending order, as specified in the query). >>>> This >>>> > means that results within a single partition are indeed ordered.** My >>>> > reading of KIP-805 suggests to me that you don't need to specify the >>>> > partition number you are querying in IQv2, which means that you can >>>> have a >>>> > valid reversed RangeQuery over a store with "multiple partitions" in >>>> it. >>>> > >>>> > Currently, IQv1 does not guarantee order of keys in this scenario. >>>> Does >>>> > IQv2 support ordering across partitions? Such an implementation would >>>> > require opening a rocksdb range scan** on multiple rocksdb instances >>>> (one >>>> > per partition), and polling the first key of each. Whether or not >>>> this is >>>> > ordered, could we please add that to the documentation? >>>> > >>>> > **(How is this implemented/guaranteed in an `inMemoryKeyValueStore`? I >>>> > don't know about that implementation). >>>> > >>>> > Colt McNealy >>>> > >>>> > *Founder, LittleHorse.dev* >>>> > >>>> > >>>> > On Tue, Oct 3, 2023 at 1:35 PM Hanyu (Peter) Zheng >>>> > wrote: >>>> > >>>> > > ok, I will update it. Thank you Matthias >>>> > > >>>> > > Sincerely, >>>> > > Hanyu >>>> > > >>>> > > On Tue, Oct 3, 2023 at 11:23 AM Matthias J. Sax >>>> > wrote: >>>> >
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Hi Bill, 1. I will update the KIP in accordance with the PR and synchronize their future updates. 2. I will use that name. 3. you mean add something about ordering at the motivation section? Sincerely, Hanyu On Tue, Oct 3, 2023 at 4:29 PM Hanyu (Peter) Zheng wrote: > Hi, Walker, > > 1. I will update the KIP in accordance with the PR and synchronize their > future updates. > 2. I will use that name. > 3. I'll provide additional details in that section. > 4. I intend to utilize rangeQuery to achieve what we're referring to as > reverseQuery. In essence, reverseQuery is merely a term. To clear up any > ambiguity, I'll make necessary adjustments to the KIP. > > Sincerely, > Hanyu > > > > On Tue, Oct 3, 2023 at 4:09 PM Hanyu (Peter) Zheng > wrote: > >> Ok, I will change it back to following the code, and update them >> together. >> >> On Tue, Oct 3, 2023 at 2:27 PM Walker Carlson >> wrote: >> >>> Hello Hanyu, >>> >>> Looking over your kip things mostly make sense but I have a couple of >>> comments. >>> >>> >>>1. You have "withDescandingOrder()". I think you mean "descending" :) >>>Also there are still a few places in the do where its called >>> "setReverse" >>>2. Also I like "WithDescendingKeys()" better >>>3. I'm not sure of what ordering guarantees we are offering. Perhaps >>> we >>>can add a section to the motivation clearly spelling out the current >>>ordering and the new offering? >>>4. When you say "use unbounded reverseQuery to achieve reverseAll" do >>>you mean "use unbounded RangeQuery to achieve reverseAll"? as far as >>> I can >>>tell we don't have a reverseQuery as a named object? >>> >>> >>> Looking good so far >>> >>> best, >>> Walker >>> >>> On Tue, Oct 3, 2023 at 2:13 PM Colt McNealy wrote: >>> >>> > Hello Hanyu, >>> > >>> > Thank you for the KIP. I agree with Matthias' proposal to keep the >>> naming >>> > convention consistent with KIP-969. I favor the `.withDescendingKeys()` >>> > name. >>> > >>> > I am curious about one thing. RocksDB guarantees that records returned >>> > during a range scan are lexicographically ordered by the bytes of the >>> keys >>> > (either ascending or descending order, as specified in the query). This >>> > means that results within a single partition are indeed ordered.** My >>> > reading of KIP-805 suggests to me that you don't need to specify the >>> > partition number you are querying in IQv2, which means that you can >>> have a >>> > valid reversed RangeQuery over a store with "multiple partitions" in >>> it. >>> > >>> > Currently, IQv1 does not guarantee order of keys in this scenario. Does >>> > IQv2 support ordering across partitions? Such an implementation would >>> > require opening a rocksdb range scan** on multiple rocksdb instances >>> (one >>> > per partition), and polling the first key of each. Whether or not this >>> is >>> > ordered, could we please add that to the documentation? >>> > >>> > **(How is this implemented/guaranteed in an `inMemoryKeyValueStore`? I >>> > don't know about that implementation). >>> > >>> > Colt McNealy >>> > >>> > *Founder, LittleHorse.dev* >>> > >>> > >>> > On Tue, Oct 3, 2023 at 1:35 PM Hanyu (Peter) Zheng >>> > wrote: >>> > >>> > > ok, I will update it. Thank you Matthias >>> > > >>> > > Sincerely, >>> > > Hanyu >>> > > >>> > > On Tue, Oct 3, 2023 at 11:23 AM Matthias J. Sax >>> > wrote: >>> > > >>> > > > Thanks for the KIP Hanyu! >>> > > > >>> > > > >>> > > > I took a quick look and it think the proposal makes sense overall. >>> > > > >>> > > > A few comments about how to structure the KIP. >>> > > > >>> > > > As you propose to not add `ReverseRangQuery` class, the code >>> example >>> > > > should go into "Rejected Alternatives" section, not in the >>> "Proposed >>> > > > Changes" section.
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Hi, Walker, 1. I will update the KIP in accordance with the PR and synchronize their future updates. 2. I will use that name. 3. I'll provide additional details in that section. 4. I intend to utilize rangeQuery to achieve what we're referring to as reverseQuery. In essence, reverseQuery is merely a term. To clear up any ambiguity, I'll make necessary adjustments to the KIP. Sincerely, Hanyu On Tue, Oct 3, 2023 at 4:09 PM Hanyu (Peter) Zheng wrote: > Ok, I will change it back to following the code, and update them together. > > On Tue, Oct 3, 2023 at 2:27 PM Walker Carlson > wrote: > >> Hello Hanyu, >> >> Looking over your kip things mostly make sense but I have a couple of >> comments. >> >> >>1. You have "withDescandingOrder()". I think you mean "descending" :) >>Also there are still a few places in the do where its called >> "setReverse" >>2. Also I like "WithDescendingKeys()" better >>3. I'm not sure of what ordering guarantees we are offering. Perhaps we >>can add a section to the motivation clearly spelling out the current >>ordering and the new offering? >>4. When you say "use unbounded reverseQuery to achieve reverseAll" do >>you mean "use unbounded RangeQuery to achieve reverseAll"? as far as I >> can >>tell we don't have a reverseQuery as a named object? >> >> >> Looking good so far >> >> best, >> Walker >> >> On Tue, Oct 3, 2023 at 2:13 PM Colt McNealy wrote: >> >> > Hello Hanyu, >> > >> > Thank you for the KIP. I agree with Matthias' proposal to keep the >> naming >> > convention consistent with KIP-969. I favor the `.withDescendingKeys()` >> > name. >> > >> > I am curious about one thing. RocksDB guarantees that records returned >> > during a range scan are lexicographically ordered by the bytes of the >> keys >> > (either ascending or descending order, as specified in the query). This >> > means that results within a single partition are indeed ordered.** My >> > reading of KIP-805 suggests to me that you don't need to specify the >> > partition number you are querying in IQv2, which means that you can >> have a >> > valid reversed RangeQuery over a store with "multiple partitions" in it. >> > >> > Currently, IQv1 does not guarantee order of keys in this scenario. Does >> > IQv2 support ordering across partitions? Such an implementation would >> > require opening a rocksdb range scan** on multiple rocksdb instances >> (one >> > per partition), and polling the first key of each. Whether or not this >> is >> > ordered, could we please add that to the documentation? >> > >> > **(How is this implemented/guaranteed in an `inMemoryKeyValueStore`? I >> > don't know about that implementation). >> > >> > Colt McNealy >> > >> > *Founder, LittleHorse.dev* >> > >> > >> > On Tue, Oct 3, 2023 at 1:35 PM Hanyu (Peter) Zheng >> > wrote: >> > >> > > ok, I will update it. Thank you Matthias >> > > >> > > Sincerely, >> > > Hanyu >> > > >> > > On Tue, Oct 3, 2023 at 11:23 AM Matthias J. Sax >> > wrote: >> > > >> > > > Thanks for the KIP Hanyu! >> > > > >> > > > >> > > > I took a quick look and it think the proposal makes sense overall. >> > > > >> > > > A few comments about how to structure the KIP. >> > > > >> > > > As you propose to not add `ReverseRangQuery` class, the code example >> > > > should go into "Rejected Alternatives" section, not in the "Proposed >> > > > Changes" section. >> > > > >> > > > For the `RangeQuery` code example, please omit all existing methods >> > etc, >> > > > and only include what will be added/changed. This make it simpler to >> > > > read the KIP. >> > > > >> > > > >> > > > nit: typo >> > > > >> > > > > the fault value is false >> > > > >> > > > Should be "the default value is false". >> > > > >> > > > >> > > > Not sure if `setReverse()` is the best name. Maybe >> > `withDescandingOrder` >> > > > (or similar, I guess `withReverseOrder` would also work) might be >> &g
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Ok, I will change it back to following the code, and update them together. On Tue, Oct 3, 2023 at 2:27 PM Walker Carlson wrote: > Hello Hanyu, > > Looking over your kip things mostly make sense but I have a couple of > comments. > > >1. You have "withDescandingOrder()". I think you mean "descending" :) >Also there are still a few places in the do where its called > "setReverse" >2. Also I like "WithDescendingKeys()" better >3. I'm not sure of what ordering guarantees we are offering. Perhaps we >can add a section to the motivation clearly spelling out the current >ordering and the new offering? >4. When you say "use unbounded reverseQuery to achieve reverseAll" do >you mean "use unbounded RangeQuery to achieve reverseAll"? as far as I > can >tell we don't have a reverseQuery as a named object? > > > Looking good so far > > best, > Walker > > On Tue, Oct 3, 2023 at 2:13 PM Colt McNealy wrote: > > > Hello Hanyu, > > > > Thank you for the KIP. I agree with Matthias' proposal to keep the naming > > convention consistent with KIP-969. I favor the `.withDescendingKeys()` > > name. > > > > I am curious about one thing. RocksDB guarantees that records returned > > during a range scan are lexicographically ordered by the bytes of the > keys > > (either ascending or descending order, as specified in the query). This > > means that results within a single partition are indeed ordered.** My > > reading of KIP-805 suggests to me that you don't need to specify the > > partition number you are querying in IQv2, which means that you can have > a > > valid reversed RangeQuery over a store with "multiple partitions" in it. > > > > Currently, IQv1 does not guarantee order of keys in this scenario. Does > > IQv2 support ordering across partitions? Such an implementation would > > require opening a rocksdb range scan** on multiple rocksdb instances (one > > per partition), and polling the first key of each. Whether or not this is > > ordered, could we please add that to the documentation? > > > > **(How is this implemented/guaranteed in an `inMemoryKeyValueStore`? I > > don't know about that implementation). > > > > Colt McNealy > > > > *Founder, LittleHorse.dev* > > > > > > On Tue, Oct 3, 2023 at 1:35 PM Hanyu (Peter) Zheng > > wrote: > > > > > ok, I will update it. Thank you Matthias > > > > > > Sincerely, > > > Hanyu > > > > > > On Tue, Oct 3, 2023 at 11:23 AM Matthias J. Sax > > wrote: > > > > > > > Thanks for the KIP Hanyu! > > > > > > > > > > > > I took a quick look and it think the proposal makes sense overall. > > > > > > > > A few comments about how to structure the KIP. > > > > > > > > As you propose to not add `ReverseRangQuery` class, the code example > > > > should go into "Rejected Alternatives" section, not in the "Proposed > > > > Changes" section. > > > > > > > > For the `RangeQuery` code example, please omit all existing methods > > etc, > > > > and only include what will be added/changed. This make it simpler to > > > > read the KIP. > > > > > > > > > > > > nit: typo > > > > > > > > > the fault value is false > > > > > > > > Should be "the default value is false". > > > > > > > > > > > > Not sure if `setReverse()` is the best name. Maybe > > `withDescandingOrder` > > > > (or similar, I guess `withReverseOrder` would also work) might be > > > > better? Would be good to align to KIP-969 proposal that suggest do > use > > > > `withDescendingKeys` methods for "reverse key-range"; if we go with > > > > `withReverseOrder` we should change KIP-969 accordingly. > > > > > > > > Curious to hear what others think about naming this consistently > across > > > > both KIPs. > > > > > > > > > > > > -Matthias > > > > > > > > > > > > On 10/3/23 9:17 AM, Hanyu (Peter) Zheng wrote: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-985%3A+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2 > > > > > > > > > > > > > > > > > > -- > > > >
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
Ok, I just talked about it with Matthias, I will change the text back to following the code, and update them together. Sincerely, Hanyu On Tue, Oct 3, 2023 at 3:49 PM Bill Bejeck wrote: > Hi Hanyu, > > Thanks for the KIP, overall it's looking good, but I have a couple of > comments > >- In the “Proposed Changes” section there's a reference to >`setReverse()` but the code example has “withDescendingOrder()” so I > think >the text needs an update to reflect the code example. >- I prefer “withDescendingKeys()” to “withDescendingOrder()” >- I also agree that we should include a section on ordering, but it >should be fairly straightforward. The “StateQueryRequest” of IQv2 > allows >users to specify a partition or partitions, so if ordering is important >they can elect to provide a single partition in the query. > > > Thanks, > Bill > > On Tue, Oct 3, 2023 at 5:27 PM Walker Carlson > > wrote: > > > Hello Hanyu, > > > > Looking over your kip things mostly make sense but I have a couple of > > comments. > > > > > >1. You have "withDescandingOrder()". I think you mean "descending" :) > >Also there are still a few places in the do where its called > > "setReverse" > >2. Also I like "WithDescendingKeys()" better > >3. I'm not sure of what ordering guarantees we are offering. Perhaps > we > >can add a section to the motivation clearly spelling out the current > >ordering and the new offering? > >4. When you say "use unbounded reverseQuery to achieve reverseAll" do > >you mean "use unbounded RangeQuery to achieve reverseAll"? as far as I > > can > >tell we don't have a reverseQuery as a named object? > > > > > > Looking good so far > > > > best, > > Walker > > > > On Tue, Oct 3, 2023 at 2:13 PM Colt McNealy wrote: > > > > > Hello Hanyu, > > > > > > Thank you for the KIP. I agree with Matthias' proposal to keep the > naming > > > convention consistent with KIP-969. I favor the `.withDescendingKeys()` > > > name. > > > > > > I am curious about one thing. RocksDB guarantees that records returned > > > during a range scan are lexicographically ordered by the bytes of the > > keys > > > (either ascending or descending order, as specified in the query). This > > > means that results within a single partition are indeed ordered.** My > > > reading of KIP-805 suggests to me that you don't need to specify the > > > partition number you are querying in IQv2, which means that you can > have > > a > > > valid reversed RangeQuery over a store with "multiple partitions" in > it. > > > > > > Currently, IQv1 does not guarantee order of keys in this scenario. Does > > > IQv2 support ordering across partitions? Such an implementation would > > > require opening a rocksdb range scan** on multiple rocksdb instances > (one > > > per partition), and polling the first key of each. Whether or not this > is > > > ordered, could we please add that to the documentation? > > > > > > **(How is this implemented/guaranteed in an `inMemoryKeyValueStore`? I > > > don't know about that implementation). > > > > > > Colt McNealy > > > > > > *Founder, LittleHorse.dev* > > > > > > > > > On Tue, Oct 3, 2023 at 1:35 PM Hanyu (Peter) Zheng > > > wrote: > > > > > > > ok, I will update it. Thank you Matthias > > > > > > > > Sincerely, > > > > Hanyu > > > > > > > > On Tue, Oct 3, 2023 at 11:23 AM Matthias J. Sax > > > wrote: > > > > > > > > > Thanks for the KIP Hanyu! > > > > > > > > > > > > > > > I took a quick look and it think the proposal makes sense overall. > > > > > > > > > > A few comments about how to structure the KIP. > > > > > > > > > > As you propose to not add `ReverseRangQuery` class, the code > example > > > > > should go into "Rejected Alternatives" section, not in the > "Proposed > > > > > Changes" section. > > > > > > > > > > For the `RangeQuery` code example, please omit all existing methods > > > etc, > > > > > and only include what will be added/changed. This make it simpler > to > > > > > read the KIP. > > &
Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
ok, I will update it. Thank you Matthias Sincerely, Hanyu On Tue, Oct 3, 2023 at 11:23 AM Matthias J. Sax wrote: > Thanks for the KIP Hanyu! > > > I took a quick look and it think the proposal makes sense overall. > > A few comments about how to structure the KIP. > > As you propose to not add `ReverseRangQuery` class, the code example > should go into "Rejected Alternatives" section, not in the "Proposed > Changes" section. > > For the `RangeQuery` code example, please omit all existing methods etc, > and only include what will be added/changed. This make it simpler to > read the KIP. > > > nit: typo > > > the fault value is false > > Should be "the default value is false". > > > Not sure if `setReverse()` is the best name. Maybe `withDescandingOrder` > (or similar, I guess `withReverseOrder` would also work) might be > better? Would be good to align to KIP-969 proposal that suggest do use > `withDescendingKeys` methods for "reverse key-range"; if we go with > `withReverseOrder` we should change KIP-969 accordingly. > > Curious to hear what others think about naming this consistently across > both KIPs. > > > -Matthias > > > On 10/3/23 9:17 AM, Hanyu (Peter) Zheng wrote: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-985%3A+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2 > > > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
[DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2
https://cwiki.apache.org/confluence/display/KAFKA/KIP-985%3A+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2 -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Re: Need generate a KIP
Thank you, Josep. Yes I also need Jira rights. On Mon, Oct 2, 2023 at 9:05 AM Josep Prat wrote: > Hi Peter, > > You are set :) Please share if you also need Jira rights. > > Best, > > On Mon, Oct 2, 2023 at 5:59 PM Hanyu (Peter) Zheng > wrote: > > > wiki ID: pzheng > > Jira ID: hanyuzheng > > > > -- > > > > [image: Confluent] <https://www.confluent.io> > > Hanyu (Peter) Zheng he/him/his > > Software Engineer Intern > > +1 (213) 431-7193 <+1+(213)+431-7193> > > Follow us: [image: Blog] > > < > > > https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog > > >[image: > > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > > <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > > <https://slackpass.io/confluentcommunity>[image: YouTube] > > <https://youtube.com/confluent> > > > > [image: Try Confluent Cloud for Free] > > < > > > https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic > > > > > > > > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud > > > <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
Need generate a KIP
wiki ID: pzheng Jira ID: hanyuzheng -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic>
