Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Hey Guozhang, Thank you for looking into the KIP. Windowed stores are addressed in another KIP. FYI, I made a change to the KIP and removed the `RawRangeQuery`. After some more thought, it doesn't provide us with many benefits (we save on one cast) which doesn't justify the cost of adding an extra query to the public interface. On Thu, Dec 9, 2021 at 9:50 PM Guozhang Wang wrote: > Hi Vicky, > > Thanks for the KIP. Just for a bit more clarification, could you elaborate > an example for windowed stores, beyond a key-value store (I think the > `myStore` is for kv-store right?). Otherwise LGTM. > > > Guozhang > > On Wed, Dec 8, 2021 at 4:18 PM Matthias J. Sax wrote: > > > Thanks for the details! > > > > I also chatted with John about it, and he filed > > https://issues.apache.org/jira/browse/KAFKA-13526 to incorporate some > > feedback as follow up work. > > > > IMHO, the hard coded query translation is not ideal and should be > > plugable. But for a v1 of IQv2 (pun intended) the hardcoded translation > > seems to be good enough. > > > > > > -Matthias > > > > On 12/8/21 9:37 AM, Vasiliki Papavasileiou wrote: > > > Hey Matthias, > > > > > > Thank you for looking into the KIP! > > > > > > We are adding raw versions of typed queries, like `RawRangeQuery` > because > > > it simplifies internal query handling since the bytes stores only > support > > > raw queries. A typed RangeQuery is handled by the `MeteredStore` which > > > creates a new `RawRangeQuery` to pass down to the wrapped stores. When > it > > > gets the result back, it deserializes the data and creates a typed > query > > > result to return to the user. So, the store's key serde are used to > > > translate typed `RangeQueries` into `RawRangeQueries` and it's value > > serde > > > are used to translate the result of the query on the way back. This > > allows > > > users to provide their own queries even if the MeteredStore has no > > > knowledge of them. > > > > > > I hope this answers your question. Let me know if you have any other > > > questions. > > > > > > Best, > > > Vicky > > > > > > > > > On Tue, Dec 7, 2021 at 12:46 AM Matthias J. Sax > > wrote: > > > > > >> Thanks for the KIP. Overall, make sense. > > >> > > >> One question: What is the purpose to `RawRangeQuery`? Seems not very > > >> user friendly. > > >> > > >> -Matthias > > >> > > >> > > >> On 11/30/21 12:48 PM, Vasiliki Papavasileiou wrote: > > >>> Thank you John! Yes, that was a typo from copying and I fixed it. > > >>> > > >>> Since there have been no more comments, I will start the vote. > > >>> > > >>> Best, > > >>> Vicky > > >>> > > >>> On Tue, Nov 30, 2021 at 5:22 AM John Roesler > > >> wrote: > > >>> > > Thanks for the KIP, Vicky! > > > > This KIP will help fill in the parity gap between IQ and > > IQv2. > > > > One thing I noticed, which looks like just a typo is that > > the value type of the proposed RangeQuery should probably be > > KeyValueIterator, right? > > > > Otherwise, it looks good to me! > > > > Thanks, > > -John > > > > On Mon, 2021-11-29 at 12:20 +, Vasiliki Papavasileiou > > wrote: > > > Hello everyone, > > > > > > I would like to start the discussion for KIP-805: Add range and > scan > > query > > > support in IQ v2 > > > > > > The KIP can be found here: > > > > > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2 > > > > > > Any suggestions are more than welcome. > > > > > > Many thanks, > > > Vicky > > > > > > >>> > > >> > > > > > > > > -- > -- Guozhang >
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Hi Vicky, Thanks for the KIP. Just for a bit more clarification, could you elaborate an example for windowed stores, beyond a key-value store (I think the `myStore` is for kv-store right?). Otherwise LGTM. Guozhang On Wed, Dec 8, 2021 at 4:18 PM Matthias J. Sax wrote: > Thanks for the details! > > I also chatted with John about it, and he filed > https://issues.apache.org/jira/browse/KAFKA-13526 to incorporate some > feedback as follow up work. > > IMHO, the hard coded query translation is not ideal and should be > plugable. But for a v1 of IQv2 (pun intended) the hardcoded translation > seems to be good enough. > > > -Matthias > > On 12/8/21 9:37 AM, Vasiliki Papavasileiou wrote: > > Hey Matthias, > > > > Thank you for looking into the KIP! > > > > We are adding raw versions of typed queries, like `RawRangeQuery` because > > it simplifies internal query handling since the bytes stores only support > > raw queries. A typed RangeQuery is handled by the `MeteredStore` which > > creates a new `RawRangeQuery` to pass down to the wrapped stores. When it > > gets the result back, it deserializes the data and creates a typed query > > result to return to the user. So, the store's key serde are used to > > translate typed `RangeQueries` into `RawRangeQueries` and it's value > serde > > are used to translate the result of the query on the way back. This > allows > > users to provide their own queries even if the MeteredStore has no > > knowledge of them. > > > > I hope this answers your question. Let me know if you have any other > > questions. > > > > Best, > > Vicky > > > > > > On Tue, Dec 7, 2021 at 12:46 AM Matthias J. Sax > wrote: > > > >> Thanks for the KIP. Overall, make sense. > >> > >> One question: What is the purpose to `RawRangeQuery`? Seems not very > >> user friendly. > >> > >> -Matthias > >> > >> > >> On 11/30/21 12:48 PM, Vasiliki Papavasileiou wrote: > >>> Thank you John! Yes, that was a typo from copying and I fixed it. > >>> > >>> Since there have been no more comments, I will start the vote. > >>> > >>> Best, > >>> Vicky > >>> > >>> On Tue, Nov 30, 2021 at 5:22 AM John Roesler > >> wrote: > >>> > Thanks for the KIP, Vicky! > > This KIP will help fill in the parity gap between IQ and > IQv2. > > One thing I noticed, which looks like just a typo is that > the value type of the proposed RangeQuery should probably be > KeyValueIterator, right? > > Otherwise, it looks good to me! > > Thanks, > -John > > On Mon, 2021-11-29 at 12:20 +, Vasiliki Papavasileiou > wrote: > > Hello everyone, > > > > I would like to start the discussion for KIP-805: Add range and scan > query > > support in IQ v2 > > > > The KIP can be found here: > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2 > > > > Any suggestions are more than welcome. > > > > Many thanks, > > Vicky > > > >>> > >> > > > -- -- Guozhang
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Thanks for the details! I also chatted with John about it, and he filed https://issues.apache.org/jira/browse/KAFKA-13526 to incorporate some feedback as follow up work. IMHO, the hard coded query translation is not ideal and should be plugable. But for a v1 of IQv2 (pun intended) the hardcoded translation seems to be good enough. -Matthias On 12/8/21 9:37 AM, Vasiliki Papavasileiou wrote: Hey Matthias, Thank you for looking into the KIP! We are adding raw versions of typed queries, like `RawRangeQuery` because it simplifies internal query handling since the bytes stores only support raw queries. A typed RangeQuery is handled by the `MeteredStore` which creates a new `RawRangeQuery` to pass down to the wrapped stores. When it gets the result back, it deserializes the data and creates a typed query result to return to the user. So, the store's key serde are used to translate typed `RangeQueries` into `RawRangeQueries` and it's value serde are used to translate the result of the query on the way back. This allows users to provide their own queries even if the MeteredStore has no knowledge of them. I hope this answers your question. Let me know if you have any other questions. Best, Vicky On Tue, Dec 7, 2021 at 12:46 AM Matthias J. Sax wrote: Thanks for the KIP. Overall, make sense. One question: What is the purpose to `RawRangeQuery`? Seems not very user friendly. -Matthias On 11/30/21 12:48 PM, Vasiliki Papavasileiou wrote: Thank you John! Yes, that was a typo from copying and I fixed it. Since there have been no more comments, I will start the vote. Best, Vicky On Tue, Nov 30, 2021 at 5:22 AM John Roesler wrote: Thanks for the KIP, Vicky! This KIP will help fill in the parity gap between IQ and IQv2. One thing I noticed, which looks like just a typo is that the value type of the proposed RangeQuery should probably be KeyValueIterator, right? Otherwise, it looks good to me! Thanks, -John On Mon, 2021-11-29 at 12:20 +, Vasiliki Papavasileiou wrote: Hello everyone, I would like to start the discussion for KIP-805: Add range and scan query support in IQ v2 The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2 Any suggestions are more than welcome. Many thanks, Vicky
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Hey Matthias, Thank you for looking into the KIP! We are adding raw versions of typed queries, like `RawRangeQuery` because it simplifies internal query handling since the bytes stores only support raw queries. A typed RangeQuery is handled by the `MeteredStore` which creates a new `RawRangeQuery` to pass down to the wrapped stores. When it gets the result back, it deserializes the data and creates a typed query result to return to the user. So, the store's key serde are used to translate typed `RangeQueries` into `RawRangeQueries` and it's value serde are used to translate the result of the query on the way back. This allows users to provide their own queries even if the MeteredStore has no knowledge of them. I hope this answers your question. Let me know if you have any other questions. Best, Vicky On Tue, Dec 7, 2021 at 12:46 AM Matthias J. Sax wrote: > Thanks for the KIP. Overall, make sense. > > One question: What is the purpose to `RawRangeQuery`? Seems not very > user friendly. > > -Matthias > > > On 11/30/21 12:48 PM, Vasiliki Papavasileiou wrote: > > Thank you John! Yes, that was a typo from copying and I fixed it. > > > > Since there have been no more comments, I will start the vote. > > > > Best, > > Vicky > > > > On Tue, Nov 30, 2021 at 5:22 AM John Roesler > wrote: > > > >> Thanks for the KIP, Vicky! > >> > >> This KIP will help fill in the parity gap between IQ and > >> IQv2. > >> > >> One thing I noticed, which looks like just a typo is that > >> the value type of the proposed RangeQuery should probably be > >> KeyValueIterator, right? > >> > >> Otherwise, it looks good to me! > >> > >> Thanks, > >> -John > >> > >> On Mon, 2021-11-29 at 12:20 +, Vasiliki Papavasileiou > >> wrote: > >>> Hello everyone, > >>> > >>> I would like to start the discussion for KIP-805: Add range and scan > >> query > >>> support in IQ v2 > >>> > >>> The KIP can be found here: > >>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2 > >>> > >>> Any suggestions are more than welcome. > >>> > >>> Many thanks, > >>> Vicky > >> > >> > > >
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Thanks for the KIP. Overall, make sense. One question: What is the purpose to `RawRangeQuery`? Seems not very user friendly. -Matthias On 11/30/21 12:48 PM, Vasiliki Papavasileiou wrote: Thank you John! Yes, that was a typo from copying and I fixed it. Since there have been no more comments, I will start the vote. Best, Vicky On Tue, Nov 30, 2021 at 5:22 AM John Roesler wrote: Thanks for the KIP, Vicky! This KIP will help fill in the parity gap between IQ and IQv2. One thing I noticed, which looks like just a typo is that the value type of the proposed RangeQuery should probably be KeyValueIterator, right? Otherwise, it looks good to me! Thanks, -John On Mon, 2021-11-29 at 12:20 +, Vasiliki Papavasileiou wrote: Hello everyone, I would like to start the discussion for KIP-805: Add range and scan query support in IQ v2 The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2 Any suggestions are more than welcome. Many thanks, Vicky
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Thank you John! Yes, that was a typo from copying and I fixed it. Since there have been no more comments, I will start the vote. Best, Vicky On Tue, Nov 30, 2021 at 5:22 AM John Roesler wrote: > Thanks for the KIP, Vicky! > > This KIP will help fill in the parity gap between IQ and > IQv2. > > One thing I noticed, which looks like just a typo is that > the value type of the proposed RangeQuery should probably be > KeyValueIterator, right? > > Otherwise, it looks good to me! > > Thanks, > -John > > On Mon, 2021-11-29 at 12:20 +, Vasiliki Papavasileiou > wrote: > > Hello everyone, > > > > I would like to start the discussion for KIP-805: Add range and scan > query > > support in IQ v2 > > > > The KIP can be found here: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2 > > > > Any suggestions are more than welcome. > > > > Many thanks, > > Vicky > >
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Thanks for the KIP, Vicky! This KIP will help fill in the parity gap between IQ and IQv2. One thing I noticed, which looks like just a typo is that the value type of the proposed RangeQuery should probably be KeyValueIterator, right? Otherwise, it looks good to me! Thanks, -John On Mon, 2021-11-29 at 12:20 +, Vasiliki Papavasileiou wrote: > Hello everyone, > > I would like to start the discussion for KIP-805: Add range and scan query > support in IQ v2 > > The KIP can be found here: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2 > > Any suggestions are more than welcome. > > Many thanks, > Vicky
[DISCUSS] KIP-805: Add range and scan query support in IQ v2
Hello everyone, I would like to start the discussion for KIP-805: Add range and scan query support in IQ v2 The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2 Any suggestions are more than welcome. Many thanks, Vicky