[jira] [Created] (KAFKA-13605) Checkpoint position in state stores

2022-01-20 Thread Patrick Stuedi (Jira)
Patrick Stuedi created KAFKA-13605:
--

 Summary: Checkpoint position in state stores
 Key: KAFKA-13605
 URL: https://issues.apache.org/jira/browse/KAFKA-13605
 Project: Kafka
  Issue Type: Improvement
Reporter: Patrick Stuedi
Assignee: Patrick Stuedi


There are cases in which a state store neither has an in-memory position built 
up nor has it gone through the state restoration process. If a store is 
persistent (i.e., RocksDB), and we stop and restart Streams, we will have 
neither of those continuity mechanisms available. This ticket is to fill in 
that gap.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


Re: [VOTE] KIP-806: Add session and window query over KV-store in IQv2

2021-12-17 Thread Patrick Stuedi
Thanks everyone for voting.

Voting passed with the following +1s:

- Luke Chen
- Guozhang Wang (binding)
- John Roesler (binding)
- Bruno Cadonna (binding)

I'll update the KIP status accordingly.

Best,
  Patrick

On Thu, Dec 16, 2021 at 2:09 PM Bruno Cadonna  wrote:

> Hi Partick,
>
> Thank you for the KIP!
>
> +1 (binding)
>
> Best,
> Bruno
>
> On 16.12.21 07:22, Luke Chen wrote:
> > Hi Patrick, John,
> >
> > Thanks for your explanation and update.
> > It looks better now.
> >
> > +1 (non-binding) from me.
> >
> > just one minor comment:
> > 10. In the example section, we are still using the variable names `lower`
> > and `upper` as before. We should update them, too. (follow the decision
> for
> > point 7 above)
> >
> > Thank you.
> > Luke
> >
> > On Thu, Dec 16, 2021 at 3:34 AM Guozhang Wang 
> wrote:
> >
> >> Thanks Patrick, John.
> >>
> >> I read through the updated KIP and it's much cleared on the scope now,
> >> appreciate that! I'm +1 overall, modulo a few minor comments:
> >>
> >> 7. The parameter names of `WindowKeyQuery#withKeyAndWindowStartRange`,
> i.e.
> >> `startTime` and `endTime` seem not updated; also for the parameter
> names of
> >> `WindowRangeQuery#withWindowStartRange`, the `earliest / latest` seems
> >> inconsistent with the existing API parameter names, how about naming
> both
> >> of them `fromWindowStartTime` and `toWindowStartTime`?
> >>
> >> 8.  `getStartTime / getEndTime` are not updated as well: should they be
> >> `getEarliestWindowStartTime` / `getLatestWindowStartTime` or `getFrom...
> >> getTo...` if you agree with 7) above?
> >>
> >> 9. "One reason we cannot use WindowKeyQuery to support
> >> SessionStore.fetch(key)": not sure I understand this part, for the new
> APIs
> >> we do not need to be following the old API's return types since they are
> >> separated, right? So if we think it's actually better for the new API
> >> mimicing `sessionStore.fetch(key)` to return a `WindowStoreIterator`,
> >> why can't we just do that?
> >>
> >>
> >> Guozhang
> >>
> >> On Wed, Dec 15, 2021 at 8:49 AM John Roesler 
> wrote:
> >>
> >>> FYI, I filed this follow-on task to explore a more general
> >>> pattern for these queries:
> >>> https://issues.apache.org/jira/browse/KAFKA-13548
> >>>
> >>> We can unblock the current scope for these queries but still
> >>> plan to revisit the API before the first release of IQv2.
> >>>
> >>> Thanks!
> >>> -John
> >>>
> >>> On Wed, 2021-12-15 at 10:34 -0600, John Roesler wrote:
> >>>> Thanks for the update, Patrick!
> >>>>
> >>>> Tl;dr: I'm +1 (binding)
> >>>>
> >>>> I just reviewed the KIP again (I hope you don't mind, I
> >>>> fixed a couple of missed renames in the text and examples).
> >>>>
> >>>> One of the design of IQv2 is to make proposing and evolving
> >>>> these queries much less onerous. Unlike adding new methods
> >>>> to all StateStore interfaces and implementations, if we want
> >>>> to make (eg) the WindowRangeQuery more flexible in the
> >>>> future, we can easily do so by just adding some builder
> >>>> methods to set the bounds independently, or by adding new
> >>>> static methods to provide different parameterizations.
> >>>>
> >>>> Therefore, even though this is not the ultimate expression
> >>>> of what we think the range queries should be, it's a
> >>>> perfectly fine starting point. The most pressing concerns to
> >>>> me were the cases where Luke and Guozhang pointed out that
> >>>> some parts of the proposal or interfaces were ambiguous,
> >>>> which looks like it's fixed now.
> >>>>
> >>>> My preference would be to go ahead and accept this proposed
> >>>> scope so that we have at least some basic key-value, window,
> >>>> and session store queries in the code base. Until we have
> >>>> those MVP queries in place, we can't really start to address
> >>>> higher-level follow-up items like:
> >>>> https://issues.apache.org/jira/browse/KAFKA-13541 (to refine
> >>>> how the serdes interplay with the queries) or
> 

Re: [VOTE] KIP-806: Add session and window query over KV-store in IQv2

2021-12-15 Thread Patrick Stuedi
1, 2021 at 7:54 AM Guozhang Wang 
> > wrote:
> > >
> > > > Hi Patrick,
> > > >
> > > > I made a pass on the KIP and have a few comments below:
> > > >
> > > > 1. The `WindowRangeQuery` has a private constructor while the
> > > > `WindowKeyQuery` has not, is that intentional?
> > > >
> > > > 2. The `WindowRangeQuery` seems not allowing to range over both
> window
> > and
> > > > key, but only window with a fixed key, in that case it seems pretty
> > much
> > > > the same as the other (ignoring the constructor), since we know we
> > would
> > > > only have a single `key` value in the returned iterator, and hence it
> > seems
> > > > returning in the form of `WindowStoreIterator` is also fine as the
> > key
> > > > is fixed and hence no need to maintain it in the returned iterator.
> I'm
> > > > wondering should we actually support ranging over keys as well in
> > > > `WindowRangeQuery`?
> > > >
> > > > 3. The KIP title mentioned both session and window, but the APIs only
> > > > involves window stores; However the return type `WindowStoreIterator`
> > is
> > > > only for window stores not session stores, so I feel we would still
> > have
> > > > some differences for session window query interface?
> > > >
> > > >
> > > > Guozhang
> > > >
> > > > On Fri, Dec 10, 2021 at 1:32 PM Patrick Stuedi
> > > > 
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I would like to start the vote for KIP-806 that adds window and
> > session
> > > > > query support to query KV-stores using IQv2.
> > > > >
> > > > > The KIP can be found here:
> > > > > https://cwiki.apache.org/confluence/x/LJaqCw
> > > > >
> > > > > Skipping the discussion phase as this KIP is following the same
> > pattern
> > > > as
> > > > > the previously submitted KIP-805 (KIP:
> > > > > https://cwiki.apache.org/confluence/x/85OqCw, Discussion:
> > > > > https://tinyurl.com/msp5mcb2). Of course concerns/comments can
> > still be
> > > > > brought up in this thread.
> > > > >
> > > > > -Patrick
> > > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> >
> >
>
> --
> -- Guozhang
>


[jira] [Created] (KAFKA-13541) Make IQv2 query/store interface type safe

2021-12-13 Thread Patrick Stuedi (Jira)
Patrick Stuedi created KAFKA-13541:
--

 Summary: Make IQv2 query/store interface type safe
 Key: KAFKA-13541
 URL: https://issues.apache.org/jira/browse/KAFKA-13541
 Project: Kafka
  Issue Type: Sub-task
Reporter: Patrick Stuedi
Assignee: Patrick Stuedi


Currently the new IQv2 interface allows applications to query state stores 
using subclasses of the Query type. Unfortunately there is currently no way 
to check that the template type of the query matches the type of the relevant 
store the query is executed on. As a consequence stores have to do a set of 
unsafe casts.

This ticket is to explore ways to make the query interface type safe where only 
type mismatches are detected at compile time.  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[VOTE] KIP-806: Add session and window query over KV-store in IQv2

2021-12-10 Thread Patrick Stuedi
Hi everyone,

I would like to start the vote for KIP-806 that adds window and session
query support to query KV-stores using IQv2.

The KIP can be found here:
https://cwiki.apache.org/confluence/x/LJaqCw

Skipping the discussion phase as this KIP is following the same pattern as
the previously submitted KIP-805 (KIP:
https://cwiki.apache.org/confluence/x/85OqCw, Discussion:
https://tinyurl.com/msp5mcb2). Of course concerns/comments can still be
brought up in this thread.

-Patrick


[jira] [Created] (KAFKA-13494) Session and Window Queries for IQv2

2021-12-01 Thread Patrick Stuedi (Jira)
Patrick Stuedi created KAFKA-13494:
--

 Summary: Session and Window Queries for IQv2
 Key: KAFKA-13494
 URL: https://issues.apache.org/jira/browse/KAFKA-13494
 Project: Kafka
  Issue Type: Sub-task
Reporter: Patrick Stuedi
Assignee: Patrick Stuedi






--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Created] (KAFKA-13493) Session and Window Queries for IQv2

2021-12-01 Thread Patrick Stuedi (Jira)
Patrick Stuedi created KAFKA-13493:
--

 Summary: Session and Window Queries for IQv2
 Key: KAFKA-13493
 URL: https://issues.apache.org/jira/browse/KAFKA-13493
 Project: Kafka
  Issue Type: Improvement
Reporter: Patrick Stuedi
Assignee: Patrick Stuedi






--
This message was sent by Atlassian Jira
(v8.20.1#820001)


Re: [VOTE} KIP-796: Interactive Query v2

2021-11-18 Thread Patrick Stuedi
+1 (non-binding), thanks John!
-Patrick

On Thu, Nov 18, 2021 at 12:27 AM John Roesler  wrote:

> Hello all,
>
> I'd like to open the vote for KIP-796, which proposes
> a revamp of the Interactive Query APIs in Kafka Streams.
>
> The proposal is here:
> https://cwiki.apache.org/confluence/x/34xnCw
>
> Thanks to all who reviewed the proposal, and thanks in
> advance for taking the time to vote!
>
> Thank you,
> -John
>


Re: [DISCUSS] KIP-796: Interactive Query v2

2021-11-15 Thread Patrick Stuedi
Hi John,

Thanks for submitting the KIP! One question I have is, assuming one
instantiates InteractiveQueryRequest via withQuery, and then later calls
getPositionBound, what will the result be? Also I noticed the Position
returning method is in InteractiveQueryRequest and InteractiveQueryResult
is named differently, any particular reason?

Best,
  Patrick


On Fri, Nov 12, 2021 at 12:29 AM John Roesler  wrote:

> Thanks for taking a look, Sophie!
>
> Ah, that was a revision error. I had initially been planning
> an Optional> with Optional.empty() meaning to
> fetch all partitions, but then decided it was needlessly
> complex and changed it to the current proposal with two
> methods:
>
> boolean isAllPartitions();
> Set getPartitions(); (which would throw an
> exception if it's an "all partitions" request).
>
> I've corrected the javadoc and also documented the
> exception.
>
> Thanks!
> -John
>
> On Thu, 2021-11-11 at 15:03 -0800, Sophie Blee-Goldman
> wrote:
> > Thanks John, I've been looking forward to this for a while now. It
> > was pretty horrifying to learn
> > how present-day IQ works  (or rather, doesn't work) with custom state
> > stores :/
> >
> > One minor cosmetic point, In the InteractiveQueryRequest class, the #
> > getPartitions
> > method has a return type of Set, but the javadocs refer to
> Optional.
> > Not
> > sure which is intended for this API, but if is supposed to be the return
> > type, do you perhaps
> > mean for it to be  Optional.ofEmpty() and Optional.of(non-empty set)
> > rather than Optional.of(empty set) and Optional.of(non-empty set) ?
> >
> > On Thu, Nov 11, 2021 at 12:03 PM John Roesler 
> wrote:
> >
> > > Hello again, all,
> > >
> > > Just bumping this discussion on a new, more flexible
> > > Interactive Query API in Kafka Streams.
> > >
> > > If there are no concerns, I'll go ahead and call a vote on
> > > Monday.
> > >
> > > Thanks!
> > > -John
> > >
> > > On Tue, 2021-11-09 at 17:37 -0600, John Roesler wrote:
> > > > Hello all,
> > > >
> > > > I'd like to start the discussion for KIP-796, which proposes
> > > > a revamp of the Interactive Query APIs in Kafka Streams.
> > > >
> > > > The proposal is here:
> > > > https://cwiki.apache.org/confluence/x/34xnCw
> > > >
> > > > I look forward to your feedback!
> > > >
> > > > Thank you,
> > > > -John
> > > >
> > >
> > >
> > >
>
>
>


Re: [VOTE] KIP-791: Add Record Metadata to State Store Context

2021-11-11 Thread Patrick Stuedi
Thanks everyone for voting.

Voting passed after 3 days with the following +1s:

- Luke Chen
- Vasiliki (Vicky) Papavasileiou
- Guozhang Wang (binding)
- John Roesler (binding)
- Bruno Cadonna (binding)

I'll update the KIP status accordingly.

Best,
  Patrick

On Wed, Nov 10, 2021 at 5:03 PM Bruno Cadonna  wrote:

> Patrick,
>
> Thank you for the KIP!
>
> +1 (binding)
>
> Best,
> Bruno
>
> On 09.11.21 16:58, John Roesler wrote:
> > +1 (binding) from me.
> >
> > Thanks, Patrick!
> >
> > On Mon, 2021-11-08 at 14:08 -0800, Guozhang Wang wrote:
> >> +1, thanks Patrick!
> >>
> >>
> >> Guozhang
> >>
> >> On Mon, Nov 8, 2021 at 5:44 AM Vasiliki Papavasileiou
> >>  wrote:
> >>
> >>> Hi Patrick,
> >>>
> >>> Having the recordMetadata available in the state stores is fundamental
> for
> >>> the consistency work and the proposed approach is reasonable.
> >>>
> >>> +1 (non-binding)
> >>>
> >>> Thank you,
> >>> Vicky
> >>>
> >>> On Mon, Nov 8, 2021 at 10:00 AM Luke Chen  wrote:
> >>>
> >>>> Hi Patrick,
> >>>> Thanks for the KIP.
> >>>> Adding RecordMetadata into StateStoreContext for offset updating makes
> >>>> sense to me.
> >>>>
> >>>> +1 (non-binding)
> >>>>
> >>>> Thank you.
> >>>> Luke
> >>>>
> >>>>
> >>>> On Mon, Nov 8, 2021 at 5:18 PM Patrick Stuedi
> >>>  >>>>>
> >>>> wrote:
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> Thanks for the feedback on KIP-791, I have updated the KIP and would
> >>> like
> >>>>> to start the voting.
> >>>>>
> >>>>> The KIP can be found here:
> >>>>> https://cwiki.apache.org/confluence/x/I5BnCw
> >>>>>
> >>>>> Please vote in this thread.
> >>>>>
> >>>>> Thanks!
> >>>>> -Patrick
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
> >
>


[VOTE] KIP-791: Add Record Metadata to State Store Context

2021-11-08 Thread Patrick Stuedi
Hi all,

Thanks for the feedback on KIP-791, I have updated the KIP and would like
to start the voting.

The KIP can be found here:
https://cwiki.apache.org/confluence/x/I5BnCw

Please vote in this thread.

Thanks!
-Patrick


[DISCUSS] KIP-791: Add Record Metadata to State Store Context

2021-11-03 Thread Patrick Stuedi
Hi everyone,

I would like to start the discussion for KIP-791: Add Record Metadata to
State Store Context.

The KIP can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-791:+Add+Record+Metadata+to+State+Store+Context

Any feedback will be highly appreciated.

Many thanks,
 Patrick


[jira] [Created] (KAFKA-13426) Add recordMetadata to StateStoreContext

2021-11-01 Thread Patrick Stuedi (Jira)
Patrick Stuedi created KAFKA-13426:
--

 Summary: Add recordMetadata to StateStoreContext
 Key: KAFKA-13426
 URL: https://issues.apache.org/jira/browse/KAFKA-13426
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Patrick Stuedi
Assignee: Patrick Stuedi


In order for state stores to provide stronger consistency in the future (e.g., 
RYW consistency) they need to be able to collect record metadata (e.g., offset 
information).

Today, we already make record metadata available in the AbstractProcessContext 
(recordMetadata()), but the call is not currently exposed through the 
StateStoreContext interface that is used by the state store. 

The task of this ticket is to expose recordMetadata in the StateStoreContext. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: [DISCUSS] KIP-763: Range queries with open endpoints

2021-09-08 Thread Patrick Stuedi
Boyang,

Thanks for your comment. The concern about whether a null-based approach
increases the chances of hidden bugs is a valid concern. In the end, the
decision to go with this approach was guided by considering the effort and
implications of changing the API towards a new range interface, versus the
likelihood of such bugs happening. Changing an API is always a
major effort. On other hand, I think the risk of hidden bugs with the
null-based API are low, for two reasons. First, there should not be any
existing application developed for the old API passing null, as null
parameters did throw an exception in the old API. New applications passing
null without the intention to do an open endpoint range query are, I would
say, unlikely, and if they happen then it's clearly the application
developer's mistake who did not read the API documentation.

-Patrick

On Sat, Aug 28, 2021 at 8:24 AM Boyang Chen 
wrote:

> Thanks Patrick for the KIP and sorry for being late to the party here. One
> thing I would like to understand is, what's the expected behavior when a
> user specifies a null key in previous versions? In the new version which
> uses null key for open-ended range queries, could it potentially hide bugs
> in the user's code when their intention was to do a bounded range query but
> accidentally set the key to null?
>
> Boyang
>
> On Wed, Jul 21, 2021 at 2:59 AM Patrick Stuedi
> 
> wrote:
>
> > Thanks John, Bruno for the valuable feedback!
> >
> > John: I had a quick look at the SessionStore and WindowStore interface.
> > While it looks like we should be able to apply similar ideas to those
> > stores, the actual interfaces are slightly different and expanding them
> for
> > open ranges may need a bit more thinking. In that sense, and to make sure
> > we will be converging with this KIP, I'd prefer to not expand the scope
> of
> > the KIP . As Bruno suggested we can always propose changes to the
> > SessionStore and WindowStore in a separate KIP. I'll add a subsection in
> > the rejected alternatives.
> >
> > @Bruno: good point, I'll add an example. Yes, all() will be equivalent to
> > range(null, null) and the implementation of all() will be a call to
> > range(null, null).
> >
> > -Patrick
> >
> > On Wed, Jul 21, 2021 at 9:12 AM Bruno Cadonna 
> wrote:
> >
> > > Thanks for the KIP, Patrick!
> > >
> > > I agree with John that your KIP is well motivated.
> > >
> > > I have just one minor feedback. Could you add store.range(null, null)
> to
> > > the example snippets with the comment that this is equivalent to all()
> > > (it is equivalent, right?)? This question about equivalence between
> > > range(null, null) and all() came up in my mind when I read the KIP and
> I
> > > think I am not the only one.
> > >
> > > Regarding expanding the KIP to SessionStore and WindowStore, I think
> you
> > > should not expand scope of the KIP. We can do the changes to the
> > > SessionStore and WindowStore in a separate KIP. But it is your call!
> > >
> > >
> > > Best,
> > > Bruno
> > >
> > > On 21.07.21 00:18, John Roesler wrote:
> > > > Thanks for the KIP, Patrick!
> > > >
> > > > I think your KIP is well motivated. It's definitely a bummer
> > > > to have to iterate over the full store as a workaround for
> > > > open-ended ranges.
> > > >
> > > > I agree with your pragmatic approach here. We have recently
> > > > had a couple of other contributions to the store APIs
> > > > (prefix and reverseRange), and the complexity of adding
> > > > those new methods far exceeded what anyone expected. I'd be
> > > > in favor of being conservative with new store APIs until we
> > > > are able to reign in that complexity somehow. Since your
> > > > proposed semantics seem reasonable, I'm in favor.
> > > >
> > > > While reviewing your KIP, it struck me that we have several
> > > > range-like APIs on:
> > > > * SessionStore
> > > > (
> > >
> >
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/ReadOnlySessionStore.java
> > > )
> > > > * WindowStore
> > > > (
> > >
> >
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/ReadOnlyWindowStore.java
> > > )
> > > > as well.
> > > >
> > > > Do you think it would be a good idea to also pro

Re: [VOTE] KIP-763: Range queries with open endpoints

2021-07-28 Thread Patrick Stuedi
Thanks everyone for voting.

The votes passes after 7 days with the following +1s:

- Luke Chen
- Matthias Sax (binding)
- Bruno Cadonna (binding)
- John Roesler (binding)
- Guozhang Wang (binding)

I'll update the KIP status accordingly.

Best,
  Patrick

On Tue, Jul 27, 2021 at 6:49 PM Guozhang Wang  wrote:

> Thank you, Patrick!
>
> The KIP looks good to me, and I also agree with the pragmatic manner. +1
> (binding)
>
> Guozhang
>
>
> On Thu, Jul 22, 2021 at 11:09 AM John Roesler  wrote:
>
> > Thank you, Patrick,
> >
> > +1 (binding) from me as well!
> >
> > Thanks,
> > -John
> >
> > On Thu, 2021-07-22 at 10:40 +0200, Bruno Cadonna wrote:
> > > Hi Patrick,
> > >
> > > Thank you for the KIP!
> > >
> > > +1 (binding)
> > >
> > > Best,
> > > Bruno
> > >
> > > On 22.07.21 03:47, Luke Chen wrote:
> > > > Hi Patrick,
> > > > I like this KIP!
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Luke
> > > >
> > > > On Thu, Jul 22, 2021 at 7:04 AM Matthias J. Sax 
> > wrote:
> > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > > On 7/21/21 1:18 PM, Patrick Stuedi wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > Thanks for the feedback on the KIP, I have updated the KIP and
> > would like
> > > > > > to start the voting.
> > > > > >
> > > > > > The KIP can be found here:
> > > > > >
> > > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-763%3A+Range+queries+with+open+endpoints
> > > > > >
> > > > > > Please vote in this thread.
> > > > > >
> > > > > > Thanks!
> > > > > > -Patrick
> > > > > >
> > > > >
> > > >
> >
> >
> >
>
> --
> -- Guozhang
>


[jira] [Created] (KAFKA-13130) Deprecate long based range queries in SessionStore

2021-07-23 Thread Patrick Stuedi (Jira)
Patrick Stuedi created KAFKA-13130:
--

 Summary: Deprecate long based range queries in SessionStore
 Key: KAFKA-13130
 URL: https://issues.apache.org/jira/browse/KAFKA-13130
 Project: Kafka
  Issue Type: Improvement
  Components: streams, streams-test-utils
Reporter: Patrick Stuedi
Assignee: Patrick Stuedi
 Fix For: 3.1.0


Migrate long based queries in ReadOnlySessionStore (fetchSession*, 
findSession*, etc.) to object based interfaces. Deprecate old long based 
interface, similar to how it was done for the ReadOnlyWindowStore.

Related KIPs:

[https://cwiki.apache.org/confluence/display/KAFKA/KIP-358%3A+Migrate+Streams+API+to+Duration+instead+of+long+ms+times]

[https://cwiki.apache.org/confluence/display/KAFKA/KIP-617%3A+Allow+Kafka+Streams+State+Stores+to+be+iterated+backwards]

 

Related Jiras:

https://issues.apache.org/jira/browse/KAFKA-12419

https://issues.apache.org/jira/browse/KAFKA-12526

https://issues.apache.org/jira/browse/KAFKA-12451

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[VOTE] KIP-763: Range queries with open endpoints

2021-07-21 Thread Patrick Stuedi
Hi all,

Thanks for the feedback on the KIP, I have updated the KIP and would like
to start the voting.

The KIP can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-763%3A+Range+queries+with+open+endpoints

Please vote in this thread.

Thanks!
-Patrick


Re: [DISCUSS] KIP-763: Range queries with open endpoints

2021-07-21 Thread Patrick Stuedi
Thanks John, Bruno for the valuable feedback!

John: I had a quick look at the SessionStore and WindowStore interface.
While it looks like we should be able to apply similar ideas to those
stores, the actual interfaces are slightly different and expanding them for
open ranges may need a bit more thinking. In that sense, and to make sure
we will be converging with this KIP, I'd prefer to not expand the scope of
the KIP . As Bruno suggested we can always propose changes to the
SessionStore and WindowStore in a separate KIP. I'll add a subsection in
the rejected alternatives.

@Bruno: good point, I'll add an example. Yes, all() will be equivalent to
range(null, null) and the implementation of all() will be a call to
range(null, null).

-Patrick

On Wed, Jul 21, 2021 at 9:12 AM Bruno Cadonna  wrote:

> Thanks for the KIP, Patrick!
>
> I agree with John that your KIP is well motivated.
>
> I have just one minor feedback. Could you add store.range(null, null) to
> the example snippets with the comment that this is equivalent to all()
> (it is equivalent, right?)? This question about equivalence between
> range(null, null) and all() came up in my mind when I read the KIP and I
> think I am not the only one.
>
> Regarding expanding the KIP to SessionStore and WindowStore, I think you
> should not expand scope of the KIP. We can do the changes to the
> SessionStore and WindowStore in a separate KIP. But it is your call!
>
>
> Best,
> Bruno
>
> On 21.07.21 00:18, John Roesler wrote:
> > Thanks for the KIP, Patrick!
> >
> > I think your KIP is well motivated. It's definitely a bummer
> > to have to iterate over the full store as a workaround for
> > open-ended ranges.
> >
> > I agree with your pragmatic approach here. We have recently
> > had a couple of other contributions to the store APIs
> > (prefix and reverseRange), and the complexity of adding
> > those new methods far exceeded what anyone expected. I'd be
> > in favor of being conservative with new store APIs until we
> > are able to reign in that complexity somehow. Since your
> > proposed semantics seem reasonable, I'm in favor.
> >
> > While reviewing your KIP, it struck me that we have several
> > range-like APIs on:
> > * SessionStore
> > (
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/ReadOnlySessionStore.java
> )
> > * WindowStore
> > (
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/ReadOnlyWindowStore.java
> )
> > as well.
> >
> > Do you think it would be a good idea to also propose a
> > similar change on those APIs? It might be nice (for exactly
> > the same reasons you documented), but it also increases the
> > scope of your work. There is one extra wrinkle I can see:
> > SessionStore has versions of the range methods that take a
> > `long` instead of an `Instant`, so `null` wouldn't work for
> > them.
> >
> > If you prefer to keep your KIP narrow in scope to just the
> > KeyValueStore, I'd also support you. In that case, it might
> > be a good idea to simply mention in the "Rejected
> > Alternatives" that you decided not to address those other
> > store APIs at this time. That way, people later on won't
> > have to wonder why those other methods didn't get updated.
> >
> > Other than that, I have no concerns!
> >
> > Thank you,
> > John
> >
> > On Mon, 2021-07-19 at 13:22 +0200, Patrick Stuedi wrote:
> >> Hi everyone,
> >>
> >> I would like to start the discussion for KIP-763: Range queries with
> open
> >> endpoints.
> >>
> >> The KIP can be found here:
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-763%3A+Range+queries+with+open+endpoints
> >>
> >> Any feedback will be highly appreciated.
> >>
> >> Many thanks,
> >>   Patrick
> >
> >
>


[DISCUSS] KIP-763: Range queries with open endpoints

2021-07-19 Thread Patrick Stuedi
Hi everyone,

I would like to start the discussion for KIP-763: Range queries with open
endpoints.

The KIP can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-763%3A+Range+queries+with+open+endpoints

Any feedback will be highly appreciated.

Many thanks,
 Patrick


Apache Kafka, permission to contribute

2021-07-19 Thread Patrick Stuedi
Hi all,

I would like to request permission to contribute to Apache Kafka. My wiki
and jira id is "pstuedi".

Thanks!
-Patrick