Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Sorry for the late reply, thanks for the review.


> About `StateStoreMigratedException`:
>
> Why is it only thrown if the state is REBALANCING? A store might be
> migrated during a rebalance, and Kafka Streams might resume back to
> RUNNING state and afterward somebody tries to use an old store handle.
> Also, if state is REBALANCING, should we throw
> `StreamThreadRebalancingException`? Hence, I think
> `StateStoreMigratedException` does only make sense during `RUNNING` state.
>

Thank you point this, already updated.


Why do we need to distinguish between `KafkaStreamsNotRunningException`
> and `StateStoreNotAvailableException`?
>

`KafkaStreamsNotRunningException` may be caused by various reasons, I think
it would be helpful that the
user can distinguish whether it is caused by the state store closed.
(Maybe I am wrong...)


Last, why do we distinguish between `KafkaStreams` instance and
> `StreamsThread`? To me, it seems we should always refer to the instance,
> because that is the level of granularity in which we enable/disable IQ atm.
>

Totally agree. Do you mean the naming of state store exceptions?
I don't have special reason to distinguish these two.
Your suggestion look more reasonable for the exception naming.


Last, for `StateStoreMigratedException`, I would add that a user need to
> rediscover the store and cannot blindly retry as the store handle is
> invalid and a new store handle must be retrieved. That is a difference
> to `StreamThreadRebalancingException` that allows for "blind" retries
> that either resolve (if the store is still on the same instance after
> rebalancing finishes, or changes to `StateStoreMigratedException` if the
> store was migrated away during rebalancing).
>

Nice, it's great! Thank you.


The KIP already updated, please take a look. :)



On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax 
wrote:

> Any update on this KIP?
>
> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> > Sorry for the late reply. The 2.4 deadline kept us quite busy.
> >
> > About `StateStoreMigratedException`:
> >
> > Why is it only thrown if the state is REBALANCING? A store might be
> > migrated during a rebalance, and Kafka Streams might resume back to
> > RUNNING state and afterward somebody tries to use an old store handle.
> > Also, if state is REBALANCING, should we throw
> > `StreamThreadRebalancingException`? Hence, I think
> > `StateStoreMigratedException` does only make sense during `RUNNING`
> state.
> >
> >
> > Why do we need to distinguish between `KafkaStreamsNotRunningException`
> > and `StateStoreNotAvailableException`?
> >
> >
> > Last, why do we distinguish between `KafkaStreams` instance and
> > `StreamsThread`? To me, it seems we should always refer to the instance,
> > because that is the level of granularity in which we enable/disable IQ
> atm.
> >
> >
> > Last, for `StateStoreMigratedException`, I would add that a user need to
> > rediscover the store and cannot blindly retry as the store handle is
> > invalid and a new store handle must be retrieved. That is a difference
> > to `StreamThreadRebalancingException` that allows for "blind" retries
> > that either resolve (if the store is still on the same instance after
> > rebalancing finishes, or changes to `StateStoreMigratedException` if the
> > store was migrated away during rebalancing).
> >
> >
> >
> > -Matthias
> >
> > On 8/9/19 10:20 AM, Vito Jeng wrote:
> >> My bad. The short link `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> >> <https://shorturl.at/CDNT9> seems incorrect.
> >>
> >> Please use the following instead: https://shorturl.at/bkKQU
> >>
> >>
> >> ---
> >> Vito
> >>
> >>
> >> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng  wrote:
> >>
> >>> Thanks, Matthias!
> >>>
> >>>> About `StreamThreadNotStartedException`:
> >>>
> >>> Thank you for explanation. I agree with your opinion.
> >>> `CompositeReadOnlyXxxStore#get()` would never throw
> >>> `StreamThreadNotStartedException`.
> >>>
> >>> For the case that corresponding thread crashes after we handed out the
> >>> store handle. We may throw `KafkaStreamsNotRunningException` or
> >>> `StateStoreMigratedException`.
> >>> In `StreamThreadStateStoreProvider`, we would throw
> >>> `KafkaStreamsNotRunningException` when stream thread is not running(
> >>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException` when
> >>> store is closed(h

Re: [VOTE] KIP-367 Introduce close(Duration) to Producer and AdminClient instead of close(long, TimeUnit)

[vito jeng]

+1



---
Vito

On Mon, Sep 10, 2018 at 4:52 PM, Dongjin Lee  wrote:

> +1. (Non-binding)
>
> On Mon, Sep 10, 2018 at 4:13 AM Matthias J. Sax 
> wrote:
>
> > Thanks a lot for the KIP.
> >
> > +1 (binding)
> >
> >
> > -Matthias
> >
> >
> > On 9/8/18 11:27 AM, Chia-Ping Tsai wrote:
> > > Hi All,
> > >
> > > I'd like to put KIP-367 to the vote.
> > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=89070496
> > >
> > > --
> > > Chia-Ping
> > >
> >
> >
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
> *github:  github.com/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> slideshare:
> www.slideshare.net/dongjinleekr
> *
>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Matthias,

Thanks for the review.
I reply separately in the following sections.


---
Vito

On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax 
wrote:

> Thanks for updating the KIP and sorry for the long pause...
>
> Seems you did a very thorough investigation of the code. It's useful to
> understand what user facing interfaces are affected.

(Some parts might be even too detailed for a KIP.)
>

I also think too detailed. Especially the section `Changes in call trace`.
Do you think it should be removed?


>
> To summarize my current understanding of your KIP, the main change is to
> introduce new exceptions that extend `InvalidStateStoreException`.
>

yep. Keep compatibility in this KIP is important things.
I think the best way is that all new exceptions extend from
`InvalidStateStoreException`.


>
> Some questions:
>
>  - Why do we need ```? Could `InvalidStateStoreException` be used for
> this purpose?
>

Does this question miss some word?


>
>  - What the superclass of `StateStoreStreamThreadNotRunningException`
> is? Should it be `InvalidStateStoreException` or `StateStoreFailException`
> ?
>
>  - Is `StateStoreClosed` a fatal or retryable exception ?
>
>
I apologize for not well written parts. I tried to modify some code in the
recent period and modify the KIP.
The modification is now complete. Please look again.


>
>
> -Matthias
>
>
> On 2/21/18 5:15 PM, vito jeng wrote:
> > Matthias,
> >
> > Sorry for not response these days.
> > I just finished it. Please have a look. :)
> >
> >
> >
> > ---
> > Vito
> >
> > On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax 
> > wrote:
> >
> >> Is there any update on this KIP?
> >>
> >> -Matthias
> >>
> >> On 1/3/18 12:59 AM, vito jeng wrote:
> >>> Matthias,
> >>>
> >>> Thank you for your response.
> >>>
> >>> I think you are right. We need to look at the state both of
> >>> KafkaStreams and StreamThread.
> >>>
> >>> After further understanding of KafkaStreams thread and state store,
> >>> I am currently rewriting the KIP.
> >>>
> >>>
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> matth...@confluent.io>
> >>> wrote:
> >>>
> >>>> Vito,
> >>>>
> >>>> Sorry for this late reply.
> >>>>
> >>>> There can be two cases:
> >>>>
> >>>>  - either a store got migrated way and thus, is not hosted an the
> >>>> application instance anymore,
> >>>>  - or, a store is hosted but the instance is in state rebalance
> >>>>
> >>>> For the first case, users need to rediscover the store. For the second
> >>>> case, they need to wait until rebalance is finished.
> >>>>
> >>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
> >>>> uses cannot query at all and thus they cannot rediscover or retry.
> >>>>
> >>>> Does this make sense?
> >>>>
> >>>> -Matthias
> >>>>
> >>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>> Matthias,
> >>>>>
> >>>>> I try to clarify some concept.
> >>>>>
> >>>>> When streams state is REBALANCING, it means the user can just plain
> >>>> retry.
> >>>>>
> >>>>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it
> >> means
> >>>>> state store migrated to another instance, the user needs to
> rediscover
> >>>> the
> >>>>> store.
> >>>>>
> >>>>> Is my understanding correct?
> >>>>>
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> >> matth...@confluent.io>
> >>>>> wrote:
> >>>>>
> >>>>>> Thanks for the KIP Vito!
> >>>>>>
> >>>>>> I agree with what Guozhang said. The original idea of the Jira was,
> to
> >>>>>> give different exceptions for different "recovery" strategies to the
> >>>> user.
> >>>>>>
> >>>>>> For example, if a store is curr

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Matthias,

Thanks for the feedback!

> It's up to you to keep the details part in the KIP or not.

Got it!

> The (incomplete) question was, if we need `StateStoreFailException` or
> if existing `InvalidStateStoreException` could be used? Do you suggest
> that `InvalidStateStoreException` is not thrown at all anymore, but only
> the new sub-classes (just to get a better understanding).

Yes. I suggest that `InvalidStateStoreException` is not thrown at all
anymore,
but only new sub-classes.

> Not sure what this sentence means:
>> The internal exception will be wrapped as category exception finally.
> Can you elaborate?

For example, `StreamThreadStateStoreProvider#stores()` will throw
`StreamThreadNotRunningException`(internal exception).
And then the internal exception will be wrapped as
`StateStoreRetryableException` or `StateStoreFailException` during the
`KafkaStreams.store()` and throw to the user.


> Can you explain the purpose of the "internal exceptions". It's unclear
to me atm why they are introduced.

Hmmm...the purpose of the "internal exceptions" is to distinguish between
the different kinds of InvalidStateStoreException.
The original idea is that we can distinguish different state store
exception for
different handling.
But to be honest, I am not quite sure this is necessary. Maybe have
some change during implementation.

Does it make sense?




---
Vito

On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax 
wrote:

> Thanks for the update Vito!
>
> It's up to you to keep the details part in the KIP or not.
>
>
> The (incomplete) question was, if we need `StateStoreFailException` or
> if existing `InvalidStateStoreException` could be used? Do you suggest
> that `InvalidStateStoreException` is not thrown at all anymore, but only
> the new sub-classes (just to get a better understanding).
>
>
> Not sure what this sentence means:
>
> > The internal exception will be wrapped as category exception finally.
>
> Can you elaborate?
>
>
> Can you explain the purpose of the "internal exceptions". It's unclear
> to me atm why they are introduced.
>
>
> -Matthias
>
> On 4/10/18 12:33 AM, vito jeng wrote:
> > Matthias,
> >
> > Thanks for the review.
> > I reply separately in the following sections.
> >
> >
> > ---
> > Vito
> >
> > On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax 
> > wrote:
> >
> >> Thanks for updating the KIP and sorry for the long pause...
> >>
> >> Seems you did a very thorough investigation of the code. It's useful to
> >> understand what user facing interfaces are affected.
> >
> > (Some parts might be even too detailed for a KIP.)
> >>
> >
> > I also think too detailed. Especially the section `Changes in call
> trace`.
> > Do you think it should be removed?
> >
> >
> >>
> >> To summarize my current understanding of your KIP, the main change is to
> >> introduce new exceptions that extend `InvalidStateStoreException`.
> >>
> >
> > yep. Keep compatibility in this KIP is important things.
> > I think the best way is that all new exceptions extend from
> > `InvalidStateStoreException`.
> >
> >
> >>
> >> Some questions:
> >>
> >>  - Why do we need ```? Could `InvalidStateStoreException` be used for
> >> this purpose?
> >>
> >
> > Does this question miss some word?
> >
> >
> >>
> >>  - What the superclass of `StateStoreStreamThreadNotRunningException`
> >> is? Should it be `InvalidStateStoreException` or
> `StateStoreFailException`
> >> ?
> >>
> >>  - Is `StateStoreClosed` a fatal or retryable exception ?
> >>
> >>
> > I apologize for not well written parts. I tried to modify some code in
> the
> > recent period and modify the KIP.
> > The modification is now complete. Please look again.
> >
> >
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 2/21/18 5:15 PM, vito jeng wrote:
> >>> Matthias,
> >>>
> >>> Sorry for not response these days.
> >>> I just finished it. Please have a look. :)
> >>>
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> matth...@confluent.io>
> >>> wrote:
> >>>
> >>>> Is there any update on this KIP?
> >>>>
> >>>> -Matthias
> >>>>
> >>>> On 1/3/18 12:59 AM, vito je

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Hi, Matthias,

Thanks for the review!

I have updated the KIP, please look again.


Thanks.


---
Vito

On Thu, Apr 19, 2018 at 9:58 PM, Matthias J. Sax 
wrote:

> Thanks for clarification! That makes sense to me.
>
> Can you update the KIP to make those suggestions explicit?
>
>
> -Matthias
>
> On 4/18/18 2:19 PM, vito jeng wrote:
> > Matthias,
> >
> > Thanks for the feedback!
> >
> >> It's up to you to keep the details part in the KIP or not.
> >
> > Got it!
> >
> >> The (incomplete) question was, if we need `StateStoreFailException` or
> >> if existing `InvalidStateStoreException` could be used? Do you suggest
> >> that `InvalidStateStoreException` is not thrown at all anymore, but only
> >> the new sub-classes (just to get a better understanding).
> >
> > Yes. I suggest that `InvalidStateStoreException` is not thrown at all
> > anymore,
> > but only new sub-classes.
> >
> >> Not sure what this sentence means:
> >>> The internal exception will be wrapped as category exception finally.
> >> Can you elaborate?
> >
> > For example, `StreamThreadStateStoreProvider#stores()` will throw
> > `StreamThreadNotRunningException`(internal exception).
> > And then the internal exception will be wrapped as
> > `StateStoreRetryableException` or `StateStoreFailException` during the
> > `KafkaStreams.store()` and throw to the user.
> >
> >
> >> Can you explain the purpose of the "internal exceptions". It's unclear
> > to me atm why they are introduced.
> >
> > Hmmm...the purpose of the "internal exceptions" is to distinguish between
> > the different kinds of InvalidStateStoreException.
> > The original idea is that we can distinguish different state store
> > exception for
> > different handling.
> > But to be honest, I am not quite sure this is necessary. Maybe have
> > some change during implementation.
> >
> > Does it make sense?
> >
> >
> >
> >
> > ---
> > Vito
> >
> > On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax 
> > wrote:
> >
> >> Thanks for the update Vito!
> >>
> >> It's up to you to keep the details part in the KIP or not.
> >>
> >>
> >> The (incomplete) question was, if we need `StateStoreFailException` or
> >> if existing `InvalidStateStoreException` could be used? Do you suggest
> >> that `InvalidStateStoreException` is not thrown at all anymore, but only
> >> the new sub-classes (just to get a better understanding).
> >>
> >>
> >> Not sure what this sentence means:
> >>
> >>> The internal exception will be wrapped as category exception finally.
> >>
> >> Can you elaborate?
> >>
> >>
> >> Can you explain the purpose of the "internal exceptions". It's unclear
> >> to me atm why they are introduced.
> >>
> >>
> >> -Matthias
> >>
> >> On 4/10/18 12:33 AM, vito jeng wrote:
> >>> Matthias,
> >>>
> >>> Thanks for the review.
> >>> I reply separately in the following sections.
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax  >
> >>> wrote:
> >>>
> >>>> Thanks for updating the KIP and sorry for the long pause...
> >>>>
> >>>> Seems you did a very thorough investigation of the code. It's useful
> to
> >>>> understand what user facing interfaces are affected.
> >>>
> >>> (Some parts might be even too detailed for a KIP.)
> >>>>
> >>>
> >>> I also think too detailed. Especially the section `Changes in call
> >> trace`.
> >>> Do you think it should be removed?
> >>>
> >>>
> >>>>
> >>>> To summarize my current understanding of your KIP, the main change is
> to
> >>>> introduce new exceptions that extend `InvalidStateStoreException`.
> >>>>
> >>>
> >>> yep. Keep compatibility in this KIP is important things.
> >>> I think the best way is that all new exceptions extend from
> >>> `InvalidStateStoreException`.
> >>>
> >>>
> >>>>
> >>>> Some questions:
> >>>>
> >>>>  - Why do we need ```? Could `InvalidStateStoreException` be used for
> >&g

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Hi, John,

Thanks for the comment!

KIP typically discuss the public interfaces, non-user-facing detail is not
necessary.
I think your opinion is correct.

But in this KIP, when a state store exception throw, the user will see the
full exception stacktrace in the log, including the internal exception.
So I think it would be better we made some description about the internal
exception in the KIP.

About the base class InvalidStateStoreException, I agree with Guozhang's
said.
We should expose it as public for people who do not want to handle the
internal exception differently.
Users can have more flexibility in how to handle errors.


Thanks.


---
Vito

On Sat, Apr 21, 2018 at 12:14 AM, John Roesler  wrote:

> Hi Vito,
>
> Thanks for the KIP!
>
> I think it's much nicer to give callers different exceptions to tell them
> whether the state store got migrated, whether it's still initializing, or
> whether there's some unrecoverable error.
>
> In the KIP, it's typically not necessary to discuss non-user-facing details
> such as what exceptions we will throw internally. The KIP is primarily to
> discuss public interface changes.
>
> You might consider simply removing all the internal details from the KIP,
> which will have the dual advantage that it makes the KIP smaller and easier
> to agree on, as well as giving you more freedom in the internal details
> when it comes to implementation.
>
> I like your decision to have your refined exceptions extend
> InvalidStateStoreException to ensure backward compatibility. Since we want
> to encourage callers to catch the more specific exceptions, and we don't
> expect to ever throw a raw InvalidStateStoreException anymore, you might
> consider adding the @Deprecated annotation to InvalidStateStoreException.
> This will gently encourage callers to migrate to the new exception and open
> the possibility of removing InvalidStateStoreException entirely in a future
> release.
>
> Thanks,
> -John
>
> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax 
> wrote:
>
> > Thanks for clarification! That makes sense to me.
> >
> > Can you update the KIP to make those suggestions explicit?
> >
> >
> > -Matthias
> >
> > On 4/18/18 2:19 PM, vito jeng wrote:
> > > Matthias,
> > >
> > > Thanks for the feedback!
> > >
> > >> It's up to you to keep the details part in the KIP or not.
> > >
> > > Got it!
> > >
> > >> The (incomplete) question was, if we need `StateStoreFailException` or
> > >> if existing `InvalidStateStoreException` could be used? Do you suggest
> > >> that `InvalidStateStoreException` is not thrown at all anymore, but
> only
> > >> the new sub-classes (just to get a better understanding).
> > >
> > > Yes. I suggest that `InvalidStateStoreException` is not thrown at all
> > > anymore,
> > > but only new sub-classes.
> > >
> > >> Not sure what this sentence means:
> > >>> The internal exception will be wrapped as category exception finally.
> > >> Can you elaborate?
> > >
> > > For example, `StreamThreadStateStoreProvider#stores()` will throw
> > > `StreamThreadNotRunningException`(internal exception).
> > > And then the internal exception will be wrapped as
> > > `StateStoreRetryableException` or `StateStoreFailException` during the
> > > `KafkaStreams.store()` and throw to the user.
> > >
> > >
> > >> Can you explain the purpose of the "internal exceptions". It's unclear
> > > to me atm why they are introduced.
> > >
> > > Hmmm...the purpose of the "internal exceptions" is to distinguish
> between
> > > the different kinds of InvalidStateStoreException.
> > > The original idea is that we can distinguish different state store
> > > exception for
> > > different handling.
> > > But to be honest, I am not quite sure this is necessary. Maybe have
> > > some change during implementation.
> > >
> > > Does it make sense?
> > >
> > >
> > >
> > >
> > > ---
> > > Vito
> > >
> > > On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> matth...@confluent.io>
> > > wrote:
> > >
> > >> Thanks for the update Vito!
> > >>
> > >> It's up to you to keep the details part in the KIP or not.
> > >>
> > >>
> > >> The (incomplete) question was, if we need `StateStoreFailException` or
> > >> if existing `InvalidStateStoreE

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Hi, Guozhang,

Thanks for the comment!



Hi, Bill,

I'll try to make some update to make the KIP better.

Thanks for the comment!


---
Vito

On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck  wrote:

> Hi Vito,
>
> Thanks for the KIP, overall it's a +1 from me.
>
> At this point, the only thing I would change is possibly removing the
> listing of all methods called by the user and the listing of all store
> types and focus on what states result in which exceptions thrown to the
> user.
>
> Thanks,
> Bill
>
> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang  wrote:
>
> > Thanks for the KIP Vito!
> >
> > I made a pass over the wiki and it looks great to me. I'm +1 on the KIP.
> >
> > About the base class InvalidStateStoreException itself, I'd actually
> > suggest we do not deprecate it but still expose it as part of the public
> > API, for people who do not want to handle these cases differently (if we
> > deprecate it then we are enforcing them to capture all three exceptions
> > one-by-one).
> >
> >
> > Guozhang
> >
> >
> > On Fri, Apr 20, 2018 at 9:14 AM, John Roesler  wrote:
> >
> > > Hi Vito,
> > >
> > > Thanks for the KIP!
> > >
> > > I think it's much nicer to give callers different exceptions to tell
> them
> > > whether the state store got migrated, whether it's still initializing,
> or
> > > whether there's some unrecoverable error.
> > >
> > > In the KIP, it's typically not necessary to discuss non-user-facing
> > details
> > > such as what exceptions we will throw internally. The KIP is primarily
> to
> > > discuss public interface changes.
> > >
> > > You might consider simply removing all the internal details from the
> KIP,
> > > which will have the dual advantage that it makes the KIP smaller and
> > easier
> > > to agree on, as well as giving you more freedom in the internal details
> > > when it comes to implementation.
> > >
> > > I like your decision to have your refined exceptions extend
> > > InvalidStateStoreException to ensure backward compatibility. Since we
> > want
> > > to encourage callers to catch the more specific exceptions, and we
> don't
> > > expect to ever throw a raw InvalidStateStoreException anymore, you
> might
> > > consider adding the @Deprecated annotation to
> InvalidStateStoreException.
> > > This will gently encourage callers to migrate to the new exception and
> > open
> > > the possibility of removing InvalidStateStoreException entirely in a
> > future
> > > release.
> > >
> > > Thanks,
> > > -John
> > >
> > > On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> matth...@confluent.io>
> > > wrote:
> > >
> > > > Thanks for clarification! That makes sense to me.
> > > >
> > > > Can you update the KIP to make those suggestions explicit?
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 4/18/18 2:19 PM, vito jeng wrote:
> > > > > Matthias,
> > > > >
> > > > > Thanks for the feedback!
> > > > >
> > > > >> It's up to you to keep the details part in the KIP or not.
> > > > >
> > > > > Got it!
> > > > >
> > > > >> The (incomplete) question was, if we need
> `StateStoreFailException`
> > or
> > > > >> if existing `InvalidStateStoreException` could be used? Do you
> > suggest
> > > > >> that `InvalidStateStoreException` is not thrown at all anymore,
> but
> > > only
> > > > >> the new sub-classes (just to get a better understanding).
> > > > >
> > > > > Yes. I suggest that `InvalidStateStoreException` is not thrown at
> all
> > > > > anymore,
> > > > > but only new sub-classes.
> > > > >
> > > > >> Not sure what this sentence means:
> > > > >>> The internal exception will be wrapped as category exception
> > finally.
> > > > >> Can you elaborate?
> > > > >
> > > > > For example, `StreamThreadStateStoreProvider#stores()` will throw
> > > > > `StreamThreadNotRunningException`(internal exception).
> > > > > And then the internal exception will be wrapped as
> > > > > `StateStoreRetryableException` or `StateStoreFailExceptio

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Matthias,

Thank you for your assistance.

> what is the status of this KIP?

Unfortunately, there is no further progress.
About seven weeks ago, I was injured in sports. I had a broken wrist on
my left wrist.
Many jobs are affected, including this KIP and implementation.


> I just re-read it, and have a couple of follow up comments. Why do we
> discuss the internal exceptions you want to add? Also, do we really need
> them? Can't we just throw the correct exception directly instead of
> wrapping it later?

I think you may be right. As I say in the previous:
"The original idea is that we can distinguish different state store
exception for different handling. But to be honest, I am not quite sure
this is necessary. Maybe have some change during implementation."

During the implementation, I also feel we maybe not need wrapper it.
We can just throw the correctly directly.


> Is `StreamThreadNotRunningException` really an retryable error?

When KafkaStream state is REBALANCING, I think it is a retryable error.

StreamThreadStateStoreProvider#stores() will throw
StreamThreadNotRunningException when StreamThread state is not RUNNING. The
user can retry until KafkaStream state is RUNNING.


> When would we throw an `StateStoreEmptyException`? The semantics is
unclear to me atm.

> When the state is RUNNING, is `StateStoreClosedException` a retryable
error?

These two comments will be answered in another mail.



---
Vito

On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax 
wrote:

> Vito,
>
> what is the status of this KIP?
>
> I just re-read it, and have a couple of follow up comments. Why do we
> discuss the internal exceptions you want to add? Also, do we really need
> them? Can't we just throw the correct exception directly instead of
> wrapping it later?
>
> When would we throw an `StateStoreEmptyException`? The semantics is
> unclear to me atm.
>
> Is `StreamThreadNotRunningException` really an retryable error?
>
> When the state is RUNNING, is `StateStoreClosedException` a retryable
> error?
>
> One more nits: ReadOnlyWindowStore got a new method #fetch(K key, long
> time); that should be added
>
>
> Overall I like the KIP but some details are still unclear. Maybe it
> might help if you open an PR in parallel?
>
>
> -Matthias
>
> On 4/24/18 8:18 AM, vito jeng wrote:
> > Hi, Guozhang,
> >
> > Thanks for the comment!
> >
> >
> >
> > Hi, Bill,
> >
> > I'll try to make some update to make the KIP better.
> >
> > Thanks for the comment!
> >
> >
> > ---
> > Vito
> >
> > On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck  wrote:
> >
> >> Hi Vito,
> >>
> >> Thanks for the KIP, overall it's a +1 from me.
> >>
> >> At this point, the only thing I would change is possibly removing the
> >> listing of all methods called by the user and the listing of all store
> >> types and focus on what states result in which exceptions thrown to the
> >> user.
> >>
> >> Thanks,
> >> Bill
> >>
> >> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang 
> wrote:
> >>
> >>> Thanks for the KIP Vito!
> >>>
> >>> I made a pass over the wiki and it looks great to me. I'm +1 on the
> KIP.
> >>>
> >>> About the base class InvalidStateStoreException itself, I'd actually
> >>> suggest we do not deprecate it but still expose it as part of the
> public
> >>> API, for people who do not want to handle these cases differently (if
> we
> >>> deprecate it then we are enforcing them to capture all three exceptions
> >>> one-by-one).
> >>>
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler 
> wrote:
> >>>
> >>>> Hi Vito,
> >>>>
> >>>> Thanks for the KIP!
> >>>>
> >>>> I think it's much nicer to give callers different exceptions to tell
> >> them
> >>>> whether the state store got migrated, whether it's still initializing,
> >> or
> >>>> whether there's some unrecoverable error.
> >>>>
> >>>> In the KIP, it's typically not necessary to discuss non-user-facing
> >>> details
> >>>> such as what exceptions we will throw internally. The KIP is primarily
> >> to
> >>>> discuss public interface changes.
> >>>>
> >>>> You might consider simply removing all the internal details from the
> >> KIP,
>

Re: [VOTE] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

[vito jeng]

Thanks for the KIP!

+1(non-binding)




---
Vito

On Sat, Jul 7, 2018 at 11:29 AM, Chia-Ping Tsai  wrote:

> hi Ismael
>
> > I think we should be thinking about
> > https://issues.apache.org/jira/browse/KAFKA-6923 at the same time.
>
> you are right. KAFKA-6923 and KAFKA-6161 are the keys to complete the
> Serializer and Deserializer. Let us add the default implementations first
> (KAFKA-6161). And then integrate the ExtendedSerializer/ExtendedDeSerializer
> to Serializer/Deserialize (KAFKA-6923)
>
> --
> Chia-Ping
>
> On 2018/07/06 15:33:18, Ismael Juma  wrote:
> > Thanks for the KIP. I think we should be thinking about
> > https://issues.apache.org/jira/browse/KAFKA-6923 at the same time.
> >
> > Ismael
> >
> > On Thu, 5 Jul 2018, 07:45 Chia-Ping Tsai,  wrote:
> >
> > > hi all,
> > >
> > > I would like to start voting on "KIP-331 Add default implementation to
> > > close() and configure() for Serializer, Deserializer and Serde"
> > >
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+
> implementation+to+close%28%29+and+configure%28%29+for+
> Serializer%2C+Deserializer+and+Serde
> > >
> > > Cheers,
> > > Chia-Ping
> > >
> >
>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Matthias,

Still in progress. I've been busy in recent weeks with my job.

But I believe I will update this KIP within few days.




---
Vito

On Tue, Dec 5, 2017 at 8:05 AM, Matthias J. Sax 
wrote:

> Vito,
>
> is there any update with regard to this KIP?
>
>
> -Matthias
>
> On 11/5/17 6:11 PM, vito jeng wrote:
> > Thanks, Guozhang and Matthias. Your comments very useful for me.
> >
> > I'll update KIP and keep going on.
> >
> >
> >
> > ---
> > Vito
> >
> > On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax 
> > wrote:
> >
> >> Thanks for the KIP Vito!
> >>
> >> I agree with what Guozhang said. The original idea of the Jira was, to
> >> give different exceptions for different "recovery" strategies to the
> user.
> >>
> >> For example, if a store is currently recreated, a user just need to wait
> >> and can query the store later. On the other hand, if a store go migrated
> >> to another instance, a user needs to rediscover the store instead of a
> >> "plain retry".
> >>
> >> Fatal errors might be a third category.
> >>
> >> Not sure if there is something else?
> >>
> >> Anyway, the KIP should contain a section that talks about this ideas and
> >> reasoning.
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>> Thanks for writing up the KIP.
> >>>
> >>> Vito, Matthias: one thing that I wanted to figure out first is what
> >>> categories of errors we want to notify the users, if we only wants to
> >>> distinguish fatal v.s. retriable then probably we should rename the
> >>> proposed StateStoreMigratedException / StateStoreClosedException
> classes.
> >>> And then from there we should list what are the possible internal
> >>> exceptions ever thrown in those APIs in the call trace, and which
> >>> exceptions should be wrapped to what others, and which ones should be
> >>> handled without re-throwing, and which ones should not be wrapped at
> all
> >>> but directly thrown to user's face.
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng 
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I'd like to start discuss KIP-216:
> >>>>
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>>>
> >>>> Please have a look.
> >>>> Thanks!
> >>>>
> >>>> ---
> >>>> Vito
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >
>
>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Thank you for your attention to this KIP and assistance.



---
Vito

On Wed, Dec 6, 2017 at 2:35 PM, Matthias J. Sax 
wrote:

> Thanks for the info. There is no hurry. Was just curious :)
>
> -Matthias
>
>
> On 12/5/17 7:34 PM, vito jeng wrote:
> > Matthias,
> >
> > Still in progress. I've been busy in recent weeks with my job.
> >
> > But I believe I will update this KIP within few days.
> >
> >
> >
> >
> > ---
> > Vito
> >
> > On Tue, Dec 5, 2017 at 8:05 AM, Matthias J. Sax 
> > wrote:
> >
> >> Vito,
> >>
> >> is there any update with regard to this KIP?
> >>
> >>
> >> -Matthias
> >>
> >> On 11/5/17 6:11 PM, vito jeng wrote:
> >>> Thanks, Guozhang and Matthias. Your comments very useful for me.
> >>>
> >>> I'll update KIP and keep going on.
> >>>
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> matth...@confluent.io>
> >>> wrote:
> >>>
> >>>> Thanks for the KIP Vito!
> >>>>
> >>>> I agree with what Guozhang said. The original idea of the Jira was, to
> >>>> give different exceptions for different "recovery" strategies to the
> >> user.
> >>>>
> >>>> For example, if a store is currently recreated, a user just need to
> wait
> >>>> and can query the store later. On the other hand, if a store go
> migrated
> >>>> to another instance, a user needs to rediscover the store instead of a
> >>>> "plain retry".
> >>>>
> >>>> Fatal errors might be a third category.
> >>>>
> >>>> Not sure if there is something else?
> >>>>
> >>>> Anyway, the KIP should contain a section that talks about this ideas
> and
> >>>> reasoning.
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>>
> >>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>> Thanks for writing up the KIP.
> >>>>>
> >>>>> Vito, Matthias: one thing that I wanted to figure out first is what
> >>>>> categories of errors we want to notify the users, if we only wants to
> >>>>> distinguish fatal v.s. retriable then probably we should rename the
> >>>>> proposed StateStoreMigratedException / StateStoreClosedException
> >> classes.
> >>>>> And then from there we should list what are the possible internal
> >>>>> exceptions ever thrown in those APIs in the call trace, and which
> >>>>> exceptions should be wrapped to what others, and which ones should be
> >>>>> handled without re-throwing, and which ones should not be wrapped at
> >> all
> >>>>> but directly thrown to user's face.
> >>>>>
> >>>>> Guozhang
> >>>>>
> >>>>>
> >>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng 
> >> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I'd like to start discuss KIP-216:
> >>>>>>
> >>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>>>>>
> >>>>>> Please have a look.
> >>>>>> Thanks!
> >>>>>>
> >>>>>> ---
> >>>>>> Vito
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Matthias,

I try to clarify some concept.

When streams state is REBALANCING, it means the user can just plain retry.

When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it means
state store migrated to another instance, the user needs to rediscover the
store.

Is my understanding correct?


---
Vito

On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax 
wrote:

> Thanks for the KIP Vito!
>
> I agree with what Guozhang said. The original idea of the Jira was, to
> give different exceptions for different "recovery" strategies to the user.
>
> For example, if a store is currently recreated, a user just need to wait
> and can query the store later. On the other hand, if a store go migrated
> to another instance, a user needs to rediscover the store instead of a
> "plain retry".
>
> Fatal errors might be a third category.
>
> Not sure if there is something else?
>
> Anyway, the KIP should contain a section that talks about this ideas and
> reasoning.
>
>
> -Matthias
>
>
> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> > Thanks for writing up the KIP.
> >
> > Vito, Matthias: one thing that I wanted to figure out first is what
> > categories of errors we want to notify the users, if we only wants to
> > distinguish fatal v.s. retriable then probably we should rename the
> > proposed StateStoreMigratedException / StateStoreClosedException classes.
> > And then from there we should list what are the possible internal
> > exceptions ever thrown in those APIs in the call trace, and which
> > exceptions should be wrapped to what others, and which ones should be
> > handled without re-throwing, and which ones should not be wrapped at all
> > but directly thrown to user's face.
> >
> > Guozhang
> >
> >
> > On Wed, Nov 1, 2017 at 11:09 PM, vito jeng  wrote:
> >
> >> Hi,
> >>
> >> I'd like to start discuss KIP-216:
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>
> >> Please have a look.
> >> Thanks!
> >>
> >> ---
> >> Vito
> >>
> >
> >
> >
>
>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Matthias,

Thank you for your response.

I think you are right. We need to look at the state both of
KafkaStreams and StreamThread.

After further understanding of KafkaStreams thread and state store,
I am currently rewriting the KIP.




---
Vito

On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax 
wrote:

> Vito,
>
> Sorry for this late reply.
>
> There can be two cases:
>
>  - either a store got migrated way and thus, is not hosted an the
> application instance anymore,
>  - or, a store is hosted but the instance is in state rebalance
>
> For the first case, users need to rediscover the store. For the second
> case, they need to wait until rebalance is finished.
>
> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
> uses cannot query at all and thus they cannot rediscover or retry.
>
> Does this make sense?
>
> -Matthias
>
> On 12/20/17 12:54 AM, vito jeng wrote:
> > Matthias,
> >
> > I try to clarify some concept.
> >
> > When streams state is REBALANCING, it means the user can just plain
> retry.
> >
> > When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it means
> > state store migrated to another instance, the user needs to rediscover
> the
> > store.
> >
> > Is my understanding correct?
> >
> >
> > ---
> > Vito
> >
> > On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax 
> > wrote:
> >
> >> Thanks for the KIP Vito!
> >>
> >> I agree with what Guozhang said. The original idea of the Jira was, to
> >> give different exceptions for different "recovery" strategies to the
> user.
> >>
> >> For example, if a store is currently recreated, a user just need to wait
> >> and can query the store later. On the other hand, if a store go migrated
> >> to another instance, a user needs to rediscover the store instead of a
> >> "plain retry".
> >>
> >> Fatal errors might be a third category.
> >>
> >> Not sure if there is something else?
> >>
> >> Anyway, the KIP should contain a section that talks about this ideas and
> >> reasoning.
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>> Thanks for writing up the KIP.
> >>>
> >>> Vito, Matthias: one thing that I wanted to figure out first is what
> >>> categories of errors we want to notify the users, if we only wants to
> >>> distinguish fatal v.s. retriable then probably we should rename the
> >>> proposed StateStoreMigratedException / StateStoreClosedException
> classes.
> >>> And then from there we should list what are the possible internal
> >>> exceptions ever thrown in those APIs in the call trace, and which
> >>> exceptions should be wrapped to what others, and which ones should be
> >>> handled without re-throwing, and which ones should not be wrapped at
> all
> >>> but directly thrown to user's face.
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng 
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I'd like to start discuss KIP-216:
> >>>>
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>>>
> >>>> Please have a look.
> >>>> Thanks!
> >>>>
> >>>> ---
> >>>> Vito
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >
>
>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Matthias,

Sorry for not response these days.
I just finished it. Please have a look. :)



---
Vito

On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax 
wrote:

> Is there any update on this KIP?
>
> -Matthias
>
> On 1/3/18 12:59 AM, vito jeng wrote:
> > Matthias,
> >
> > Thank you for your response.
> >
> > I think you are right. We need to look at the state both of
> > KafkaStreams and StreamThread.
> >
> > After further understanding of KafkaStreams thread and state store,
> > I am currently rewriting the KIP.
> >
> >
> >
> >
> > ---
> > Vito
> >
> > On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax 
> > wrote:
> >
> >> Vito,
> >>
> >> Sorry for this late reply.
> >>
> >> There can be two cases:
> >>
> >>  - either a store got migrated way and thus, is not hosted an the
> >> application instance anymore,
> >>  - or, a store is hosted but the instance is in state rebalance
> >>
> >> For the first case, users need to rediscover the store. For the second
> >> case, they need to wait until rebalance is finished.
> >>
> >> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
> >> uses cannot query at all and thus they cannot rediscover or retry.
> >>
> >> Does this make sense?
> >>
> >> -Matthias
> >>
> >> On 12/20/17 12:54 AM, vito jeng wrote:
> >>> Matthias,
> >>>
> >>> I try to clarify some concept.
> >>>
> >>> When streams state is REBALANCING, it means the user can just plain
> >> retry.
> >>>
> >>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it
> means
> >>> state store migrated to another instance, the user needs to rediscover
> >> the
> >>> store.
> >>>
> >>> Is my understanding correct?
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> matth...@confluent.io>
> >>> wrote:
> >>>
> >>>> Thanks for the KIP Vito!
> >>>>
> >>>> I agree with what Guozhang said. The original idea of the Jira was, to
> >>>> give different exceptions for different "recovery" strategies to the
> >> user.
> >>>>
> >>>> For example, if a store is currently recreated, a user just need to
> wait
> >>>> and can query the store later. On the other hand, if a store go
> migrated
> >>>> to another instance, a user needs to rediscover the store instead of a
> >>>> "plain retry".
> >>>>
> >>>> Fatal errors might be a third category.
> >>>>
> >>>> Not sure if there is something else?
> >>>>
> >>>> Anyway, the KIP should contain a section that talks about this ideas
> and
> >>>> reasoning.
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>>
> >>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>> Thanks for writing up the KIP.
> >>>>>
> >>>>> Vito, Matthias: one thing that I wanted to figure out first is what
> >>>>> categories of errors we want to notify the users, if we only wants to
> >>>>> distinguish fatal v.s. retriable then probably we should rename the
> >>>>> proposed StateStoreMigratedException / StateStoreClosedException
> >> classes.
> >>>>> And then from there we should list what are the possible internal
> >>>>> exceptions ever thrown in those APIs in the call trace, and which
> >>>>> exceptions should be wrapped to what others, and which ones should be
> >>>>> handled without re-throwing, and which ones should not be wrapped at
> >> all
> >>>>> but directly thrown to user's face.
> >>>>>
> >>>>> Guozhang
> >>>>>
> >>>>>
> >>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng 
> >> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I'd like to start discuss KIP-216:
> >>>>>>
> >>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>>>>>
> >>>>>> Please have a look.
> >>>>>> Thanks!
> >>>>>>
> >>>>>> ---
> >>>>>> Vito
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi, Matthias & Vinoth,

Thanks for the feedback.

> What is still unclear to me is, what we gain by having both
> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> state store is closed on not -- I can't query it anyway? Maybe I miss
> something thought?

Yes, both `StreamsNotRunningException` and
`StateStoreNotAvailableException` are fatal exception.
But `StateStoreNotAvailableException` is fatal exception about state store
related.
I think it would be helpful that if user need to distinguish these two
different case to handle it.

I'm not very sure, does that make sense?


---
Vito


On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar  wrote:

> +1 on merging `StreamsNotRunningException` and
> `StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO
> its best to have these exceptions be about the state store (and not streams
> state), to easier understanding.
>
> Additionally, KIP-535 allows for querying of state stores in rebalancing
> state. So do we need the StreamsRebalancingException?
>
>
> On 2020/01/09 03:38:11, "Matthias J. Sax"  wrote:
> > Sorry that I dropped the ball on this...
> >
> > Thanks for updating the KIP. Overall LGTM now. Feel free to start a VOTE
> > thread.
> >
> > What is still unclear to me is, what we gain by having both
> > `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
> > exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> > NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> > state store is closed on not -- I can't query it anyway? Maybe I miss
> > something thought?
> >
> >
> > -Matthias
> >
> >
> > On 11/3/19 6:07 PM, Vito Jeng wrote:
> > > Sorry for the late reply, thanks for the review.
> > >
> > >
> > >> About `StateStoreMigratedException`:
> > >>
> > >> Why is it only thrown if the state is REBALANCING? A store might be
> > >> migrated during a rebalance, and Kafka Streams might resume back to
> > >> RUNNING state and afterward somebody tries to use an old store handle.
> > >> Also, if state is REBALANCING, should we throw
> > >> `StreamThreadRebalancingException`? Hence, I think
> > >> `StateStoreMigratedException` does only make sense during `RUNNING`
> state.
> > >>
> > >
> > > Thank you point this, already updated.
> > >
> > >
> > > Why do we need to distinguish between `KafkaStreamsNotRunningException`
> > >> and `StateStoreNotAvailableException`?
> > >>
> > >
> > > `KafkaStreamsNotRunningException` may be caused by various reasons, I
> think
> > > it would be helpful that the
> > > user can distinguish whether it is caused by the state store closed.
> > > (Maybe I am wrong...)
> > >
> > >
> > > Last, why do we distinguish between `KafkaStreams` instance and
> > >> `StreamsThread`? To me, it seems we should always refer to the
> instance,
> > >> because that is the level of granularity in which we enable/disable
> IQ atm.
> > >>
> > >
> > > Totally agree. Do you mean the naming of state store exceptions?
> > > I don't have special reason to distinguish these two.
> > > Your suggestion look more reasonable for the exception naming.
> > >
> > >
> > > Last, for `StateStoreMigratedException`, I would add that a user need
> to
> > >> rediscover the store and cannot blindly retry as the store handle is
> > >> invalid and a new store handle must be retrieved. That is a difference
> > >> to `StreamThreadRebalancingException` that allows for "blind" retries
> > >> that either resolve (if the store is still on the same instance after
> > >> rebalancing finishes, or changes to `StateStoreMigratedException` if
> the
> > >> store was migrated away during rebalancing).
> > >>
> > >
> > > Nice, it's great! Thank you.
> > >
> > >
> > > The KIP already updated, please take a look. :)
> > >
> > >
> > >
> > > On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax  >
> > > wrote:
> > >
> > >> Any update on this KIP?
> > >>
> > >> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> > >>> Sorry for the late reply. The 2.4 deadline kep

[VOTE] KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi, all,

I would like to start the vote for KIP-216.

Currently, IQ throws InvalidStateStoreException for any types of error.
With this KIP, user can distinguish different types of error.

KIP is here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors

Thanks

---
Vito
-- 


---
Vito

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

lidStateStoreException
> * drop StateStoreNotAvailableException in favor of
> StreamsNotRunningException
> * clearly document on all public methods which exceptions
> need to be handled
>
> How do you feel about this?
> Thanks,
> -John
>
> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> > Thanks for KIP Vito.
> >
> > Overall the KIP LGTM, but I'd have to agree with others on merging the
> > `StreamsNotRunningException` and `StateStoreNotAvailableException`
> classes.
> >
> > Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
> > NOT_RUNNING || ERROR` I'm not even sure how we could distinguish when to
> > use the different
> > exceptions.  Maybe a good middle ground would be to have a detailed
> > exception message.
> >
> > The KIP freeze is close, so I think if we can agree on this, we can wrap
> up
> > the voting soon.
> >
> > Thanks,
> > Bill
> >
> > On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax 
> > wrote:
> >
> > > Vito,
> > >
> > > It's still unclear to me what the advantage is, to have both
> > > `StreamsNotRunningException` and `StateStoreNotAvailableException`?
> > >
> > > For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING / ERROR`
> > > and thus, for a user point of view, why does it matter if the store is
> > > closed on not? I don't understand why/how this information would be
> > > useful? Do you have a concrete example in mind how a user would react
> > > differently to both exceptions?
> > >
> > >
> > > @Vinoth: about `StreamsRebalancingException` -- to me, it seems best to
> > > actually do this on a per-query basis, ie, have an overload
> > > `KafkaStreams#store(...)` that takes a boolean flag that allow to
> > > _disable_ the exception and opt-in to query a active store during
> > > recovery. However, as KIP-535 actually introduces this change in
> > > behavior, I think KIP-216 should not cover this, but KIP-535 should be
> > > updated. I'll follow up on the other KIP thread to raise this point.
> > >
> > >
> > > -Matthias
> > >
> > > On 1/11/20 12:26 AM, Vito Jeng wrote:
> > > > Hi, Matthias & Vinoth,
> > > >
> > > > Thanks for the feedback.
> > > >
> > > >> What is still unclear to me is, what we gain by having both
> > > >> `StreamsNotRunningException` and `StateStoreNotAvailableException`.
> Both
> > > >> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN
> /
> > > >> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> > > >> state store is closed on not -- I can't query it anyway? Maybe I
> miss
> > > >> something thought?
> > > >
> > > > Yes, both `StreamsNotRunningException` and
> > > > `StateStoreNotAvailableException` are fatal exception.
> > > > But `StateStoreNotAvailableException` is fatal exception about state
> > > store
> > > > related.
> > > > I think it would be helpful that if user need to distinguish these
> two
> > > > different case to handle it.
> > > >
> > > > I'm not very sure, does that make sense?
> > > >
> > > >
> > > > ---
> > > > Vito
> > > >
> > > >
> > > > On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar 
> > > wrote:
> > > >
> > > >> +1 on merging `StreamsNotRunningException` and
> > > >> `StateStoreNotAvailableException`, both exceptions are fatal
> anyway. IMO
> > > >> its best to have these exceptions be about the state store (and not
> > > streams
> > > >> state), to easier understanding.
> > > >>
> > > >> Additionally, KIP-535 allows for querying of state stores in
> rebalancing
> > > >> state. So do we need the StreamsRebalancingException?
> > > >>
> > > >>
> > > >> On 2020/01/09 03:38:11, "Matthias J. Sax" 
> > > wrote:
> > > >>> Sorry that I dropped the ball on this...
> > > >>>
> > > >>> Thanks for updating the KIP. Overall LGTM now. Feel free to start a
> > > VOTE
> > > >>> thread.
> > > >>>
> > > >>> What is still unclear to me is, what we gain by having both
> > > >>> `StreamsNotRunningException` and `Stat

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi John,

About `StreamsNotStartedException is strange` --
The original idea came from Matthias, two years ago. :)
You can reference here:
https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc5...@confluent.io%3e

About omitting the categorization --
It looks reasonable. I'm fine with omitting the categorization but not very
sure it is a good choice.
Does any other folks provide opinion?


Hi, folks,

Just update the KIP-216, please take a look.

---
Vito


On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng  wrote:

>
> Hi, folks,
>
> Thank you suggestion, really appreciate it. :)
> I understand your concern. I'll merge StreamsNotRunningException and
> StateStoreNotAvailableException.
>
>
> ---
> Vito
>
>
> On Thu, Jan 16, 2020 at 6:22 AM John Roesler  wrote:
>
>> Hey Vito,
>>
>> Yes, thanks for the KIP. Sorry the discussion has been so long.
>> Hopefully, we can close it out soon.
>>
>> I agree we can drop StreamsNotRunningException in favor of
>> just StateStoreNotAvailableException.
>>
>> Unfortunately, I have some higher-level concerns. The value
>> of these exceptions is that they tell you how to handle the
>> various situations that can arise while querying a distributed
>> data store.
>>
>> Ideally, as a caller, I should be able to just catch "retriable" or
>> "fatal" and handle them appropriately. Otherwise, there's no
>> point in having categories, and we should just have all the
>> exceptions extend InvalidStateStoreException.
>>
>> Presently, it's not possible to tell from just the
>> "retriable"/"fatal" distinction what to do. You  can tell
>> from the descriptions of the various exceptions. E.g.:
>>
>> Retriable:
>>  * StreamsRebalancingException: the exact same call
>> should just be retried until the rebalance is complete
>>  * StateStoreMigratedException: the store handle is
>> now invalid, so you need to re-discover the instance
>> and get a new handle on that instance. In other words,
>> the query itself may be valid, but the particular method
>> invocation on this particular instance has encountered
>> a fatal exception.
>>
>> Fatal:
>>  * UnknownStateStoreException: this is truly fatal. No amount
>> of retrying or re-discovering is going to get you a handle on a
>> store that doesn't exist in the cluster.
>>  * StateStoreNotAvailableException: this is actually recoverable,
>> since the store might exist in the cluster, but isn't available on
>> this particular instance (which is shut down or whatever).
>>
>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
>> with omitting the categorization and just having 5 subclasses
>> of InvalidStateStoreException. Each of them would tell you
>> how to handle them, and it's not too many to really
>> understand and handle each one.
>>
>> If you really want to have a middle tier, I'd recommend:
>> * RetryableStateStoreException: the exact same call
>> should be repeated.
>> * RecoverableStateStoreException: the store handle
>> should be discarded and the caller should re-discover
>> the location of the store and repeat the query on the
>> correct instance.
>> * FatalStateStoreException: the query/request is totally
>> invalid and will never succeed.
>>
>> However, attempting to categorize the proposed exceptions
>> reveals even problems with this categorization:
>> Retriable:
>> * StreamsRebalancingException
>> Recoverable:
>> * StateStoreMigratedException
>> * StreamsNotRunningException
>> Fatal:
>> * UnknownStateStoreException
>>
>> But StreamsNotStartedException is strange... It means that
>> one code path got a handle on a specific KafkaStreams object
>> instance and sent it a query before another code path
>> invoked the start() method on the exact same object instance.
>> It seems like the most likely scenario is that whoever wrote
>> the program just forgot to call start() before querying, in
>> which case, retrying isn't going to help, and a fatal exception
>> is more appropriate. I.e., it sounds like a "first 15 minutes
>> experience" problem, and making it fatal would be more
>> helpful. Even in a production context, there's no reason not
>> to sequence your application startup such that you don't
>> accept queries until after Streams is started. Thus, I guess
>> I'd categorize it under "

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Thanks Bill, John and Matthias. Glad you guys joined this discussion.
I got a lot out of the discussion.

I would like to update KIP-216 base on John's suggestion to remove the
category.


---
Vito


On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax 
wrote:

> > Nevertheless, if we omit the categorization, it’s moot.
>
> Ack.
>
> I am fine to remove the middle tier. As John pointed out, it might be
> weird to have only one concrete exception type per category. We can also
> explain in detail how to handle each exception in their JavaDocs.
>
>
> -Matthias
>
> On 1/16/20 6:38 AM, Bill Bejeck wrote:
> > Vito,
> >
> > Thanks for the updates, the KIP LGTM.
> >
> > -Bill
> >
> > On Wed, Jan 15, 2020 at 11:31 PM John Roesler 
> wrote:
> >
> >> Hi Vito,
> >>
> >> Haha, your archive game is on point!
> >>
> >> What Matthias said in that email is essentially what I figured was the
> >> rationale. It makes sense, but the point I was making is that this
> really
> >> doesn’t seem like a good way to structure a production app. On the other
> >> hand, considering the exception fatal has a good chance of avoiding a
> >> frustrating debug session if you just forgot to call start.
> >>
> >> Nevertheless, if we omit the categorization, it’s moot.
> >>
> >> It would be easy to add a categorization layer later if we want it, but
> >> not very easy to change it if we get it wrong.
> >>
> >> Thanks for your consideration!
> >> -John
> >>
> >> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
> >>> Hi John,
> >>>
> >>> About `StreamsNotStartedException is strange` --
> >>> The original idea came from Matthias, two years ago. :)
> >>> You can reference here:
> >>>
> >>
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc5...@confluent.io%3e
> >>>
> >>> About omitting the categorization --
> >>> It looks reasonable. I'm fine with omitting the categorization but not
> >> very
> >>> sure it is a good choice.
> >>> Does any other folks provide opinion?
> >>>
> >>>
> >>> Hi, folks,
> >>>
> >>> Just update the KIP-216, please take a look.
> >>>
> >>> ---
> >>> Vito
> >>>
> >>>
> >>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng  wrote:
> >>>
> >>>>
> >>>> Hi, folks,
> >>>>
> >>>> Thank you suggestion, really appreciate it. :)
> >>>> I understand your concern. I'll merge StreamsNotRunningException and
> >>>> StateStoreNotAvailableException.
> >>>>
> >>>>
> >>>> ---
> >>>> Vito
> >>>>
> >>>>
> >>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler 
> >> wrote:
> >>>>
> >>>>> Hey Vito,
> >>>>>
> >>>>> Yes, thanks for the KIP. Sorry the discussion has been so long.
> >>>>> Hopefully, we can close it out soon.
> >>>>>
> >>>>> I agree we can drop StreamsNotRunningException in favor of
> >>>>> just StateStoreNotAvailableException.
> >>>>>
> >>>>> Unfortunately, I have some higher-level concerns. The value
> >>>>> of these exceptions is that they tell you how to handle the
> >>>>> various situations that can arise while querying a distributed
> >>>>> data store.
> >>>>>
> >>>>> Ideally, as a caller, I should be able to just catch "retriable" or
> >>>>> "fatal" and handle them appropriately. Otherwise, there's no
> >>>>> point in having categories, and we should just have all the
> >>>>> exceptions extend InvalidStateStoreException.
> >>>>>
> >>>>> Presently, it's not possible to tell from just the
> >>>>> "retriable"/"fatal" distinction what to do. You  can tell
> >>>>> from the descriptions of the various exceptions. E.g.:
> >>>>>
> >>>>> Retriable:
> >>>>>  * StreamsRebalancingException: the exact same call
> >>>>> should just be retried until the rebalance is complete
> >>>>>  * StateStoreMigratedExcep

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi, folks,

Just update the KIP, please take a look.

Thanks!

---
Vito


On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng  wrote:

> Thanks Bill, John and Matthias. Glad you guys joined this discussion.
> I got a lot out of the discussion.
>
> I would like to update KIP-216 base on John's suggestion to remove the
> category.
>
>
> ---
> Vito
>
>
> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax 
> wrote:
>
>> > Nevertheless, if we omit the categorization, it’s moot.
>>
>> Ack.
>>
>> I am fine to remove the middle tier. As John pointed out, it might be
>> weird to have only one concrete exception type per category. We can also
>> explain in detail how to handle each exception in their JavaDocs.
>>
>>
>> -Matthias
>>
>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
>> > Vito,
>> >
>> > Thanks for the updates, the KIP LGTM.
>> >
>> > -Bill
>> >
>> > On Wed, Jan 15, 2020 at 11:31 PM John Roesler 
>> wrote:
>> >
>> >> Hi Vito,
>> >>
>> >> Haha, your archive game is on point!
>> >>
>> >> What Matthias said in that email is essentially what I figured was the
>> >> rationale. It makes sense, but the point I was making is that this
>> really
>> >> doesn’t seem like a good way to structure a production app. On the
>> other
>> >> hand, considering the exception fatal has a good chance of avoiding a
>> >> frustrating debug session if you just forgot to call start.
>> >>
>> >> Nevertheless, if we omit the categorization, it’s moot.
>> >>
>> >> It would be easy to add a categorization layer later if we want it, but
>> >> not very easy to change it if we get it wrong.
>> >>
>> >> Thanks for your consideration!
>> >> -John
>> >>
>> >> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
>> >>> Hi John,
>> >>>
>> >>> About `StreamsNotStartedException is strange` --
>> >>> The original idea came from Matthias, two years ago. :)
>> >>> You can reference here:
>> >>>
>> >>
>> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc5...@confluent.io%3e
>> >>>
>> >>> About omitting the categorization --
>> >>> It looks reasonable. I'm fine with omitting the categorization but not
>> >> very
>> >>> sure it is a good choice.
>> >>> Does any other folks provide opinion?
>> >>>
>> >>>
>> >>> Hi, folks,
>> >>>
>> >>> Just update the KIP-216, please take a look.
>> >>>
>> >>> ---
>> >>> Vito
>> >>>
>> >>>
>> >>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng 
>> wrote:
>> >>>
>> >>>>
>> >>>> Hi, folks,
>> >>>>
>> >>>> Thank you suggestion, really appreciate it. :)
>> >>>> I understand your concern. I'll merge StreamsNotRunningException and
>> >>>> StateStoreNotAvailableException.
>> >>>>
>> >>>>
>> >>>> ---
>> >>>> Vito
>> >>>>
>> >>>>
>> >>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler 
>> >> wrote:
>> >>>>
>> >>>>> Hey Vito,
>> >>>>>
>> >>>>> Yes, thanks for the KIP. Sorry the discussion has been so long.
>> >>>>> Hopefully, we can close it out soon.
>> >>>>>
>> >>>>> I agree we can drop StreamsNotRunningException in favor of
>> >>>>> just StateStoreNotAvailableException.
>> >>>>>
>> >>>>> Unfortunately, I have some higher-level concerns. The value
>> >>>>> of these exceptions is that they tell you how to handle the
>> >>>>> various situations that can arise while querying a distributed
>> >>>>> data store.
>> >>>>>
>> >>>>> Ideally, as a caller, I should be able to just catch "retriable" or
>> >>>>> "fatal" and handle them appropriately. Otherwise, there's no
>> >>>>> point in having categories, and we should just have all the
>> >>>>> exceptions extend Inval

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Thanks Matthias.

The KIP is about InvalidStateStoreException.
I pick `StateStoreNotAvailableException` because it may be more intuitive
than `StreamsNotRunningException`.

No matter which one picked, it's good to me.

---
Vito


On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax 
wrote:

> Thanks for updating the KIP!
>
> One last comment/question: you kept `StateStoreNotAvailableException` in
> favor of `StreamsNotRunningException` (to merge both as suggested).
>
> I am wondering, if it might be better to keep
> `StreamsNotRunningException` instead of
> `StateStoreNotAvailableException`, because this exception is thrown if
> Streams is in state PENDING_SHUTDOWN / NOT_RUNNING / ERROR ?
>
>
>
> -Matthias
>
> On 1/17/20 9:56 PM, John Roesler wrote:
> > Thanks, Vito. I've just cast my vote.
> > -John
> >
> > On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
> >> Hi, folks,
> >>
> >> Just update the KIP, please take a look.
> >>
> >> Thanks!
> >>
> >> ---
> >> Vito
> >>
> >>
> >> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng  wrote:
> >>
> >>> Thanks Bill, John and Matthias. Glad you guys joined this discussion.
> >>> I got a lot out of the discussion.
> >>>
> >>> I would like to update KIP-216 base on John's suggestion to remove the
> >>> category.
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>>
> >>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax  >
> >>> wrote:
> >>>
> >>>>> Nevertheless, if we omit the categorization, it’s moot.
> >>>>
> >>>> Ack.
> >>>>
> >>>> I am fine to remove the middle tier. As John pointed out, it might be
> >>>> weird to have only one concrete exception type per category. We can
> also
> >>>> explain in detail how to handle each exception in their JavaDocs.
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
> >>>>> Vito,
> >>>>>
> >>>>> Thanks for the updates, the KIP LGTM.
> >>>>>
> >>>>> -Bill
> >>>>>
> >>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler 
> >>>> wrote:
> >>>>>
> >>>>>> Hi Vito,
> >>>>>>
> >>>>>> Haha, your archive game is on point!
> >>>>>>
> >>>>>> What Matthias said in that email is essentially what I figured was
> the
> >>>>>> rationale. It makes sense, but the point I was making is that this
> >>>> really
> >>>>>> doesn’t seem like a good way to structure a production app. On the
> >>>> other
> >>>>>> hand, considering the exception fatal has a good chance of avoiding
> a
> >>>>>> frustrating debug session if you just forgot to call start.
> >>>>>>
> >>>>>> Nevertheless, if we omit the categorization, it’s moot.
> >>>>>>
> >>>>>> It would be easy to add a categorization layer later if we want it,
> but
> >>>>>> not very easy to change it if we get it wrong.
> >>>>>>
> >>>>>> Thanks for your consideration!
> >>>>>> -John
> >>>>>>
> >>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
> >>>>>>> Hi John,
> >>>>>>>
> >>>>>>> About `StreamsNotStartedException is strange` --
> >>>>>>> The original idea came from Matthias, two years ago. :)
> >>>>>>> You can reference here:
> >>>>>>>
> >>>>>>
> >>>>
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc5...@confluent.io%3e
> >>>>>>>
> >>>>>>> About omitting the categorization --
> >>>>>>> It looks reasonable. I'm fine with omitting the categorization but
> not
> >>>>>> very
> >>>>>>> sure it is a good choice.
> >>>>>>> Does any other folks provide opinion?
> >>>>>>>
> >>>>>>>
> >>>>>>> Hi, folks,
> >>>>&g

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Got it, thanks Matthias.

---
Vito


On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax 
wrote:

> Thanks Vito.
>
> I am also ok with either name. Just a personal slight preference, but
> not a important.
>
>
> -Matthias
>
> On 1/21/20 6:52 PM, Vito Jeng wrote:
> > Thanks Matthias.
> >
> > The KIP is about InvalidStateStoreException.
> > I pick `StateStoreNotAvailableException` because it may be more intuitive
> > than `StreamsNotRunningException`.
> >
> > No matter which one picked, it's good to me.
> >
> > ---
> > Vito
> >
> >
> > On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax 
> > wrote:
> >
> >> Thanks for updating the KIP!
> >>
> >> One last comment/question: you kept `StateStoreNotAvailableException` in
> >> favor of `StreamsNotRunningException` (to merge both as suggested).
> >>
> >> I am wondering, if it might be better to keep
> >> `StreamsNotRunningException` instead of
> >> `StateStoreNotAvailableException`, because this exception is thrown if
> >> Streams is in state PENDING_SHUTDOWN / NOT_RUNNING / ERROR ?
> >>
> >>
> >>
> >> -Matthias
> >>
> >> On 1/17/20 9:56 PM, John Roesler wrote:
> >>> Thanks, Vito. I've just cast my vote.
> >>> -John
> >>>
> >>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
> >>>> Hi, folks,
> >>>>
> >>>> Just update the KIP, please take a look.
> >>>>
> >>>> Thanks!
> >>>>
> >>>> ---
> >>>> Vito
> >>>>
> >>>>
> >>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng 
> wrote:
> >>>>
> >>>>> Thanks Bill, John and Matthias. Glad you guys joined this discussion.
> >>>>> I got a lot out of the discussion.
> >>>>>
> >>>>> I would like to update KIP-216 base on John's suggestion to remove
> the
> >>>>> category.
> >>>>>
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>>
> >>>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <
> matth...@confluent.io
> >>>
> >>>>> wrote:
> >>>>>
> >>>>>>> Nevertheless, if we omit the categorization, it’s moot.
> >>>>>>
> >>>>>> Ack.
> >>>>>>
> >>>>>> I am fine to remove the middle tier. As John pointed out, it might
> be
> >>>>>> weird to have only one concrete exception type per category. We can
> >> also
> >>>>>> explain in detail how to handle each exception in their JavaDocs.
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
> >>>>>>> Vito,
> >>>>>>>
> >>>>>>> Thanks for the updates, the KIP LGTM.
> >>>>>>>
> >>>>>>> -Bill
> >>>>>>>
> >>>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler  >
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Vito,
> >>>>>>>>
> >>>>>>>> Haha, your archive game is on point!
> >>>>>>>>
> >>>>>>>> What Matthias said in that email is essentially what I figured was
> >> the
> >>>>>>>> rationale. It makes sense, but the point I was making is that this
> >>>>>> really
> >>>>>>>> doesn’t seem like a good way to structure a production app. On the
> >>>>>> other
> >>>>>>>> hand, considering the exception fatal has a good chance of
> avoiding
> >> a
> >>>>>>>> frustrating debug session if you just forgot to call start.
> >>>>>>>>
> >>>>>>>> Nevertheless, if we omit the categorization, it’s moot.
> >>>>>>>>
> >>>>>>>> It would be easy to add a categorization layer later if we want
> it,
> >> but
> >>>>>>>> not very easy to change it if we get it wrong.
> >>>>>>>>
> >>>>>>

Re: [VOTE] KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Thanks to everyone for voting.

The KIP-216 has been accepted, with 3 binding votes from
Bill, John and Matthias and 1 non-binding votes.


---
Vito


On Thu, Jan 23, 2020 at 1:46 PM Matthias J. Sax 
wrote:

> +1 (binding)
>
> -Matthias
>
> On 1/17/20 9:35 PM, John Roesler wrote:
> > Thanks for the KIP!
> >
> > I'm +1 (binding)
> >
> > Thanks,
> > -John
> >
> > On Thu, Jan 16, 2020, at 08:46, Bill Bejeck wrote:
> >> Thanks for the KIP.
> >>
> >> +1 (binding)
> >>
> >> -Bill
> >>
> >> On Tue, Jan 14, 2020 at 9:41 AM Navinder Brar
> >>  wrote:
> >>
> >>> +1 (non-binding) With a small comment which was mentioned by Vinoth as
> >>> well. Did we fix on the flag for StreamsRebalancingException, I don't
> see
> >>> it in the KIP.
> >>> -Navinder
> >>>
> >>>
> >>> On Tuesday, 14 January, 2020, 08:00:11 pm IST, Vito Jeng <
> >>> v...@is-land.com.tw> wrote:
> >>>
> >>>  Hi, all,
> >>>
> >>> I would like to start the vote for KIP-216.
> >>>
> >>> Currently, IQ throws InvalidStateStoreException for any types of error.
> >>> With this KIP, user can distinguish different types of error.
> >>>
> >>> KIP is here:
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>>
> >>> Thanks
> >>>
> >>> ---
> >>> Vito
> >>> --
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>
>
>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi, folks,

KIP-562(KAFKA-9445) already merged three days ago.

I have updated KIP-216 to reflect the KIP-562.
The main change is to introduce a new exception
`InvalidStateStorePartitionException`, will be thrown when user requested
partition not available.

Please take a look and any feedback is welcome.
Thanks Matthias for the reminder.

---
Vito


On Thu, Jan 23, 2020 at 2:13 PM Vito Jeng  wrote:

> Got it, thanks Matthias.
>
> ---
> Vito
>
>
> On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax 
> wrote:
>
>> Thanks Vito.
>>
>> I am also ok with either name. Just a personal slight preference, but
>> not a important.
>>
>>
>> -Matthias
>>
>> On 1/21/20 6:52 PM, Vito Jeng wrote:
>> > Thanks Matthias.
>> >
>> > The KIP is about InvalidStateStoreException.
>> > I pick `StateStoreNotAvailableException` because it may be more
>> intuitive
>> > than `StreamsNotRunningException`.
>> >
>> > No matter which one picked, it's good to me.
>> >
>> > ---
>> > Vito
>> >
>> >
>> > On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax 
>> > wrote:
>> >
>> >> Thanks for updating the KIP!
>> >>
>> >> One last comment/question: you kept `StateStoreNotAvailableException`
>> in
>> >> favor of `StreamsNotRunningException` (to merge both as suggested).
>> >>
>> >> I am wondering, if it might be better to keep
>> >> `StreamsNotRunningException` instead of
>> >> `StateStoreNotAvailableException`, because this exception is thrown if
>> >> Streams is in state PENDING_SHUTDOWN / NOT_RUNNING / ERROR ?
>> >>
>> >>
>> >>
>> >> -Matthias
>> >>
>> >> On 1/17/20 9:56 PM, John Roesler wrote:
>> >>> Thanks, Vito. I've just cast my vote.
>> >>> -John
>> >>>
>> >>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
>> >>>> Hi, folks,
>> >>>>
>> >>>> Just update the KIP, please take a look.
>> >>>>
>> >>>> Thanks!
>> >>>>
>> >>>> ---
>> >>>> Vito
>> >>>>
>> >>>>
>> >>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng 
>> wrote:
>> >>>>
>> >>>>> Thanks Bill, John and Matthias. Glad you guys joined this
>> discussion.
>> >>>>> I got a lot out of the discussion.
>> >>>>>
>> >>>>> I would like to update KIP-216 base on John's suggestion to remove
>> the
>> >>>>> category.
>> >>>>>
>> >>>>>
>> >>>>> ---
>> >>>>> Vito
>> >>>>>
>> >>>>>
>> >>>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <
>> matth...@confluent.io
>> >>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>>> Nevertheless, if we omit the categorization, it’s moot.
>> >>>>>>
>> >>>>>> Ack.
>> >>>>>>
>> >>>>>> I am fine to remove the middle tier. As John pointed out, it might
>> be
>> >>>>>> weird to have only one concrete exception type per category. We can
>> >> also
>> >>>>>> explain in detail how to handle each exception in their JavaDocs.
>> >>>>>>
>> >>>>>>
>> >>>>>> -Matthias
>> >>>>>>
>> >>>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
>> >>>>>>> Vito,
>> >>>>>>>
>> >>>>>>> Thanks for the updates, the KIP LGTM.
>> >>>>>>>
>> >>>>>>> -Bill
>> >>>>>>>
>> >>>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <
>> vvcep...@apache.org>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Hi Vito,
>> >>>>>>>>
>> >>>>>>>> Haha, your archive game is on point!
>> >>>>>>>>
>> >>>>>>>> What Matthias said in that email is essentially what I figured
>> was
>> >> the
>> >>>>>>>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi, Matthias and Navinder,

I have a question about the valid partition in
StreamThreadStateStoreProvider.

In the StreamThreadStateStoreProvider#createKeyTaskId(storeName, partition):
https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L103

We pass an integer as partition and then use this partition to create
TaskId instance in the topic group while loop. How do we make sure the
partition is valid? If we pass a correct storeName and a invalid partition
into createKeyTaskId() , it still looks can be created a new TaskId and
would not throw InvalidStateStorePartitionException.

I guess this would cause a NullPointerException at line #62 because this
keyTaskId cannot found in the task list.
https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L62

Does this right? or there something wrong with me?

---
Vito


On Wed, Feb 5, 2020 at 2:53 AM Navinder Brar
 wrote:

> Thanks Vito, for incorporating this. Makes sense.
>
> -Navinder
>
>
> On Wednesday, February 5, 2020, 12:17 AM, Matthias J. Sax <
> mj...@apache.org> wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Thanks Vito!
>
> That makes sense to me.
>
>
> On 2/1/20 11:29 PM, Vito Jeng wrote:
> > Hi, folks,
> >
> > KIP-562(KAFKA-9445) already merged three days ago.
> >
> > I have updated KIP-216 to reflect the KIP-562. The main change is
> > to introduce a new exception `InvalidStateStorePartitionException`,
> > will be thrown when user requested partition not available.
> >
> > Please take a look and any feedback is welcome. Thanks Matthias for
> > the reminder.
> >
> > --- Vito
> >
> >
> > On Thu, Jan 23, 2020 at 2:13 PM Vito Jeng 
> > wrote:
> >
> >> Got it, thanks Matthias.
> >>
> >> --- Vito
> >>
> >>
> >> On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax
> >>  wrote:
> >>
> >>> Thanks Vito.
> >>>
> >>> I am also ok with either name. Just a personal slight
> >>> preference, but not a important.
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 1/21/20 6:52 PM, Vito Jeng wrote:
> >>>> Thanks Matthias.
> >>>>
> >>>> The KIP is about InvalidStateStoreException. I pick
> >>>> `StateStoreNotAvailableException` because it may be more
> >>> intuitive
> >>>> than `StreamsNotRunningException`.
> >>>>
> >>>> No matter which one picked, it's good to me.
> >>>>
> >>>> --- Vito
> >>>>
> >>>>
> >>>> On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax
> >>>>  wrote:
> >>>>
> >>>>> Thanks for updating the KIP!
> >>>>>
> >>>>> One last comment/question: you kept
> >>>>> `StateStoreNotAvailableException`
> >>> in
> >>>>> favor of `StreamsNotRunningException` (to merge both as
> >>>>> suggested).
> >>>>>
> >>>>> I am wondering, if it might be better to keep
> >>>>> `StreamsNotRunningException` instead of
> >>>>> `StateStoreNotAvailableException`, because this exception
> >>>>> is thrown if Streams is in state PENDING_SHUTDOWN /
> >>>>> NOT_RUNNING / ERROR ?
> >>>>>
> >>>>>
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 1/17/20 9:56 PM, John Roesler wrote:
> >>>>>> Thanks, Vito. I've just cast my vote. -John
> >>>>>>
> >>>>>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
> >>>>>>> Hi, folks,
> >>>>>>>
> >>>>>>> Just update the KIP, please take a look.
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>> --- Vito
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng
> >>>>>>> 
> >>> wrote:
> >>>>>>>
> >>>>>>>> Thanks Bill, John and Matthias. Glad you guys joined
> >>>>>>>> this
> >>> discussion.

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Thanks Navinder and Matthias.

---
Vito


On Sat, Feb 22, 2020 at 8:12 PM Matthias J. Sax  wrote:

> Good find Vito!
>
> What Navinder says makes sense -- as there is no RC for 2.5.0 yet, I
> took the liberty to do a HOTFIX PR so we can address the issue in 2.5.0
> already.
>
> https://github.com/apache/kafka/pull/8158
>
>
> -Matthias
>
> On 2/21/20 11:11 PM, Navinder Brar wrote:
> > Hi Vito,
> >
> > I checked the code and I think you are right. If a user provides a
> wrong partition there will be NPE at
> tasks.get(keyTaskId).getStore(storeName) as that task is not available
> at this machine.
> >
> > I think we split the line:
>
> https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L62
> into 2 parts and check tasks.get(keyTaskId) separately. If it is null,
> we can throw an InvalidPartitionException. WDYS?
> >
> > Thanks,
> > Navinder
> >
> >
> > On Saturday, 22 February, 2020, 06:22:14 am IST, Vito Jeng
>  wrote:
> >
> >  Hi, Matthias and Navinder,
> >
> > I have a question about the valid partition in
> > StreamThreadStateStoreProvider.
> >
> > In the StreamThreadStateStoreProvider#createKeyTaskId(storeName,
> partition):
> >
>
> https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L103
> >
> > We pass an integer as partition and then use this partition to create
> > TaskId instance in the topic group while loop. How do we make sure the
> > partition is valid? If we pass a correct storeName and a invalid
> partition
> > into createKeyTaskId() , it still looks can be created a new TaskId and
> > would not throw InvalidStateStorePartitionException.
> >
> > I guess this would cause a NullPointerException at line #62 because this
> > keyTaskId cannot found in the task list.
> >
>
> https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L62
> >
> > Does this right? or there something wrong with me?
> >
> > ---
> > Vito
> >
> >
> > On Wed, Feb 5, 2020 at 2:53 AM Navinder Brar
> >  wrote:
> >
> >> Thanks Vito, for incorporating this. Makes sense.
> >>
> >> -Navinder
> >>
> >>
> >> On Wednesday, February 5, 2020, 12:17 AM, Matthias J. Sax <
> >> mj...@apache.org> wrote:
> >>
> > Thanks Vito!
> >
> > That makes sense to me.
> >
> >
> > On 2/1/20 11:29 PM, Vito Jeng wrote:
> >>>> Hi, folks,
> >>>>
> >>>> KIP-562(KAFKA-9445) already merged three days ago.
> >>>>
> >>>> I have updated KIP-216 to reflect the KIP-562. The main change is
> >>>> to introduce a new exception `InvalidStateStorePartitionException`,
> >>>> will be thrown when user requested partition not available.
> >>>>
> >>>> Please take a look and any feedback is welcome. Thanks Matthias for
> >>>> the reminder.
> >>>>
> >>>> --- Vito
> >>>>
> >>>>
> >>>> On Thu, Jan 23, 2020 at 2:13 PM Vito Jeng 
> >>>> wrote:
> >>>>
> >>>>> Got it, thanks Matthias.
> >>>>>
> >>>>> --- Vito
> >>>>>
> >>>>>
> >>>>> On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax
> >>>>>  wrote:
> >>>>>
> >>>>>> Thanks Vito.
> >>>>>>
> >>>>>> I am also ok with either name. Just a personal slight
> >>>>>> preference, but not a important.
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>> On 1/21/20 6:52 PM, Vito Jeng wrote:
> >>>>>>> Thanks Matthias.
> >>>>>>>
> >>>>>>> The KIP is about InvalidStateStoreException. I pick
> >>>>>>> `StateStoreNotAvailableException` because it may be more
> >>>>>> intuitive
> >>>>>>> than `StreamsNotRunningException`.
> >>>>>>>
> >>>>>>> No matter which one picked, it's good to me.
> >&

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Matthias,

> Sorry to hear! Get well soon!

I'm back! :)

Base on many suggestion and discussion, I already rewrite the KIP.
Please take a look:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors


> I see. If this is the intention, than I would suggest to have two (or
>  maybe three) different exceptions:
>  - StreamThreadRebalancingException (retryable)
>  - StreamThreadNotRunning (not retryable -- thrown if in state
> PENDING_SHUTDOWN or DEAD
>  - maybe StreamThreadNotStarted (for state CREATED)
> The last one is tricky and could also be merged into one of the first
> two, depending if you want to argue that it's retryable or not. (Just
> food for though -- not sure what others think.)

I agree with your suggestion, really helpful.


> Overall I like the KIP but some details are still unclear. Maybe it
> might help if you open an PR in parallel?

It would be a lot of help for further discussion.
I will complete the implementation in a few days and then I will create a
PR.


---
Vito


On Mon, Jun 25, 2018 at 8:25 AM Matthias J. Sax 
wrote:

> Sorry to hear! Get well soon!
>
> It's not a big deal if the KIP stalls a little bit. Feel free to pick it
> up again when you find time.
>
> >>> Is `StreamThreadNotRunningException` really an retryable error?
> >>
> >> When KafkaStream state is REBALANCING, I think it is a retryable error.
> >>
> >> StreamThreadStateStoreProvider#stores() will throw
> >> StreamThreadNotRunningException when StreamThread state is not RUNNING.
> The
> >> user can retry until KafkaStream state is RUNNING.
>
> I see. If this is the intention, than I would suggest to have two (or
> maybe three) different exceptions:
>
>  - StreamThreadRebalancingException (retryable)
>  - StreamThreadNotRunning (not retryable -- thrown if in state
> PENDING_SHUTDOWN or DEAD
>  - maybe StreamThreadNotStarted (for state CREATED)
>
> The last one is tricky and could also be merged into one of the first
> two, depending if you want to argue that it's retryable or not. (Just
> food for though -- not sure what others think.)
>
>
>
> -Matthias
>
> On 6/22/18 8:06 AM, vito jeng wrote:
> > Matthias,
> >
> > Thank you for your assistance.
> >
> >> what is the status of this KIP?
> >
> > Unfortunately, there is no further progress.
> > About seven weeks ago, I was injured in sports. I had a broken wrist on
> > my left wrist.
> > Many jobs are affected, including this KIP and implementation.
> >
> >
> >> I just re-read it, and have a couple of follow up comments. Why do we
> >> discuss the internal exceptions you want to add? Also, do we really need
> >> them? Can't we just throw the correct exception directly instead of
> >> wrapping it later?
> >
> > I think you may be right. As I say in the previous:
> > "The original idea is that we can distinguish different state store
> > exception for different handling. But to be honest, I am not quite sure
> > this is necessary. Maybe have some change during implementation."
> >
> > During the implementation, I also feel we maybe not need wrapper it.
> > We can just throw the correctly directly.
> >
> >
> >> Is `StreamThreadNotRunningException` really an retryable error?
> >
> > When KafkaStream state is REBALANCING, I think it is a retryable error.
> >
> > StreamThreadStateStoreProvider#stores() will throw
> > StreamThreadNotRunningException when StreamThread state is not RUNNING.
> The
> > user can retry until KafkaStream state is RUNNING.
> >
> >
> >> When would we throw an `StateStoreEmptyException`? The semantics is
> > unclear to me atm.
> >
> >> When the state is RUNNING, is `StateStoreClosedException` a retryable
> > error?
> >
> > These two comments will be answered in another mail.
> >
> >
> >
> > ---
> > Vito
> >
> > On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax 
> > wrote:
> >
> >> Vito,
> >>
> >> what is the status of this KIP?
> >>
> >> I just re-read it, and have a couple of follow up comments. Why do we
> >> discuss the internal exceptions you want to add? Also, do we really need
> >> them? Can't we just throw the correct exception directly instead of
> >> wrapping it later?
> >>
> >> When would we throw an `StateStoreEmptyException`? The semantics is
> >> unclear to me atm.
> >>
> >> Is `StreamThreadNotRunningException` really an ret

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Hi John,

Thanks for reviewing the KIP.

> I didn't follow the addition of a new method to the QueryableStoreType
> interface. Can you elaborate why this is necessary to support the new
> exception types?

To support the new exception types, I would check stream state in the
following classes:
  - CompositeReadOnlyKeyValueStore class
  - CompositeReadOnlySessionStore class
  - CompositeReadOnlyWindowStore class
  - DelegatingPeekingKeyValueIterator class

It is also necessary to keep backward compatibility. So I plan passing
stream
instance to QueryableStoreType instance during KafkaStreams#store()
invoked.
It looks a most simple way, I think.

It is why I add a new method to the QueryableStoreType interface. I can
understand
that we should try to avoid adding the public api method. However, at the
moment
I have no better ideas.

Any thoughts?


> Also, looking over your KIP again, it seems valuable to introduce
> "retriable store exception" and "fatal store exception" marker interfaces
> that the various exceptions can mix in. It would be nice from a usability
> perspective to be able to just log and retry on any "retriable" exception
> and log and shutdown on any fatal exception.

I agree that this is valuable to the user.
I'll update the KIP.


Thanks


---
Vito


On Tue, Oct 9, 2018 at 2:30 AM John Roesler  wrote:

> Hi Vito,
>
> I'm glad to hear you're well again!
>
> I didn't follow the addition of a new method to the QueryableStoreType
> interface. Can you elaborate why this is necessary to support the new
> exception types?
>
> Also, looking over your KIP again, it seems valuable to introduce
> "retriable store exception" and "fatal store exception" marker interfaces
> that the various exceptions can mix in. It would be nice from a usability
> perspective to be able to just log and retry on any "retriable" exception
> and log and shutdown on any fatal exception.
>
> Thanks,
> -John
>
> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang  wrote:
>
> > Thanks for the explanation, that makes sense.
> >
> >
> > Guozhang
> >
> >
> > On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax 
> > wrote:
> >
> > > The scenario I had I mind was, that KS is started in one thread while a
> > > second thread has a reference to the object to issue queries.
> > >
> > > If a query is issue before the "main thread" started KS, and the "query
> > > thread" knows that it will eventually get started, it can retry. On the
> > > other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
> impossible
> > > to issue any query against it now or in the future and thus the error
> is
> > > not retryable.
> > >
> > >
> > > -Matthias
> > >
> > > On 6/25/18 10:15 AM, Guozhang Wang wrote:
> > > > I'm wondering if StreamThreadNotStarted could be merged into
> > > > StreamThreadNotRunning, because I think users' handling logic for the
> > > third
> > > > case would be likely the same as the second. Do you have some
> scenarios
> > > > where users may want to handle them differently?
> > > >
> > > > Guozhang
> > > >
> > > > On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
> > matth...@confluent.io>
> > > > wrote:
> > > >
> > > >> Sorry to hear! Get well soon!
> > > >>
> > > >> It's not a big deal if the KIP stalls a little bit. Feel free to
> pick
> > it
> > > >> up again when you find time.
> > > >>
> > > >>>>> Is `StreamThreadNotRunningException` really an retryable error?
> > > >>>>
> > > >>>> When KafkaStream state is REBALANCING, I think it is a retryable
> > > error.
> > > >>>>
> > > >>>> StreamThreadStateStoreProvider#stores() will throw
> > > >>>> StreamThreadNotRunningException when StreamThread state is not
> > > >> RUNNING. The
> > > >>>> user can retry until KafkaStream state is RUNNING.
> > > >>
> > > >> I see. If this is the intention, than I would suggest to have two
> (or
> > > >> maybe three) different exceptions:
> > > >>
> > > >>  - StreamThreadRebalancingException (retryable)
> > > >>  - StreamThreadNotRunning (not retryable -- thrown if in state
> > > >> PENDING_SHUTDOWN or DEAD
> > > >>  - maybe StreamThreadNotStarted (for s

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Just open a PR for further discussion:
https://github.com/apache/kafka/pull/5814

Any suggestion is welcome.
Thanks!

---
Vito


On Thu, Oct 11, 2018 at 12:14 AM vito jeng  wrote:

> Hi John,
>
> Thanks for reviewing the KIP.
>
> > I didn't follow the addition of a new method to the QueryableStoreType
> > interface. Can you elaborate why this is necessary to support the new
> > exception types?
>
> To support the new exception types, I would check stream state in the
> following classes:
>   - CompositeReadOnlyKeyValueStore class
>   - CompositeReadOnlySessionStore class
>   - CompositeReadOnlyWindowStore class
>   - DelegatingPeekingKeyValueIterator class
>
> It is also necessary to keep backward compatibility. So I plan passing
> stream
> instance to QueryableStoreType instance during KafkaStreams#store()
> invoked.
> It looks a most simple way, I think.
>
> It is why I add a new method to the QueryableStoreType interface. I can
> understand
> that we should try to avoid adding the public api method. However, at the
> moment
> I have no better ideas.
>
> Any thoughts?
>
>
> > Also, looking over your KIP again, it seems valuable to introduce
> > "retriable store exception" and "fatal store exception" marker interfaces
> > that the various exceptions can mix in. It would be nice from a usability
> > perspective to be able to just log and retry on any "retriable" exception
> > and log and shutdown on any fatal exception.
>
> I agree that this is valuable to the user.
> I'll update the KIP.
>
>
> Thanks
>
>
> ---
> Vito
>
>
> On Tue, Oct 9, 2018 at 2:30 AM John Roesler  wrote:
>
>> Hi Vito,
>>
>> I'm glad to hear you're well again!
>>
>> I didn't follow the addition of a new method to the QueryableStoreType
>> interface. Can you elaborate why this is necessary to support the new
>> exception types?
>>
>> Also, looking over your KIP again, it seems valuable to introduce
>> "retriable store exception" and "fatal store exception" marker interfaces
>> that the various exceptions can mix in. It would be nice from a usability
>> perspective to be able to just log and retry on any "retriable" exception
>> and log and shutdown on any fatal exception.
>>
>> Thanks,
>> -John
>>
>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang  wrote:
>>
>> > Thanks for the explanation, that makes sense.
>> >
>> >
>> > Guozhang
>> >
>> >
>> > On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax > >
>> > wrote:
>> >
>> > > The scenario I had I mind was, that KS is started in one thread while
>> a
>> > > second thread has a reference to the object to issue queries.
>> > >
>> > > If a query is issue before the "main thread" started KS, and the
>> "query
>> > > thread" knows that it will eventually get started, it can retry. On
>> the
>> > > other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
>> impossible
>> > > to issue any query against it now or in the future and thus the error
>> is
>> > > not retryable.
>> > >
>> > >
>> > > -Matthias
>> > >
>> > > On 6/25/18 10:15 AM, Guozhang Wang wrote:
>> > > > I'm wondering if StreamThreadNotStarted could be merged into
>> > > > StreamThreadNotRunning, because I think users' handling logic for
>> the
>> > > third
>> > > > case would be likely the same as the second. Do you have some
>> scenarios
>> > > > where users may want to handle them differently?
>> > > >
>> > > > Guozhang
>> > > >
>> > > > On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
>> > matth...@confluent.io>
>> > > > wrote:
>> > > >
>> > > >> Sorry to hear! Get well soon!
>> > > >>
>> > > >> It's not a big deal if the KIP stalls a little bit. Feel free to
>> pick
>> > it
>> > > >> up again when you find time.
>> > > >>
>> > > >>>>> Is `StreamThreadNotRunningException` really an retryable error?
>> > > >>>>
>> > > >>>> When KafkaStream state is REBALANCING, I think it is a retryable
>> > > error.
>> > > >>>>
>> > > >>>> StreamThreadStat

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi, Matthias,

Sorry for the late reply.

> I am wondering what the semantic impact/change is, if we introduce
> `RetryableStateStoreException` and `FatalStateStoreException` that both
> inherit from it. While I like the introduction of both from a high level
> point of view, I just want to make sure it's semantically sound and
> backward compatible. Atm, I think it's fine, but I want to point it out
> such that everybody can think about this, too, so we can verify that
> it's a natural evolving API change.

Thank you for pointing this out. This's really important for public API.

Just when I was replying to you, I found that KIP needs some modify.
I will fix it ASAP, and then let's continue the discussion.

---
Vito


On Wed, Nov 7, 2018 at 7:06 AM Matthias J. Sax 
wrote:

> Hey Vito,
>
> I saw that you updated your PR, but did not reply to my last comments.
> Any thoughts?
>
>
> -Matthias
>
> On 10/19/18 10:34 AM, Matthias J. Sax wrote:
> > Glad to have you back Vito :)
> >
> > Some follow up thoughts:
> >
> >  - the current `InvalidStateStoreException` is documents as being
> > sometimes retry-able. From the JavaDocs:
> >
> >> These exceptions may be transient [...] Hence, it is valid to backoff
> and retry when handling this exception.
> >
> > I am wondering what the semantic impact/change is, if we introduce
> > `RetryableStateStoreException` and `FatalStateStoreException` that both
> > inherit from it. While I like the introduction of both from a high level
> > point of view, I just want to make sure it's semantically sound and
> > backward compatible. Atm, I think it's fine, but I want to point it out
> > such that everybody can think about this, too, so we can verify that
> > it's a natural evolving API change.
> >
> >  - StateStoreClosedException:
> >
> >> will be wrapped to StateStoreMigratedException or
> StateStoreNotAvailableException later.
> >
> > Can you clarify the cases (ie, when will it be wrapped with the one or
> > the other)?
> >
> >  - StateStoreIsEmptyException:
> >
> > I don't understand the semantic of this exception. Maybe it's a naming
> > issue?
> >
> >> will be wrapped to StateStoreMigratedException or
> StateStoreNotAvailableException later.
> >
> > Also, can you clarify the cases (ie, when will it be wrapped with the
> > one or the other)?
> >
> >
> > I am also wondering, if we should introduce a fatal exception
> > `UnkownStateStoreException` to tell users that they passed in an unknown
> > store name?
> >
> >
> >
> > -Matthias
> >
> >
> >
> > On 10/17/18 8:14 PM, vito jeng wrote:
> >> Just open a PR for further discussion:
> >> https://github.com/apache/kafka/pull/5814
> >>
> >> Any suggestion is welcome.
> >> Thanks!
> >>
> >> ---
> >> Vito
> >>
> >>
> >> On Thu, Oct 11, 2018 at 12:14 AM vito jeng  wrote:
> >>
> >>> Hi John,
> >>>
> >>> Thanks for reviewing the KIP.
> >>>
> >>>> I didn't follow the addition of a new method to the QueryableStoreType
> >>>> interface. Can you elaborate why this is necessary to support the new
> >>>> exception types?
> >>>
> >>> To support the new exception types, I would check stream state in the
> >>> following classes:
> >>>   - CompositeReadOnlyKeyValueStore class
> >>>   - CompositeReadOnlySessionStore class
> >>>   - CompositeReadOnlyWindowStore class
> >>>   - DelegatingPeekingKeyValueIterator class
> >>>
> >>> It is also necessary to keep backward compatibility. So I plan passing
> >>> stream
> >>> instance to QueryableStoreType instance during KafkaStreams#store()
> >>> invoked.
> >>> It looks a most simple way, I think.
> >>>
> >>> It is why I add a new method to the QueryableStoreType interface. I can
> >>> understand
> >>> that we should try to avoid adding the public api method. However, at
> the
> >>> moment
> >>> I have no better ideas.
> >>>
> >>> Any thoughts?
> >>>
> >>>
> >>>> Also, looking over your KIP again, it seems valuable to introduce
> >>>> "retriable store exception" and "fatal store exception" marker
> interfaces
> >>>> that the various exceptions can mix in. It would be nice from 

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi, Matthias,

KIP already updated.

> - StateStoreClosedException:
>   will be wrapped to StateStoreMigratedException or
StateStoreNotAvailableException later.
> Can you clarify the cases (ie, when will it be wrapped with the one or
the other)?

For example, in the implementation(CompositeReadOnlyKeyValueStore#get), we
get all stores first, and then call ReadOnlyKeyValueStore#get to get value
in every store iteration.

When calling ReadOnlyKeyValueStore#get, the StateStoreClosedException will
be thrown if the state store is not open.
We need catch StateStoreClosedException and wrap it in different exception
type:
  * If the stream's state is CREATED, we wrap StateStoreClosedException
with StreamThreadNotStartedException. User can retry until to RUNNING.
  * If the stream's state is RUNNING / REBALANCING, the state store should
be migrated, we wrap StateStoreClosedException with
StateStoreMigratedException. User can rediscover the state store.
  * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
stream thread is not available, we wrap StateStoreClosedException with
StateStoreNotAvailableException. User cannot retry when this exception is
thrown.


> - StateStoreIsEmptyException:
>  I don't understand the semantic of this exception. Maybe it's a naming
issue?

I think yes. :)
Does `EmptyStateStoreException` is better ? (already updated in the KIP)


> - StateStoreIsEmptyException:
> will be wrapped to StateStoreMigratedException or
StateStoreNotAvailableException later.
> Also, can you clarify the cases (ie, when will it be wrapped with the one
or the other)?

For example, in the implementation (CompositeReadOnlyKeyValueStore#get), we
call StateStoreProvider#stores (WrappingStoreProvider#stores) to get all
stores. EmptyStateStoreException will be thrown when cannot find any store
and then we need catch it and wrap it in different exception type:
  * If the stream's state is CREATED, we wrap EmptyStateStoreException with
StreamThreadNotStartedException. User can retry until to RUNNING.
  * If the stream's state is RUNNING / REBALANCING, the state store should
be migrated, we wrap EmptyStateStoreException with
StateStoreMigratedException. User can rediscover the state store.
  * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
stream thread is not available, we wrap EmptyStateStoreException with
StateStoreNotAvailableException. User cannot retry when this exception is
thrown.

I hope the above reply can clarify.

The last one that was not replied was:

> I am also wondering, if we should introduce a fatal exception
> `UnkownStateStoreException` to tell users that they passed in an unknown
> store name?

Until now, unknown state store is not thinking about in the KIP.
I believe it would be very useful for users.

Looking at the related code(WrappingStoreProvider#stores),
I found that I can't distinguish between the state store was migrated or an
unknown state store.

Any thoughts?

---
Vito



On Sun, Nov 11, 2018 at 5:31 PM Vito Jeng  wrote:

> Hi, Matthias,
>
> Sorry for the late reply.
>
> > I am wondering what the semantic impact/change is, if we introduce
> > `RetryableStateStoreException` and `FatalStateStoreException` that both
> > inherit from it. While I like the introduction of both from a high level
> > point of view, I just want to make sure it's semantically sound and
> > backward compatible. Atm, I think it's fine, but I want to point it out
> > such that everybody can think about this, too, so we can verify that
> > it's a natural evolving API change.
>
> Thank you for pointing this out. This's really important for public API.
>
> Just when I was replying to you, I found that KIP needs some modify.
> I will fix it ASAP, and then let's continue the discussion.
>
> ---
> Vito
>
>
> On Wed, Nov 7, 2018 at 7:06 AM Matthias J. Sax 
> wrote:
>
>> Hey Vito,
>>
>> I saw that you updated your PR, but did not reply to my last comments.
>> Any thoughts?
>>
>>
>> -Matthias
>>
>> On 10/19/18 10:34 AM, Matthias J. Sax wrote:
>> > Glad to have you back Vito :)
>> >
>> > Some follow up thoughts:
>> >
>> >  - the current `InvalidStateStoreException` is documents as being
>> > sometimes retry-able. From the JavaDocs:
>> >
>> >> These exceptions may be transient [...] Hence, it is valid to backoff
>> and retry when handling this exception.
>> >
>> > I am wondering what the semantic impact/change is, if we introduce
>> > `RetryableStateStoreException` and `FatalStateStoreException` that both
>> > inherit from it. While I like the introduction of both from a high level
>> > point of view, I just wan

[DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Hi,

I'd like to start discuss KIP-216:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors

Please have a look.
Thanks!

---
Vito

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[vito jeng]

Thanks, Guozhang and Matthias. Your comments very useful for me.

I'll update KIP and keep going on.



---
Vito

On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax 
wrote:

> Thanks for the KIP Vito!
>
> I agree with what Guozhang said. The original idea of the Jira was, to
> give different exceptions for different "recovery" strategies to the user.
>
> For example, if a store is currently recreated, a user just need to wait
> and can query the store later. On the other hand, if a store go migrated
> to another instance, a user needs to rediscover the store instead of a
> "plain retry".
>
> Fatal errors might be a third category.
>
> Not sure if there is something else?
>
> Anyway, the KIP should contain a section that talks about this ideas and
> reasoning.
>
>
> -Matthias
>
>
> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> > Thanks for writing up the KIP.
> >
> > Vito, Matthias: one thing that I wanted to figure out first is what
> > categories of errors we want to notify the users, if we only wants to
> > distinguish fatal v.s. retriable then probably we should rename the
> > proposed StateStoreMigratedException / StateStoreClosedException classes.
> > And then from there we should list what are the possible internal
> > exceptions ever thrown in those APIs in the call trace, and which
> > exceptions should be wrapped to what others, and which ones should be
> > handled without re-throwing, and which ones should not be wrapped at all
> > but directly thrown to user's face.
> >
> > Guozhang
> >
> >
> > On Wed, Nov 1, 2017 at 11:09 PM, vito jeng  wrote:
> >
> >> Hi,
> >>
> >> I'd like to start discuss KIP-216:
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>
> >> Please have a look.
> >> Thanks!
> >>
> >> ---
> >> Vito
> >>
> >
> >
> >
>
>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

` -- why can it be wrapped as
> `StreamThreadNotStartedException` ? It seems that the later would only
> be thrown by `KafkaStreams#store()` and thus would be throw directly. A
> closed-exception should only happen after a store was successfully
> retrieved but cannot be queried any longer? Hence, converting/wrapping
> it into a `StateStoreMigratedException` make sense. I am also not sure,
> when a closed-exception would be wrapped by a
> `StateStoreNotAvailableException` (implying my understanding as describe
> above)?
>
> Same questions about `EmptyStateStoreException`.
>
> Thinking about both internal exceptions twice, I am wondering if it
> makes sense to have both internal exceptions at all? I have the
> impression that it make only sense to wrap them with a
> `StateStoreMigragedException`, but if they are wrapped into the same
> exception all the time, we can just remove both and throw
> `StateStoreMigratedException` directly?
>
>
> Last point: Why do we need to add?
>
> > QueryableStoreType#setStreams(KafkaStreams streams);
>
> John asked this question already and you replied to it. But I am not
> sure what your answer means. Can you explain it in more detail?
>
>
>
> Thanks for your patience on this KIP!
>
>
>
> -Matthias
>
>
>
>
>
>
> On 11/11/18 4:55 AM, Vito Jeng wrote:
> > Hi, Matthias,
> >
> > KIP already updated.
> >
> >> - StateStoreClosedException:
> >>   will be wrapped to StateStoreMigratedException or
> > StateStoreNotAvailableException later.
> >> Can you clarify the cases (ie, when will it be wrapped with the one or
> > the other)?
> >
> > For example, in the implementation(CompositeReadOnlyKeyValueStore#get),
> we
> > get all stores first, and then call ReadOnlyKeyValueStore#get to get
> value
> > in every store iteration.
> >
> > When calling ReadOnlyKeyValueStore#get, the StateStoreClosedException
> will
> > be thrown if the state store is not open.
> > We need catch StateStoreClosedException and wrap it in different
> exception
> > type:
> >   * If the stream's state is CREATED, we wrap StateStoreClosedException
> > with StreamThreadNotStartedException. User can retry until to RUNNING.
> >   * If the stream's state is RUNNING / REBALANCING, the state store
> should
> > be migrated, we wrap StateStoreClosedException with
> > StateStoreMigratedException. User can rediscover the state store.
> >   * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
> > stream thread is not available, we wrap StateStoreClosedException with
> > StateStoreNotAvailableException. User cannot retry when this exception is
> > thrown.
> >
> >
> >> - StateStoreIsEmptyException:
> >>  I don't understand the semantic of this exception. Maybe it's a naming
> > issue?
> >
> > I think yes. :)
> > Does `EmptyStateStoreException` is better ? (already updated in the KIP)
> >
> >
> >> - StateStoreIsEmptyException:
> >> will be wrapped to StateStoreMigratedException or
> > StateStoreNotAvailableException later.
> >> Also, can you clarify the cases (ie, when will it be wrapped with the
> one
> > or the other)?
> >
> > For example, in the implementation (CompositeReadOnlyKeyValueStore#get),
> we
> > call StateStoreProvider#stores (WrappingStoreProvider#stores) to get all
> > stores. EmptyStateStoreException will be thrown when cannot find any
> store
> > and then we need catch it and wrap it in different exception type:
> >   * If the stream's state is CREATED, we wrap EmptyStateStoreException
> with
> > StreamThreadNotStartedException. User can retry until to RUNNING.
> >   * If the stream's state is RUNNING / REBALANCING, the state store
> should
> > be migrated, we wrap EmptyStateStoreException with
> > StateStoreMigratedException. User can rediscover the state store.
> >   * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
> > stream thread is not available, we wrap EmptyStateStoreException with
> > StateStoreNotAvailableException. User cannot retry when this exception is
> > thrown.
> >
> > I hope the above reply can clarify.
> >
> > The last one that was not replied was:
> >
> >> I am also wondering, if we should introduce a fatal exception
> >> `UnkownStateStoreException` to tell users that they passed in an unknown
> >> store name?
> >
> > Until now, unknown state store is not thinking about in the KIP.
> > I believe it would be very useful for users.
> >
&g

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

the same
> exception all the time, we can just remove both and throw
> `StateStoreMigratedException` directly?
>
>
> Last point: Why do we need to add?
>
> > QueryableStoreType#setStreams(KafkaStreams streams);
>
> John asked this question already and you replied to it. But I am not
> sure what your answer means. Can you explain it in more detail?
>
>
>
> Thanks for your patience on this KIP!
>
>
>
> -Matthias
>
>
>
>
>
>
> On 11/11/18 4:55 AM, Vito Jeng wrote:
> > Hi, Matthias,
> >
> > KIP already updated.
> >
> >> - StateStoreClosedException:
> >>   will be wrapped to StateStoreMigratedException or
> > StateStoreNotAvailableException later.
> >> Can you clarify the cases (ie, when will it be wrapped with the one or
> > the other)?
> >
> > For example, in the implementation(CompositeReadOnlyKeyValueStore#get),
> we
> > get all stores first, and then call ReadOnlyKeyValueStore#get to get
> value
> > in every store iteration.
> >
> > When calling ReadOnlyKeyValueStore#get, the StateStoreClosedException
> will
> > be thrown if the state store is not open.
> > We need catch StateStoreClosedException and wrap it in different
> exception
> > type:
> >   * If the stream's state is CREATED, we wrap StateStoreClosedException
> > with StreamThreadNotStartedException. User can retry until to RUNNING.
> >   * If the stream's state is RUNNING / REBALANCING, the state store
> should
> > be migrated, we wrap StateStoreClosedException with
> > StateStoreMigratedException. User can rediscover the state store.
> >   * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
> > stream thread is not available, we wrap StateStoreClosedException with
> > StateStoreNotAvailableException. User cannot retry when this exception is
> > thrown.
> >
> >
> >> - StateStoreIsEmptyException:
> >>  I don't understand the semantic of this exception. Maybe it's a naming
> > issue?
> >
> > I think yes. :)
> > Does `EmptyStateStoreException` is better ? (already updated in the KIP)
> >
> >
> >> - StateStoreIsEmptyException:
> >> will be wrapped to StateStoreMigratedException or
> > StateStoreNotAvailableException later.
> >> Also, can you clarify the cases (ie, when will it be wrapped with the
> one
> > or the other)?
> >
> > For example, in the implementation (CompositeReadOnlyKeyValueStore#get),
> we
> > call StateStoreProvider#stores (WrappingStoreProvider#stores) to get all
> > stores. EmptyStateStoreException will be thrown when cannot find any
> store
> > and then we need catch it and wrap it in different exception type:
> >   * If the stream's state is CREATED, we wrap EmptyStateStoreException
> with
> > StreamThreadNotStartedException. User can retry until to RUNNING.
> >   * If the stream's state is RUNNING / REBALANCING, the state store
> should
> > be migrated, we wrap EmptyStateStoreException with
> > StateStoreMigratedException. User can rediscover the state store.
> >   * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
> > stream thread is not available, we wrap EmptyStateStoreException with
> > StateStoreNotAvailableException. User cannot retry when this exception is
> > thrown.
> >
> > I hope the above reply can clarify.
> >
> > The last one that was not replied was:
> >
> >> I am also wondering, if we should introduce a fatal exception
> >> `UnkownStateStoreException` to tell users that they passed in an unknown
> >> store name?
> >
> > Until now, unknown state store is not thinking about in the KIP.
> > I believe it would be very useful for users.
> >
> > Looking at the related code(WrappingStoreProvider#stores),
> > I found that I can't distinguish between the state store was migrated or
> an
> > unknown state store.
> >
> > Any thoughts?
> >
> > ---
> > Vito
> >
> >
> >
> > On Sun, Nov 11, 2018 at 5:31 PM Vito Jeng  wrote:
> >
> >> Hi, Matthias,
> >>
> >> Sorry for the late reply.
> >>
> >>> I am wondering what the semantic impact/change is, if we introduce
> >>> `RetryableStateStoreException` and `FatalStateStoreException` that both
> >>> inherit from it. While I like the introduction of both from a high
> level
> >>> point of view, I just want to make sure it's semantically sound and
> >>> backward compatible. Atm, I t

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi, Mattias,

Just completed the modification of KIP, please take a look when you are
available.

---
Vito


On Wed, Jul 3, 2019 at 9:07 PM Vito Jeng  wrote:

> Hi, Matthias,
>
> This is second part.
>
> > For the internal exceptions:
> >
> > `StateStoreClosedException` -- why can it be wrapped as
> > `StreamThreadNotStartedException` ? It seems that the later would only
> > be thrown by `KafkaStreams#store()` and thus would be throw directly.
>
> Both `StateStoreClosedException` and `EmptyStateStoreException` not can be
> wrapped as `StreamThreadNotStartedException`.
> This is a mistaken written in the previous KIP. Thank you point this.
>
> > A closed-exception should only happen after a store was successfully
> > retrieved but cannot be queried any longer? Hence, converting/wrapping
> > it into a `StateStoreMigratedException` make sense. I am also not sure,
> > when a closed-exception would be wrapped by a
> > `StateStoreNotAvailableException` (implying my understanding as describe
> > above)?
> >
> > Same questions about `EmptyStateStoreException`.
> >
> > Thinking about both internal exceptions twice, I am wondering if it
> > makes sense to have both internal exceptions at all? I have the
> > impression that it make only sense to wrap them with a
> > `StateStoreMigragedException`, but if they are wrapped into the same
> > exception all the time, we can just remove both and throw
> > `StateStoreMigratedException` directly?
>
> After deeper thinking, I think you are right. It seems we can throw
> `StateStoreMigratedException` directly.
> So that we can remove `StateStoreClosedException`,
> `EmptyStateStoreException` and `StateStoreNotAvailableException`.
> Will update the KIP.
>
> BTW, if we remove above three exceptions, the
> `StreamThreadNotRunningException` will be the only one sub class extends
> from FatalStateStoreException.
> Should we remove `StreamThreadNotRunningException` and throw
> `FatalStateStoreException` directly ?
>
> > Last point: Why do we need to add?
> > QueryableStoreType#setStreams(KafkaStreams streams);
> > John asked this question already and you replied to it. But I am not
> > sure what your answer means. Can you explain it in more detail?
>
> The main purpose is to pass the KafkaStreams reference into
> CompositeReadOnlyKeyValueStore / CompositeReadOnlySessionStore/
> CompositeReadOnlyWindowStore instance.
> We need check KafkaStreams state to warp InvalidStateStoreException in to
> other exception(e.g., StateStoreMigratedException) when the user accesses
> these read-only stores.
>
> The original thought is to add `setStreams` method in to
> QueryableStoreType. But now I think I find a better way during recent days.
> This way does not need to change any public interface. So we can skip this
> question. :)
>
>
> I will update the KIP based on our discussion.
> Thank you for help to finish the KIP!
>
> ---
> Vito
>
>
> On Thu, Jun 6, 2019 at 8:23 AM Matthias J. Sax 
> wrote:
>
>> Hi Vito,
>>
>> sorry for dropping this discussion on the floor a while back. I was just
>> re-reading the KIP and discussion thread, and I think it is shaping out
>> nicely!
>>
>> I like the overall hierarchy of the exception classes.
>>
>> Some things are still not 100% clear:
>>
>>
>> You listed all methods that may throw an `InvalidStateStoreException`
>> atm. For the new exceptions, can any exception be thrown by any method?
>> It might help to understand this relationship better.
>>
>> For example, StreamThreadNotStartedException, seems to only make sense
>> for `KafkaStreams#store()`?
>>
>>
>> For `StreamThreadNotRunningException` should we rename it to
>> `KafkaStreamsNotRunningException` ?
>>
>>
>> The description of `StreamThreadNotRunningException` and
>> `StateStoreNotAvailableException` seems to be the same? From my
>> understandng, the description makes sense for
>> `StreamThreadNotRunningException` -- for
>> `StateStoreNotAvailableException` I was expecting/inferring from the
>> name, that it would be thrown if no such store exists in the topology at
>> all (ie, user passed in a invalid/wrong store name). For this case, this
>> exception should be thrown only from `KafkaStreams#store()` ?
>>
>>
>> For the internal exceptions:
>>
>> `StateStoreClosedException` -- why can it be wrapped as
>> `StreamThreadNotStartedException` ? It seems that the later would only
>> be thrown by `KafkaStreams#store()` and thus would be throw directly. A
>> closed-exception should only ha

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Thanks, Matthias!

> About `StreamThreadNotStartedException`:

Thank you for explanation. I agree with your opinion.
`CompositeReadOnlyXxxStore#get()` would never throw
`StreamThreadNotStartedException`.

For the case that corresponding thread crashes after we handed out the
store handle. We may throw `KafkaStreamsNotRunningException` or
`StateStoreMigratedException`.
In `StreamThreadStateStoreProvider`, we would throw
`KafkaStreamsNotRunningException` when stream thread is not running(
https://shorturl.at/CDNT9) or throw `StateStoreMigratedException` when
store is closed(https://shorturl.at/hrvAN). So I think we do not need to
add a new type for this case. Does that make sense?


> About `KafkaStreamsNotRunningException` vs
`StreamThreadNotRunningException`:

I understand your point. I rename `StreamThreadNotRunningException` to
`KafkaStreamsNotRunningException`.


About check unknown state store names:
Thank you for the hint. I add a new type `UnknownStateStoreException` for
this case.


> Also, we should still have fatal exception
`StateStoreNotAvailableException`? Not sure why you remove it?

Thank you point this, already add it again.

The KIP already updated, please take a look.

---
Vito


On Tue, Jul 30, 2019 at 6:04 AM Matthias J. Sax 
wrote:

> Any update on this KIP Vito?
>
>
> On 7/11/19 4:26 PM, Matthias J. Sax wrote:
> > Thanks Vito! I think the KIP shapes out nicely!
> >
> >
> > To answer the open question you raised (I also adjust my answers based
> > on the latest KIP update)
> >
> >
> >
> > About `StreamThreadNotStartedException`: I understand what you pointed
> > out. However, I think we can consider the following: If a thread is not
> > started yet, and `KafkaStreams#store()` throw this exception, we would
> > not return a `CompositeReadOnlyXxxStore` to the user. Hence, `get()`
> > cannot be called. And if we return `CompositeReadOnlyXxxStore` the
> > thread was started and `get()` would never hit the condition to throw
> > the exception? Or do I miss something (this part of the logic is a
> > little tricky...)
> >
> > However, thinking about it, what could happen IMHO is, that the
> > corresponding thread crashes after we handed out the store handle. For
> > this case, it would make sense to throw an exception from `get()` but it
> > would be a different one IMHO. Maybe we need a new type
> > (`StreamThreadDeadException` or similar?) or we should reuse
> > `StoreMigratedException` because if a thread dies we would migrate the
> > store to another thread. (The tricky part might be, to detect this
> > condition correctly -- not 100% sure atm how we could do this.)
> >
> > What do you think about this?
> >
> >
> >
> > About `KafkaStreamsNotRunningException` vs
> > `StreamThreadNotRunningException` -- I see your point. Atm, I think we
> > don't allow querying at all if KafkaStreams is not in state RUNNING
> > (correct me if I am wrong). Hence, if there is an instance with 2
> > thread, and 1 thread is actually up and ready, but the other thread is
> > not, you cannot query anything. Only if both threads are in state
> > RUNNING we allow to query. It might be possible to change the code to
> > allow querying if a thread is ready independent from the other threads.
> > For this case, the name you suggest would make more sense. But I
> > _think_, that the current behavior is different and thus,
> > `KafkaStreamsNotRunningException` seems to reflect the current behavior
> > better? -- I also want to add that we are talking about a fatal
> > exception -- if a thread crashes, we would migrate the store to another
> > thread and it would not be fatal, but the store can be re-discovered.
> > Only if all thread would die, if would be fatal -- however, for this
> > case KafakStreams would transit to DEAD anyway.
> >
> >
> >
> >> When the user passes a store name to `KafkaStreams#store()`, does there
> >> have a way that distinguish the store name is "a wrong name" or
> "migrated"
> >> during `QueryableStoreProvider#getStore()`?
> >> From my current understanding, I cannot distinguish these two.
> >
> > This should be possible. In the private KafkaStreams constructor, we
> > have access to `InternalTopologyBuilder` that can give us all known
> > store names. Hence, we can get a set of all known store names, keep them
> > as a member variable and use in `KafkaStreams#store()` in an initial
> > check if the store name is valid or not.
> >
> >
> >
> >> Should we remove `StreamThreadNotRunningException` and throw
> >> `FatalStateStoreException` directly ?
>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

My bad. The short link `https://shorturl.at/CDNT9`
<https://shorturl.at/CDNT9> seems incorrect.

Please use the following instead: https://shorturl.at/bkKQU


---
Vito


On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng  wrote:

> Thanks, Matthias!
>
> > About `StreamThreadNotStartedException`:
>
> Thank you for explanation. I agree with your opinion.
> `CompositeReadOnlyXxxStore#get()` would never throw
> `StreamThreadNotStartedException`.
>
> For the case that corresponding thread crashes after we handed out the
> store handle. We may throw `KafkaStreamsNotRunningException` or
> `StateStoreMigratedException`.
> In `StreamThreadStateStoreProvider`, we would throw
> `KafkaStreamsNotRunningException` when stream thread is not running(
> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException` when
> store is closed(https://shorturl.at/hrvAN). So I think we do not need to
> add a new type for this case. Does that make sense?
>
>
> > About `KafkaStreamsNotRunningException` vs
> `StreamThreadNotRunningException`:
>
> I understand your point. I rename `StreamThreadNotRunningException` to
> `KafkaStreamsNotRunningException`.
>
>
> About check unknown state store names:
> Thank you for the hint. I add a new type `UnknownStateStoreException` for
> this case.
>
>
> > Also, we should still have fatal exception
> `StateStoreNotAvailableException`? Not sure why you remove it?
>
> Thank you point this, already add it again.
>
> The KIP already updated, please take a look.
>
> ---
> Vito
>