I think the current proposal looks good to me. One minor suggestion I have is to consider keeping the same error message:
Failing batch since transaction was aborted When we were running into this issue in Streams and accidentally rethrowing the KafkaException as fatal, we ended up checking the specific error message of the KafkaException and swallowing the exception if it was equivalent to the above. Obviously this was pretty hacky (hence the motivation for this KIP) and luckily we found a way around this, but it makes me wonder if any applications out there might be doing the same. So maybe we should reuse the old error message just in case? Besides that, this KIP LGTM On Thu, Sep 3, 2020 at 5:23 AM Gokul Srinivas <apa...@nym3r0s.cc> wrote: > All, > > Gentle reminder - any comments on the line of thinking I mentioned in > the last email? I've updated the Exception to be named > "TransactionAbortedException" on the KIP confluence page. > > -Gokul > > On 01-09-2020 18:34, Gokul Srinivas wrote: > > Matthias, Sophie, Jason, > > > > Took another pass at understanding the internals and it seems to me > > like we should be extending the `ApiException` rather than the > > `RetriableException`. > > > > The check in question > > ===================== > > > > Do we abort any undrained batches that are present on this transaction > > if the transaction is in an aborting state? And, if we do, what would > > be the reason sent back to the user for aborting these batches? > > > > Logic for this > > ============== > > > > If the transaction `isAborting` and `hasAbortableError` and the > > `lastError()` is not empty -> then there has been some error which > > will cause / has caused the transaction to abort and this *is* a > > runtime exception. This same exception should be sent back to the user. > > > > If the transaction `isAborting` and `lastError()` is empty -> then for > > some unknown reason (maybe even user initiated, according to the > > tests), the transaction manager has started to abort the transaction. > > In this case, the newly proposed exception should be sent back to the > > user. > > > > Reasoning > > ========= > > > > Prima facie - I do not think this is a `RetriableException`. > > > > If the user has chosen to abort this transaction, then it would be up > > to the user to choose whether to retry the exception, in which case it > > is /*not*/ a `RetriableException`. > > > > If there is a case where the transaction manager has no error, but has > > started to abort the exception, we still do not retry the transaction, > > rather we abort any undrained batches - in which case, it is /*still > > not*/ a `RetriableException`. > > > > Does that sound right? > > > > -Gokul > > > > On 29-08-2020 01:17, Jason Gustafson wrote: > >> Hi Gokul, > >> > >> Thanks, I think it makes sense to use a separate exception type. +1 on > >> Sophie's suggestion for `TransactionAbortedException`. > >> > >> Extending from `RetriableException` seems reasonable as well. I guess > >> the > >> only question is whether it's safe to catch it as a `RetriableException` > >> and apply common retry logic. For a transactional producer, my > >> expectation > >> is that the application would abort the transaction and retry it. > >> However, > >> if the transaction is already being aborted, maybe it would be better to > >> skip the abort. It might be helpful to have an example which shows > >> how we > >> expect applications to handle this. > >> > >> Thanks, > >> Jason > >> > >> > >> > >> > >> > >> > >> On Thu, Aug 27, 2020 at 6:25 PM Sophie Blee-Goldman > >> <sop...@confluent.io> > >> wrote: > >> > >>> Hey Gokul, thanks for taking up this KIP! > >>> > >>> I agree with Matthias that directly extending KafkaException may not be > >>> ideal, > >>> and we should instead extend APIException or RetriableException. Of the > >>> two, > >>> I think APIException would be more appropriate. My understanding is > >>> that > >>> RetriableException is generally reserved for internally retriable > >>> exceptions > >>> whereas APIException is used for pseudo-fatal exceptions that > >>> require some > >>> user input as to how to proceed (eg ProducerFencedException) > >>> > >>> I also agree that the name could be a bit more concise. My personal > >>> vote > >>> would be for "TransactionAbortedException" which seems a bit more > >>> grammatically aligned with the other exceptions in Kafka. > >>> > >>> Cheers, > >>> Sophie > >>> > >>> On Thu, Aug 27, 2020 at 6:01 PM Matthias J. Sax <mj...@apache.org> > >>> wrote: > >>> > >>>> Thanks for the KIP. Looks good overall. > >>>> > >>>> However, I am wondering if the new exception should extend > >>>> `KafkaException`? It seems, extending `ApiException` or maybe even > >>>> `RetriableException` might be better? > >>>> > >>>> About the name itself. I would prefer something simpler like > >>>> `AbortedTransactionException`. > >>>> > >>>> Thoughts? > >>>> > >>>> > >>>> -Matthias > >>>> > >>>> > >>>> On 8/27/20 10:24 AM, Gokul Srinivas wrote: > >>>>> Hello all, > >>>>> > >>>>> I would like to propose the following KIP to throw a new non-fatal > >>>>> exception whilst aborting transactions with non-flushed data. This > >>>>> will > >>>>> help users distinguish non-fatal errors and potentially retry the > >>> batch. > >>>>> *Issue *- https://issues.apache.org/jira/browse/KAFKA-10186 > >>>>> <https://issues.apache.org/jira/browse/KAFKA-10186> > >>>>> > >>>>> *KIP *- > >>>>> > >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-654:+Aborted+transaction+with+non-flushed+data+should+throw+a+non-fatal+exception > >>> > >>>>> < > >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-654:+Aborted+transaction+with+non-flushed+data+should+throw+a+non-fatal+exception > >>> > >>>>> > >>>>> > >>>>> Please let me know how best we can proceed with this. > >>>>> > >>>>> -Gokul > >>>>> > >>>>> > >>>> > > >
