Chris' point about upgrades is valid. An existing configuration will now have additional behavior. We should clearly call this out in the kip, and whenever they are prepared -- the release notes. It's a bit crummy when upgrading, but I do think it's better than introducing a new configuration in the long term.
On Mon, Nov 15, 2021 at 2:52 PM Knowles Atchison Jr <katchiso...@gmail.com> wrote: > Chris, > > Thank you for the feedback. I can certainly update the KIP to state that > once exactly one support is in place, the task would be failed even if > error.tolerance were set to all. Programmatically it would still require > PRs to be merged to build on top of. I also liked my original > implementation of the hook as it gave the connector writers the most > flexibility in handling producer errors. I changed the original > implementation as the progression/changes still supported my use case and I > thought it would move this process along faster. > > Knowles > > On Thu, Nov 11, 2021 at 3:43 PM Chris Egerton <chr...@confluent.io.invalid > > > wrote: > > > Hi Knowles, > > > > I think this looks good for the most part but I'd still like to see an > > explicit mention in the KIP (and proposed doc/Javadoc changes) that > states > > that, with exactly-once support enabled, producer exceptions that result > > from failures related to exactly-once support (including but not limited > to > > ProducerFencedExcecption instances ( > > > > > https://kafka.apache.org/30/javadoc/org/apache/kafka/common/errors/ProducerFencedException.html > > )) > > will not be skipped even with "errors.tolerance" set to "all", and will > > instead unconditionally cause the task to fail. Your proposal that > > "WorkerSourceTask could check the configuration before handing off the > > records and exception to this function" seems great as long as we update > > "handing off the records and exceptions to this function" to the > > newly-proposed behavior of "logging the exception and continuing to poll > > the task for data". > > > > I'm also a little bit wary of updating the existing "errors.tolerance" > > configuration to have new behavior that users can't opt out of without > also > > opting out of the current behavior they get with "errors.tolerance" set > to > > "all", but I think I've found a decent argument in favor of it. One > thought > > that came to mind is whether this use case was originally considered when > > KIP-298 was being discussed. However, it appears that KAFKA-8586 ( > > https://issues.apache.org/jira/browse/KAFKA-8586), the fix for which > > caused > > tasks to fail on non-retriable, asynchronous producer exceptions instead > of > > logging them and continuing, was discovered over a full year after the > > changes for KIP-298 (https://github.com/apache/kafka/pull/5065) were > > merged. I suspect that the current proposal aligns nicely with the > original > > design intent of KIP-298, and that if KAFKA-8586 were discovered before > or > > during discussion for KIP-298, non-retriable, asynchronous producer > > exceptions would have been included in its scope. With that in mind, > > although it may cause issues for some niche use cases, I think that this > is > > a valid change and would be worth the tradeoff of potentially > complicating > > life for a small number of users. I'd be interested in Arjun's thoughts > on > > this though (as he designed and implemented KIP-298), and if this > analysis > > is agreeable, we may want to document that information in the KIP as well > > to strengthen our case for not introducing a new configuration property > and > > instead making this behavior tied to the existing "errors.tolerance" > > property with no opt-out besides using a new value for that property. > > > > My last thought is that, although it may be outside the scope of this > KIP, > > I believe your original proposal of giving tasks a hook to handle > > downstream exceptions is actually quite valid. The DLQ feature for sink > > connectors is an extremely valuable one as it prevents data loss when > > "errors.tolerance" is set to "all" by allowing users to reprocess > > problematic records at a later date without stopping the flow of data in > > their connector entirely. As others have noted, it's difficult if not > > outright impossible to provide a Kafka DLQ topic for source connectors > with > > the same guarantees, and so allowing source connectors the option of > > storing problematic records back in the system that they came from seems > > like a reasonable alternative. I think we're probably past the point of > > making that happen in this KIP, but I don't believe the changes you've > > proposed make that any harder in the future than it is now (which is > > great!), and I wanted to voice my general support for a mechanism like > this > > in case you or someone following along think it'd be worth it to pursue > at > > a later date. > > > > Thanks for your KIP and thanks for your patience with the process! > > > > Cheers, > > > > Chris > > > > On Fri, Nov 5, 2021 at 8:26 AM Knowles Atchison Jr < > katchiso...@gmail.com> > > wrote: > > > > > Good morning, > > > > > > If there is no additional feedback, I am going to call a vote for this > > KIP > > > on Monday. > > > > > > Knowles > > > > > > On Tue, Nov 2, 2021 at 10:00 AM Knowles Atchison Jr < > > katchiso...@gmail.com > > > > > > > wrote: > > > > > > > Third time's the charm. > > > > > > > > I've added a getter for the RetryWithToleranceOperator to get the > > > > ToleranceType. I've updated WorkerSourceTask to check this setting to > > see > > > > if it is ToleranceType.ALL. > > > > > > > > Setting "errors.tolerance" to "all" solves both problems: > > > > > > > > 1. Use an existing configuration > > > > 2. Moves the configuration back to the connector/task level instead > of > > at > > > > the connect worker level. > > > > > > > > I've updated the KIP and PR. > > > > > > > > Additional thoughts and feedback are welcome. > > > > > > > > Knowles > > > > > > > > On Mon, Nov 1, 2021 at 2:00 AM Arjun Satish <arjun.sat...@gmail.com> > > > > wrote: > > > > > > > >> Looks really nice. Thanks for the changes. Couple of suggestions: > > > >> > > > >> 1. Can we reuse any of the existing configs, instead of introducing > a > > > new > > > >> one? I’m wondering if the error.tolerance configuration’s scope can > be > > > >> increased to include produce errors as well. That’ll help us keep > > number > > > >> of > > > >> configs in check. Effectively, if error.tolerance is set to all, > then > > > the > > > >> behavior would be like how you describe the worker would ignore > > producer > > > >> errors. > > > >> > > > >> 2. If we do choose to have a new config, could you please call out > the > > > >> possible values it can take in the kip? > > > >> > > > >> Thanks again! > > > >> > > > >> Best, > > > >> > > > >> > > > >> On Fri, Oct 29, 2021 at 9:53 AM Knowles Atchison Jr < > > > >> katchiso...@gmail.com> > > > >> wrote: > > > >> > > > >> > Arjun, > > > >> > > > > >> > Thank you for your feedback, I have updated the KIP. > > > >> > > > > >> > This solution is more elegant than my original proposal; however, > > > after > > > >> > working on the implementation, we have now pushed the > configuration > > > from > > > >> > the connector/task itself back to the connect worker. All tasks > > > running > > > >> on > > > >> > the worker would share this ignore producer exception > configuration > > > >> flag. > > > >> > This works for my use cases where I cannot envision setting this > for > > > >> only > > > >> > one type of connector we have, but this does take the choice out > of > > > the > > > >> > hands of the connector developer. I suppose that is for the best, > > in a > > > >> > vacuum only the worker should have a say in how it handles message > > > >> > production. > > > >> > > > > >> > Additional thoughts and feedback are welcome. > > > >> > > > > >> > Knowles > > > >> > > > > >> > On Thu, Oct 28, 2021 at 10:54 AM Arjun Satish < > > arjun.sat...@gmail.com > > > > > > > >> > wrote: > > > >> > > > > >> > > Yes, that makes sense. And it fits in very nicely with the > current > > > >> error > > > >> > > handling framework. > > > >> > > > > > >> > > On Thu, Oct 28, 2021 at 10:39 AM Knowles Atchison Jr < > > > >> > > katchiso...@gmail.com> > > > >> > > wrote: > > > >> > > > > > >> > > > That would work. I originally thought that it would be > confusing > > > to > > > >> > > > overload that function when a Record that wasn't actually > > written, > > > >> but > > > >> > > > looking at SourceTask more closely, in > > commitRecord(SourceRecord, > > > >> > > > RecordMetadata), the RecordMetadata is set to null in the > event > > > of a > > > >> > > > filtered transformation so the framework is already doing this > > in > > > a > > > >> > > certain > > > >> > > > regard. > > > >> > > > > > > >> > > > Knowles > > > >> > > > > > > >> > > > On Thu, Oct 28, 2021 at 10:29 AM Arjun Satish < > > > >> arjun.sat...@gmail.com> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > To ack the message back to the source system, we already > have > > a > > > >> > > > > commitRecord method. Once the bad record is handled by > > skip/dlq, > > > >> we > > > >> > > could > > > >> > > > > just call commitRecord() on it? > > > >> > > > > > > > >> > > > > On Thu, Oct 28, 2021 at 9:35 AM Knowles Atchison Jr < > > > >> > > > katchiso...@gmail.com > > > >> > > > > > > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > Hi Chris, > > > >> > > > > > > > > >> > > > > > Thank you for your reply! > > > >> > > > > > > > > >> > > > > > It is a clarity error regarding the javadoc. I am not > > > >> operationally > > > >> > > > > > familiar with all of the exceptions Kafka considers > > > >> non-retriable, > > > >> > > so I > > > >> > > > > > pulled the list from Callback.java: > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/apache/kafka/blob/1afe2a5190e9c98e38c84dc793f4303ea51bc19b/clients/src/main/java/org/apache/kafka/clients/producer/Callback.java#L35 > > > >> > > > > > to be an illustrative example of the types of exceptions > > that > > > >> would > > > >> > > > kill > > > >> > > > > > the connector outright. Any exception thrown during the > > > producer > > > >> > > write > > > >> > > > > will > > > >> > > > > > be passed to this handler. I will update the KIP/PR to be > > more > > > >> > clear > > > >> > > on > > > >> > > > > > this matter. > > > >> > > > > > > > > >> > > > > > You raise an excellent point, how should the framework > > protect > > > >> the > > > >> > > > > > connector or developer from themselves? If a connector > > enables > > > >> > > > > exactly-once > > > >> > > > > > semantics, it would make sense to me to have the task > > killed. > > > >> The > > > >> > > > > framework > > > >> > > > > > should enforce this type of misconfiguration that would > > break > > > >> the > > > >> > > > > internal > > > >> > > > > > semantics of KIP-618. WorkerSourceTask could check the > > > >> > configuration > > > >> > > > > before > > > >> > > > > > handing off the records and exception to this function, > fail > > > >> > initial > > > >> > > > > > configuration check, or something of that nature. > > > >> > > > > > > > > >> > > > > > Hi Arjun, > > > >> > > > > > > > > >> > > > > > Thank you for your response! > > > >> > > > > > > > > >> > > > > > My specific use case is our custom JMS connector. We ack > > back > > > to > > > >> > the > > > >> > > > jms > > > >> > > > > > broker once Kafka commits the record. We thread out our > JMS > > > >> > consumer > > > >> > > > such > > > >> > > > > > that I would need access to the SourceRecord to confirm we > > are > > > >> > going > > > >> > > to > > > >> > > > > > throw away the message. > > > >> > > > > > > > > >> > > > > > Skipping such records, writing some log messages, and/or > > > writing > > > >> > some > > > >> > > > > error > > > >> > > > > > context to a DLQ would cover most if not all of the use > > cases > > > I > > > >> > > > envision. > > > >> > > > > > > > > >> > > > > > "discard.message.on.producer.exception": "true" > > > >> > > > > > > > > >> > > > > > or some equivalent would get my personal use case 99% of > the > > > way > > > >> > > > there. I > > > >> > > > > > would still need some kind of callback from inside the > > > connector > > > >> > with > > > >> > > > the > > > >> > > > > > Source Record to successfully ack back to my source > system. > > > >> > > > > > > > > >> > > > > > I have updated the KIP regarding the callback being > executed > > > in > > > >> a > > > >> > > > > different > > > >> > > > > > thread than poll(). > > > >> > > > > > > > > >> > > > > > Knowles > > > >> > > > > > > > > >> > > > > > On Thu, Oct 28, 2021 at 2:02 AM Arjun Satish < > > > >> > arjun.sat...@gmail.com > > > >> > > > > > > >> > > > > > wrote: > > > >> > > > > > > > > >> > > > > > > Hi Knowles, > > > >> > > > > > > > > > >> > > > > > > Thanks for the KIP! > > > >> > > > > > > > > > >> > > > > > > Could you please call out some use-cases on what the > > source > > > >> > > > connectors > > > >> > > > > > > would do when they hit such exceptions? I'm wondering if > > we > > > >> would > > > >> > > > need > > > >> > > > > to > > > >> > > > > > > do anything other than skipping such records, writing > some > > > log > > > >> > > > > messages, > > > >> > > > > > > and/or writing some error context to a DLQ? > > > >> > > > > > > > > > >> > > > > > > One of the goals for Connect was to abstract away > > > intricacies > > > >> of > > > >> > > > Kafka > > > >> > > > > > > topics, clients etc, so that connectors could focus on > the > > > >> > external > > > >> > > > > > systems > > > >> > > > > > > themselves. Ideally, we'd want to see if we could call > out > > > the > > > >> > most > > > >> > > > > > common > > > >> > > > > > > cases and handle them in the framework itself, instead > of > > > >> > > delegating > > > >> > > > > them > > > >> > > > > > > back to the connector. This way, instead of the new API, > > > we'd > > > >> > > > probably > > > >> > > > > > > introduce some more configuration options, but they > could > > be > > > >> > > > applicable > > > >> > > > > > to > > > >> > > > > > > all the connectors that are out there. > > > >> > > > > > > > > > >> > > > > > > Also, If the above mentioned are the most common uses, > > then > > > we > > > >> > > could > > > >> > > > > > apply > > > >> > > > > > > KIP-298 (with some adjustments) to source connectors for > > > >> > > > non-retriable > > > >> > > > > > > producer errors. > > > >> > > > > > > > > > >> > > > > > > If we decide to go with the API you are referring to > > though, > > > >> > would > > > >> > > > the > > > >> > > > > > > preTransformation record suffice? SMTs can be causing > the > > > >> actual > > > >> > > > issues > > > >> > > > > > > (for example, changing the topic name) that cause these > > > >> > > non-retriable > > > >> > > > > > > exceptions. The new callback might be receiving > > insufficient > > > >> > > context > > > >> > > > to > > > >> > > > > > do > > > >> > > > > > > any corrective action. > > > >> > > > > > > > > > >> > > > > > > In the documentation for the new API, we might want to > > > specify > > > >> > that > > > >> > > > > this > > > >> > > > > > > callback will be called from a different thread than the > > > ones > > > >> > > calling > > > >> > > > > > > poll(). So any shared objects must be protected > > > appropriately. > > > >> > > > > > > > > > >> > > > > > > Cheers, > > > >> > > > > > > > > > >> > > > > > > On Wed, Oct 27, 2021 at 7:01 PM Chris Egerton > > > >> > > > > > <chr...@confluent.io.invalid > > > >> > > > > > > > > > > >> > > > > > > wrote: > > > >> > > > > > > > > > >> > > > > > > > Hi Knowles, > > > >> > > > > > > > > > > >> > > > > > > > Thanks for the KIP. I may have more to say later but > > > there's > > > >> > one > > > >> > > > > thing > > > >> > > > > > > I'd > > > >> > > > > > > > like to make sure to share now. In the Javadocs for > the > > > >> > proposed > > > >> > > > > > > > SourceTask::ignoreNonRetriableProducerException > method, > > > >> > > > > > > > the InvalidProducerEpochException exception class is > > > >> included > > > >> > as > > > >> > > an > > > >> > > > > > > example > > > >> > > > > > > > of a non-retriable exception that may cause the new > > > >> SourceTask > > > >> > > > method > > > >> > > > > > to > > > >> > > > > > > be > > > >> > > > > > > > invoked. This exception should only arise if the > source > > > >> task's > > > >> > > > > producer > > > >> > > > > > > is > > > >> > > > > > > > a transactional producer, which is currently never the > > > case > > > >> > and, > > > >> > > > once > > > >> > > > > > > > KIP-618 ( > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-618 > > > >> > > > ) > > > >> > > > > is > > > >> > > > > > > > merged, will only be the case when the task is running > > > with > > > >> > > > > > exactly-once > > > >> > > > > > > > support. I wonder if it's safe to allow connectors to > > > >> discard > > > >> > > this > > > >> > > > > > > > exception when they're running with exactly-once > > support, > > > >> or if > > > >> > > the > > > >> > > > > > task > > > >> > > > > > > > should still be unconditionally failed in that case? > > > >> > > > > > > > > > > >> > > > > > > > Cheers, > > > >> > > > > > > > > > > >> > > > > > > > Chris > > > >> > > > > > > > > > > >> > > > > > > > On Wed, Oct 27, 2021 at 5:39 PM John Roesler < > > > >> > > vvcep...@apache.org> > > > >> > > > > > > wrote: > > > >> > > > > > > > > > > >> > > > > > > > > Hi Knowles, > > > >> > > > > > > > > > > > >> > > > > > > > > Thanks for the reply! That all sounds reasonable to > > me, > > > >> and > > > >> > > > > > > > > that's a good catch regarding the SourceRecord. > > > >> > > > > > > > > > > > >> > > > > > > > > Thanks, > > > >> > > > > > > > > -John > > > >> > > > > > > > > > > > >> > > > > > > > > On Wed, 2021-10-27 at 15:32 -0400, Knowles Atchison > Jr > > > >> > > > > > > > > wrote: > > > >> > > > > > > > > > John, > > > >> > > > > > > > > > > > > >> > > > > > > > > > Thank you for the response and feedback! > > > >> > > > > > > > > > > > > >> > > > > > > > > > I originally started my first pass with the > > > >> > > > > ProducerRecord<byte[], > > > >> > > > > > > > > byte[]>. > > > >> > > > > > > > > > For our connector, we need some of the information > > out > > > >> of > > > >> > the > > > >> > > > > > > > > SourceRecord > > > >> > > > > > > > > > to ack our source system. If I had the actual > > > >> > > ProducerRecord, I > > > >> > > > > > would > > > >> > > > > > > > > have > > > >> > > > > > > > > > to convert it back before I would be able to do > > > anything > > > >> > > useful > > > >> > > > > > with > > > >> > > > > > > > it. > > > >> > > > > > > > > I > > > >> > > > > > > > > > think there is merit in providing both records as > > > >> > parameters > > > >> > > to > > > >> > > > > > this > > > >> > > > > > > > > > callback. Then connector writers can decide which > of > > > the > > > >> > > > > > > > representations > > > >> > > > > > > > > of > > > >> > > > > > > > > > the data is most useful to them. I also noticed > that > > > in > > > >> my > > > >> > > PR I > > > >> > > > > was > > > >> > > > > > > > > sending > > > >> > > > > > > > > > the SourceRecord post transformation, when we > really > > > >> should > > > >> > > be > > > >> > > > > > > sending > > > >> > > > > > > > > the > > > >> > > > > > > > > > preTransformRecord. > > > >> > > > > > > > > > > > > >> > > > > > > > > > The Streams solution to this is very interesting. > > > Given > > > >> the > > > >> > > > > nature > > > >> > > > > > > of a > > > >> > > > > > > > > > connector, to me it makes the most sense for the > api > > > >> call > > > >> > to > > > >> > > be > > > >> > > > > > part > > > >> > > > > > > of > > > >> > > > > > > > > > that task rather than an external class that is > > > >> > configurable. > > > >> > > > > This > > > >> > > > > > > > allows > > > >> > > > > > > > > > the connector to use state it may have at the time > > to > > > >> > inform > > > >> > > > > > > decisions > > > >> > > > > > > > on > > > >> > > > > > > > > > what to do with these producer exceptions. > > > >> > > > > > > > > > > > > >> > > > > > > > > > I have updated the KIP and PR. > > > >> > > > > > > > > > > > > >> > > > > > > > > > Knowles > > > >> > > > > > > > > > > > > >> > > > > > > > > > On Wed, Oct 27, 2021 at 1:03 PM John Roesler < > > > >> > > > > vvcep...@apache.org> > > > >> > > > > > > > > wrote: > > > >> > > > > > > > > > > > > >> > > > > > > > > > > Good morning, Knowles, > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > Thanks for the KIP! > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > To address your latest questions, it is fine to > > call > > > >> for > > > >> > a > > > >> > > > > > > > > > > vote if a KIP doesn't generate much discussion. > > > Either > > > >> > the > > > >> > > > > > > > > > > KIP was just not controversial enough for anyone > > to > > > >> > > comment, > > > >> > > > > > > > > > > in which case a vote is appropriate; or no one > had > > > >> time > > > >> > to > > > >> > > > > > > > > > > review it, in which case, calling for a vote > might > > > be > > > >> > more > > > >> > > > > > > > > > > provacative and elicit a response. > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > As far as pinging people directly, one idea > would > > be > > > >> to > > > >> > > look > > > >> > > > > > > > > > > at the git history (git blame/praise) for the > > files > > > >> > you're > > > >> > > > > > > > > > > changing to see which committers have recently > > been > > > >> > > > > > > > > > > involved. Those are the folks who are most > likely > > to > > > >> have > > > >> > > > > > > > > > > valuable feedback on your proposal. It might not > > be > > > >> > > > > > > > > > > appropriate to directly email them, but I have > > seen > > > >> KIP > > > >> > > > > > > > > > > discussions before that requested feedback from > > > >> people by > > > >> > > > > > > > > > > name. It's probably not best to lead with that, > > but > > > >> since > > > >> > > no > > > >> > > > > > > > > > > one has responded so far, it might not hurt. I'm > > > sure > > > >> > that > > > >> > > > > > > > > > > the reason they haven't noticed your KIP is just > > > that > > > >> > they > > > >> > > > > > > > > > > are so busy it slipped their radar. They might > > > >> actually > > > >> > > > > > > > > > > appreciate a more direct ping at this point. > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > I'm happy to review, but as a caveat, I don't > have > > > >> much > > > >> > > > > > > > > > > experience with using or maintaining Connect, so > > > >> caveat > > > >> > > > > > > > > > > emptor as far as my review goes. > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > First of all, thanks for the well written KIP. > > > Without > > > >> > much > > > >> > > > > > > > > > > context, I was able to understand the motivation > > and > > > >> > > > > > > > > > > proposal easily just by reading your document. > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > I think your proposal is a good one. It seems > like > > > it > > > >> > would > > > >> > > > > > > > > > > be pretty obvious as a user what (if anything) > to > > do > > > >> with > > > >> > > > > > > > > > > the proposed method. > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > For your reference, this proposal reminds me of > > > these > > > >> > > > > > > > > > > capabilities in Streams: > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/errors/DeserializationExceptionHandler.java > > > >> > > > > > > > > > > and > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/errors/ProductionExceptionHandler.java > > > >> > > > > > > > > > > . > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > I'm not sure if there's value in bringing your > > > >> proposed > > > >> > > > > > > > > > > interface closer to that pattern or not. Streams > > and > > > >> > > Connect > > > >> > > > > > > > > > > are quite different domains after all. At > least, I > > > >> wanted > > > >> > > > > > > > > > > you to be aware of them so you could consider > the > > > >> > > > > > > > > > > alternative API strategy they present. > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > Regardless, I do wonder if it would be helpful > to > > > also > > > >> > > > > > > > > > > include the actual ProducerRecord we tried to > > send, > > > >> since > > > >> > > > > > > > > > > there's a non-trivial transformation that takes > > > place > > > >> to > > > >> > > > > > > > > > > convert the SourceRecord into a ProducerRecord. > > I'm > > > >> not > > > >> > > sure > > > >> > > > > > > > > > > what people would do with it, exactly, but it > > might > > > be > > > >> > > > > > > > > > > helpful in deciding what to do about the > > exception, > > > or > > > >> > > maybe > > > >> > > > > > > > > > > even in understanding the exception. > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > Those are the only thoughts that come to my > mind! > > > >> Thanks > > > >> > > > > > > > > > > again, > > > >> > > > > > > > > > > -John > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > On Wed, 2021-10-27 at 09:16 -0400, Knowles > > Atchison > > > Jr > > > >> > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > Good morning, > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > Bumping this thread. Is there someone specific > > on > > > >> the > > > >> > > > Connect > > > >> > > > > > > > > framework > > > >> > > > > > > > > > > > team that I should ping? Is it appropriate to > > just > > > >> > call a > > > >> > > > > vote? > > > >> > > > > > > All > > > >> > > > > > > > > > > source > > > >> > > > > > > > > > > > connectors are dead in the water without a way > > to > > > >> > handle > > > >> > > > > > producer > > > >> > > > > > > > > write > > > >> > > > > > > > > > > > exceptions. Thank you. > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > Knowles > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > On Mon, Oct 18, 2021 at 8:33 AM Christopher > > > Shannon > > > >> < > > > >> > > > > > > > > > > > christopher.l.shan...@gmail.com> wrote: > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > I also would find this feature useful to > > handle > > > >> > errors > > > >> > > > > > better, > > > >> > > > > > > > does > > > >> > > > > > > > > > > anyone > > > >> > > > > > > > > > > > > have any comments or feedback? > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > On Mon, Oct 11, 2021 at 8:52 AM Knowles > > Atchison > > > >> Jr < > > > >> > > > > > > > > > > katchiso...@gmail.com > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Good morning, > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Bumping this for visibility. I would like > > this > > > >> to > > > >> > go > > > >> > > > into > > > >> > > > > > the > > > >> > > > > > > > > next > > > >> > > > > > > > > > > > > release. > > > >> > > > > > > > > > > > > > KIP freeze is Friday. > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Any comments and feedback are welcome. > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Knowles > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Oct 5, 2021 at 4:24 PM Knowles > > > Atchison > > > >> Jr > > > >> > < > > > >> > > > > > > > > > > > > katchiso...@gmail.com> > > > >> > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Hello all, > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > I would like to discuss the following > KIP: > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-779%3A+Allow+Source+Tasks+to+Handle+Producer+Exceptions > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > The main purpose is to allow Source > Tasks > > > the > > > >> > > ability > > > >> > > > > to > > > >> > > > > > > see > > > >> > > > > > > > > > > underlying > > > >> > > > > > > > > > > > > > > Producer Exceptions and decide what to > do > > > >> rather > > > >> > > than > > > >> > > > > > being > > > >> > > > > > > > > > > killed. In > > > >> > > > > > > > > > > > > > our > > > >> > > > > > > > > > > > > > > use cases we would want to log/write off > > > some > > > >> > > > > information > > > >> > > > > > > and > > > >> > > > > > > > > > > continue > > > >> > > > > > > > > > > > > > > processing. > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > PR is here: > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > https://github.com/apache/kafka/pull/11382 > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Any comments and feedback are welcome. > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Knowles > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
