Hi Randall Thanks for your feedback
1) user can override: yes they can but they most surely won't know about it. I didn't know about this improvement until I got on twitter and exchanged with Ismael. I would say most users don't even know Kafka connect is running with acks=all. My understanding behind the philosophy of Kafka connect was that users only worry about writing a connector and the framework makes the whole ETL safe. In that regards I think it's important to increase the level of safety by preventing network duplicates (I don't think anyone is against not having duplicates) and at the same time increasing performance by having more in flight requests while keeping ordering guarantees (I don't think anyone is against that either). So the behaviour changes but I don't see any drawbacks to it. 1*) I'm very much allergic to introducing more Configs, but if the community desires we can control that behaviour explicitly with a new config and default the behaviour to false. It would give the users an easier opt in and eventually we'll flip the config default to true 2) is there an easy way to clearly detect if a broker is running a specific version of the API. If so, I don't mind including an if statement for a conditional worker configuration and that would solve backwards compatibility? Thanks Stephane On Wed., 20 Jun. 2018, 10:54 pm Randall Hauch, <rha...@gmail.com> wrote: > Thanks for starting this conversation, Stephane. I have a few questions. > > The worker already accepts nearly all producer properties already, and all > `producer.*` properties override any hard-coded properties defined in > `Worker.java`. So isn't it currently possible for a user to define these > properties in their worker configuration if they want? > > Second, wouldn't this change the default behavior for existing worker > configurations that have not overridden these properties? IOW, we would > need to address the migration path to ensure backward compatibility. > > Third, the KIP mentions but does not really address the problem of running > workers against pre-1.0 Kafka clusters. That definitely is something that > happens frequently, so what is the planned approach for addressing this > compatibility concern? > > All of these factors are likely why this has not yet been addressed to > date: it's already possible for users to enable this feature, but doing it > by default has compatibility concerns. > > Thoughts? > > Best regards, > > Randall > > > On Wed, Jun 20, 2018 at 1:17 AM, Stephane Maarek < > steph...@simplemachines.com.au> wrote: > > > KIP link: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 318%3A+Make+Kafka+Connect+Source+idempotent > > > > > > By looking at the code, it seems Worker.java is where the magic happens, > > but do we need to also operate changes to KafkaBasedLog.java (~line 241) > ? > > > > Love to hear your thoughts! > > >