Re: [DISCUSS] KIP-318: Make Kafka Connect Source idempotent

2018-06-20 Thread Stephane Maarek
Hi Randall Thanks for your feedback 1) user can override: yes they can but they most surely won't know about it. I didn't know about this improvement until I got on twitter and exchanged with Ismael. I would say most users don't even know Kafka connect is running with acks=all. My understanding

Re: [DISCUSS] KIP-318: Make Kafka Connect Source idempotent

2018-06-20 Thread Randall Hauch
Thanks for starting this conversation, Stephane. I have a few questions. The worker already accepts nearly all producer properties already, and all `producer.*` properties override any hard-coded properties defined in `Worker.java`. So isn't it currently possible for a user to define these

[DISCUSS] KIP-318: Make Kafka Connect Source idempotent

2018-06-20 Thread Stephane Maarek
KIP link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-318%3A+Make+Kafka+Connect+Source+idempotent By looking at the code, it seems Worker.java is where the magic happens, but do we need to also operate changes to KafkaBasedLog.java (~line 241) ? Love to hear your thoughts!