Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
Hi Jason, Thanks for the comments: 1. I have also updated the KIP to indicate that the DuplicateSequenceException could be the new error code returned in the producer callback as a result of tightening up the semantics of the OutOfOrderSequenceException. 2. I think returning the message format version in the TopicMetadata of the MetadatResponse is a great idea. It is general enough, and could be put to other uses in the future. Also, it is more efficient to send the right message format to the broker to begin with, so that's another thing going for this solution. I have updated the KIP to reflect this change. Tom: I have update the KIP so that the 'safe' value of 'enable.idempotence' is converted to 'requested' everywhere. Thanks, Apurva On Tue, Sep 5, 2017 at 11:17 AM, Jason Gustafson wrote: > The proposal looks good. Two minor comments: > > 1. Can we call out how we handle the duplicate case? This is a change in > behavior since we currently raise OutOfOrderSequence in this case. > > 2. Instead of passing through `idempotenceLevel` in the ProduceRequest, I > wonder if we should have a field for the minimum required message format. > When using enable.idempotence=required, we could set the minimum required > version to v2. For enable.idempotence=requested, we could use v0. The > advantage is that we may find other uses for a more general field in the > future. Alternatively, maybe we really should be returning the message > format version of each topic in the TopicMetadata response. A nice bonus of > doing so is that it gives the producer the ability to craft the right > format version ahead of time and avoid the need for conversion on the > broker. > > Thanks, > Jason > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta wrote: > > > Hi, > > > > In the discussion of KIP-185 (enable idempotence by default), we > discovered > > some shortcomings of the existing idempotent producer implementation. > > Fixing these issues requires changes to the ProduceRequest and > > ProduceResponse protocols as well as changes to the values of the > > 'enable.idempotence' producer config. > > > > Hence, I split off those changes into another KIP so as to decouple the > two > > issues. Please have a look at my follow up KIP below: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled > > > > KIP-185 depends on KIP-192, and I hope to make progress on the latter > > independently. > > > > Thanks, > > Apurva > > >
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
Agreed, we should make it consistent across the various pages. On Tue, Sep 5, 2017 at 10:06 PM, Ted Yu wrote: > Sounds good. > > Please update the KIP and: > > https://cwiki.apache.org/confluence/display/KAFKA/ > Kafka+Exactly+Once+-+Dealing+with+older+message+formats+ > when+idempotence+is+enabled > > On Tue, Sep 5, 2017 at 2:03 PM, Ismael Juma wrote: > > > Hi Ted, > > > > The current proposal has 3 options: requested, required, off ("safe" was > in > > an earlier proposal). I think these convey the meaning more clearly IMO. > > > > Ismael > > > > On 5 Sep 2017 9:22 pm, "Ted Yu" wrote: > > > > > For enable.idempotence=safe, it seems giving user impression that > > > idempotence > > > would be safe. > > > > > > However, since it really means best effort, the 'safety' is debatable. > > > > > > Why not just call the new mode besteffort ? > > > > > > Cheers > > > > > > On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma > wrote: > > > > > > > If we add the message format version (a topic config) in the response > > of > > > > TopicMetadata, we should consider adding the max message bytes as > well. > > > > That would allow us to later improve the implementation of KIP-126 to > > > split > > > > the batch _before_ sending. > > > > > > > > Ismael > > > > > > > > On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson > > > > wrote: > > > > > > > > > The proposal looks good. Two minor comments: > > > > > > > > > > 1. Can we call out how we handle the duplicate case? This is a > change > > > in > > > > > behavior since we currently raise OutOfOrderSequence in this case. > > > > > > > > > > 2. Instead of passing through `idempotenceLevel` in the > > > ProduceRequest, I > > > > > wonder if we should have a field for the minimum required message > > > format. > > > > > When using enable.idempotence=required, we could set the minimum > > > required > > > > > version to v2. For enable.idempotence=requested, we could use v0. > The > > > > > advantage is that we may find other uses for a more general field > in > > > the > > > > > future. Alternatively, maybe we really should be returning the > > message > > > > > format version of each topic in the TopicMetadata response. A nice > > > bonus > > > > of > > > > > doing so is that it gives the producer the ability to craft the > right > > > > > format version ahead of time and avoid the need for conversion on > the > > > > > broker. > > > > > > > > > > Thanks, > > > > > Jason > > > > > > > > > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > In the discussion of KIP-185 (enable idempotence by default), we > > > > > discovered > > > > > > some shortcomings of the existing idempotent producer > > implementation. > > > > > > Fixing these issues requires changes to the ProduceRequest and > > > > > > ProduceResponse protocols as well as changes to the values of the > > > > > > 'enable.idempotence' producer config. > > > > > > > > > > > > Hence, I split off those changes into another KIP so as to > decouple > > > the > > > > > two > > > > > > issues. Please have a look at my follow up KIP below: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled > > > > > > > > > > > > KIP-185 depends on KIP-192, and I hope to make progress on the > > latter > > > > > > independently. > > > > > > > > > > > > Thanks, > > > > > > Apurva > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
Sounds good. Please update the KIP and: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Exactly+Once+-+Dealing+with+older+message+formats+when+idempotence+is+enabled On Tue, Sep 5, 2017 at 2:03 PM, Ismael Juma wrote: > Hi Ted, > > The current proposal has 3 options: requested, required, off ("safe" was in > an earlier proposal). I think these convey the meaning more clearly IMO. > > Ismael > > On 5 Sep 2017 9:22 pm, "Ted Yu" wrote: > > > For enable.idempotence=safe, it seems giving user impression that > > idempotence > > would be safe. > > > > However, since it really means best effort, the 'safety' is debatable. > > > > Why not just call the new mode besteffort ? > > > > Cheers > > > > On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma wrote: > > > > > If we add the message format version (a topic config) in the response > of > > > TopicMetadata, we should consider adding the max message bytes as well. > > > That would allow us to later improve the implementation of KIP-126 to > > split > > > the batch _before_ sending. > > > > > > Ismael > > > > > > On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson > > > wrote: > > > > > > > The proposal looks good. Two minor comments: > > > > > > > > 1. Can we call out how we handle the duplicate case? This is a change > > in > > > > behavior since we currently raise OutOfOrderSequence in this case. > > > > > > > > 2. Instead of passing through `idempotenceLevel` in the > > ProduceRequest, I > > > > wonder if we should have a field for the minimum required message > > format. > > > > When using enable.idempotence=required, we could set the minimum > > required > > > > version to v2. For enable.idempotence=requested, we could use v0. The > > > > advantage is that we may find other uses for a more general field in > > the > > > > future. Alternatively, maybe we really should be returning the > message > > > > format version of each topic in the TopicMetadata response. A nice > > bonus > > > of > > > > doing so is that it gives the producer the ability to craft the right > > > > format version ahead of time and avoid the need for conversion on the > > > > broker. > > > > > > > > Thanks, > > > > Jason > > > > > > > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > In the discussion of KIP-185 (enable idempotence by default), we > > > > discovered > > > > > some shortcomings of the existing idempotent producer > implementation. > > > > > Fixing these issues requires changes to the ProduceRequest and > > > > > ProduceResponse protocols as well as changes to the values of the > > > > > 'enable.idempotence' producer config. > > > > > > > > > > Hence, I split off those changes into another KIP so as to decouple > > the > > > > two > > > > > issues. Please have a look at my follow up KIP below: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled > > > > > > > > > > KIP-185 depends on KIP-192, and I hope to make progress on the > latter > > > > > independently. > > > > > > > > > > Thanks, > > > > > Apurva > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
Hi Ted, The current proposal has 3 options: requested, required, off ("safe" was in an earlier proposal). I think these convey the meaning more clearly IMO. Ismael On 5 Sep 2017 9:22 pm, "Ted Yu" wrote: > For enable.idempotence=safe, it seems giving user impression that > idempotence > would be safe. > > However, since it really means best effort, the 'safety' is debatable. > > Why not just call the new mode besteffort ? > > Cheers > > On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma wrote: > > > If we add the message format version (a topic config) in the response of > > TopicMetadata, we should consider adding the max message bytes as well. > > That would allow us to later improve the implementation of KIP-126 to > split > > the batch _before_ sending. > > > > Ismael > > > > On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson > > wrote: > > > > > The proposal looks good. Two minor comments: > > > > > > 1. Can we call out how we handle the duplicate case? This is a change > in > > > behavior since we currently raise OutOfOrderSequence in this case. > > > > > > 2. Instead of passing through `idempotenceLevel` in the > ProduceRequest, I > > > wonder if we should have a field for the minimum required message > format. > > > When using enable.idempotence=required, we could set the minimum > required > > > version to v2. For enable.idempotence=requested, we could use v0. The > > > advantage is that we may find other uses for a more general field in > the > > > future. Alternatively, maybe we really should be returning the message > > > format version of each topic in the TopicMetadata response. A nice > bonus > > of > > > doing so is that it gives the producer the ability to craft the right > > > format version ahead of time and avoid the need for conversion on the > > > broker. > > > > > > Thanks, > > > Jason > > > > > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta > > wrote: > > > > > > > Hi, > > > > > > > > In the discussion of KIP-185 (enable idempotence by default), we > > > discovered > > > > some shortcomings of the existing idempotent producer implementation. > > > > Fixing these issues requires changes to the ProduceRequest and > > > > ProduceResponse protocols as well as changes to the values of the > > > > 'enable.idempotence' producer config. > > > > > > > > Hence, I split off those changes into another KIP so as to decouple > the > > > two > > > > issues. Please have a look at my follow up KIP below: > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled > > > > > > > > KIP-185 depends on KIP-192, and I hope to make progress on the latter > > > > independently. > > > > > > > > Thanks, > > > > Apurva > > > > > > > > > >
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
For enable.idempotence=safe, it seems giving user impression that idempotence would be safe. However, since it really means best effort, the 'safety' is debatable. Why not just call the new mode besteffort ? Cheers On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma wrote: > If we add the message format version (a topic config) in the response of > TopicMetadata, we should consider adding the max message bytes as well. > That would allow us to later improve the implementation of KIP-126 to split > the batch _before_ sending. > > Ismael > > On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson > wrote: > > > The proposal looks good. Two minor comments: > > > > 1. Can we call out how we handle the duplicate case? This is a change in > > behavior since we currently raise OutOfOrderSequence in this case. > > > > 2. Instead of passing through `idempotenceLevel` in the ProduceRequest, I > > wonder if we should have a field for the minimum required message format. > > When using enable.idempotence=required, we could set the minimum required > > version to v2. For enable.idempotence=requested, we could use v0. The > > advantage is that we may find other uses for a more general field in the > > future. Alternatively, maybe we really should be returning the message > > format version of each topic in the TopicMetadata response. A nice bonus > of > > doing so is that it gives the producer the ability to craft the right > > format version ahead of time and avoid the need for conversion on the > > broker. > > > > Thanks, > > Jason > > > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta > wrote: > > > > > Hi, > > > > > > In the discussion of KIP-185 (enable idempotence by default), we > > discovered > > > some shortcomings of the existing idempotent producer implementation. > > > Fixing these issues requires changes to the ProduceRequest and > > > ProduceResponse protocols as well as changes to the values of the > > > 'enable.idempotence' producer config. > > > > > > Hence, I split off those changes into another KIP so as to decouple the > > two > > > issues. Please have a look at my follow up KIP below: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled > > > > > > KIP-185 depends on KIP-192, and I hope to make progress on the latter > > > independently. > > > > > > Thanks, > > > Apurva > > > > > >
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
If we add the message format version (a topic config) in the response of TopicMetadata, we should consider adding the max message bytes as well. That would allow us to later improve the implementation of KIP-126 to split the batch _before_ sending. Ismael On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson wrote: > The proposal looks good. Two minor comments: > > 1. Can we call out how we handle the duplicate case? This is a change in > behavior since we currently raise OutOfOrderSequence in this case. > > 2. Instead of passing through `idempotenceLevel` in the ProduceRequest, I > wonder if we should have a field for the minimum required message format. > When using enable.idempotence=required, we could set the minimum required > version to v2. For enable.idempotence=requested, we could use v0. The > advantage is that we may find other uses for a more general field in the > future. Alternatively, maybe we really should be returning the message > format version of each topic in the TopicMetadata response. A nice bonus of > doing so is that it gives the producer the ability to craft the right > format version ahead of time and avoid the need for conversion on the > broker. > > Thanks, > Jason > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta wrote: > > > Hi, > > > > In the discussion of KIP-185 (enable idempotence by default), we > discovered > > some shortcomings of the existing idempotent producer implementation. > > Fixing these issues requires changes to the ProduceRequest and > > ProduceResponse protocols as well as changes to the values of the > > 'enable.idempotence' producer config. > > > > Hence, I split off those changes into another KIP so as to decouple the > two > > issues. Please have a look at my follow up KIP below: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled > > > > KIP-185 depends on KIP-192, and I hope to make progress on the latter > > independently. > > > > Thanks, > > Apurva > > >
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
The proposal looks good. Two minor comments: 1. Can we call out how we handle the duplicate case? This is a change in behavior since we currently raise OutOfOrderSequence in this case. 2. Instead of passing through `idempotenceLevel` in the ProduceRequest, I wonder if we should have a field for the minimum required message format. When using enable.idempotence=required, we could set the minimum required version to v2. For enable.idempotence=requested, we could use v0. The advantage is that we may find other uses for a more general field in the future. Alternatively, maybe we really should be returning the message format version of each topic in the TopicMetadata response. A nice bonus of doing so is that it gives the producer the ability to craft the right format version ahead of time and avoid the need for conversion on the broker. Thanks, Jason On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta wrote: > Hi, > > In the discussion of KIP-185 (enable idempotence by default), we discovered > some shortcomings of the existing idempotent producer implementation. > Fixing these issues requires changes to the ProduceRequest and > ProduceResponse protocols as well as changes to the values of the > 'enable.idempotence' producer config. > > Hence, I split off those changes into another KIP so as to decouple the two > issues. Please have a look at my follow up KIP below: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled > > KIP-185 depends on KIP-192, and I hope to make progress on the latter > independently. > > Thanks, > Apurva >