Yes that sound likely, if you changed the number of partitions then the hashing of the key's will change destination. You need to either clear the data (ie change retention to very small and roll the logs) or recreate the topic.
/svante Den fre 17 maj 2019 kl 12:32 skrev Nitay Kufert <nita...@ironsrc.com>: > I would like to add that during this time I played around with my > kafka-streams application, including several resets of the application. > I think maybe I caused it by not deleting all the messages from the input > topic of the streams application before increasing the number of partitions > and resetting. > If this sounds like a possible explanation (It does to me) - than I guess > this whole threads is redundant hehe :) > > On Fri, May 17, 2019 at 1:01 PM Nitay Kufert <nita...@ironsrc.com> wrote: > > > Hey all, > > > > I am trying to understand a situation I came across and can't find an > > explanation... > > > > In my compacted topic I have 32 partitions, and when searching the files > I > > have found that a message is found on 2 different partitions (4 and 16). > > It seems that the "right" partition is 16 because it keeps updating the > > value, while the message on partition 4 is never updated and contains an > > out-of-date value. > > This topic is an output of my kafka-stream application. > > > > The problem I get is when I consume the whole topic, I get the "bad" > > message last and this will cause the value I am seeing to be wrong > > (out-of-date). > > I can solve it programmatically but I am not sure this situation should > > even happen (maybe I am wrong making this assumption?) > > > > Here is a snippet of the values I get when consuming and filtering for > the > > desired message (in *bold* you can see the "bad" message): > > > > CreateTime:1557847043142 unique_key_123 14351.7399916489 > > CreateTime:1557848585761 unique_key_123 14352.6399916489 > > CreateTime:1557849467135 unique_key_123 14353.0899916489 > > CreateTime:1557849565972 unique_key_123 14356.8399916489 > > CreateTime:1557850075281 unique_key_123 14357.5199916489 > > CreateTime:1557850272513 unique_key_123 14358.1999916489 > > CreateTime:1557850323990 unique_key_123 14358.6499916489 > > CreateTime:1557850919067 unique_key_123 14360.8999916489 > > CreateTime:1557851600162 unique_key_123 14361.5799916489 > > CreateTime:1557852735188 unique_key_123 14367.4299916489 > > CreateTime:1557852870033 unique_key_123 14369.6799916489 > > CreateTime:1557852913943 unique_key_123 14372.8299916489 > > CreateTime:1557853111931 unique_key_123 14375.9799916489 > > CreateTime:1557853257402 unique_key_123 14376.4299916489 > > CreateTime:1557853654326 unique_key_123 14378.6799916489 > > CreateTime:1557854126273 unique_key_123 14382.2799916489 > > CreateTime:1557854882530 unique_key_123 14387.6799916489 > > CreateTime:1557855665133 unique_key_123 14392.8599916489 > > CreateTime:1557856458505 unique_key_123 14394.6099916489 > > CreateTime:1557856814818 unique_key_123 14398.2099916489 > > CreateTime:1557857560956 unique_key_123 14400.4599916489 > > CreateTime:1557858721649 unique_key_123 14404.0599916489 > > CreateTime:1557859251577 unique_key_123 14405.8099916489 > > CreateTime:1557859736903 unique_key_123 14409.4099916489 > > *CreateTime:1556637853098 unique_key_123 9954.93* > > CreateTime:1557860847333 unique_key_123 14411.4099916489 > > CreateTime:1557861171424 unique_key_123 14413.2099916489 > > CreateTime:1557861411332 unique_key_123 14413.8899916489 > > CreateTime:1557861459234 unique_key_123 14414.5699916489 > > CreateTime:1557861550401 unique_key_123 14422.6699916489 > > CreateTime:1557861709925 unique_key_123 14425.3699916489 > > CreateTime:1557862790701 unique_key_123 14427.6199916489 > > CreateTime:1557863450236 unique_key_123 14428.0699916489 > > CreateTime:1557865167738 unique_key_123 14433.0699916489 > > CreateTime:1557865378520 unique_key_123 14435.3199916489 > > CreateTime:1557865441654 unique_key_123 14437.0699916489 > > CreateTime:1557865503081 unique_key_123 14437.5199916489 > > CreateTime:1557865907130 unique_key_123 14438.1999916489 > > CreateTime:1557867190954 unique_key_123 14440.6799916489 > > CreateTime:1557867416951 unique_key_123 14441.1299916489 > > CreateTime:1557869616387 unique_key_123 14443.6099916489 > > CreateTime:1557870569098 unique_key_123 14445.3599916489 > > CreateTime:1557871319283 unique_key_123 14450.3599916489 > > CreateTime:1557872731646 unique_key_123 14451.2599916489 > > > > > > I would really appreciate an explanation or reassurance that this is not > > expected behavior. > > > > Let me know if I can supply more information > > > > Thanks! > > -- > > > > Nitay Kufert > > Backend Developer > > [image: ironSource] <http://www.ironsrc.com> > > > > email nita...@ironsrc.com > > mobile +972-54-5480021 > > fax +972-77-5448273 > > skype nitay.kufert.ssa > > 9 Ehad Ha'am st. Tel- Aviv > > ironsrc.com <http://www.ironsrc.com> > > [image: linkedin] <https://www.linkedin.com/company/ironsource> [image: > > twitter] <https://twitter.com/ironsource> [image: facebook] > > <https://www.facebook.com/ironSource> [image: googleplus] > > <https://plus.google.com/+ironsrc> > > This email (including any attachments) is for the sole use of the > intended > > recipient and may contain confidential information which may be protected > > by legal privilege. If you are not the intended recipient, or the > employee > > or agent responsible for delivering it to the intended recipient, you are > > hereby notified that any use, dissemination, distribution or copying of > > this communication and/or its content is strictly prohibited. If you are > > not the intended recipient, please immediately notify us by reply email > or > > by telephone, delete this email and destroy any copies. Thank you. > > > > > -- > > Nitay Kufert > Backend Developer > [image: ironSource] <http://www.ironsrc.com> > > email nita...@ironsrc.com > mobile +972-54-5480021 > fax +972-77-5448273 > skype nitay.kufert.ssa > 9 Ehad Ha'am st. Tel- Aviv > ironsrc.com <http://www.ironsrc.com> > [image: linkedin] <https://www.linkedin.com/company/ironsource> [image: > twitter] <https://twitter.com/ironsource> [image: facebook] > <https://www.facebook.com/ironSource> [image: googleplus] > <https://plus.google.com/+ironsrc> > This email (including any attachments) is for the sole use of the intended > recipient and may contain confidential information which may be protected > by legal privilege. If you are not the intended recipient, or the employee > or agent responsible for delivering it to the intended recipient, you are > hereby notified that any use, dissemination, distribution or copying of > this communication and/or its content is strictly prohibited. If you are > not the intended recipient, please immediately notify us by reply email or > by telephone, delete this email and destroy any copies. Thank you. >
