Looks like you are right, although I am not sure I understand why. :)
The outlined scenario is indeed part of a test suite, which goes through the
following stages (with some temporal separation between each),
1/ Create a new topic (with some guid in the name), with three partitions
2/ Launch the three consumers (all part of the same consumer group, which is
fixed at compile time).
3/ Fire the five messages
Adding 'props.put("auto.offset.reset", "smallest");' to my list of settings
seems to resolve the hanging issue I mentioned.
The line
>>
If this is set to largest, the consumer may lose some messages when the number
of partitions, for the topics it subscribes to, changes on the broker. To
prevent data loss during partition addition, set auto.offset.reset to smallest"
<<
seems to suggest that problems would only occur if the number of partitions
changes on the broker, but that is set to (and stays at) 3.
Whilst I'm quite interested in finding out what is going on here, I'm in any
case quite thrilled that at least it seems to work now. :) Thanks!
-J
-----Original Message-----
From: noah [mailto:[email protected]]
Sent: 23 September 2015 09:44
To: [email protected]
Subject: Re: high level consumer timeout?
Assuming this is a test case with a new topic/consumer groups for each run, do
you have auto.offset.reset=smallest? This happens to me constantly in tests
because my consumers end up missing the first message since the default is
largest (in which case auto commit is a red herring.)
On Wed, Sep 23, 2015 at 3:17 AM Joris Peeters <[email protected]>
wrote:
> I have a topic with three partitions, to which I send 5 messages (very
> rapidly after one another) and they get partitioned well ([1,2,2] as
> it happens).
> I'm launching three identical high level java customers (kafka 8.2.1),
> single-threaded, to consume those messages. Once a message is
> received, the consumers have to do a fair amount of work, which
> typically takes a few minutes. Right now I'm committing the offset
> after every message has been completed (so I have auto.commit.enable to false
> *).
>
> What I am seeing, with this setup, is that the three consumers each
> work through their first message and then just hang (in the subsequent
> "it.hasNext()"). The remaining two messages do not arrive. When, on
> the other hand, I set auto.commit.enable to true, everything goes well
> and the three consumers churn through the five messages without trouble.
>
> This makes me suspect there is some sort of heartbeat issue going on,
> that is triggered by the auto.commit? We don't want to auto-commit in
> this case, though, so does anyone know of a different solution that could be
> tried?
>
> My situation sounds somewhat similar to
> http://grokbase.com/t/kafka/users/14ck70w38q/kafka-consumer-session-ti
> meouts
> - but I am not sure if anyone got to the bottom of that.
>
> As always, any help would be much appreciated!
> -Joris.
>
>
> (*) These are the consumer properties I have set explicitly:
>
> props.put("auto.commit.interval.ms", "1000");
> props.put("auto.commit.enable", "false"); // or true.
> props.put("offsets.storage", "kafka");
>
>
>
>
> Winton Capital Management Limited ("Winton") is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> authorised and regulated by the Financial Conduct Authority in the
> United Kingdom, registered as an investment adviser with the US
> Securities and Exchange Commission, registered with the US Commodity
> Futures Trading Commission and a member of the National Futures
> Association in the United States.
>
> This communication, including any attachments, is confidential and may
> be privileged. This email is for use by the intended recipient only.
> If you receive it in error, please notify the sender and delete it.
> You should not copy or disclose all or any part of this email.
>
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering
> materials and consult their own advisers with respect to investment
> decisions and inform themselves as to applicable legal requirements,
> exchange control regulations and taxes in the countries of their
> citizenship, residence or domicile. Past performance is not indicative of
> future results.
>
> Winton takes reasonable steps to ensure the accuracy and integrity of
> its communications, including emails. However Winton accepts no
> liability for any materials transmitted. Emails are not secure and
> cannot be guaranteed to be error free.
>
Winton Capital Management Limited (“Winton”) is a limited company registered in
England and Wales with its registered offices at 16 Old Bailey, London, EC4M
7EG (Registered Company No. 3311531). Winton is authorised and regulated by the
Financial Conduct Authority in the United Kingdom, registered as an investment
adviser with the US Securities and Exchange Commission, registered with the US
Commodity Futures Trading Commission and a member of the National Futures
Association in the United States.
This communication, including any attachments, is confidential and may be
privileged. This email is for use by the intended recipient only. If you
receive it in error, please notify the sender and delete it. You should not
copy or disclose all or any part of this email.
This email does not constitute an offer or solicitation and nothing contained
in this email constitutes, and should not be construed as, investment advice.
Prospective investors should request offering materials and consult their own
advisers with respect to investment decisions and inform themselves as to
applicable legal requirements, exchange control regulations and taxes in the
countries of their citizenship, residence or domicile. Past performance is not
indicative of future results.
Winton takes reasonable steps to ensure the accuracy and integrity of its
communications, including emails. However Winton accepts no liability for any
materials transmitted. Emails are not secure and cannot be guaranteed to be
error free.
