Try setting max.block.ms low (defaults to 6) and see if you get
exceptions thrown. Given that you're retrying three times, it means each
message could be blocking for 3 minutes if you're overloading the broker,
which is my gut feeling.
Can you please describe your cluster?
On 5 Sep. 2018
Okay so acks will be 1 then. Anything in the broker logs when this occurs?
On Wed, 5 Sep. 2018, 3:32 pm LEE, Tung-Yu, wrote:
> We didn't specify the acks in our producer.properties. I guess it will be
> the default value?
>
> Liam Clarke 於 2018年9月5日 週三 11:09 寫道：
>
> > What are your producer
We didn't specify the acks in our producer.properties. I guess it will be
the default value?
Liam Clarke 於 2018年9月5日 週三 11:09 寫道：
> What are your producer acks configured to?
>
> On Wed, 5 Sep. 2018, 2:46 pm LEE, Tung-Yu, wrote:
>
> > Actually we are not sure how often the blocking happens.
What are your producer acks configured to?
On Wed, 5 Sep. 2018, 2:46 pm LEE, Tung-Yu, wrote:
> Actually we are not sure how often the blocking happens. Whenever it
> happens, it seems never to stop until we kill the process.
>
>
>
> We start 12 processes at the same time using KafkaProducer to
Would I be able to run a Kafka streams application contains join
KStream-KStream that consumes and produces to a Kafka cluster that has a formal
process to provision topics creation? Since there are behind scene creation of
and topics need to be created for join
operation. These topics are
Actually we are not sure how often the blocking happens. Whenever it
happens, it seems never to stop until we kill the process.
We start 12 processes at the same time using KafkaProducer to send data
which is a csv containing around 80,000 records every minute. (crontab)
Like this:
*/1 * *
> the cold storage is eventually consistent with Kafka
Correct, and yes this arrangement is not 100% robust, but no system is. At
least in my experience, this DR plan has been effective at rebuilding and
fixing data stores downstream of Kafka, which I submit is a more common
issue than total loss
Hello,
We currently use Kafka 1.0.2 and find that sometimes when using
KafkaProducer.send(), it blocked and didn't throw any exception.
Some code snippets and configuration are as fellows.
Any feedback is welcomed, thank you.
Tung-Yu
# code snippets
Producer producer =
Follow-up question:
3. If this is indeed an issue, how does it happen? What makes the
broker go into that state?
On Tue, Sep 4, 2018 at 10:14 AM Kostas Christidis wrote:
>
> I'm reasonably certain I've bumped into a case where a broker in my
> cluster is serving bad metadata to clients, making
I'm reasonably certain I've bumped into a case where a broker in my
cluster is serving bad metadata to clients, making them think that a
certain partition is leaderless.
This is similar to what's described here (end of the post):
https://github.com/Shopify/sarama/issues/665#issuecomment-223128098
Thank you for your answer Ryanne. It’s always a pleasure to be presented with
such unambiguous advice. It really gives you something to work with :).
To any other readers, I am very interested in hearing of other approaches to DR
in Kafka.
Ryanne, I agree with your statement as to the
Hello,
I have made necessary changes as per the original discussion thread, and
would like to put it for votes.
Thank you very much for your suggestion and guidance so far.
Regards,
When you say "Blocked" - for how long did the blocking happened? Also, have
you considered "Flushing" for immediate availability?
On Tue, 4 Sep 2018 at 11:03, LEE, Tung-Yu wrote:
> Hello,
>
> We currently use Kafka 1.0.2 and find that sometimes when using
> KafkaProducer.send(), it blocked
Hello,
We currently use Kafka 1.0.2 and find that sometimes when using
KafkaProducer.send(), it blocked and didn't throw any exception.
Some code snippets and configuration are as fellows.
Any feedback is welcomed, thank you.
Tung-Yu
# code snippets
Producer producer =
Thank you for the reference but I don't see how it answers my question.
Basically, I am looking for a Java call that I can make from the producer to
determine the maximum message size set on the broker.
Assume a process is starting and it has the connection information for the
broker so it
15 matches
Mail list logo
