Re: Random continuous TimeoutException with Topic not present on one KafkaProducer out of many in multithreaded env

e operation. > > Regards, > Deepak > > From: Luke Chen > Sent: 07 June 2022 11:46 > To: Deepak Jain > Cc: users@kafka.apache.org > Subject: Re: Random continuous TimeoutException with Topic not present on > one KafkaProducer out of many in multithreaded env > >

RE: Random continuous TimeoutException with Topic not present on one KafkaProducer out of many in multithreaded env

though the topic is created and used but it still throws this exception and fails the operation. Regards, Deepak From: Luke Chen Sent: 07 June 2022 11:46 To: Deepak Jain Cc: users@kafka.apache.org Subject: Re: Random continuous TimeoutException with Topic not present on one KafkaProducer out of

Re: Random continuous TimeoutException with Topic not present on one KafkaProducer out of many in multithreaded env

Hi, > > Thanks for the quick reply. > > We are already using the config max.block.ms (alongwith with other > recommended config like request.timeout.ms and others). Although the > value we are using is very less at 250 ms but since we have 5 different > KafkaProducer running i

RE: Random continuous TimeoutException with Topic not present on one KafkaProducer out of many in multithreaded env

Hi, Thanks for the quick reply. We are already using the config max.block.ms (alongwith with other recommended config like request.timeout.ms and others). Although the value we are using is very less at 250 ms but since we have 5 different KafkaProducer running in each individual thread out

Re: Random continuous TimeoutException with Topic not present on one KafkaProducer out of many in multithreaded env

en' ; Alap Patwardhan < > a...@cumulus-systems.com>; Bhushan Patil < > bhushan.pa...@cumulus-systems.com> > Subject: Random continuous TimeoutException with Topic not present on one > KafkaProducer out of many in multithreaded env > > Hello Everyone, > > We are u

RE: Random continuous TimeoutException with Topic not present on one KafkaProducer out of many in multithreaded env

ushan Patil Subject: Random continuous TimeoutException with Topic not present on one KafkaProducer out of many in multithreaded env Hello Everyone, We are using Kafka 2.8.1 Broker/Client system in our prod env. Getting following exception randomly after 1 hour or so for one Realtime transfer fr

Random continuous TimeoutException with Topic not present on one KafkaProducer out of many in multithreaded env

) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:885) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:773) We are using multithreaded KafkaProducer with their each unique topic sending data to single broker. Here, we

Re: GCP Dataproc - getting error in importing KafkaProducer

0.2* > > *python3* > Python 3.7.3 (default, Apr 3 2021, 20:42:31) > [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux > Type "help", "copyright", "credits" or "license" for more information. > *>>> from kafka import KafkaProducer

Re: GCP Dataproc - getting error in importing KafkaProducer

-python-2.0.2 *pip list|grep kafka* *kafka-python 2.0.2* *python3* Python 3.7.3 (default, Apr 3 2021, 20:42:31) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux Type "help", "copyright", "credits" or "license" for more informati

GCP Dataproc - getting error in importing KafkaProducer

Hello All, I've a GCP Dataproc cluster, and i'm running a Spark StructuredStreaming job on this. I'm trying to use KafkaProducer to push aggregated data into a Kafka topic, however when i import KafkaProducer (from kafka import KafkaProducer), it gives error ``` Traceback (m

KafkaProducer Metadata versioning

Hi I wonder how the KafkaProducer Metadata version works. I have some KafkaProducers in Production that started failing and will not recover after a topic was recreated (deleted and created again). Those are shared producers, so I stopped publication of that topic before the procedure, but I

Re: KafkaProducer - Oversized batches

t; > > > > On Sat, Dec 7, 2019 at 7:59 AM Tomoyuki Saito > wrote:> > > > > > > > > > Hi,> > > > > >> > > > > > ## Questions> > > > > >> > > > > > 1. Any possible way to make sure to av

Re: KafkaProducer - Oversized batches

gt; > > > > On Sat, Dec 7, 2019 at 7:59 AM Tomoyuki Saito wrote:> > > > > > > > Hi,> > > > >> > > > > ## Questions> > > > >> > > > > 1. Any possible way to make sure to avoid batch split, or oversized>

Re: KafkaProducer - Oversized batches

e following> > > > PR: https://github.com/apache/kafka/pull/6469 (kafka#6469)> > > >> > > > ## Background> > > >> > > > `FlinkKafkaProducer` expects that callbacks for sent records will be> > > > executed after `KafkaProducer#flush`, whi

Re: KafkaProducer - Oversized batches

a#6469)> > >> > > ## Background> > >> > > `FlinkKafkaProducer` expects that callbacks for sent records will be> > > executed after `KafkaProducer#flush`, while executing snapshot. But,> > > it does not hold when batch split happens, as explained in PR

Re: KafkaProducer - Oversized batches

t; > `FlinkKafkaProducer` expects that callbacks for sent records will be > executed after `KafkaProducer#flush`, while executing snapshot. But, > it does not hold when batch split happens, as explained in PR > kafka#6469. We are observing IllegalStateException for this issue, > with

KafkaProducer - Oversized batches

records will be executed after `KafkaProducer#flush`, while executing snapshot. But, it does not hold when batch split happens, as explained in PR kafka#6469. We are observing IllegalStateException for this issue, with Flink deployment. https://github.com/apache/flink/blob/release-1.9.1/flink

Basic KafkaProducer failing

Hello All, I am trying to push some data into a topic using KafkaProducer API. But the initialization is giving a NullPointer Exception.Tried debugging but could not figure it out. Kafka Maven version is 2.2.0 Below is the code & Exception: > import java.util.Properties; &

How to get best concurrency performance while using KafkaProducer?

KafkaProducer javadoc <https://archive.apache.org/dist/kafka/1.1.0/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html> says that using single instance will give faster performance than using multiple instances "*The producer is thread safe and sharing a single producer inst

Re: KafkaProducer (0.9): Single producer multiple topics.

ate: Friday, August 4, 2017 at 12:45 PM To: "users@kafka.apache.org<mailto:users@kafka.apache.org>" mailto:users@kafka.apache.org>> Subject: KafkaProducer (0.9): Single producer multiple topics. Hello, We’re currently using one producer per topic. We’re considering switchin

KafkaProducer (0.9): Single producer multiple topics.

Hello, We’re currently using one producer per topic. We’re considering switching to one producer for multiple topics for obvious reasons. Now, is there a case where one topic might be slower for some reason, and not the others? And if there is such a case, how would it play out with a single pr

KafkaProducer error

Hello , I am trying to run a kafka producer through Scala client ( lntellij ID) and i have included the kafka client jar files. When i run i get error as class not found :kafkaProducer Www.github.com/axeish/kafkawatcher Thanks

KafkaProducer overriding security.protocol config value and unable to connect to Broker

Hi everyone, I am facing an issue when writing to Kafka broker. I am instantiating a KafkaProducer by passing it configuration properties, on a secured Kafka Broker with SASL_PLAINTEXT security.protocol. I can see that I am passing the right security.protocol when instantiating the Producer, but

Re: Simple KafkaProducer to handle multiple requests or not

producer. Per request comes > > different set of records. > > > > I am using only one instance of kafka producer to push the records to > kafka > > cluster. Is that good way to use kafka producer? As it is mentioned in > > documentation, the kafkaproducer can b

Re: Simple KafkaProducer to handle multiple requests or not

that has kafka producer. Per request comes > different set of records. > > I am using only one instance of kafka producer to push the records to kafka > cluster. Is that good way to use kafka producer? As it is mentioned in > documentation, the kafkaproducer can be shared across multiple t

Doubts regarding KafkaProducer implemetation

Hi, We have three brokers in a cluster with replication factor is 3. We are using Kafka-0.10.0.1. We see some failures on metadata timeout exceptions while producing. We have configured retries=3 and max in flight request=1. After comparing with the old scala Producer code found that in new Produc

Simple KafkaProducer to handle multiple requests or not

push the records to kafka cluster. Is that good way to use kafka producer? As it is mentioned in documentation, the kafkaproducer can be shared across multiple threads. Or should be there one kafka producer created to handle one request? Is there any best practice documents/guidelines to follow

Re: Retries not being run in KafkaProducer

Search the mailing list I wrote about it. I forked and applied the commit that fixed that. I'm waiting for a few months for 0.9.0.2 but still nothing- don't know what is taking so long if such an important bug is in the code. On Tue, 20 Dec 2016 at 22:55 Andrew Clarkson < andrew.clark...@rallyhealt

Retries not being run in KafkaProducer

Hi All, We're using the "org.apache.kafka.kafka-clients" library version "0.9.0.1". We're running 0.9.0.1 brokers. I've been tracking down "org.apache.kafka.common.errors.TimeoutException: Batch Expired" issues in our producer. Based on my local testing, it looks like it doesn't actually retry m

KafkaProducer

Hi, all. I use the kafka which the version is 0.8.2.1. When partiton transferring， KafkaProducer needs about 5mins to recovery, so do you know how to fix it. Thanks!!!

KafkaProducer 0.9.0.1 Client - Async `send` stops sending

I am running a map-reduce job to queue data from HDFS into Kafka. The mappers simply open a file and senda the data to Kafka using KafkaProducer 0.9.0.1 client. The issue I am debugging is that the Kafka Producer async send fails to `send` to brokers after sometime. I was able to capture the

Re: kafkaproducer send blocks until broker is available

This is a known issue and is being tracked in this JIRA https://issues.apache.org/jira/browse/KAFKA-3539 -Jaikiran On Saturday 10 September 2016 12:20 AM, Peter Sinoros Szabo wrote: Hi, I'd like to use the Java Kafka producer in a non-blocking async mode. My assuptions were that until the new

kafkaproducer send blocks until broker is available

Hi, I'd like to use the Java Kafka producer in a non-blocking async mode. My assuptions were that until the new message can fit into the producer's memory, it will queue up those messages and send out once the broker is available. I tested a simple case when I am sending messages using KafkaPro

Getting a list of consumers using KafkaConsumer/KafkaProducer

Is there a way to get a list of all consumer groups and consumer group offsets using either KafkaConsumer or KafkaProducer (or some other method) in the new Java client? Best regards, Marko www.kafkatool.com

Re: KafkaProducer block on send

t;> >>>> Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat >>>> Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor >>>> Twitter : @ppatierno >>>> Linkedin : paolopatierno >>>> Blog : DevExperience >>>> >>

Re: KafkaProducer block on send

gt; > > > > > Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat > > > Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor > > > Twitter : @ppatierno > > > Linkedin : paolopatierno > > > Blog : DevExperience > > > > > >

Re: KafkaProducer block on send

Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor > > Twitter : @ppatierno > > Linkedin : paolopatierno > > Blog : DevExperience > > > >> Subject: Re: KafkaProducer block on send > >> From: ozhurakou...@hortonworks.com > >> To: users@kafk

RE: KafkaProducer block on send

It's sad that after almost one month it's still "unassigned" :-( Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog : DevExperience > Subject: Re:

Re: KafkaProducer block on send

soft Azure Advisor > Twitter : @ppatierno > Linkedin : paolopatierno > Blog : DevExperience > >> Subject: Re: KafkaProducer block on send >> From: ozhurakou...@hortonworks.com >> To: users@kafka.apache.org >> Date: Wed, 4 May 2016 14:47:25 + >> >&g

Re: KafkaProducer block on send

Linkedin : paolopatierno Blog : DevExperience Subject: Re: KafkaProducer block on send From: ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com> To: users@kafka.apache.org<mailto:users@kafka.apache.org> Date: Mon, 11 Apr 2016 19:42:17 + Dana Thanks for the e

RE: KafkaProducer block on send

xperience > From: ppatie...@live.com > To: users@kafka.apache.org > Subject: RE: KafkaProducer block on send > Date: Wed, 4 May 2016 07:24:25 + > > Hi Oleg, > > can you share the JIRA link here because I totally agree with you. > For me the send() should be total

RE: KafkaProducer block on send

(IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog : DevExperience > Subject: Re: KafkaProducer block on send > From: ozhurakou...@hortonworks.com > To: users@kafka.apache.org > Date: Mon, 11 Ap

Re: KafkaProducer NullPointerException

. We just added our new tracing utility (uses > SLF4J, LOG4J2) which has a couple of loggers that use Kafka appenders. Now > we get a null pointer exception when we try to create KafkaProducer. If we > remove all loggers from the Log4j2.xml it successfully constructs and > proceeds as befo

Re: KafkaProducer stuck in send() call

Thank you Vinay Will give it a shot and see if it happens again. Oleg > On Apr 21, 2016, at 12:26 PM, vinay sharma wrote: > > Hi Oleg, > > There are a few configurations of kafka producer that are worth considering > in your case.:- ProducerConfig.RETRIES_CONFIG > > it's java doc says "Setti

Re: KafkaProducer stuck in send() call

Hi Oleg, There are a few configurations of kafka producer that are worth considering in your case.:- ProducerConfig.RETRIES_CONFIG it's java doc says "Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this r

KafkaProducer stuck in send() call

So we have the following scenario Kafka broker (Kerberized) goes into the state of unresponsiveness while KafkaProducer issued a send() call. Regardless of the reasons why the broker ends up in this state my question is really about KafkaProducer. It continues to retry (see stack trace below

Re: KafkaProducer Retries in .9.0.1

t; > > side errors and network errors. > > > Try changing "message.max.bytes" broker config property for simulating > > > broker side error. > > > > > > > > > > > > > > > > > > > > > On Wed, Apr 6, 20

Re: KafkaProducer Retries in .9.0.1

>> > > > On Wed, Apr 6, 2016 at 12:01 AM, Manikumar Reddy < >> > > > manikumar.re...@gmail.com> >> > > > wrote: >> > > > >> > > > > Hi, >> > > > > >> > > > > Producer message size validat

Re: KafkaProducer Retries in .9.0.1

t; > > > > > > > Hi, > > > > > > > > > > Producer message size validation checks ("buffer.memory", > > > > > "max.request.size" ) happens before > > > > > batching and sending messages. Retry mechanism is applica

Re: KafkaProducer Retries in .9.0.1

; > Hi, > > > > > > > > Producer message size validation checks ("buffer.memory", > > > > "max.request.size" ) happens before > > > > batching and sending messages. Retry mechanism is applicable for > > broker > > > > side

Re: KafkaProducer Retries in .9.0.1

.size" ) happens before > > > batching and sending messages. Retry mechanism is applicable for > broker > > > side errors and network errors. > > > Try changing "message.max.bytes" broker config property for simulating > > > broker side error. > > > > &

Re: KafkaProducer Retries in .9.0.1

> broker side error. > > > > > > > > > > > > > > On Wed, Apr 6, 2016 at 9:53 AM, christopher palm > wrote: > > > > > Hi All, > > > > > > I am working with the KafkaProducer using the properties below, > > > so

KafkaProducer NullPointerException

Hi, We have a Kafka producer app that participates in the larger system. It worked fine sending messages. We just added our new tracing utility (uses SLF4J, LOG4J2) which has a couple of loggers that use Kafka appenders. Now we get a null pointer exception when we try to create KafkaProducer

Re: KafkaProducer 0.9.0.1 continually sends metadata requests

iginal Message- > From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of Ismael > Juma > Sent: 12 April 2016 16:24 > To: users@kafka.apache.org > Subject: Re: KafkaProducer 0.9.0.1 continually sends metadata requests > > Hi Jonathan, > > That makes sense,

RE: KafkaProducer 0.9.0.1 continually sends metadata requests

users@kafka.apache.org Subject: Re: KafkaProducer 0.9.0.1 continually sends metadata requests Hi Jonathan, That makes sense, thanks. Phil, do you know if your metadata requests are succeeding? Retrying every retry.backoff.ms is expected if they fail, but not if they succeed. In terms of posting a comment to

Re: KafkaProducer 0.9.0.1 continually sends metadata requests

; > > > phil.luckhu...@encycle.com> wrote: > > > > > >> Thanks Jonathan, I didn't spot that JIRA. > > >> > > >> Phil > > >> > > >> -Original Message- > > >> From: Jonathan Bond [mailto:jb...@netflix.com.INVALID] >

RE: KafkaProducer 0.9.0.1 continually sends metadata requests

Ismael, I don't see how I can add any information to KAFKA-3358 as I haven't got an account? Phil -Original Message- From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of Ismael Juma Sent: 12 April 2016 14:59 To: users@kafka.apache.org Subject: Re: KafkaProduc

Re: KafkaProducer 0.9.0.1 continually sends metadata requests

:17 PM, Phil Luckhurst < > > phil.luckhu...@encycle.com> wrote: > > > >> Thanks Jonathan, I didn't spot that JIRA. > >> > >> Phil > >> > >> -Original Message- > >> From: Jonathan Bond [mailto:jb...@netflix.com.INVALID

Re: KafkaProducer 0.9.0.1 continually sends metadata requests

il.luckhu...@encycle.com> wrote: > >> Thanks Jonathan, I didn't spot that JIRA. >> >> Phil >> >> -Original Message- >> From: Jonathan Bond [mailto:jb...@netflix.com.INVALID] >> Sent: 12 April 2016 14:08 >> To: users@kafka.apache.org &

Re: KafkaProducer 0.9.0.1 continually sends metadata requests

...@netflix.com.INVALID] > Sent: 12 April 2016 14:08 > To: users@kafka.apache.org > Subject: Re: KafkaProducer 0.9.0.1 continually sends metadata requests > > Phil, > In our case this bug placed significant load on our brokers. We raised a > bug https://issues.apache.org/j

RE: KafkaProducer 0.9.0.1 continually sends metadata requests

Thanks Jonathan, I didn't spot that JIRA. Phil -Original Message- From: Jonathan Bond [mailto:jb...@netflix.com.INVALID] Sent: 12 April 2016 14:08 To: users@kafka.apache.org Subject: Re: KafkaProducer 0.9.0.1 continually sends metadata requests Phil, In our case this bug p

Re: KafkaProducer 0.9.0.1 continually sends metadata requests

= > [Partition(topic = phil-pa-1-device-update, partition = 0, leader = 0, > replicas = [0,], isr = [0,]]) > > These metadata requests continue to be sent every 100ms (retry.backoff.ms) > until we stop the process. > > This only seems to happen if the Kafka

KafkaProducer 0.9.0.1 continually sends metadata requests

ition(topic = phil-pa-1-device-update, partition = 0, leader = 0, replicas = [0,], isr = [0,]]) These metadata requests continue to be sent every 100ms (retry.backoff.ms) until we stop the process. This only seems to happen if the KafkaProducer instance is created but not used to publish a me

Re: KafkaProducer block on send

re Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog : DevExperience Subject: KafkaProducer block on send From: ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com> To: users@kafka.apache.org<mailto:users@kafka.apache.org> Date: Thu, 7 Apr 2016 13:04:49 + I know it’s been

RE: KafkaProducer block on send

on Windows Embedded & IoTMicrosoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog : DevExperience > Subject: KafkaProducer block on send > From: ozhurakou...@hortonworks.com > To: users@kafka.apache.org > Date: Thu, 7 Apr 2016 13:04:49 + > > I know it’s been dis

RE: KafkaProducer block on send

din : paolopatierno Blog : DevExperience > Subject: KafkaProducer block on send > From: ozhurakou...@hortonworks.com > To: users@kafka.apache.org > Date: Thu, 7 Apr 2016 13:04:49 + > > I know it’s been discussed before, but that conversation never really > concluded with any r

KafkaProducer block on send

KafkaProducer: @Override public Future send(ProducerRecord record, Callback callback) { try { // first make sure the metadata for the topic is available long waitedOnMetadataMs = waitOnMetadata(record.topic(), this.maxBlockTimeMs); . . . } By definition the method that

Re: KafkaProducer Retries in .9.0.1

;max.request.size" ) happens before > > batching and sending messages. Retry mechanism is applicable for broker > > side errors and network errors. > > Try changing "message.max.bytes" broker config property for simulating > > broker side error. > > > &g

Re: KafkaProducer Retries in .9.0.1

> > > > > On Wed, Apr 6, 2016 at 9:53 AM, christopher palm wrote: > > > Hi All, > > > > I am working with the KafkaProducer using the properties below, > > so that the producer keeps trying to send upon failure on Kafka .9.0.1. > > I am forcing a

Re: KafkaProducer Retries in .9.0.1

r simulating broker side error. On Wed, Apr 6, 2016 at 9:53 AM, christopher palm wrote: > Hi All, > > I am working with the KafkaProducer using the properties below, > so that the producer keeps trying to send upon failure on Kafka .9.0.1. > I am forcing a failure by setting

KafkaProducer Retries in .9.0.1

Hi All, I am working with the KafkaProducer using the properties below, so that the producer keeps trying to send upon failure on Kafka .9.0.1. I am forcing a failure by setting my buffersize smaller than my payload,which causes the expected exception below. I don't see the producer ret

Re: KafkaProducer "send" blocks on first attempt with Kafka server offline

urrent design. > > > > In any event, there is a configuration that you can tweak to set the max > > time the producer will spend blocking in send(): max.block.ms > > > > -Dana > > > > > >> On Tue, Mar 29, 2016 at 7:26 PM, Steven Wu > wrote: >

Re: KafkaProducer "send" blocks on first attempt with Kafka server offline

ng a Future should never block. I have brought >> this up when 0.8.2 was first released for new Java producer. >> >> As Oleg said, KafkaProducer can also block if metadata is not fetched. This >> is probably more often than offline broker, because metadata is loaded &

Re: KafkaProducer "send" blocks on first attempt with Kafka server offline

uld never block. I have brought > this up when 0.8.2 was first released for new Java producer. > > As Oleg said, KafkaProducer can also block if metadata is not fetched. This > is probably more often than offline broker, because metadata is loaded > lazily when there is a first send att

Re: KafkaProducer "send" blocks on first attempt with Kafka server offline

I also agree that returning a Future should never block. I have brought this up when 0.8.2 was first released for new Java producer. As Oleg said, KafkaProducer can also block if metadata is not fetched. This is probably more often than offline broker, because metadata is loaded lazily when

Re: KafkaProducer "send" blocks on first attempt with Kafka server offline

I agree and considering that send(..) method returns Future one would argue it must never block, otherwise what’s the point of returning Future if you remove user’s ability to control how long are they willing to wait and what to do when certain types of exception arise. Nevertheless it does and

KafkaProducer "send" blocks on first attempt with Kafka server offline

Hello, as documentation says, the KafkaProducer.send() method is asynchronous and it just returns immediately.I found out that it's not so true when the Kafka server it's trying to connect isn't online.Of course it happens only on the first send() method invocation. It means that if the Kafka se

Re: Flush Messages in KafkaProducer Buffer

ot; parameter). > > > > On Wed, Nov 25, 2015 at 3:58 AM, Muqtafi Akhmad > > wrote: > > > > > Hello guys, > > > > > > I am using KafkaProducer > > (org.apache.kafka.clients.producer.KafkaProducer) > > > to send messages to Kaf

Re: Flush Messages in KafkaProducer Buffer

: > > > Hello guys, > > > > I am using KafkaProducer > (org.apache.kafka.clients.producer.KafkaProducer) > > to send messages to Kafka broker. I set BUFFER_MEMORY_CONFIG to some MBs. > > If there is a case where we need to shutdown kafka producer when there > ar

Re: Flush Messages in KafkaProducer Buffer

In 0.9.0, close() has a timeout parameter that allows specifying how long to wait for the in-flight messages to complete (definition of complete depends on value of "acks" parameter). On Wed, Nov 25, 2015 at 3:58 AM, Muqtafi Akhmad wrote: > Hello guys, > > I am

Flush Messages in KafkaProducer Buffer

Hello guys, I am using KafkaProducer (org.apache.kafka.clients.producer.KafkaProducer) to send messages to Kafka broker. I set BUFFER_MEMORY_CONFIG to some MBs. If there is a case where we need to shutdown kafka producer when there are messages in producer's buffer, is there any way to

Re: new KafkaProducer hangs when no write permissions to topic

hanks, Jason On Mon, Nov 9, 2015 at 8:31 AM, Zhuo Liu wrote: > Hello everyone, > > > I am using 0.8.2 KafkaProducer in storm to publish to a topic. > > I create with no write acls, > > then use the new producer to try to publish and it will hang. > > > I guess w

new KafkaProducer hangs when no write permissions to topic

Hello everyone, I am using 0.8.2 KafkaProducer in storm to publish to a topic. I create with no write acls, then use the new producer to try to publish and it will hang. I guess we would expect some exception instead. Best Regards, Zhuo Liu Yahoo Inc. Ph.D., CSSE of Auburn University http

Re: KafkaProducer recovery/restart if broker dies

in tomcat writes to kafka > >* 3 node kafka cluster > >* kafka 0.8.2 > >* new producer > > > >The producer config: > > > >acks=1 > >compression.type=snappy > >retries=0 > >batch_size=32768 > >buffer.memory=67108864 > >linger.ms=1500 > >metadata.fetch.timeout.ms=5000 > >timeout.ms= 1500 > >retry.backoff.ms=1 > >reconnect.backoff.ms=1 > > > >I can poll our Zookeeper and check if all brokers are alive, but I think > >KafkaProducer checks it already. > > > >Alexey > >

Re: KafkaProducer recovery/restart if broker dies

t; > >acks=1 > >compression.type=snappy > >retries=0 > >batch_size=32768 > >buffer.memory=67108864 > >linger.ms=1500 > >metadata.fetch.timeout.ms=5000 > >timeout.ms= 1500 > >retry.backoff.ms=1 > >reconnect.backoff.ms=1 > > > >I can poll our Zookeeper and check if all brokers are alive, but I think > >KafkaProducer checks it already. > > > >Alexey > >

Re: KafkaProducer recovery/restart if broker dies

> >acks=1 >compression.type=snappy >retries=0 >batch_size=32768 >buffer.memory=67108864 >linger.ms=1500 >metadata.fetch.timeout.ms=5000 >timeout.ms= 1500 >retry.backoff.ms=1 >reconnect.backoff.ms=1 > >I can poll our Zookeeper and check if all brokers are alive, but I think >KafkaProducer checks it already. > >Alexey

KafkaProducer recovery/restart if broker dies

retry.backoff.ms=1 reconnect.backoff.ms=1 I can poll our Zookeeper and check if all brokers are alive, but I think KafkaProducer checks it already. Alexey

Re: blocking KafkaProducer call

Great !!. The sync mode and those properties are orthogonal. Thanks, Mayuresh On Wed, Apr 1, 2015 at 10:46 AM, sunil kalva wrote: > thanks grant > by changing linger.ms=0,batch.size=1, problem solved > > > On Wed, Apr 1, 2015 at 11:11 PM, Grant Henke wrote: > > > They can. You can read more a

Re: blocking KafkaProducer call

thanks grant by changing linger.ms=0,batch.size=1, problem solved On Wed, Apr 1, 2015 at 11:11 PM, Grant Henke wrote: > They can. You can read more about configuring the new java producer here: > http://kafka.apache.org/documentation.html#newproducerconfigs > > Thanks, > Grant > > On Wed, Apr 1

Re: blocking KafkaProducer call

They can. You can read more about configuring the new java producer here: http://kafka.apache.org/documentation.html#newproducerconfigs Thanks, Grant On Wed, Apr 1, 2015 at 12:34 PM, sunil kalva wrote: > Does these config params has effect when i try to simulate "sync" mode by > not passing cal

Re: blocking KafkaProducer call

Does these config params has effect when i try to simulate "sync" mode by not passing callback ? On Wed, Apr 1, 2015 at 10:32 PM, Mayuresh Gharat wrote: > Whats your "linger.ms" and "batch.size" ? > > Thanks, > > Mayuresh > > On Wed, Apr 1, 2015 at 5:51 AM, sunil kalva wrote: > > > I am trying

Re: blocking KafkaProducer call

Whats your "linger.ms" and "batch.size" ? Thanks, Mayuresh On Wed, Apr 1, 2015 at 5:51 AM, sunil kalva wrote: > I am trying to simulate "sync" call using following code, > > try { > > Future send = producer.send(new > ProducerRecord("the-topic", "key".getBytes(), > "value".getBytes())).get

Re: blocking KafkaProducer call

I am trying to simulate "sync" call using following code, try { Future send = producer.send(new ProducerRecord("the-topic", "key".getBytes(), "value".getBytes())).get(); send.get(); System.out.println("Time = " + (System.currentTimeMillis() - b)); } catch (Exception e) { } And i

Re: blocking KafkaProducer call

thanks ghenke, that was a quick response. I will test and will let you know if i have some questions. On Tue, Mar 31, 2015 at 11:17 PM, Grant Henke wrote: > I think you are looking at is this section: > > > If you want to simulate a simple blocking call you can do the following: > > > > producer

Re: blocking KafkaProducer call

I think you are looking at is this section: > If you want to simulate a simple blocking call you can do the following: > > producer.send(new ProducerRecord("the-topic", > "key".getBytes(), "value".getBytes())).get(); > > What that is doing is calling .get() on the Future returned by the send meth

blocking KafkaProducer call

Hi According to this http://kafka.apache.org/082/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html documentation, if i don't pass callback it will work as blocking call, Does it mean that message will be immediately sent to kafka cluster and all possible exceptions will be thr

Re: KafkaProducer javadoc Valid configuration strings link correct?

Hi Joe, I think this is a bug on the javadoc, it should be pointing to the newproducerconfigs, and the java producer only accept these configs. Guozhang On Sat, Mar 21, 2015 at 5:29 PM, Joseph Lawson wrote: > Hi everyone, > > > I was reviewing the javadocs for the 082 and 083 Kafka Producer (

KafkaProducer javadoc Valid configuration strings link correct?

Hi everyone, I was reviewing the javadocs for the 082 and 083 Kafka Producer (http://kafka.apache.org/082/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html and http://kafka.apache.org/083/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html)

Re: KafkaProducer from kafka.clients hangs when some partitions are not available

Jun, > > Can you also take a look at the second problem I am having? > > > > > I am trying to test how KafkaProducer behaves with topic replication > > > factor > > > > = 1 > > > > > > > >1. One broker is offline BEFORE KafkaP

Re: KafkaProducer from kafka.clients hangs when some partitions are not available

Jun, Can you also take a look at the second problem I am having? > > > I am trying to test how KafkaProducer behaves with topic replication > > factor > > > = 1 > > > > > >1. One broker is offline BEFORE KafkaProducer starts sending > messa

  1   2   >