Hello With Respect,
Here I met a problem when use scala API to send/receive data to/from kafka
brokers. I write a very simple producer and consumer code(just like the
official examples), I found the code with Java API can work correctly, but the
code with Scala API may lost data. Here is detai
If producer doesn't get a response, retries but both produce-requests
succeeded, you will get duplicates. Kafka does not have a Idempotent
Producer.
On Fri, Nov 13, 2015 at 4:35 AM, Prabhjot Bharaj
wrote:
> Hi Gwen,
>
> If producer cant get a response but the message got committed, Because of
>
Hi Gwen,
If producer cant get a response but the message got committed, Because of
retries, can this producer be committing messages more than once??
I'm trying to see this problem from the point of view of Idempotent Producer
Thanks,
Prabhjot
On Nov 13, 2015 1:26 PM, "Hawin Jiang" wrote:
> Tha
Thanks Gwen for your excellent slides
I will test it again based on your suggestions.
Best regards
Hawin
On Thu, Nov 12, 2015 at 6:35 PM, Gwen Shapira wrote:
> Hi,
>
> First, here's a handy slide-deck on avoiding data loss in Kafka:
>
> http://www.slideshare.net/gwenshap/kafka-reliability-w
Hi,
First, here's a handy slide-deck on avoiding data loss in Kafka:
http://www.slideshare.net/gwenshap/kafka-reliability-when-it-absolutely-positively-has-to-be-there
Note configuration parameters like the number of retries.
Also, it looks like you are sending data to Kafka asynchronously, but
Hi Pradeep
Here is my configuration
# Producer Basics #
# list of brokers used for bootstrapping knowledge about the rest of the
cluster
# format: host1:port1,host2:port2 ...
metadata.broker.list=localhost:9092
# name of the partitioner c
i have 3 brokers;
the ack configuration is -1(all), meaning a message is sent successfully only
after getting every broker's ack;
is this a bug?
At 2015-11-12 21:08:49, "Pradeep Gollakota" wrote:
>What is your producer configuration? Specifically, how many acks are you
>requesting from Ka
What is your producer configuration? Specifically, how many acks are you
requesting from Kafka?
On Thu, Nov 12, 2015 at 2:03 AM, jinxing wrote:
> in kafka_0.8.3.0:
> kafkaProducer = new KafkaProducer<>(properties, new ByteArraySerializer(),
> new ByteArraySerializer());
> kafkaProducer.flush();
in kafka_0.8.3.0:
kafkaProducer = new KafkaProducer<>(properties, new ByteArraySerializer(),
new ByteArraySerializer());
kafkaProducer.flush();
you can call the flush after sending every few messages;
At 2015-11-12 17:36:24, "Hawin Jiang" wrote:
>Hi Prabhjot
>
>The messages are "Thread1_ka
Hi Prabhjot
The messages are "Thread1_kafka_1" and "Thread2_kafka_1". Something like
that.
For GetOffsetShell report below:
[kafka@dn-01 bin]$ ./kafka-run-class.sh kafka.tools.GetOffsetShell
--broker-list dn-01:9092 --time -1 --topic kafka-test
kafka-test:0:12529261
@Jinxing
Can you share
there is a flush api of the producer, you can call this to prevent messages
lost;
maybe it can help;
At 2015-11-12 16:43:54, "Hawin Jiang" wrote:
>Hi Jinxing
>
>I don't think we can resolve this issue by increasing producers. if I
>increased more producers, it should lost more message
yes, increase producer will trigger more messages lost;
that's what I wanted to say;
At 2015-11-12 16:43:54, "Hawin Jiang" wrote:
>Hi Jinxing
>
>I don't think we can resolve this issue by increasing producers. if I
>increased more producers, it should lost more messages.
>
>I just test two
for example the issue can be as below:
from the producer side, say 1000 messages is sent, and no exception found from
the callback;
from the kafka broker side, the log size only increase 900;
as a result only 900 messages received from the consumer;
conclusion:
the problem is we cannot
Hi,
Just to confirm that the number of messages produced are what you are
seeing, What does GetOffsetShell report for this topic ?
Regards,
Prabhjot
On Thu, Nov 12, 2015 at 2:13 PM, Hawin Jiang wrote:
> Hi Jinxing
>
> I don't think we can resolve this issue by increasing producers. if I
> in
Hi Jinxing
I don't think we can resolve this issue by increasing producers. if I
increased more producers, it should lost more messages.
I just test two producers.
Thread Producer 1 has 83064 messages in producer side and 82273 messages in
consumer side
Thread Producer 2 has 89844 messages in p
maybe there some changes in 0.9.0.0;
but still you can try increase producer sending rate, and see if there are
message lost but no exception;
note that, to increase the producer sending rate, you must have enough producer
'power';
in my case, I have 50 producer sending message at the same
Hi Jinxing
I am using kafka_2.10-0.9.0.0-SNAPSHOT. I have downloaded source code and
installed it last week.
I saw 97446 messages have been sent to kafka successfully.
So far, I have not found any failed messages.
Best regards
Hawin
On Thu, Nov 12, 2015 at 12:07 AM, jinxing wrote:
> Hi,
Hi All
I have sent messages to Kafka for one minute. I found 97446 messages in
producer side and 96896 messages in consumer side for Case 1.
I also tried case 2. I have faced the same issues. The number is not match
between producer and consumer.
Can someone take a look at this issue?
Thanks.
The following is my kafka cluster setting:
Kafka version: 2.8.0-0.8.0
5 Zookeepers
16 kafka nodes
90 partitions for each of topic
RC: 3
We got a wired issue for Kafka restarting.
1) modify retention hours in server properties (from 72 hours to 168 hours)
2) Rolling restart Kafka nodes one by one
Hello Chen,
You can look into brokers for "message size too large" exceptions if you
cannot access the producer logs (both of them should have this in their log
files). Also which ack mode are your producer using?
Guozhang
On Mon, Oct 27, 2014 at 10:31 AM, Mark Roberts wrote:
> I think it will
I think it will depend on how your producer application logs things, but
yes I have historically seen exceptions in the producer logs when they
exceed the max message size.
-Mark
On Mon, Oct 27, 2014 at 10:19 AM, Chen Wang
wrote:
> Hello folks,
> I recently noticed our message amount in kafka s
Hello folks,
I recently noticed our message amount in kafka seems to have dropped
significantly. I didn't see any exception on my consumer side. Since
producer is not within my control, I am trying to get some guidance on how
I could debug this issue.
Our individual message size recently have inc
22 matches
Mail list logo
