Yes, your approach works. I am not sure if we should take this as default
solution, though. User can have a simple wrapper + customized rebalance
listener. The tricky part is that the rebalance listener might need
different implementations. So it looks the current API provides enough
simplicity and
Thanks Jiangjie
So you agree that with the modified ConsumerIterator.next() code, the high
level consumer becomes at-least-once, even with auto-commit enabled? That
is what I really want to know.
I'll have a look at the rebalancing code. I think I understand: during
rebalancing, with auto-commit
kafka-producer-perf-test.sh --broker-list 192.168.1.107:9092 --message-size
600 --messages 17000 --topic jun8
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
[2015-06-19 14:07:46,196] WARN Failed to send producer req
Yes, you are right. Will update the patch
On Mon, Jun 22, 2015 at 12:16 PM Jiangjie Qin
wrote:
> Should we still store the value bytes when logAsString is set to TRUE and
> only store the length when logAsString is set to FALSE.
>
> On 6/21/15, 7:29 PM, "tao xiao" wrote:
>
> >The patch I submitt
Should we still store the value bytes when logAsString is set to TRUE and
only store the length when logAsString is set to FALSE.
On 6/21/15, 7:29 PM, "tao xiao" wrote:
>The patch I submitted did the what you suggested. It store the size only
>and print it out when error occurs.
>
>On Mon, Jun 2
The patch I submitted did the what you suggested. It store the size only
and print it out when error occurs.
On Mon, Jun 22, 2015 at 5:26 AM Jiangjie Qin
wrote:
> Yes, we can expose a user callback in MM, just like we did for rebalance
> listener.
> I still think ErrorLoggingCallback needs some
I am using kafka_2.10 now. My consumer is tuck in infinite loop at
highlighted line.
Is there any issue with code?
public static void main(String[] args) {
Properties props = new Properties();
props.put("zookeeper.connect","zkaddress:2181");
props.put("group.id", "testgroup");
props.put("zookeeper
Hmm, it might be a little bit difficult to tell what happened without
looking at your test code.
Can you try with the settings I mentioned? We can try to use only two
threads here:
1. Producer thread: produce some messages -> stop/start Kafka server ->
produce some more messages.
2. Consumer thread
Yes, we can expose a user callback in MM, just like we did for rebalance
listener.
I still think ErrorLoggingCallback needs some change, though. Can we only
store the value bytes when logAsString is set to true? That looks more
reasonable to me.
Jiangjie (Becket) Qin
On 6/21/15, 3:02 AM, "tao xia
Sushant,
You are using kafka clients new consumer api. It looks like you want to
use high-level consumer api?. If so you need use following kafka core lib as
the dependency
org.apache.kafka
kafka_2.10
0.8.2.1
https://cwiki.apache.org/confluence/display/KAFKA/Consumer
which is the latest jar to be used for kafka java client.
As in
org.apache.kafka
kafka-clients
0.8.2.1
In class org.apache.kafka.clients.consumer.KafkaConsumer
public Map> poll(long timeout) {
// TODO Auto-generated method stub
return null;
}
poll method retur
The re-balancing that you speak of, doesn't happen, as already noted in the
question if one consumer is listening on all partitions, then if second
consumer starts, it will not receive any message at all, until the first
one fails.
Is there any reference link for this re-balancing behaviour?
On W
On Sun, Jun 21, 2015 at 1:10 AM Jiangjie Qin
wrote:
> Hey Noah,
>
> Carl is right about the offset. The offset to be commit should be the
> largest-consumed-offset + 1. But this should not break the at least once
> guarantee.
> From what I can see, your consumer should not skip messages. Just to
Yes, I agree with that. It is even better if we can supply our own
callback. For people who want to view the content of message when failure
they still can do so
On Sun, Jun 21, 2015 at 2:20 PM Guozhang Wang wrote:
> Hi Tao / Jiangjie,
>
> I think a better fix here may be not letting MirrorMaker
14 matches
Mail list logo
