Hi
After modification, my program run for 3 days without problem. Maximum
checkpoint
duration was 6 seconds. (Before modification, it took several minutes)
I think the issue was fixed with it.
Regards,
Hionori
2016-06-15 22:51 GMT+09:00 Robert Metzger :
> Hi,
>
> I've looked at this issue alrea
Thank you for the reply.
I thought that was what I found, but unfortunately wasn't.
The previous messages still be consumed while the consumer has been
re-executed with a few shutdown time...
2016-07-12 9:54 GMT+09:00 Snehal Nagmote :
> Hello *,*
>
> If I understand your question correctly ,
Hello *,*
If I understand your question correctly , what you are looking for is a
setting in consumer which will only read latest messages .
auto.offset.reset = latest
This way , when you start new consumer group , it will always start reading
from the recent offset,
Thanks,
Snehal
On 1
Hello.
Generally, a Kafka Consumer consumes stored messages from Kafka Broker(s)
when the Consumer has been executed.
I, however, want to create a function that only consumes incoming messages
after executing, instead of consuming the previously stored messages as I
mentioned above, for real-time
On Wed, Jun 29, 2016 at 01:30:34PM +0100, Tom Crayford wrote:
> I think you'll be far better off using EBS and Kafka's inbuilt distribution
> than NFS mounts. Kafka's designed for distributing data natively, and not
> for NFS style mounts.
>
I did some initial benchmarking of amazon EFS and agree
On Fri, Jun 17, 2016 at 08:13:52PM +0200, Ismael Juma wrote:
> Hi Harsha,
>
> Comments below.
>
> On Fri, Jun 17, 2016 at 7:48 PM, Harsha wrote:
>
> > Hi Ismael,
> > "Are you saying that you are aware of many Kafka users still
> > using Java 7
> > > who would be ready to upgrade
On Fri, Jun 17, 2016 at 12:02:17AM +0200, jan.o...@wooga.net wrote:
>
> Hi Ismael,
>
> Unfortunately Java 8 doesn't play nice with FreeBSD. We have seen a lot of
> JVM crashes running our 0.9 brokers on Java 8... Java 7 on the other hand is
> totally stable.
>
> Until these issues have been a
Hi,
It would be very great if you can link my repo. Thank very much.
2016-07-11 18:26 GMT+02:00 Guozhang Wang :
> Thanks Florian!
>
>
> Do you mind if I link your repo in the Kafka Streams eco-system page?
>
>
> Guozhang
>
> On Mon, Jul 11, 2016 at 1:37 AM, Michael Noll
> wrote:
>
> > Thanks fo
Oh, I am talking about another memory leak. the offheap memory leak we had
experienced. Which is about Direct Buffer memory. the callstack as below.
ReplicaFetcherThread.warn - [ReplicaFetcherThread-4-1463989770], Error in
fetch kafka.server.ReplicaFetcherThread$FetchRequest@7f4c1657. Possible
cau
Hi (I'm the author of that ticket):
>From my understanding limiting MaxDirectMemory won't workaround this memory
leak. The leak is inside the JVM's implementation, not in normal direct
buffers. On one of our brokers with this issue, we're seeing the JVM report
1.2GB of heap, and 128MB of offheap m
please refer (KAFKA-3933)
a workaround is -XX:MaxDirectMemorySize=1024m
if your callstack has direct buffer issues.(effectively off heap memory)
On Wed, May 11, 2016 at 9:50 AM, Russ Lavoie wrote:
> Good Afternoon,
>
> I am currently trying to do a rolling upgrade from Kafka 0.8.2.1 to 0.9.0.1
>
That depends on the data and your consumers. It’s not necessarily an issue at
all. That said, we prefer to have consumers read off a single topic but our
data is high frequency and easily hashed across partitions so we have hundreds
of consumers reading off some topics.
I would try to partition
EMR is a product optimized for elastic scaling of clusters - using amazon's
pre-defined AMIs. Hadoop and Spark work well on this, but not so much
Kafka, which isn't so easy to automatically add/remove nodes from. The best
bet is still to use EC2 with your own Kafka AMIs or predefined third party
on
Thanks Florian!
Do you mind if I link your repo in the Kafka Streams eco-system page?
Guozhang
On Mon, Jul 11, 2016 at 1:37 AM, Michael Noll wrote:
> Thanks for sharing, Florian!
>
> -Michael
>
>
> On Fri, Jul 8, 2016 at 6:48 PM, Florian Hussonnois
> wrote:
>
> > Hi All,
> >
> > Since a few
Hi, many queueing systems always have concerns on their scheduling policy (i.e.
FCFS, PS)
I have two questions on it.
(1) What is the scheduling policy of Kafka in serving requests from producers
and consumers?
(2) How many requests can be served simultaneously? (Is this related with
num.io.thr
Hi, many queueing systems always have concerns on their scheduling policy (i.e.
FCFS, PS)
I have two questions on it.
(1) What is the scheduling policy of Kafka in serving requests from producers
and consumers?
(2) How many requests can be served simultaneously? (Is this related with
num.io.thr
Hi All:
I update broker version from 0.8.2 to 0.10.0 and set
"compression.type=snappy" in server.properties. Version of producers and
consumers is still 0.8.2. I expect all the new data received by brokers is
stored compressedly in log files. But the result is in contrast。
My que
Hello,
we have a kafka cluster of 20 brokers (v0.8.2.1), and we are repeatedly
running into trouble in a maintenance scenario. Each broker node uses 2 HDs
to store the logs in our case (topic replication is 3 all over)
Typical maintenance scenario is that one of the disks on a node fails, so
we s
Thanks a lot Micheal.
I used WallClockTimeStampExtractor for now.
Thanks,
Vivek
> On Jul 8, 2016, at 1:25 AM, Michael Noll wrote:
>
> Vivek,
>
> in this case you should manually embed a timestamp within the payload of
> the produced messages (e.g. as a Long field in an Avro-encoded message
> v
You could set the auto.commit.interval.ms to a lower value, in your example
it is 10 seconds, which can be a lot of messages. I don't really see how it
could be prevented any further, since offset's can only committed by
consumer to the partitions they are assigned to. I do believe there is some
wo
Using the 0.8.2.1 client.
Is it possible to statistically minimize the possibility of duplication in
this scenario or has this behavior been corrected in a later client
version? Or is the test flawed?
https://gist.github.com/mluban/03a5c0d9221182e6ddbc37189c4d3eb0
Hello Jan,
Thanks for the information and also offer. I might come back to that
some day, but for now we seem to have located the issue:
the consumer was noch consuming as expected. Hence the consumer group is
not yet known to kafka and can not be seen.
Cheers
Jörg
On 11.07.2016 14:07, Jan O
Hi Joerg,
We recently built a tool for fetching current offsets (per partition) for a
given consumer group. Also for setting the group to a specific offset.
It even allows to reset a consumer group to a given timestamp by running bisect
(binary search) on the messages.
Unfortunately we're usin
Hello!
We recently switched to Kafka 0.9.0.1 and currently I don't seem to be
able to figure out how to read the consumer offsets via cli. We are
using the 0.9.0.1 new consumer and are storing the offsets in kafka.
Status:
kafka-consumer-offset-checker.sh is old and deprecated, points to
kaf
Hi kafka folks,
I've written some code to test Kafka brokers using the Kafka 0.9 API (I
need Kerberos support).
It appears the 0.9 API is trapping broker connection refused exception at a
lower level and retrying in a loop (visible only at debug log level), when
I actually want it to fail and rai
Thanks for sharing, Florian!
-Michael
On Fri, Jul 8, 2016 at 6:48 PM, Florian Hussonnois
wrote:
> Hi All,
>
> Since a few weeks I'm working for fun on a CEP library on top of
> KafkaStreams.
> There is some progress and I think my project start to look something, or
> at least I hope ;)
>
> ht
Kafka Server: kafka_2.11-0.9.0.1, Kafka Client:0.9.0.1
2 Kafka brokders(with 1 zookeeper node) as below:
bootstrap.servers=kafka-node0:9092,kafka-node1:9092
then kill the kafka-node0 ,and when the consumer code run at:
ConsumerRecords records = consumer.poll(100);
got the messages below:
15:20:
27 matches
Mail list logo
