Re: Multiple streaming jobs on the same topic

2016-04-01 Thread R Krishna
Then, can you specify a size/percentage of cache per consumer group? On Apr 1, 2016 9:09 AM, "Cees de Groot" wrote: > One of Kafka's design ideas is to keep data in the JVM to a minimum, > offloading caching to the OS. So on the Kafka level, there's pretty much > not much you can do - the old dat

Re: Multiple streaming jobs on the same topic

2016-04-01 Thread Cees de Groot
One of Kafka's design ideas is to keep data in the JVM to a minimum, offloading caching to the OS. So on the Kafka level, there's pretty much not much you can do - the old data is buffered by the system (has to be to be read into userspace) and thus this reduces the amount of cache available to the

Multiple streaming jobs on the same topic

2016-04-01 Thread Mayur Mohite
Hi, We have a kafka cluster running in production and there are two spark streaming job (J1 and J2) that fetches the data from the same topic. We noticed that if one of the two jobs (say J1) starts reading data from old offset (that job failed for 2 hours and when we started the job after fixing