Re: Kafka Multiple Consumer Group for Same Topic
I don't have benchmarks but multiple consumer groups are possible. For Kafka the performance should be similar or close to as having multiple consumers using a single group. _ From: Senthil Kumar Sent: Tuesday, January 24, 2017 10:38 PM Subject: Kafka Multiple Consumer Group for Same Topic To: Cc: Hi Team , Sorry if the same question asked already in this group ! Say we have topic => ad_events .. I want to read events from ad_events topic and send it to two different systems... This can be achieved by creating two Consume Groups.. Example : Consumer Group SYS1 with 10 threads Consume Group SYS2 with 10 threads Would like to know having two different Consumer Groups will impact performance of Kafka Read ?? Also want to see the *Benchmarking Result*( Numbers ) of Single Topic Read with *One Consumer Group* Vs Single Topic with *Two/Three Consumer Group*.. Cheers, Senthil
Re: Storing Kafka Message JSON to deep storage like S3
Why not just have a parallel consumer read all messages from whichever topics you're interested in and store them wherever you want to? You don't need to "backup" Kafka messages. _ From: Aseem Bansal Sent: Tuesday, December 6, 2016 4:55 PM Subject: Storing Kafka Message JSON to deep storage like S3 To: Hi Has anyone done a storage of Kafka JSON messages to deep storage like S3. We are looking to back up all of our raw Kafka JSON messages for Exploration. S3, HDFS, MongoDB come to mind initially. I know that it can be stored in kafka itself but storing them in Kafka itself does not seem like a good option as we won't be able to query it and the configurations of machines containing kafka will have to be increased as we go. Something like S3 we won't have to manage.
Re: Need to add & remove consumers dynamically in diffrent group and consume consecutively.
Do you have only one partition in the topic? The way Kafka works is that all messages are first distributed into partitions in the topic and then the consumers are distributed among them and they read them sequentially. If you have only one partition in the topic, all your messages will be in it but only one consumer can read messages per partition. If you have multiple partitions, you can have that many consumers reading from it in parallel. Let me know if that wasn't clear. _ From: Kaushil Rambhia/ MUM/CORP/ ENGINEERING Sent: Wednesday, October 19, 2016 8:40 PM Subject: Need to add & remove consumers dynamically in diffrent group and consume consecutively. To: Hi guys, i am using apache kafka with phprd kafka, i want to know how can i use multiple Kafka consumers on same partition from different groups to consume message parallel, say if consumer are c1,c2,c3 consuming single partition 0, than if c1 is consuming from 0 offset than c2 should start from 1 and c3 from 2 and if any new consumer comes up it should start from latest i.e 4th offset which is yet to be consumed by any consumer. So in short all consumers should consume consecutively from kafka partition and should not consumed same message again no matter any new consumer is added or removed from different group.It should automatically select consecutive offsets. It would be good if anyone can recommend any solution or can this be achieved by using kafka with some other tools like spark or something else. -- Regards, Kaushil Rambhia -- DISCLAIMER: The contents of this message may be legally privileged and confidential and are for the use of the intended recipient(s) only. It should not be read, copied and used by anyone other than the intended recipient(s). If you have received this message in error, please immediately notify the sender, preserve its confidentiality and delete it. Before opening any attachments please check them for viruses and defects.
Re: Using Kafka without persisting message to disk
I'd second Tom here. Create a ram disk or just let Kafka write to disk. Use compression and batch messages and the OS fscache would take care of the rest. Kafka is pretty fast and you probably won't notice. _ From: Tom Crayford Sent: Friday, July 15, 2016 12:12 AM Subject: Re: Using Kafka without persisting message to disk To: Users Hi Jack, No, kafka doesn't support not writing to disk. If you're really 100% sure of yourself you could use a ramdisk and mount Kafka on it, but that's not supported. I'd recommend "just" writing to disk, it's plenty fast enough for nearly all use cases. Thanks Tom Crayford Heroku Kafka On Thu, Jul 14, 2016 at 7:33 PM, Jack Huang wrote: > Hi all, > > Is there a way to make a topic to be stored in memory only and not writing > to disk? If not, what's the best way to minimize writing to disk? For this > application we only need the notion of partitions and a short retention > time (1hr or so) from Kafka. We want to use Kafka because we want to keep > the flexibility to add persistence back if we need to. > > Thanks, > Jack >
Re: Kafka as master data store
This topic comes up often on this list. Kafka can be used as a datastore if that’s what your application wants with the caveat that Kafka isn’t designed to keep data around forever. There is a default retention time after which older data gets deleted. The high level consumer essentially reads data as a stream and while you can do sort of random access with the low level consumer, its not ideal. > On 15-Feb-2016, at 10:26 PM, Ted Swerve wrote: > > Hello, > > Is it viable to use infinite-retention Kafka topics as a master data > store? I'm not talking massive volumes of data here, but still potentially > extending into tens of terabytes. > > Are there any drawbacks or pitfalls to such an approach? It seems like a > compelling design, but there seem to be mixed messages about its > suitability for this kind of role. > > Regards, > Ted
Re: Multiple instances of HL Consumer
You can have the same consumer id and Kafka will balance partitions across the two instances automatically. When one of them dies the partitions are rebalanced and assigned to the remaining alive consumers. _ From: Panda, Samaresh Sent: Friday, June 5, 2015 7:42 pm Subject: Multiple instances of HL Consumer To: I've a HL consumer receiving messages using four threads (four partitions). This is a stand-alone Java client. For fail-safe reasons, I want to run another instance of the exact same Java client in a different box. Here are my questions: > Can I keep the same consumer group name or it must be different for the 2nd > instance? > If same consumer group, will the 2nd client receive same set of messages > again? > In general what's the best practice to designing fail-safe clients? Thanks Sam
