Re: Partition Count Dilemma
Hi there, sorry just checking the user list for "parallel consumer" - you can use Confluent's Parallel Consumer client library to increase your concurrency far beyond the number of partitions you have available. For example in your case - you can process 1,000's of messages in parallel without touching the partition count. Have a look here: https://www.confluent.io/confluent-accelerators/#parallel-consumer Latest talk here: https://www.confluent.io/events/kafka-summit-europe-2021/introducing-confluent-labs-parallel-consumer-client/ Github here: https://github.com/confluentinc/parallel-consumer On Thu, Mar 21, 2019 at 9:18 AM Jérémy Thulliez wrote: > In your case, > > Increase the number of partitions on an existing topic will have an impact > on how the message will be dispatched into partitions. > > I mean if you use the default partitionner, and your message have key, the > algorithm is 'hash(key) % number of partitions' > > That means that a message with the same key, won't go into the same > partition, before and after the change. > > IMO, It was important to notice, in case you need ordering by key. > > Jeremy > -- <https://www.confluent.io> Antony Stubbs Follow us: Blog <https://confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog> • Slack <https://slackpass.io/confluentcommunity> • Twitter <https://twitter.com/ConfluentInc> • YouTube <https://youtube.com/confluent>
Re: Confluent's parallel consumer
Hi Israel, Pushkar, everyone! Sorry I didn't notice this message. Feel free to collaborate on the github discussions page about the project too. And yes Israel is correct - offsets are handled correctly regardless of what modes you choose. Offsets are tracked individually, and committed to kafka inside the commit metadata payload - there is no message loss. Here's a conversation on the topic in the discussion forum: https://github.com/confluentinc/parallel-consumer/discussions/135#discussioncomment-1050437 Let me know if I can help with anything else! Cheers, Antony. On Thu, Jul 15, 2021 at 11:50 PM Israel Ekpo wrote: > Hi Pushkar, > > Based on what I understand about the library, I don't think you need to > worry about data loss because there are mechanisms in place to track which > offsets have been processed in the event something goes wrong during > processing. > > If the processing client goes offline or is unresponsive and has to be > resumed or restarted, it should restart from where it left off since the > offsets are being tracked. > > > https://www.confluent.io/events/kafka-summit-europe-2021/introducing-confluent-labs-parallel-consumer-client/ > > This should be true regardless of the ordering strategy you select. > > Take a look at the talk from the most recent Kafka Summit for additional > details on this > > The github documentation for the project also covers this briefly > > > https://github.com/confluentinc/parallel-consumer/blob/master/README.adoc#commit-mode > > I hope this was helpful. If you still have additional questions, please > continue to bring it up. > > Thanks. > > On Thu, Jul 15, 2021 at 11:21 AM Pushkar Deole > wrote: > > > It is consumer client node that has received events and is processing > > those... > > > > On Thu, Jul 15, 2021 at 8:49 PM Israel Ekpo > wrote: > > > > > Hi Pushkar, > > > > > > When you use the term “node/instance” are you referring to the Kafka > > > Brokers or the consuming clients that are retrieving events from the > > > broker? > > > > > > Please could you elaborate/clarify? > > > > > > On Thu, Jul 15, 2021 at 10:00 AM Pushkar Deole > > > wrote: > > > > > > > Well... with key-level ordering, i am mainly concerned about event > > loss, > > > if > > > > any, in below mentioned scenario: > > > > > > > > 1. since event1 with key1 and event2 with key2 are both part of the > > same > > > > partition1 > > > > 2. key1 event has offset 30 while key2 has offset 40 > > > > 3. key2 is processed by background thread and offset committed which > is > > > 40 > > > > 4. before key1 gets processed by background thread, the instance/node > > > goes > > > > down > > > > 5. partition1 gets rebalanced to node2 and start processing ahead of > > > offset > > > > 40, thus losing key1 > > > > > > > > > > > > > > > > On Thu, Jul 15, 2021 at 7:18 PM Israel Ekpo > > > wrote: > > > > > > > > > Hi Pushkar, > > > > > > > > > > If you are selecting key-based ordering, you should not be > concerned > > > > about > > > > > the other keys from the same partitions being committed first > > > > > > > > > > If that is a concern for your use cases then you should go with > > > partition > > > > > based ordering to ensure that the events are processed in the > > sequence > > > > they > > > > > are picked up from the topic partition. > > > > > > > > > > For commit mode, you have the asynchronous, synchronous and > > > transactional > > > > > modes. I think if you are concerned with the order of commits you > > > should > > > > > look into the last two modes. > > > > > > > > > > My recommendation would be to go with the partition based ordering > > with > > > > > synchronous commits to start with. > > > > > > > > > > > > > > > > > > > > On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole < > pdeole2...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi All, and Antony (author of below article) > > > > > > > > > > > > i came across this article which seemed interesting: Introducing > > > > > > Confluent’s Parallel Consumer Message Processing Cl
Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo
> > > > > > > @Boyang: as mentioned by Bruno, can we maybe add black/white > options > > > for > > > > > both proposals, too? > > > > > > > > > > I also agree that Design B is not ideal with regard to the Kafka > > logo. > > > > > Would it be possible to change Design B accordingly? > > > > > > > > > > I am not a font expert, but the fonts in both design are different > > and > > > I > > > > > am wondering if there is an official Apache Kafka font that we > should > > > > > reuse to make sure that the logos align -- I would expect that both > > > > > logos (including "Apache Kafka" and "Kafka Streams" names) will be > > used > > > > > next to each other and it would look awkward if the font differs. > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > On 8/18/20 11:28 AM, Navinder Brar wrote: > > > > > > Hi, > > > > > > Thanks for the KIP, really like the idea. I am +1(non-binding) > on A > > > > > mainly because I felt like you have to tilt your head to realize > the > > > > > otter's head in B. > > > > > > Regards,Navinder > > > > > > > > > > > > On Tuesday, 18 August, 2020, 11:44:20 pm IST, Guozhang Wang < > > > > > wangg...@gmail.com> wrote: > > > > > > > > > > > > I'm leaning towards design B primarily because it reminds me of > > the > > > > > Firefox > > > > > > logo which I like a lot. But I also share Adam's concern that it > > > should > > > > > > better not obscure the Kafka logo --- so if we can tweak a bit to > > fix > > > > it > > > > > my > > > > > > vote goes to B, otherwise A :) > > > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > On Tue, Aug 18, 2020 at 9:48 AM Bruno Cadonna < > br...@confluent.io> > > > > > wrote: > > > > > > > > > > > >> Thanks for the KIP! > > > > > >> > > > > > >> I am +1 (non-binding) for A. > > > > > >> > > > > > >> I would also like to hear opinions whether the logo should be > > > > colorized > > > > > >> or just black and white. > > > > > >> > > > > > >> Best, > > > > > >> Bruno > > > > > >> > > > > > >> > > > > > >> On 15.08.20 16:05, Adam Bellemare wrote: > > > > > >>> I prefer Design B, but given that I missed the discussion > > thread, I > > > > > think > > > > > >>> it would be better without the Otter obscuring any part of the > > > Kafka > > > > > >> logo. > > > > > >>> > > > > > >>> On Thu, Aug 13, 2020 at 6:31 PM Boyang Chen < > > > > > reluctanthero...@gmail.com> > > > > > >>> wrote: > > > > > >>> > > > > > >>>> Hello everyone, > > > > > >>>> > > > > > >>>> I would like to start a vote thread for KIP-657: > > > > > >>>> > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-657%3A+Add+Customized+Kafka+Streams+Logo > > > > > >>>> > > > > > >>>> This KIP is aiming to add a new logo for the Kafka Streams > > > library. > > > > > And > > > > > >> we > > > > > >>>> prepared two candidates with a cute otter. You could look up > the > > > KIP > > > > > to > > > > > >>>> find those logos. > > > > > >>>> > > > > > >>>> > > > > > >>>> Please post your vote against these two customized logos. For > > > > > example, I > > > > > >>>> would vote for *design-A (binding)*. > > > > > >>>> > > > > > >>>> This vote thread shall be open for one week to collect enough > > > votes > > > > to > > > > > >> call > > > > > >>>> for a winner. Still, feel free to post any question you may > have > > > > > >> regarding > > > > > >>>> this KIP, thanks! > > > > > >>>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Michael G. Noll > > Principal Technologist, Office of the CTO > > <https://www.confluent.io> > -- <https://www.confluent.io> Antony Stubbs Principal Consulting Engineer / Solutions Architect Follow us: Blog <https://confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog> • Slack <https://slackpass.io/confluentcommunity> • Twitter <https://twitter.com/ConfluentInc> • YouTube <https://youtube.com/confluent>
