Hi Giannis Thanks for the reply some partitions have way more data than others?
Yes, some of the partitions are overloaded. Say 5 out of 10 were overloaded. We are now using Default partitioner, the key is device unique identifier. What is the number of your devices? how many messages are you processing? As of now there are 1000 devices. We need to process 1000 messages per sec. (This message count will grow in near future) We are using apache storm now and in the plan to migrate to Flink. Can you please throw some light on backpressure? We need to know is there any way to achieve the no collision on device data with scalable angle and unique channel? Your advice/help will be appreciated. Thanks and regards Karthick. On Fri, Sep 15, 2023 at 6:58 PM Giannis Polyzos <ipolyzos...@gmail.com> wrote: > Hi Karthick, > > on a high level seems like a data skew issue and some partitions have way > more data than others? > What is the number of your devices? how many messages are you processing? > Most of the things you share above sound like you are looking for > suggestions around load distribution for Kafka. i.e number of partitions, > how to distribute your device data etc. > It would be good to also share what your flink job is doing as I don't see > anything mentioned around that.. are you observing back pressure in the > Flink UI? > > Best > > On Fri, Sep 15, 2023 at 3:46 PM Karthick <ibmkarthickma...@gmail.com> > wrote: > >> Dear Apache Flink Community, >> >> >> >> I am writing to urgently address a critical challenge we've encountered >> in our IoT platform that relies on Apache Kafka and real-time data >> processing. We believe this issue is of paramount importance and may have >> broad implications for the community. >> >> >> >> In our IoT ecosystem, we receive data streams from numerous devices, each >> uniquely identified. To maintain data integrity and ordering, we've >> meticulously configured a Kafka topic with ten partitions, ensuring that >> each device's data is directed to its respective partition based on its >> unique identifier. This architectural choice has proven effective in >> maintaining data order, but it has also unveiled a significant problem: >> >> >> >> *One device's data processing slowness is interfering with other devices' >> data, causing a detrimental ripple effect throughout our system.* >> >> To put it simply, when a single device experiences processing delays, it >> acts as a bottleneck within the Kafka partition, leading to delays in >> processing data from other devices sharing the same partition. This issue >> undermines the efficiency and scalability of our entire data processing >> pipeline. >> >> Additionally, I would like to highlight that we are currently using the >> default partitioner for choosing the partition of each device's data. If >> there are alternative partitioning strategies that can help alleviate this >> problem, we are eager to explore them. >> >> We are in dire need of a high-scalability solution that not only ensures >> each device's data processing is independent but also prevents any >> interference or collisions between devices' data streams. Our primary >> objectives are: >> >> 1. *Isolation and Independence:* We require a strategy that guarantees >> one device's processing speed does not affect other devices in the same >> Kafka partition. In other words, we need a solution that ensures the >> independent processing of each device's data. >> >> >> 2. *Open-Source Implementation:* We are actively seeking pointers to >> open-source implementations or references to working solutions that address >> this specific challenge within the Apache ecosystem or any existing >> projects, libraries, or community-contributed solutions that align with our >> requirements would be immensely valuable. >> >> We recognize that many Apache Flink users face similar issues and may >> have already found innovative ways to tackle them. We implore you to share >> your knowledge and experiences on this matter. Specifically, we are >> interested in: >> >> *- Strategies or architectural patterns that ensure independent >> processing of device data.* >> >> *- Insights into load balancing, scalability, and efficient data >> processing across Kafka partitions.* >> >> *- Any existing open-source projects or implementations that address >> similar challenges.* >> >> >> >> We are confident that your contributions will not only help us resolve >> this critical issue but also assist the broader Apache Flink community >> facing similar obstacles. >> >> >> >> Please respond to this thread with your expertise, solutions, or any >> relevant resources. Your support will be invaluable to our team and the >> entire Apache Flink community. >> >> Thank you for your prompt attention to this matter. >> >> >> Thanks & Regards >> >> Karthick. >> >
