Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka
Yes but you have more than 1 POS terminal per location so you still don't need 20,000 partitions. Just one per location. How many locations do you have? In doesn’t matter anyway since you can build a Kafka cluster with up to 200,000 partitions if you use the latest versions of Kafka. https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions “As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions” -hans > On Apr 1, 2019, at 2:02 AM, Alexander Kuterin wrote: > > Thanks, Hans! > We use location specific SKU pricing and send specific price lists to the > specific POS terminal. > > пн, 1 апр. 2019 г., 3:01 Hans Jespersen : > >> Doesn’t every one of the 20,000 POS terminals want to get the same price >> list messages? If so then there is no need for 20,000 partitions. >> >> -hans >> >>> On Mar 31, 2019, at 7:24 PM, >> wrote: >>> >>> Hello! >>> >>> >>> >>> I ask for your help in connection with the my recent task: >>> >>> - Price lists are delivered to 20,000 points of sale with a frequency of >> <10 >>> price lists per day. >>> >>> - The order in which the price lists follow is important. It is also >>> important that the price lists are delivered to the point of sale online. >>> >>> - At each point of sale, an agent application is deployed, which >> processes >>> the received price lists. >>> >>> >>> >>> This task is not particularly difficult. Help in solving the task is not >>> required. >>> >>> >>> >>> The difficulty is that Kafka in our company is a new "silver bullet", and >>> the project manager requires me to implement the following technical >>> decision: >>> >>> deploy 20,000 Kafka consumer instances (one instance for each point of >> sale) >>> for one topic partitioned into 20,000 partitions - one partition per >>> consumer. >>> >>> Technical problems obtained in experiments with this technical decision >> do >>> not convince him. >>> >>> >>> >>> Please give me references to the books/documents/blogposts. which clearly >>> shows that Kafka not intended for this way to use (references to other >>> anti-patterns/pitfalls will be useful). >>> >>> My own attempts to find such references were unsuccessful. >>> >>> >>> >>> Thank you! >>> >>> >>> >>
Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka
Thanks, Hans! We use location specific SKU pricing and send specific price lists to the specific POS terminal. пн, 1 апр. 2019 г., 3:01 Hans Jespersen : > Doesn’t every one of the 20,000 POS terminals want to get the same price > list messages? If so then there is no need for 20,000 partitions. > > -hans > > > On Mar 31, 2019, at 7:24 PM, > wrote: > > > > Hello! > > > > > > > > I ask for your help in connection with the my recent task: > > > > - Price lists are delivered to 20,000 points of sale with a frequency of > <10 > > price lists per day. > > > > - The order in which the price lists follow is important. It is also > > important that the price lists are delivered to the point of sale online. > > > > - At each point of sale, an agent application is deployed, which > processes > > the received price lists. > > > > > > > > This task is not particularly difficult. Help in solving the task is not > > required. > > > > > > > > The difficulty is that Kafka in our company is a new "silver bullet", and > > the project manager requires me to implement the following technical > > decision: > > > > deploy 20,000 Kafka consumer instances (one instance for each point of > sale) > > for one topic partitioned into 20,000 partitions - one partition per > > consumer. > > > > Technical problems obtained in experiments with this technical decision > do > > not convince him. > > > > > > > > Please give me references to the books/documents/blogposts. which clearly > > shows that Kafka not intended for this way to use (references to other > > anti-patterns/pitfalls will be useful). > > > > My own attempts to find such references were unsuccessful. > > > > > > > > Thank you! > > > > > > >
Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka
Yes, we have location specific SKU pricing. пн, 1 апр. 2019 г., 4:08 Steve Howard : > Do you have location specific SKU pricing? > > On Sun, Mar 31, 2019, 7:25 PM > > Hello! > > > > > > > > I ask for your help in connection with the my recent task: > > > > - Price lists are delivered to 20,000 points of sale with a frequency of > > <10 > > price lists per day. > > > > - The order in which the price lists follow is important. It is also > > important that the price lists are delivered to the point of sale online. > > > > - At each point of sale, an agent application is deployed, which > processes > > the received price lists. > > > > > > > > This task is not particularly difficult. Help in solving the task is not > > required. > > > > > > > > The difficulty is that Kafka in our company is a new "silver bullet", and > > the project manager requires me to implement the following technical > > decision: > > > > deploy 20,000 Kafka consumer instances (one instance for each point of > > sale) > > for one topic partitioned into 20,000 partitions - one partition per > > consumer. > > > > Technical problems obtained in experiments with this technical decision > do > > not convince him. > > > > > > > > Please give me references to the books/documents/blogposts. which clearly > > shows that Kafka not intended for this way to use (references to other > > anti-patterns/pitfalls will be useful). > > > > My own attempts to find such references were unsuccessful. > > > > > > > > Thank you! > > > > > > > > >
Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka
Do you have location specific SKU pricing? On Sun, Mar 31, 2019, 7:25 PM Hello! > > > > I ask for your help in connection with the my recent task: > > - Price lists are delivered to 20,000 points of sale with a frequency of > <10 > price lists per day. > > - The order in which the price lists follow is important. It is also > important that the price lists are delivered to the point of sale online. > > - At each point of sale, an agent application is deployed, which processes > the received price lists. > > > > This task is not particularly difficult. Help in solving the task is not > required. > > > > The difficulty is that Kafka in our company is a new "silver bullet", and > the project manager requires me to implement the following technical > decision: > > deploy 20,000 Kafka consumer instances (one instance for each point of > sale) > for one topic partitioned into 20,000 partitions - one partition per > consumer. > > Technical problems obtained in experiments with this technical decision do > not convince him. > > > > Please give me references to the books/documents/blogposts. which clearly > shows that Kafka not intended for this way to use (references to other > anti-patterns/pitfalls will be useful). > > My own attempts to find such references were unsuccessful. > > > > Thank you! > > > >
Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka
I don’t want to be a downer, but because kafka is relatively new, the reference material you seek probably doesn’t exist. Kafka is flexible and can be made to work in many different scenarios — not all of the ideal. It sounds like you’ve already reached a conclusion that kafka is the wrong solution for your requirements. Please share with us the evidence that you used to reach this conclusion. It would be helpful if you described the technical problems you encountered in your experiments so that others can give their opinion on whether can can be resolved or whether they are deal-breakers. -- Peter > On Mar 31, 2019, at 4:24 PM, wrote: > > Hello! > > > > I ask for your help in connection with the my recent task: > > - Price lists are delivered to 20,000 points of sale with a frequency of <10 > price lists per day. > > - The order in which the price lists follow is important. It is also > important that the price lists are delivered to the point of sale online. > > - At each point of sale, an agent application is deployed, which processes > the received price lists. > > > > This task is not particularly difficult. Help in solving the task is not > required. > > > > The difficulty is that Kafka in our company is a new "silver bullet", and > the project manager requires me to implement the following technical > decision: > > deploy 20,000 Kafka consumer instances (one instance for each point of sale) > for one topic partitioned into 20,000 partitions - one partition per > consumer. > > Technical problems obtained in experiments with this technical decision do > not convince him. > > > > Please give me references to the books/documents/blogposts. which clearly > shows that Kafka not intended for this way to use (references to other > anti-patterns/pitfalls will be useful). > > My own attempts to find such references were unsuccessful. > > > > Thank you! > > >
Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka
Doesn’t every one of the 20,000 POS terminals want to get the same price list messages? If so then there is no need for 20,000 partitions. -hans > On Mar 31, 2019, at 7:24 PM, wrote: > > Hello! > > > > I ask for your help in connection with the my recent task: > > - Price lists are delivered to 20,000 points of sale with a frequency of <10 > price lists per day. > > - The order in which the price lists follow is important. It is also > important that the price lists are delivered to the point of sale online. > > - At each point of sale, an agent application is deployed, which processes > the received price lists. > > > > This task is not particularly difficult. Help in solving the task is not > required. > > > > The difficulty is that Kafka in our company is a new "silver bullet", and > the project manager requires me to implement the following technical > decision: > > deploy 20,000 Kafka consumer instances (one instance for each point of sale) > for one topic partitioned into 20,000 partitions - one partition per > consumer. > > Technical problems obtained in experiments with this technical decision do > not convince him. > > > > Please give me references to the books/documents/blogposts. which clearly > shows that Kafka not intended for this way to use (references to other > anti-patterns/pitfalls will be useful). > > My own attempts to find such references were unsuccessful. > > > > Thank you! > > >
Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka
Hello! I ask for your help in connection with the my recent task: - Price lists are delivered to 20,000 points of sale with a frequency of <10 price lists per day. - The order in which the price lists follow is important. It is also important that the price lists are delivered to the point of sale online. - At each point of sale, an agent application is deployed, which processes the received price lists. This task is not particularly difficult. Help in solving the task is not required. The difficulty is that Kafka in our company is a new "silver bullet", and the project manager requires me to implement the following technical decision: deploy 20,000 Kafka consumer instances (one instance for each point of sale) for one topic partitioned into 20,000 partitions - one partition per consumer. Technical problems obtained in experiments with this technical decision do not convince him. Please give me references to the books/documents/blogposts. which clearly shows that Kafka not intended for this way to use (references to other anti-patterns/pitfalls will be useful). My own attempts to find such references were unsuccessful. Thank you!
