Do you need to partition by userid? For example, does the order need to be preserved for each users' messages?
-Dave -----Original Message----- From: Francisco Lopes [mailto:chico.lo...@gmail.com] Sent: Tuesday, June 14, 2016 1:28 PM To: users@kafka.apache.org Subject: Evenly process messages from a partition key Hello, I'm processing events from several users but one user should not affect the other user processing throughput. My initial idea was: one single topic partitioned by the userId. So if I have 5000 users and 1000 topics, each partition will receive messages from 5 users. For 100 workers, each worker will take care of 10 partitions and thus each worker will take care of 50 users. But then 1 user that sends a burst of messages will affect the other 49 users. Without using 1 partition per user, which would bring a lot of workers (and idle workers!), does anyone have any suggestion on how to evenly process these messages? Thank you. Regards, Francisco This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
