Hi again Joe, I would suggest keeping things simple and using a single topic and a single partition with replication in the beginning if possible. You can split your topic up into multiple topics later if needed. You can split up your topic into more partitions later if needed.
If you need strict ordering of the events I suggest you use a partition key from the start in your producers. Cheers, Jens On Tue, Jan 19, 2016 at 6:43 PM, Joe San <codeintheo...@gmail.com> wrote: > Hi Jens, > > Thanks for your suggestion. So here is a bit more text: > > I have a List of signals for a specific device: > > Map[DeviceId, List[Signal]], where Signal is case class Signal(name: > String, value: String) > > Now I will write a producer to connect to the device and read these signals > at constant intervals. I want to know how I should create my topic assuming > that I have say 10 devices to start with, later scaling it up to 100000 > such devices. > > How should I model my topic? Should I create one topic per device? > > Thanks and Regards, > Joe > > On Tue, Jan 19, 2016 at 4:58 PM, Jens Rantil <jens.ran...@tink.se> wrote: > > > Hi Joe, > > > > I think you are leaving out all your requirements to be able to get > decent > > answer from anyone here. > > > > On Tue, Jan 19, 2016 at 8:47 AM, Joe San <codeintheo...@gmail.com> > wrote: > > > > > I soon realized that this is a doomed approach. > > > > > > > Could you elaborate? Why would it be doomed? > > > > > > > Is there a better way to model my topic so that I could > > > guarantee replication and high throughput and scale if I need to add > > 10,000 > > > of those devices? Any suggestions? > > > > > > > What does "high throughput and scale" mean? What is "high throughput" for > > you? What is "scale" for you? > > > > Let's start with a simple proposal: How about having a single topic with > a > > single partition and you replicate it to two other brokers? You can > > increase the number of partitions in the future and you have your > > "guaranteed" (whatever that means) replication. > > > > Cheers, > > Jens > > > > -- > > Jens Rantil > > Backend engineer > > Tink AB > > > > Email: jens.ran...@tink.se > > Phone: +46 708 84 18 32 > > Web: www.tink.se > > > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin > > < > > > http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary > > > > > Twitter <https://twitter.com/tink> > > > -- Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook <https://www.facebook.com/#!/tink.se> Linkedin <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary> Twitter <https://twitter.com/tink>
