Re: Apache Kafka Topic Design

[Jens Rantil]

Hi again Joe,

I would suggest keeping things simple and using a single topic and a single
partition with replication in the beginning if possible. You can split your
topic up into multiple topics later if needed. You can split up your topic
into more partitions later if needed.

If you need strict ordering of the events I suggest you use a partition key
from the start in your producers.

Cheers,
Jens

On Tue, Jan 19, 2016 at 6:43 PM, Joe San  wrote:

> Hi Jens,
>
> Thanks for your suggestion. So here is a bit more text:
>
> I have a List of signals for a specific device:
>
> Map[DeviceId, List[Signal]], where Signal is case class Signal(name:
> String, value: String)
>
> Now I will write a producer to connect to the device and read these signals
> at constant intervals. I want to know how I should create my topic assuming
> that I have say 10 devices to start with, later scaling it up to 10
> such devices.
>
> How should I model my topic? Should I create one topic per device?
>
> Thanks and Regards,
> Joe
>
> On Tue, Jan 19, 2016 at 4:58 PM, Jens Rantil  wrote:
>
> > Hi Joe,
> >
> > I think you are leaving out all your requirements to be able to get
> decent
> > answer from anyone here.
> >
> > On Tue, Jan 19, 2016 at 8:47 AM, Joe San 
> wrote:
> >
> > > I soon realized that this is a doomed approach.
> > >
> >
> > Could you elaborate? Why would it be doomed?
> >
> >
> > > Is there a better way to model my topic so that I could
> > > guarantee replication and high throughput and scale if I need to add
> > 10,000
> > > of those devices? Any suggestions?
> > >
> >
> > What does "high throughput and scale" mean? What is "high throughput" for
> > you? What is "scale" for you?
> >
> > Let's start with a simple proposal: How about having a single topic with
> a
> > single partition and you replicate it to two other brokers? You can
> > increase the number of partitions in the future and you have your
> > "guaranteed" (whatever that means) replication.
> >
> > Cheers,
> > Jens
> >
> > --
> > Jens Rantil
> > Backend engineer
> > Tink AB
> >
> > Email: jens.ran...@tink.se
> > Phone: +46 708 84 18 32
> > Web: www.tink.se
> >
> > Facebook  Linkedin
> > <
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > >
> >  Twitter 
> >
>



-- 
Jens Rantil
Backend engineer
Tink AB

Email: jens.ran...@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook  Linkedin

 Twitter 

Re: Apache Kafka Topic Design

[Joe San]

Hi Jens,

Thanks for your suggestion. So here is a bit more text:

I have a List of signals for a specific device:

Map[DeviceId, List[Signal]], where Signal is case class Signal(name:
String, value: String)

Now I will write a producer to connect to the device and read these signals
at constant intervals. I want to know how I should create my topic assuming
that I have say 10 devices to start with, later scaling it up to 10
such devices.

How should I model my topic? Should I create one topic per device?

Thanks and Regards,
Joe

On Tue, Jan 19, 2016 at 4:58 PM, Jens Rantil  wrote:

> Hi Joe,
>
> I think you are leaving out all your requirements to be able to get decent
> answer from anyone here.
>
> On Tue, Jan 19, 2016 at 8:47 AM, Joe San  wrote:
>
> > I soon realized that this is a doomed approach.
> >
>
> Could you elaborate? Why would it be doomed?
>
>
> > Is there a better way to model my topic so that I could
> > guarantee replication and high throughput and scale if I need to add
> 10,000
> > of those devices? Any suggestions?
> >
>
> What does "high throughput and scale" mean? What is "high throughput" for
> you? What is "scale" for you?
>
> Let's start with a simple proposal: How about having a single topic with a
> single partition and you replicate it to two other brokers? You can
> increase the number of partitions in the future and you have your
> "guaranteed" (whatever that means) replication.
>
> Cheers,
> Jens
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.ran...@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook  Linkedin
> <
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> >
>  Twitter 
>