inline

On Mon, Sep 5, 2016 at 12:00 AM, F21 <[email protected]> wrote:

> Hi all,
>
> I am currently looking at using Kafka as a "message bus" for an event
> store. I plan to have all my events written into HBase for permanent
> storage and then have a reader/writer that reads from HBase to push them
> into kafka.


> In terms of kafka, I plan to set it to keep all messages indefinitely.
> That way, if any consumers need to rebuild their views or if new consumers
> are created, they can just read from the stream to rebuild the views.
>

Kafka isn't designed at all for permanent message storage, except for
compacted topics. I suggest you rethink this, unless compacted topics work
for you (Kafka is not designed to keep unbounded amounts of data for
unbounded amounts of time, simply to provide messaging and replay over
short, bounded windows).


>
> I plan to use domain-driven design and will use the concept of aggregates
> in the system. An example of an aggregate might be a customer. All events
> for a given aggregate needs to be delivered in order. In the case of kafka,
> I would need to over partition the system by a lot, as any changes in the
> number of partitions could result in messages that were bound for a given
> partition being pushed into a newly created partition. Are there any issues
> if I create a new partition every time an aggregate is created? In a system
> with a large amount of aggregates, this will result in millions or hundreds
> of millions of partitions. Will this cause performance issues?
>

Yes.

Kafka is designed to support hundreds to thousands of partitions per
machine, not millions (and there is an upper bound per cluster which is
well below one million). I suggest you rethink this and likely use a
standard "hash based partitioning" scheme.


>
> Cheers,
>
> Francis
>
>

Reply via email to