Creating new consumers after data has been discarded

2016-02-24 Thread Ted Swerve
Hello, One of the big attractions of Kafka for me was the ability to write new consumers of topics that would then be able to connect to a topic and replay all the previous events. However, most of the time, Kafka appears to be used with a retention period - presumably in such cases, the events

Re: Kafka as master data store

2016-02-16 Thread Ted Swerve
s > and cold storage. > > > > On 15 Feb 2016, at 21:31, Ted Swerve <ted.swe...@gmail.com> wrote: > > > > HI Ben, Sharninder, > > > > Thanks for your responses, I appreciate it. > > > > Ben - thanks for the tips on settings. A backup could cer

Re: Kafka as master data store

2016-02-15 Thread Ted Swerve
sort of random access with the > low level consumer, its not ideal. > > > > > On 15-Feb-2016, at 10:26 PM, Ted Swerve <ted.swe...@gmail.com> wrote: > > > > Hello, > > > > Is it viable to use infinite-retention Kafka topics as a master data > > st

Kafka as master data store

2016-02-15 Thread Ted Swerve
Hello, Is it viable to use infinite-retention Kafka topics as a master data store? I'm not talking massive volumes of data here, but still potentially extending into tens of terabytes. Are there any drawbacks or pitfalls to such an approach? It seems like a compelling design, but there seem to

Kafka as master data store

2016-02-15 Thread Ted Swerve
Hello, Is it viable to use infinite-retention Kafka topics as a master data store? I'm not talking massive volumes of data here, but still potentially extending into tens of terabytes. Are there any drawbacks or pitfalls to such an approach? It seems like a compelling design, but there seem to