Hi Thiago, Have you had a chance to work on the Kinesis support for Samza? I've had a few people ask me about it, so perhaps there's an opportunity for folks to collaborate. There's definitely interest.
I've also got an engineering contact for Kinesis at Amazon who is interested in helping. I don't have time to work on this myself, but if you want to ping me off-list I can put you in touch. (Anyone else lurking on this list who's interested in Kinesis?) Best, Martin On 7 Oct 2014, at 21:40, Thiago Souza <[email protected]> wrote: > Hi Chris, > > Thanks for the clarification! I'll provide more information as soon as I > get something implemented. > > Regards > > On Tue, Oct 7, 2014 at 5:31 PM, Chris Riccomini < > [email protected]> wrote: > >> Hey Thiago, >> >> You can return null for these APIs. If you do so, it's up to the >> underlying SystemConsumer implementation to determine what to do, since it >> doesn't know where to start reading from. In Kafka's case, it just falls >> back to the auto.offset.reset setting in Kafka's consumer config API, >> which defaults to the most recent message. In Kinesis' case, you could >> have the SystemConsumer start reading from anywhere that makes sense. >> >> Cheers, >> Chris >> >> On 10/7/14 1:26 PM, "Thiago Souza" <[email protected]> wrote: >> >>> Hello all, >>> >>> I've been doing a little research on this. >>> >>> It seems that the first obstacle would be with >>> "SystemAdmin.getOffsetsAfter" as it looks like that Amazon Kinesis does >>> not >>> provide this information out of the box. >>> >>> Regards >>> >>> On Tue, Oct 7, 2014 at 5:02 PM, Scott Clasen <[email protected]> wrote: >>> >>>> +1 >>>> >>>> On Tue, Oct 7, 2014 at 12:51 PM, Martin Kleppmann <[email protected] >>> >>>> wrote: >>>> >>>>> I've personally not used Kinesis before, but as far as I know it's >>>> very >>>>> similar to Kafka (it is also based on a total order of messages per >>>>> partition), so Samza should be able to support it well. >>>>> >>>>> If you want to develop a SystemConsumer/SystemProducer for Kinesis, it >>>>> would be an awesome contribution to Samza :) >>>>> >>>>> Thanks, >>>>> Martin >>>>> >>>>> On 7 Oct 2014, at 15:35, Thiago Souza <[email protected]> wrote: >>>>> >>>>>> Hi there, >>>>>> >>>>>> I'm quite new to Samza and currently evaluating it for a new >>>> project. >>>>>> >>>>>> Does it make any sense to implement an alternative to Kafka based on >>>>> Amazon >>>>>> Kinesis? >>>>>> >>>>>> Thanks, >>>>>> Thiago Souza >>>>> >>>>> >>>> >> >>
