ases if I use HDFS for storage ?
>>
>> Thank you very much.
>>
>> Regards,
>> Raja.
>>
>> From: "hsy...@gmail.com"
>> Reply-To: "users@apex.apache.org"
>> Date: Monday, June 6, 2016 at 7:13 PM
>>
>> To: "users@a
l] ?
>
> Will the application latency greatly decreases if I use HDFS for storage ?
>
> Thank you very much.
>
> Regards,
> Raja.
>
> From: "hsy...@gmail.com"
> Reply-To: "users@apex.apache.org"
> Date: Monday, June 6, 2016 at 7:13 PM
>
> T
3 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>"
mailto:users@apex.apache.org>>
Subject: Re: kafka offset commit
Raja,
Not exactly, Apex actually stores offsets as part of the operator state, And
state of the operator are checkpointed internally and periodically
, to apex is not storing offsets status at any location ?
> Like how Storm stores in Zookeeper ?
>
>
> Regards,
> Raja.
>
> From: "hsy...@gmail.com"
> Reply-To: "users@apex.apache.org"
> Date: Monday, June 6, 2016 at 6:42 PM
>
> To: "u
t;mailto:users@apex.apache.org>"
mailto:users@apex.apache.org>>
Date: Monday, June 6, 2016 at 6:42 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>"
mailto:users@apex.apache.org>>
Subject: Re: kafka offset commit
Hey Raja,
For 0.8, you have to implement OffsetM
offsets
into either zookeeper(0.8.2 client has API to do that) or any other
datastore like DB or HDFS. And also you have to implement the method
loadInitialOffsets to load back offset you want.
You are welcome to contribute a default implementation using buildin kafka
offset commit request API for
sers@apex.apache.org<mailto:users@apex.apache.org>"
mailto:users@apex.apache.org>>
Subject: Re: kafka offset commit
Hi Raja,
Which Kafka version are you using?
With the new 0.9 connector there is no need for the offset manager:
https://github.com/apache/apex-malhar/tree/master/kafka/
Hi Raja,
Which Kafka version are you using?
With the new 0.9 connector there is no need for the offset manager:
https://github.com/apache/apex-malhar/tree/master/kafka/src/main/java/org/apache/apex/malhar/kafka
Thanks,
Thomas
On Mon, Jun 6, 2016 at 3:06 PM, Raja.Aravapalli
wrote:
> Hi
>
> C
Hi
Can someone please help me understand, where will the offsets be stored when
consuming with “KafkaSinglePortStringInputOperator” ?
And, how to handle restarts ?
I worked with Storm earlier, Storm maintains the offsets in zookeeper and
client id is maintained for every consumer, using whic
