Hi group
I've got this problem while doing maintenance of our samza job, that job
has been running for a while and processed nearly 350 million data,
we have 40 partitions with 10s checkpoint commit interval, this leads to a
checkpoint topic with 18308702 messages. 10s interval is probably too
sh
Hi,
Is your checkpoint topic log compacted? That may help in reducing the size
of the log.
On Sat, May 7, 2016 at 2:35 AM, Liu Bo wrote:
> Hi group
>
> I've got this problem while doing maintenance of our samza job, that job
> has been running for a while and processed nearly 350 million data,
Thanks for the reply Jagadish, it's really a good point.
I check the kafka configuration, log compaction is enabled, but other
compaction related settings are all default at the server side.
The checkpoint topic has a topic level config which set the cleanup policy
to compact (done by samza while
Hi, Bo,
I embedded my answers in-between:
On Sun, May 8, 2016 at 9:00 PM, Liu Bo wrote:
> The other thing is log retention is set to 24 hour or 30GB. But seems not
> working for checkpoint topic. As all the *.log file are there unlike the
> data topic which only has recent ones.
>
>
When your t
Hi Yi
Thanks a log the reply and the hint. It's more about a kafka issue, the
lucky thing is that Samza experts "happens to" be kafka experts. :)
I just checked the cleaner log and found out we run to this issue:
https://issues.apache.org/jira/browse/KAFKA-1641
the log cleaner stop for about a mo
Hi, Bo,
I am glad that your issue is resolved! As for the upgrade to Kafka 0.9, you
should not need to wait if you want to upgrade broker version. The bug is
on the broker side, hence, if you upgrade Kafka broker version to 0.9, you
will bring in the fix. And Kafka broker 0.9 supports client with
