Hi Yi My colleague found that samza automatically set log compaction when creating the checkpointing topic.
Topic:__samza_checkpoint_ver_1_for_xxx_1 PartitionCount:1 ReplicationFactor:3 Configs:segment.bytes=26214400,cleanup.policy=compact Topic: __samza_checkpoint_ver_1_for_xxx_1 Partition: 0 Leader: 66 Replicas: 66,24,65 Isr: 24,65,66 So, the problem is not log-compaction. On Tue, Nov 3, 2015 at 8:33 PM, Yi Pan <nickpa...@gmail.com> wrote: > Hi, Bae, > > Where did you see this log? Is it in JobRunner? Or AppMaster? Or > SamzaContainer? > > There are a few factors that may have the impact: > 1. How many system stream partitions you have as the input? And how many > tasks are there? > 2. Did you set your checkpoint topic as log-compact topic in Kafka? The > topic size would be much smaller if log compaction is turned on. > > Regards > > -Yi > > On Tue, Nov 3, 2015 at 3:59 PM, Bae, Jae Hyeon <metac...@gmail.com> wrote: > > > Hi Samza Dev > > > > Do you know why the following job is taking too long? > > > > 2015-11-03 23:58:17 KafkaCheckpointManager [INFO] Get latest offset > 3386930 > > for topic __samza_checkpoint_ver_1_for_xxx_1 and partition 0. > > > > This is seriously slowing down development. How can I fix this problem? > > > > Thank you > > Best, Jae > > >
