Also, I am using Samza 0.9.1. On Wed, Nov 4, 2015 at 12:18 AM, Bae, Jae Hyeon <metac...@gmail.com> wrote:
> Hi Yi > > There are 8 partitions in the input topic. The size of checkpoint topic is > 26 MB. > > segment.bytes26214400cleanup.policycompact > > On Tue, Nov 3, 2015 at 11:12 PM, Yi Pan <nickpa...@gmail.com> wrote: > >> Hi, Jae, >> >> That's correct. I mentioned that just to confirm that checkpoint topic >> should be log-compacted. Then, the next question is: what's the size of >> the >> data in the checkpoint topic and how many input topic partitions you have >> in the job? >> >> It would also be helpful if you can share which version of samza you are >> using. >> >> Thanks! >> >> -Yi >> >> On Tue, Nov 3, 2015 at 11:03 PM, Bae, Jae Hyeon <metac...@gmail.com> >> wrote: >> >> > Hi Yi >> > >> > My colleague found that samza automatically set log compaction when >> > creating the checkpointing topic. >> > >> > Topic:__samza_checkpoint_ver_1_for_xxx_1 PartitionCount:1 >> > ReplicationFactor:3 >> Configs:segment.bytes=26214400,cleanup.policy=compact >> > >> > Topic: __samza_checkpoint_ver_1_for_xxx_1 Partition: 0 Leader: 66 >> Replicas: >> > 66,24,65 Isr: 24,65,66 >> > >> > So, the problem is not log-compaction. >> > >> > >> > >> > On Tue, Nov 3, 2015 at 8:33 PM, Yi Pan <nickpa...@gmail.com> wrote: >> > >> > > Hi, Bae, >> > > >> > > Where did you see this log? Is it in JobRunner? Or AppMaster? Or >> > > SamzaContainer? >> > > >> > > There are a few factors that may have the impact: >> > > 1. How many system stream partitions you have as the input? And how >> many >> > > tasks are there? >> > > 2. Did you set your checkpoint topic as log-compact topic in Kafka? >> The >> > > topic size would be much smaller if log compaction is turned on. >> > > >> > > Regards >> > > >> > > -Yi >> > > >> > > On Tue, Nov 3, 2015 at 3:59 PM, Bae, Jae Hyeon <metac...@gmail.com> >> > wrote: >> > > >> > > > Hi Samza Dev >> > > > >> > > > Do you know why the following job is taking too long? >> > > > >> > > > 2015-11-03 23:58:17 KafkaCheckpointManager [INFO] Get latest offset >> > > 3386930 >> > > > for topic __samza_checkpoint_ver_1_for_xxx_1 and partition 0. >> > > > >> > > > This is seriously slowing down development. How can I fix this >> problem? >> > > > >> > > > Thank you >> > > > Best, Jae >> > > > >> > > >> > >> > >
