Re: lag for a specific partitions on newly added host
The new host has been in place for over a week. Lag is still high on partitions that exist on that new host. Should I attempt another reassign? Thanks -jeremy > On Dec 13, 2016, at 5:43 PM, Apurva Mehta wrote: > > How did you add the host and when did you measure the lag? If you used the > reassign-partitions script, it will move partitions to the new host, but > the data copy will take time. in that period, those partitions will lag. > However, once the reassign-partitions script finishes, the partitions on > the new replica should be caught up and no longer demonstrate any lag. > >> On Fri, Dec 9, 2016 at 8:58 PM, Jeremy Hansen wrote: >> >> Here’s the topic description: >> >> Topic:blumfrub PartitionCount:15 ReplicationFactor:5 Configs: >>Topic: blumfrub Partition: 0Leader: 1001Replicas: >> 1001,0,1,2,3 Isr: 0,1001,1,2,3 >>Topic: blumfrub Partition: 1Leader: 0 Replicas: >> 0,1,2,3,4 Isr: 0,1,2,3,4 >>Topic: blumfrub Partition: 2Leader: 1 Replicas: >> 1,2,3,4,1001 Isr: 1001,1,2,3,4 >>Topic: blumfrub Partition: 3Leader: 2 Replicas: >> 2,3,4,1001,0 Isr: 0,1001,2,3,4 >>Topic: blumfrub Partition: 4Leader: 0 Replicas: >> 3,4,1001,0,1 Isr: 0,1001,1,3,4 >>Topic: blumfrub Partition: 5Leader: 4 Replicas: >> 4,1001,0,1,2 Isr: 0,1001,1,2,4 >>Topic: blumfrub Partition: 6Leader: 1001Replicas: >> 1001,1,2,3,4 Isr: 1001,1,2,3,4 >>Topic: blumfrub Partition: 7Leader: 0 Replicas: >> 0,2,3,4,1001 Isr: 0,1001,2,3,4 >>Topic: blumfrub Partition: 8Leader: 1 Replicas: >> 1,3,4,1001,0 Isr: 0,1001,1,3,4 >>Topic: blumfrub Partition: 9Leader: 2 Replicas: >> 2,4,1001,0,1 Isr: 0,1001,1,2,4 >>Topic: blumfrub Partition: 10 Leader: 0 Replicas: >> 3,1001,0,1,2 Isr: 0,1001,1,2,3 >>Topic: blumfrub Partition: 11 Leader: 4 Replicas: >> 4,0,1,2,3 Isr: 0,1,2,3,4 >>Topic: blumfrub Partition: 12 Leader: 1001Replicas: >> 1001,2,3,4,0 Isr: 0,1001,2,3,4 >>Topic: blumfrub Partition: 13 Leader: 0 Replicas: >> 0,3,4,1001,1 Isr: 0,1001,1,3,4 >>Topic: blumfrub Partition: 14 Leader: 1 Replicas: >> 1,4,1001,0,2 Isr: 0,1001,1,2,4 >> >> 1001 is the new broker. >> >> -jeremy >> >> >> >>> On Dec 9, 2016, at 8:55 PM, Jeremy Hansen wrote: >>> >>> I added a new host to kafka. Partitions that fall on this new host have >> a very high lag and I’m trying to understand why this would be and how to >> fix it. >>> >>> iloveconsuming blumfrub 0 >> 5434682 7416933 1982251 iloveconsuming_kf0001.host. >> com-1481225033576-47393b55-0 >>> iloveconsuming blumfrub 1 >> 7416828 7416875 47 iloveconsuming_kf0001.host. >> com-1481225033769-17152bca-0 >>> iloveconsuming blumfrub 2 >> 7416799 7416848 49 iloveconsuming_kf0001.host. >> com-1481225033791-77a30285-0 >>> iloveconsuming blumfrub 3 >> 7416898 7416903 5 iloveconsuming_kf0001.host. >> com-1481225033822-d088f844-0 >>> iloveconsuming blumfrub 4 >> 7416891 7416925 34 iloveconsuming_kf0001.host. >> com-1481225033846-78f8e5b5-0 >>> iloveconsuming blumfrub 5 >> 7416843 7416883 40 iloveconsuming_kf0001.host. >> com-1481225033869-54027178-0 >>> iloveconsuming blumfrub 6 >> 5434720 7416896 1982176 iloveconsuming_kf0001.host. >> com-1481225033891-cc3f6bf6-0 >>> iloveconsuming blumfrub 7 >> 7416896 7416954 58 iloveconsuming_kf0001.host. >> com-1481225033915-79b49de8-0 >>> iloveconsuming blumfrub 8 >> 7416849 7416898 49 iloveconsuming_kf0001.host. >> com-1481225033939-1fe784c0-0 >>> iloveconsuming blumfrub 9 >> 7416898 7416917 19 iloveconsuming_kf0001.host. >> com-1481225033961-40cc3185-0 >>> iloveconsuming blumfrub 10 >> 35457186
Re: lag for a specific partitions on newly added host
Here’s the topic description: Topic:blumfrub PartitionCount:15 ReplicationFactor:5 Configs: Topic: blumfrub Partition: 0Leader: 1001Replicas: 1001,0,1,2,3 Isr: 0,1001,1,2,3 Topic: blumfrub Partition: 1Leader: 0 Replicas: 0,1,2,3,4 Isr: 0,1,2,3,4 Topic: blumfrub Partition: 2Leader: 1 Replicas: 1,2,3,4,1001 Isr: 1001,1,2,3,4 Topic: blumfrub Partition: 3Leader: 2 Replicas: 2,3,4,1001,0 Isr: 0,1001,2,3,4 Topic: blumfrub Partition: 4Leader: 0 Replicas: 3,4,1001,0,1 Isr: 0,1001,1,3,4 Topic: blumfrub Partition: 5Leader: 4 Replicas: 4,1001,0,1,2 Isr: 0,1001,1,2,4 Topic: blumfrub Partition: 6Leader: 1001Replicas: 1001,1,2,3,4 Isr: 1001,1,2,3,4 Topic: blumfrub Partition: 7Leader: 0 Replicas: 0,2,3,4,1001 Isr: 0,1001,2,3,4 Topic: blumfrub Partition: 8Leader: 1 Replicas: 1,3,4,1001,0 Isr: 0,1001,1,3,4 Topic: blumfrub Partition: 9Leader: 2 Replicas: 2,4,1001,0,1 Isr: 0,1001,1,2,4 Topic: blumfrub Partition: 10 Leader: 0 Replicas: 3,1001,0,1,2 Isr: 0,1001,1,2,3 Topic: blumfrub Partition: 11 Leader: 4 Replicas: 4,0,1,2,3 Isr: 0,1,2,3,4 Topic: blumfrub Partition: 12 Leader: 1001Replicas: 1001,2,3,4,0 Isr: 0,1001,2,3,4 Topic: blumfrub Partition: 13 Leader: 0 Replicas: 0,3,4,1001,1 Isr: 0,1001,1,3,4 Topic: blumfrub Partition: 14 Leader: 1 Replicas: 1,4,1001,0,2 Isr: 0,1001,1,2,4 1001 is the new broker. -jeremy > On Dec 9, 2016, at 8:55 PM, Jeremy Hansen wrote: > > I added a new host to kafka. Partitions that fall on this new host have a > very high lag and I’m trying to understand why this would be and how to fix > it. > > iloveconsuming blumfrub 0 5434682 >7416933 1982251 > iloveconsuming_kf0001.host.com-1481225033576-47393b55-0 > iloveconsuming blumfrub 1 7416828 >7416875 47 > iloveconsuming_kf0001.host.com-1481225033769-17152bca-0 > iloveconsuming blumfrub 2 7416799 >7416848 49 > iloveconsuming_kf0001.host.com-1481225033791-77a30285-0 > iloveconsuming blumfrub 3 7416898 >7416903 5 > iloveconsuming_kf0001.host.com-1481225033822-d088f844-0 > iloveconsuming blumfrub 4 7416891 >7416925 34 > iloveconsuming_kf0001.host.com-1481225033846-78f8e5b5-0 > iloveconsuming blumfrub 5 7416843 >7416883 40 > iloveconsuming_kf0001.host.com-1481225033869-54027178-0 > iloveconsuming blumfrub 6 5434720 >7416896 1982176 > iloveconsuming_kf0001.host.com-1481225033891-cc3f6bf6-0 > iloveconsuming blumfrub 7 7416896 >7416954 58 > iloveconsuming_kf0001.host.com-1481225033915-79b49de8-0 > iloveconsuming blumfrub 8 7416849 >7416898 49 > iloveconsuming_kf0001.host.com-1481225033939-1fe784c0-0 > iloveconsuming blumfrub 9 7416898 >7416917 19 > iloveconsuming_kf0001.host.com-1481225033961-40cc3185-0 > iloveconsuming blumfrub 10 35457186 >3545722135 > iloveconsuming_kf0001.host.com-1481225033998-a817062e-0 > iloveconsuming blumfrub 11 7416866 >7416909 43 > iloveconsuming_kf0001.host.com-1481225034020-7a15999e-0 > iloveconsuming blumfrub 12 5434739 >7416907 1982168 > iloveconsuming_kf0001.host.com-1481225034043-badde97c-0 > iloveconsuming blumfrub 13 7416818 >7416865 47 > iloveconsuming_kf0001.host.com-1481225034066-6e77e3dc-0 > iloveconsuming blumfrub 14 7416901 >7416947 46 > iloveconsuming_kf0002.host.com-1481225107317-32355c51-0 > > Why do the partitions that fall on the newly added host so lagged? > > Thanks > -jeremy
lag for a specific partitions on newly added host
I added a new host to kafka. Partitions that fall on this new host have a very high lag and I’m trying to understand why this would be and how to fix it. iloveconsuming blumfrub 0 5434682 7416933 1982251 iloveconsuming_kf0001.host.com-1481225033576-47393b55-0 iloveconsuming blumfrub 1 7416828 7416875 47 iloveconsuming_kf0001.host.com-1481225033769-17152bca-0 iloveconsuming blumfrub 2 7416799 7416848 49 iloveconsuming_kf0001.host.com-1481225033791-77a30285-0 iloveconsuming blumfrub 3 7416898 7416903 5 iloveconsuming_kf0001.host.com-1481225033822-d088f844-0 iloveconsuming blumfrub 4 7416891 7416925 34 iloveconsuming_kf0001.host.com-1481225033846-78f8e5b5-0 iloveconsuming blumfrub 5 7416843 7416883 40 iloveconsuming_kf0001.host.com-1481225033869-54027178-0 iloveconsuming blumfrub 6 5434720 7416896 1982176 iloveconsuming_kf0001.host.com-1481225033891-cc3f6bf6-0 iloveconsuming blumfrub 7 7416896 7416954 58 iloveconsuming_kf0001.host.com-1481225033915-79b49de8-0 iloveconsuming blumfrub 8 7416849 7416898 49 iloveconsuming_kf0001.host.com-1481225033939-1fe784c0-0 iloveconsuming blumfrub 9 7416898 7416917 19 iloveconsuming_kf0001.host.com-1481225033961-40cc3185-0 iloveconsuming blumfrub 10 35457186 3545722135 iloveconsuming_kf0001.host.com-1481225033998-a817062e-0 iloveconsuming blumfrub 11 7416866 7416909 43 iloveconsuming_kf0001.host.com-1481225034020-7a15999e-0 iloveconsuming blumfrub 12 5434739 7416907 1982168 iloveconsuming_kf0001.host.com-1481225034043-badde97c-0 iloveconsuming blumfrub 13 7416818 7416865 47 iloveconsuming_kf0001.host.com-1481225034066-6e77e3dc-0 iloveconsuming blumfrub 14 7416901 7416947 46 iloveconsuming_kf0002.host.com-1481225107317-32355c51-0 Why do the partitions that fall on the newly added host so lagged? Thanks -jeremy
Re: rebalancing partitions across a jbod
After running, I see partitions on the new volumes that were added to my brokers. Both mount points are listed in log.dirs comma separated. After running kafka-reassign-partitions.sh, I now see partitions in the second additional volume. -jeremy > On Oct 20, 2016, at 10:30 PM, Manikumar wrote: > > kafka-reassign-partitions.sh is for migrating data to new/other brokers. > How it is useful for migrating data within same broker? > you reassigned some partitions to other broker and moved back to original > broker? > > On Fri, Oct 21, 2016 at 10:46 AM, Jeremy Hansen wrote: > >> So I seemed to have solved this issue just by running >> kafka-reassign-partitions.sh. I now see even distribution of partitions >> across both disk volumes. Does this make sense? >> >> Thanks >> -jeremy >> >>> On Oct 20, 2016, at 10:07 PM, Manikumar >> wrote: >>> >>> Kafka does not automatically migrate the the existing partition data to >> new >>> volumes. Only new >>> partitions will placed on on new volumes. >>> >>> For now, you can manually copy some the partition dirs(careful with >>> checkpoint files) to new disk, >>> or you can increase the partitions. >>> >>> Or we can just delete the old disk contents and add new disk to >> log.dirs, >>> kafka replicates the the >>> data from other brokers and places on available disks. This may take time >>> depending on data size. >>> >>> JIRA related to auto migration: https://issues.apache.org/ >>> jira/browse/KAFKA-1689 >>> >>> On Fri, Oct 21, 2016 at 3:01 AM, Jeremy Hansen >> wrote: >>> >>>> I’ve read several posts on this but it seemed to be old versions of >>>> kafka. I wanted to get the latest proper way to do this. >>>> >>>> If I add a volume to my brokers running kafka 0.10.0 and I’ve adjusted >>>> log.dirs with the new volume mount point, what do I have to do to >> rebalance >>>> so kafka takes advantage of the new volume? Also, does it make sense to >>>> increase partitions when adding an additional volume? >>>> >>>> Thank you >>>> -jeremy >> >>
Re: rebalancing partitions across a jbod
So I seemed to have solved this issue just by running kafka-reassign-partitions.sh. I now see even distribution of partitions across both disk volumes. Does this make sense? Thanks -jeremy > On Oct 20, 2016, at 10:07 PM, Manikumar wrote: > > Kafka does not automatically migrate the the existing partition data to new > volumes. Only new > partitions will placed on on new volumes. > > For now, you can manually copy some the partition dirs(careful with > checkpoint files) to new disk, > or you can increase the partitions. > > Or we can just delete the old disk contents and add new disk to log.dirs, > kafka replicates the the > data from other brokers and places on available disks. This may take time > depending on data size. > > JIRA related to auto migration: https://issues.apache.org/ > jira/browse/KAFKA-1689 > > On Fri, Oct 21, 2016 at 3:01 AM, Jeremy Hansen wrote: > >> I’ve read several posts on this but it seemed to be old versions of >> kafka. I wanted to get the latest proper way to do this. >> >> If I add a volume to my brokers running kafka 0.10.0 and I’ve adjusted >> log.dirs with the new volume mount point, what do I have to do to rebalance >> so kafka takes advantage of the new volume? Also, does it make sense to >> increase partitions when adding an additional volume? >> >> Thank you >> -jeremy
rebalancing partitions across a jbod
I’ve read several posts on this but it seemed to be old versions of kafka. I wanted to get the latest proper way to do this. If I add a volume to my brokers running kafka 0.10.0 and I’ve adjusted log.dirs with the new volume mount point, what do I have to do to rebalance so kafka takes advantage of the new volume? Also, does it make sense to increase partitions when adding an additional volume? Thank you -jeremy
