someone can help to analysis it?
> 在 2017年11月10日,上午11:08,Json Tu <kafka...@126.com> 写道:
>
> I‘m so sorry for my poor english.
>
> what I really means is my broker machine is configured as 8 core 16G. but my
> jvm configure is as below.
> java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20
> -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC
> -Djava.awt.headless=true -Xloggc:/xx/yy/kafkaServer-gc.log -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=10M
> -XX:+HeapDumpOnOutOfMemoryError.
>
> we have 30+ clusters with this jvm configure, and are deployed on the machine
> which configured as 8 core 16G. compare to other clusters, the current
> cluster have more than 5 times partitions than other clusters.
> when we restart other clusters, there is no such phenomenon.
>
> may be some metrics or logs can leads to find root cause of this phenomenon.
> Looking forward to more suggestions.
>
>
>> 在 2017年11月9日,下午9:59,John Yost <hokiege...@gmail.com> 写道:
>>
>> I've seen this before and it was due to long GC pauses due in large part to
>> a memory heap > 8 GB.
>>
>> --John
>>
>> On Thu, Nov 9, 2017 at 8:17 AM, Json Tu <kafka...@126.com> wrote:
>>
>>> Hi,
>>> we have a kafka cluster which is made of 6 brokers, with 8 cpu and
>>> 16G memory on each broker’s machine, and we have about 1600 topics in the
>>> cluster,about 1700 partitions’ leader and 1600 partitions' replica on each
>>> broker.
>>> when we restart a normal broke, we find that there are 500+
>>> partitions shrink and expand frequently when restart the broker,
>>> there are many logs as below.
>>>
>>> [2017-11-09 17:05:51,173] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:22,047] INFO Partition [Yelp,5] on broker 4759726:
>>> Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:28,634] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:44,658] INFO Partition [Yelp,5] on broker 4759726:
>>> Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:47,611] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:07:19,703] INFO Partition [Yelp,5] on broker 4759726:
>>> Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:07:26,811] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> …
>>>
>>>
>>> and repeat shrink and expand after 30 minutes which is the default
>>> value of leader.imbalance.check.interval.seconds, and at that time
>>> we can find the log of controller’s auto rebalance,which can leads some
>>> partition’s leader change to this restarted broker.
>>> we have no shrink and expand when our cluster is running except when
>>> we restart it,so replica.fetch.thread.num is 1,and it seems enough.
>>>
>>> we can reproduce it at each restart,can someone give some suggestions.
>>> thanks before.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>