This sound exactly similar to what I experienced in the similar scenario. Can you please take a look at the File System time stamp of the actual log files on one of the broker hosts ?
For me when I restarted the new brokers with version 0.10.0 it changes to current ts. Meaning if I have set 48 hrs of retention window, the file which is about to get deleted is stamped with latest ts so it will be scheduled to be deleted 48 hrs later. More times you restart the broker it will push deletion operation further into future. So this will lead to additional data retained on the disk. I posted the same question to which I was told it should be fixed in 0.10.1 but I observed same behaviour. Apparently the change from 0.8 to 0.10 is that, the file ts of the log chunk is not more taken into consideration but the time stamp with in the message as set by new protocol is used and it is backward compatible, meaning 0.8 logs files will be treated for their actual file TS. But does not look like in practical. Here are solutions I used. 1. Reduce the retention period if you can so the disk will still hold the logs till they expire in near future after which things become smooth 2. Use log.retention.bytes option for brokers to limit size of the logs retained. My request is that this should be clearly added to documentation of upgrade. Hope this helps. On Sun, Jun 11, 2017 at 7:46 PM, ext-gfenol...@eramet-sln.nc < ext-gfenol...@eramet-sln.nc> wrote: > Hello, > > Since we upgraded from Kafka 0.8 to Kafka0.10.1, our brokers are using > 100% of our disk capacity (100% util in iostat), with from 100MB/s to > 1000MB/s constant read stats. > > It’s been half a week now, and usage is not decreasing. Note that we > didn’t experience that before the upgrade. > > > > There’s nothing particularly helpful in the logs (except the fact that we > have corrupted index files at startup that kafka recreated corrupted again, > but it was already present in Kafka 0.8). > > It leads to a fairly high amount of CPU iowait, in all our environments. > > Even weirder thing is that the usage is not exactly the same on all > brokers. In first one, we have 100MB/s, 200MB/s for the 2nd one, and > 1000MB/s for the 3rd one, all of them having the same datastore, same > disk speed. > > > > With htop, I can see that only one Java thread is doing that, but I’m not > sure how to gather much info about it. > > > > Can you help ? > > Thanks, > > > > Guillaume > > > > > > *Guillaume FENOLLAR* > > Prestataire pour DSI/ESI > > Société Le Nickel - SLN > > Courriel : ext-gfenol...@eramet-sln.nc > > Site internet : www.sln.nc > > Page Facebook : LeNickel.SLN > > > > > > > ------------------------------ > CONFIDENTIALITE > L'information contenue dans ce courrier électronique et ses pièces jointes > est confidentielle, et est établie à l'intention exclusive de ses > destinataires. Dans le cas où ce message ne vous serait pas destiné, nous > vous remercions de bien vouloir en aviser immédiatement l'émetteur et de > procéder à sa suppression. Toutes copies, diffusions ou accès non autorisés > à ce message sont interdits à toutes personnes, autre que le(s) > destinataire(s). Un courrier électronique est susceptible d’altération ou > de falsification et peut entrainer des pertes et/ou la destruction de > données. Le Groupe ERAMET et/ou ses filiales déclinent toute responsabilité > en la matière. En conséquence ce courrier électronique ainsi que ses pièces > jointes sont utilisés à votre propre risque. > > CONFIDENTIALITY > The information contained in this e-mail and any accompanying documents is > confidential or otherwise protected from disclosure. If you are not the > intended recipient, please immediately alert the sender by reply e-mail and > delete this message and any attachments. Any copy, dissemination or > unauthorized access of the contents of this message by anyone other than > the intended recipient is strictly prohibited. E-mails may be susceptible > to falsification or alteration and cause data corruption and/or loss of > data. ERAMET and/or any of its subsidiaries decline any liability resulting > from the consequences thereof. Therefore, this e-mail and any attachments > are used at your own risk. >
