For what is worth, 0.9.0.0 and above don't accept messages with no key for compacted topics.
Ismael On Mon, Jun 12, 2017 at 10:17 PM, ext-gfenol...@eramet-sln.nc < ext-gfenol...@eramet-sln.nc> wrote: > Hello Milind, and thank you for your answer. > > We just found the solution to our problem. > Apparently, some developers were sending bad messages with no key, so the > log cleaner couldn’t wipe them properly and crashed at startup. The topics > were set in compact mode. > We have set the compact.policy to delete instead and after the cleaning, > the problem disappeared, and we were back to a normal IO operation. > > I consider this as a serious bug, as we’re running on SSDs and several > weeks or less of this 100% usage can easily break this kind of hardware. > > Guillaume > > > ______________________________________________ > FENOLLAR Guillaume (SLN-EXT) > Prestataire pour DSI/ESI > Société Le Nickel - SLN > > > > Courriel : ext-gfenol...@eramet-sln.nc > Site internet : www.sln.nc > Page Facebook : LeNickel.SLN > ______________________________________________ > > -----Message d'origine----- > De : Milind Vaidya [mailto:kava...@gmail.com] > Envoyé : mardi 13 juin 2017 05:17 > À : users@kafka.apache.org > Objet : Re: Kafka 0.10.1 cluster using 100% disk usage (reads) > > This sound exactly similar to what I experienced in the similar scenario. > > Can you please take a look at the File System time stamp of the actual log > files on one of the broker hosts ? > > For me when I restarted the new brokers with version 0.10.0 it changes to > current ts. Meaning if I have set 48 hrs of retention window, the file > which is about to get deleted is stamped with latest ts so it will be > scheduled to be deleted 48 hrs later. More times you restart the broker it > will push deletion operation further into future. So this will lead to > additional data retained on the disk. > > I posted the same question to which I was told it should be fixed in > 0.10.1 but I observed same behaviour. > Apparently the change from 0.8 to 0.10 is that, the file ts of the log > chunk is not more taken into consideration but the time stamp with in the > message as set by new protocol is used and it is backward compatible, > meaning 0.8 logs files will be treated for their actual file TS. But does > not look like in practical. > > Here are solutions I used. > > 1. Reduce the retention period if you can so the disk will still hold the > logs till they expire in near future after which things become smooth 2. > Use log.retention.bytes option for brokers to limit size of the logs > retained. > > My request is that this should be clearly added to documentation of > upgrade. > > Hope this helps. > > > On Sun, Jun 11, 2017 at 7:46 PM, ext-gfenol...@eramet-sln.nc < > ext-gfenol...@eramet-sln.nc> wrote: > > > Hello, > > > > Since we upgraded from Kafka 0.8 to Kafka0.10.1, our brokers are using > > 100% of our disk capacity (100% util in iostat), with from 100MB/s to > > 1000MB/s constant read stats. > > > > It’s been half a week now, and usage is not decreasing. Note that we > > didn’t experience that before the upgrade. > > > > > > > > There’s nothing particularly helpful in the logs (except the fact that > > we have corrupted index files at startup that kafka recreated > > corrupted again, but it was already present in Kafka 0.8). > > > > It leads to a fairly high amount of CPU iowait, in all our environments. > > > > Even weirder thing is that the usage is not exactly the same on all > > brokers. In first one, we have 100MB/s, 200MB/s for the 2nd one, and > > 1000MB/s for the 3rd one, all of them having the same datastore, same > > disk speed. > > > > > > > > With htop, I can see that only one Java thread is doing that, but I’m > > not sure how to gather much info about it. > > > > > > > > Can you help ? > > > > Thanks, > > > > > > > > Guillaume > > > > > > > > > > > > *Guillaume FENOLLAR* > > > > Prestataire pour DSI/ESI > > > > Société Le Nickel - SLN > > > > Courriel : ext-gfenol...@eramet-sln.nc > > > > Site internet : www.sln.nc > > > > Page Facebook : LeNickel.SLN > > > > > > > > > > > > > > ------------------------------ > > CONFIDENTIALITE > > L'information contenue dans ce courrier électronique et ses pièces > > jointes est confidentielle, et est établie à l'intention exclusive de > > ses destinataires. Dans le cas où ce message ne vous serait pas > > destiné, nous vous remercions de bien vouloir en aviser immédiatement > > l'émetteur et de procéder à sa suppression. Toutes copies, diffusions > > ou accès non autorisés à ce message sont interdits à toutes personnes, > > autre que le(s) destinataire(s). Un courrier électronique est > > susceptible d’altération ou de falsification et peut entrainer des > > pertes et/ou la destruction de données. Le Groupe ERAMET et/ou ses > > filiales déclinent toute responsabilité en la matière. En conséquence > > ce courrier électronique ainsi que ses pièces jointes sont utilisés à > votre propre risque. > > > > CONFIDENTIALITY > > The information contained in this e-mail and any accompanying > > documents is confidential or otherwise protected from disclosure. If > > you are not the intended recipient, please immediately alert the > > sender by reply e-mail and delete this message and any attachments. > > Any copy, dissemination or unauthorized access of the contents of this > > message by anyone other than the intended recipient is strictly > > prohibited. E-mails may be susceptible to falsification or alteration > > and cause data corruption and/or loss of data. ERAMET and/or any of > > its subsidiaries decline any liability resulting from the consequences > > thereof. Therefore, this e-mail and any attachments are used at your own > risk. > > > ________________________________ > CONFIDENTIALITE > L'information contenue dans ce courrier électronique et ses pièces jointes > est confidentielle, et est établie à l'intention exclusive de ses > destinataires. Dans le cas où ce message ne vous serait pas destiné, nous > vous remercions de bien vouloir en aviser immédiatement l'émetteur et de > procéder à sa suppression. Toutes copies, diffusions ou accès non autorisés > à ce message sont interdits à toutes personnes, autre que le(s) > destinataire(s). Un courrier électronique est susceptible d’altération ou > de falsification et peut entrainer des pertes et/ou la destruction de > données. Le Groupe ERAMET et/ou ses filiales déclinent toute responsabilité > en la matière. En conséquence ce courrier électronique ainsi que ses pièces > jointes sont utilisés à votre propre risque. > > CONFIDENTIALITY > The information contained in this e-mail and any accompanying documents is > confidential or otherwise protected from disclosure. If you are not the > intended recipient, please immediately alert the sender by reply e-mail and > delete this message and any attachments. Any copy, dissemination or > unauthorized access of the contents of this message by anyone other than > the intended recipient is strictly prohibited. E-mails may be susceptible > to falsification or alteration and cause data corruption and/or loss of > data. ERAMET and/or any of its subsidiaries decline any liability resulting > from the consequences thereof. Therefore, this e-mail and any attachments > are used at your own risk. >
