If target table is dropped then you can remove its hints but there could be more hints from other table. If it has tables of your interest , then I won't comment on truncating hints.
Size of hints depends on Kafka load , looks like you had overloaded the cluster during data load and not hints are just recovering from it. I would say wait until cluster comes to normal state. May be some other expert can suggest an alternate. Sent from my iPhone > On May 16, 2017, at 8:50 AM, varun saluja <saluj...@gmail.com> wrote: > > Thanks for update. > I could see lot of io waits. This causing Gc and mutation drops . > But as i mentioned we do not have high load for now. Hint replays are > creating such high disk I/O. > compactionstats show very high hint bytes like 780gb around. Is this normal? > > Just mentioning we are using flash disks. > > In such case, if i run truncatehints , will it remove or decrease size of > hints bytes in compaction stats. I can trigger repair therafter. > Please let me know if any recommendation on same. > > Also , table which we dumped from kafka which created this much hints and > compaction pendings is also dropped today. Because we have to redump table > again once cluster is stable. > > Regards, > Varun > > Sent from my iPhone > >> On 16-May-2017, at 6:59 PM, Nitan Kainth <ni...@bamlabs.com> wrote: >> >> Yes but it means data has to be replicated using repair. >> >> Hints are out come of unhealthy nodes, focus on finding why you have >> mutation drops, is it node, io or network etc. ideally you shouldn't see >> increasing hints all the time. >> >> Sent from my iPhone >> >>> On May 16, 2017, at 7:58 AM, varun saluja <saluj...@gmail.com> wrote: >>> >>> Hi Nitan, >>> >>> Thanks for response. >>> >>> Yes, I could see mutation drops and increase count in system.hints. Is >>> there any way , i can proceed to truncate hints like using nodetool >>> truncatehints. >>> >>> >>> Regards, >>> Varun Saluja >>> >>>> On 16 May 2017 at 17:52, Nitan Kainth <ni...@bamlabs.com> wrote: >>>> Do you see mutation drops? >>>> Select count from system.hints; is it increasing? >>>> >>>> Sent from my iPhone >>>> >>>>> On May 16, 2017, at 5:52 AM, varun saluja <saluj...@gmail.com> wrote: >>>>> >>>>> Hi Experts, >>>>> >>>>> We are facing issue on production cluster. Compaction on system.hint >>>>> table is running from last 2 days. >>>>> >>>>> >>>>> pending tasks: 1 >>>>> compaction type keyspace table completed total >>>>> unit progress >>>>> Compaction system hints 20623021829 877874092407 >>>>> bytes 2.35% >>>>> Active compaction remaining time : 0h27m15s >>>>> >>>>> >>>>> Active compaction remaining time shows in minutes. But, this is job is >>>>> running like indefinitely. >>>>> >>>>> We have 3 node cluster V 2.1.7. And we ran write intensive job last week >>>>> on particular table. >>>>> Compaction on this table finished but hint table size is growing >>>>> continuously. >>>>> >>>>> Can someone Please help me. >>>>> >>>>> >>>>> Thanks & Regards, >>>>> Varun Saluja >>>>> >>>
