I have to join this thread for .... obvious reasons ;) Bojan - you'll get better help if you share your configs. If you are monitoring Flume with something, sharing various metrics/charts will help, too. You could also run Flume under a Java profiler and see what's eating your heap.
Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Thu, Dec 4, 2014 at 4:40 PM, Ahmed Vila <[email protected]> wrote: > Zdravo Bojane :) > > Flume is well designed and it shouldn't eat up the memory. On the other > hand, miss-configuration can effectively bring server to a crawl and > eventually produce events loss. > Pasting your configuration in here, along with basic hardware info behind > it and a size of your single event in bytes would be helpful. > > The most common things to blame is using inappropriately large memory > channel size for a given amount of memory, transaction size, HDFS sink > batch size etc. because all of them are stored in memory. > > Anyway, you can achieve graceful restart by changing flume's configuration > file modification time - basically just touching it. > It will sense a change and as a result close sources, sinks and channels, > and start them again without overhead of booting up JVM. > That should trigger java garbage collector to clean up resources > associated to those closed instances of sources, sinks and channels. > As a result, you might loose some events if you're using memory channel > since I think it doesn't have a shutdown procedure. Sink should flush batch > to the HDFS, but that also should be tested. > > Also, tweaking Java GC could be of a help, but I never had a need to do so > with Flume. > > > On Thu, Dec 4, 2014 at 9:18 PM, Bojan Kostić <[email protected]> > wrote: > >> Hi, >> >> I have a problem with my flume setup. Overtime they just take too much >> memory. And i need them to restart every now and then, I searched web and i >> did not found any clue how to fix this. Some people blame HDFS... >> For now i just kill process with TERM signal and then wait for couple of >> minutes to shutdown. Now i wish to do this automatically every day. But i >> don't want to lose logs. Is there a way to do this? I checked flume-ng >> script and there is only start. I could write my own sh script which will >> send TERM signal and then check for flume process and if there is none >> start again. But first i want to check is there some smarter way to do this. >> >> Best >> Bojan >> > > > > -- > > Best regards, > Ahmed Vila | Senior software developer > DevLogic | Sarajevo | Bosnia and Herzegovina > > Office : +387 33 942 123 > Mobile: +387 62 139 348 > > Website: www.devlogic.eu > E-mail : [email protected] > --------------------------------------------------------------------- > This e-mail and any attachment is for authorised use by the intended > recipient(s) only. This email contains confidential information. It should > not be copied, disclosed to, retained or used by, any party other than the > intended recipient. Any unauthorised distribution, dissemination or copying > of this E-mail or its attachments, and/or any use of any information > contained in them, is strictly prohibited and may be illegal. If you are > not an intended recipient then please promptly delete this e-mail and any > attachment and all copies and inform the sender directly via email. Any > emails that you send to us may be monitored by systems or persons other > than the named communicant for the purposes of ascertaining whether the > communication complies with the law and company policies. > > --------------------------------------------------------------------- > This e-mail and any attachment is for authorised use by the intended > recipient(s) only. This email contains confidential information. It should > not be copied, disclosed to, retained or used by, any party other than the > intended recipient. Any unauthorised distribution, dissemination or copying > of this E-mail or its attachments, and/or any use of any information > contained in them, is strictly prohibited and may be illegal. If you are > not an intended recipient then please promptly delete this e-mail and any > attachment and all copies and inform the sender directly via email. Any > emails that you send to us may be monitored by systems or persons other > than the named communicant for the purposes of ascertaining whether the > communication complies with the law and company policies.
