Re: Giant logfile again
> The remaining lines are only messages like shown in the last line: >Skipping over finished thread 1 of 1: EHT 05944 ... > rocketed the log file size up to 20GByte in less than 2 hours. Its an extremely rare case which I noticed about a month ago. I checked in a preventive measure (I dont quite know why it happened) in r4055. I merged the change to the 0.2.x branch too but it was too late for 0.2.18. It should not happen anymore. - dBera -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Giant logfile again
Hi all, I've got another huge 20GByte logfile (beagle 0.2.18 on openSUSE 10.3, x86_64): Below are the last lines with meaning. There was no activity for 30 minutes and a shutdown was initiated. A thread was aborted, see line 7 in the printout below. Until that point the logfile was only 5MByte. The remaining lines are only messages like shown in the last line: Skipping over finished thread 1 of 1: EHT 05944 ... rocketed the log file size up to 20GByte in less than 2 hours. Any idea what went wrong ? BTW, it stopped at 20GByte size because I've moved my $HOME/.beagle/Log to a dedicated 20G partition. I've seen already log files larger than 200GByte - see an earlier thread in this mailing list. Rgds, Stephan. 20071117 13:09:41.6888 05941 IndexH DEBUG: Helper Size: VmRSS=31.5 MB, size=2.23, 30.7% 20071117 13:09:45.2019 05941 IndexH DEBUG: Helper Size: VmRSS=33.2 MB, size=2.34, 33.5% 20071117 13:09:45.6528 05941 IndexH DEBUG: FileSystemIndex optimized in 46.53s 20071117 13:09:48.2106 05941 IndexH DEBUG: Helper Size: VmRSS=33.7 MB, size=2.38, 34.4% 20071117 13:39:45.8631 05941 IndexH DEBUG: No activity for 30.0 minutes, shutting down 20071117 13:39:45.8650 05941 IndexH INFO: Shutdown requested 20071117 13:39:45.8667 05941 IndexH DEBUG: Thread aborted: EHT 05944 [05941 IndexHelper] Beagle.Daemon.Server:Run 20071117 13:39:45.8667 05941 IndexH DEBUG: at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_thread_interruption_checkpoint () 20071117 13:39:45.8667 05941 IndexH DEBUG: at (wrapper managed-to-native) System.Globalization.CultureInfo:construct_datetime_format () 20071117 13:39:45.8667 05941 IndexH DEBUG: at System.Globalization.CultureInfo.get_DateTimeFormat () [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: at System.Globalization.DateTimeFormatInfo.get_CurrentInfo () [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: at System.Globalization.DateTimeFormatInfo.GetInstance (IFormatProvider provider) [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: at System.DateTime.ToString (System.String format, IFormatProvider fp) [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: at System.String.FormatHelper (System.Text.StringBuilder result, IFormatProvider provider, System.String format, System.Object[] args) [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: at System.Text.StringBuilder.AppendFormat (IFormatProvider provider, System.String format, System.Object[] args) [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: at System.Text.StringBuilder.AppendFormat (System.String format, System.Object arg0, System.Object arg1) [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: at Beagle.Util.Log.WriteLine (LogLevel level, System.String format, System.Object[] args, System.Exception ex) [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: at Beagle.Util.Log.Debug (System.String message, System.Object[] args) [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: at Beagle.Util.Logger.Debug (System.String message, System.Object[] args) [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: at Beagle.Daemon.Server.Run () [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: at Beagle.Util.ExceptionHandlingThread.ThreadStarted () [0x0] 20071117 13:39:45.8667 05941 IndexH DEBUG: 20071117 13:39:45.9036 05941 IndexH DEBUG: All workers have finished. Exiting main loop. 20071117 13:39:45.9047 05941 IndexH DEBUG: Skipping over finished thread 1 of 2: EHT 05944 [05941 IndexHelper] Beagle.Daemon.Server:Run 20071117 13:39:45.9048 05941 IndexH DEBUG: Joining thread 2 of 2: EHT 05946 [05941 IndexHelper] Beagle.IndexHelper.IndexHelperTool:DaemonMonitorWorker 20071117 13:39:46.7391 05941 IndexH DEBUG: Skipping over finished thread 1 of 1: EHT 05944 [05941 IndexHelper] Beagle.Daemon.Server:Run ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi, On 10/14/07, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > Ya. Its possibly because of a lower inotify max_watches limit. It used to > print too many warning messages in the past, even without --debug (besides > causing other problems). > > BTW, this problem was fixed in the next bugfix release, 0.2.18. And you should > up the max_watches limit; the value in /proc/sys/fs/inotify/max_user_watches > should be significantly larger that the number of directories in your home > directory. (OpenSUSE 10.3 possibly does that ... not sure). Yeah, we made inotify (and a higher max_user_watches) a requirement in 0.2.18. openSUSE 10.3 has the number of watches increased to 65535 in /etc/sysctl.conf. Other distros should follow suit. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi, On 10/13/07, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > I was referring to this part: > // FIXME: We always turn on full debugging output! We are > //still debugging this code, after all... > //arg_debug ? LogLevel.Debug : LogLevel.Warn, > LogLevel.Debug, > ... > > which basically ignores the --debug. I think it is high time we respect the > users decision by default. > > Did you mean all distributions (I know about OpenSUSE and Fedora, maybe Ubuntu > also does) know about this and patch the code with a higher loglevel ? Yes, I know that openSUSE, Fedora, and Ubuntu all do this. Packages from the openSUSE Build Service (which I also maintain w/ the stock openSUSE Beagle package) don't have this patch, though, and that is intentional. A lot of the people who are running the build service package were asked to upgrade to it from whatever shipped with 10.2 to ensure that bugs were fixed. This was the easiest way to do that. > I don't think all distributions know about this; even if they do, what purpose > does it serve us. Only users who install from source will ever see large > logfiles and thereby figure out there is some problem. There's a tradeoff here, to be sure, and my assumption was that users who build from source are more likely to be savvy and more able and willing to help us with debugging if they ran into a problem. It's no small task to build from source and the majority of people won't do it. > What I was suggesting was a way around this: respect the --debug flag and > allow turning on the debug during runtime (so that if a user is suspicious of > something, debug can be turned on and log file inspected). That way > distributions wont have to patch it and we can still ask users for debugging > information even if they do. Sure, and we have this today with most of the Beagle instances out there, since most users just use packages. They can run with --debug, which turns up the debug level, and they can send the process SIGUSR1 to raise debugging to the equivalent to --debug at runtime. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Debajyoti Bera wrote: > BTW, this problem was fixed in the next bugfix release, 0.2.18. I've upgraded yesterday to 0.2.18 and will keep an eye on it. > And you should > up the max_watches limit; the value in /proc/sys/fs/inotify/max_user_watches > should be significantly larger that the number of directories in your home > directory. Hmm, the limit is set to 8k, the number of directories in my home is about the double of that. > (OpenSUSE 10.3 possibly does that ... not sure). I'll check this asap. I've just got the 10.3 DVDs on my desk. Regards, Stephan. ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
> > Did you mean all distributions (I know about OpenSUSE and Fedora, maybe > > Ubuntu also does) know about this and patch the code with a higher > > loglevel ? I don't think all distributions know about this; even if they > > do, what purpose does it serve us. Only users who install from source > > will ever see large logfiles and thereby figure out there is some > > problem. > > A little check here seems to confirm your guess: > The concerned Linux box runs beagle 0.2.17 built by openSUSE Build Service > (at least rpm -qi says so), it was started only with the --bg flag (no > --debug flag !) and it still does produce the large log files. Ya. Its possibly because of a lower inotify max_watches limit. It used to print too many warning messages in the past, even without --debug (besides causing other problems). BTW, this problem was fixed in the next bugfix release, 0.2.18. And you should up the max_watches limit; the value in /proc/sys/fs/inotify/max_user_watches should be significantly larger that the number of directories in your home directory. (OpenSUSE 10.3 possibly does that ... not sure). - dBera -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi Debajyoti, Debajyoti Bera wrote: > Did you mean all distributions (I know about OpenSUSE and Fedora, maybe > Ubuntu > also does) know about this and patch the code with a higher loglevel ? I > don't think all distributions know about this; even if they do, what purpose > does it serve us. Only users who install from source will ever see large > logfiles and thereby figure out there is some problem. A little check here seems to confirm your guess: The concerned Linux box runs beagle 0.2.17 built by openSUSE Build Service (at least rpm -qi says so), it was started only with the --bg flag (no --debug flag !) and it still does produce the large log files. > What I was suggesting was a way around this: respect the --debug flag and > allow turning on the debug during runtime (so that if a user is suspicious of > something, debug can be turned on and log file inspected). I'd vote for this suggestion. Kind regards, Stephan. ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
> > This is a good question. I am in favour of turning off the debug flag > > in the "release" build (i.e. the installed one). Instead debugging can > > be turned on when running uninstalled or by using some environment > > variable. Though debugging helps and some hard to find bugs were found > > because some users had the debugging enabled, the cost users pay is > > too much. > > All the major distributions which ship Beagle turn the log level down > by default; you have to pass --debug into the daemon to override it. > I'm not a fan of turning it off for releases, I was referring to this part: // FIXME: We always turn on full debugging output! We are //still debugging this code, after all... //arg_debug ? LogLevel.Debug : LogLevel.Warn, LogLevel.Debug, ... which basically ignores the --debug. I think it is high time we respect the users decision by default. Did you mean all distributions (I know about OpenSUSE and Fedora, maybe Ubuntu also does) know about this and patch the code with a higher loglevel ? I don't think all distributions know about this; even if they do, what purpose does it serve us. Only users who install from source will ever see large logfiles and thereby figure out there is some problem. What I was suggesting was a way around this: respect the --debug flag and allow turning on the debug during runtime (so that if a user is suspicious of something, debug can be turned on and log file inspected). That way distributions wont have to patch it and we can still ask users for debugging information even if they do. - dBera -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi, On 10/13/07, D Bera <[EMAIL PROTECTED]> wrote: > > First, there are always old log files floating around in the Log directory. > > Do we really need them ? Log files older than 14 days are removed automatically by Beagle, IIRC. > This is a good question. I am in favour of turning off the debug flag > in the "release" build (i.e. the installed one). Instead debugging can > be turned on when running uninstalled or by using some environment > variable. Though debugging helps and some hard to find bugs were found > because some users had the debugging enabled, the cost users pay is > too much. All the major distributions which ship Beagle turn the log level down by default; you have to pass --debug into the daemon to override it. I'm not a fan of turning it off for releases, and indeed in this case it probably wouldn't have helped. I'd be willing to bet that the vast majority of those 140 gigs are WARN or ERROR level messages, which are logged in all cases. Moreover, there is a level of degrees here. If a log file is 140 gigs, something is wrong. There is probably looping going on there, etc. I wouldn't expect an index helper log file to get anywhere near one gig in normal usage. (It would be restarted before it got anywhere near that.) So it's actually kind of a good thing, because they point the user to a problem that might only have otherwise been manifested in missed files, CPU pegging, etc. This way we can at least identify is the issue. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi, > First, there are always old log files floating around in the Log directory. > Do we really need them ? This is a good question. I am in favour of turning off the debug flag in the "release" build (i.e. the installed one). Instead debugging can be turned on when running uninstalled or by using some environment variable. Though debugging helps and some hard to find bugs were found because some users had the debugging enabled, the cost users pay is too much. Besides, which production ready software has debugging turned on ;-) ? > Second, the current log file should be limited in size. I've found the > following Java code snippet on the Web: > http://searchdomino.techtarget.com/tip/0,289483,sid4_gci1110735,00.html > This could give an idea how to do it (not tested !). > > However, if it reaches a certain size, it should be deleted. And a new one > created. Or keep one or more (but limited in number) in compressed backups. > Similar what happens to the /var/log/messages on a Linux box. There are bz2 > compressed backups. I've just tried it with one of my beagle IndexHelper > log files: 8MB down to 300k, compress ratio > 20. I thought about this too and I am still debating if this is a good idea. Old log files are automatically deleted, so if a large log file appears then its mostly due to some looping bug (all the known ones are fixed but there might be new ones) in which case the single file will keep on growing and growing ... how much compression will help ?! The other approach, i.e. creating a new file and removing/compressing the old one - how will that work too in above situation when the user is caught in some looping bug ?! One view we can adopt is not treat the symptom but eliminate all sources of looping, but but ... we are trying our best and corner cases seem to pop up from time to time :( Personally, I prefer the first option (turn off debug by default with ways to turn it on while beagled is running). - dBera -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi Debajyoti, Debajyoti Bera wrote: > The reason of the large log file is probably a known issue (and already > fixed). But now this makes me thinking ... how to cap the log file size ?! > > - dBera > IMO, there are two points to consider: First, there are always old log files floating around in the Log directory. Do we really need them ? Second, the current log file should be limited in size. I've found the following Java code snippet on the Web: http://searchdomino.techtarget.com/tip/0,289483,sid4_gci1110735,00.html This could give an idea how to do it (not tested !). However, if it reaches a certain size, it should be deleted. And a new one created. Or keep one or more (but limited in number) in compressed backups. Similar what happens to the /var/log/messages on a Linux box. There are bz2 compressed backups. I've just tried it with one of my beagle IndexHelper log files: 8MB down to 300k, compress ratio > 20. Regards, Stephan. ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
> Wow, 140+ GByte. Oops!!! > Can the size of those log files limited without putting > them into an extra partition ? > > Might be this a known issue. My beagle is a bit older, > still version 0.2.17. The reason of the large log file is probably a known issue (and already fixed). But now this makes me thinking ... how to cap the log file size ?! - dBera -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Giant logfile
Hi all, I've just noticed that I've run out of HD space. Looking around I've found in my $HOME/.beagle/Log: ... -rw-r- 1 steve users 146672805359 2007-10-10 04:41 2007-10-09-15-38-19-IndexHelper ... Wow, 140+ GByte. Can the size of those log files limited without putting them into an extra partition ? Might be this a known issue. My beagle is a bit older, still version 0.2.17. Regards, Stephan. ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
