Re: Giant logfile
> > Did you mean all distributions (I know about OpenSUSE and Fedora, maybe > > Ubuntu also does) know about this and patch the code with a higher > > loglevel ? I don't think all distributions know about this; even if they > > do, what purpose does it serve us. Only users who install from source > > will ever see large logfiles and thereby figure out there is some > > problem. > > A little check here seems to confirm your guess: > The concerned Linux box runs beagle 0.2.17 built by openSUSE Build Service > (at least rpm -qi says so), it was started only with the --bg flag (no > --debug flag !) and it still does produce the large log files. Ya. Its possibly because of a lower inotify max_watches limit. It used to print too many warning messages in the past, even without --debug (besides causing other problems). BTW, this problem was fixed in the next bugfix release, 0.2.18. And you should up the max_watches limit; the value in /proc/sys/fs/inotify/max_user_watches should be significantly larger that the number of directories in your home directory. (OpenSUSE 10.3 possibly does that ... not sure). - dBera -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi Debajyoti, Debajyoti Bera wrote: > Did you mean all distributions (I know about OpenSUSE and Fedora, maybe > Ubuntu > also does) know about this and patch the code with a higher loglevel ? I > don't think all distributions know about this; even if they do, what purpose > does it serve us. Only users who install from source will ever see large > logfiles and thereby figure out there is some problem. A little check here seems to confirm your guess: The concerned Linux box runs beagle 0.2.17 built by openSUSE Build Service (at least rpm -qi says so), it was started only with the --bg flag (no --debug flag !) and it still does produce the large log files. > What I was suggesting was a way around this: respect the --debug flag and > allow turning on the debug during runtime (so that if a user is suspicious of > something, debug can be turned on and log file inspected). I'd vote for this suggestion. Kind regards, Stephan. ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
> > This is a good question. I am in favour of turning off the debug flag > > in the "release" build (i.e. the installed one). Instead debugging can > > be turned on when running uninstalled or by using some environment > > variable. Though debugging helps and some hard to find bugs were found > > because some users had the debugging enabled, the cost users pay is > > too much. > > All the major distributions which ship Beagle turn the log level down > by default; you have to pass --debug into the daemon to override it. > I'm not a fan of turning it off for releases, I was referring to this part: // FIXME: We always turn on full debugging output! We are //still debugging this code, after all... //arg_debug ? LogLevel.Debug : LogLevel.Warn, LogLevel.Debug, ... which basically ignores the --debug. I think it is high time we respect the users decision by default. Did you mean all distributions (I know about OpenSUSE and Fedora, maybe Ubuntu also does) know about this and patch the code with a higher loglevel ? I don't think all distributions know about this; even if they do, what purpose does it serve us. Only users who install from source will ever see large logfiles and thereby figure out there is some problem. What I was suggesting was a way around this: respect the --debug flag and allow turning on the debug during runtime (so that if a user is suspicious of something, debug can be turned on and log file inspected). That way distributions wont have to patch it and we can still ask users for debugging information even if they do. - dBera -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: GSoC Weekly Report
> Sorry, I was unclear. By "removing sqlite2" I meant simply removing > it as an option from configure.in and requiring only sqlite3, not > removing the codepaths from the cut-and-pasted code. Then, at some > point in the future, porting over to Mono's own Mono.Data.Sqlite. What to do with our local changes to Mono.Data.SqliteClient ? I always get confused with them. Dont even know what are those changes and why are they there :-/ (it has something to with threading and locking) ? - dBera PS: Mannn... I love these Liberation fonts... can't stop reading the same mail ten times :P -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Just notice, Kerry/Beagle is a Massive Memory Hog
> > > That is a total of 14 and I using over 300 Megs for beagle. (ouch) > > > > You are lucky that 14 instances are using only 300 Megs - thats about > > "20" Megs a piece. Normally instances are about 30 Meg each. > I'm not denying that there's a bug here -- certainly why you have 14 > instances of it running is puzzling -- but memory usage is often > misunderstood on Linux. He emailed me a htop screenshot showing the different instances, and it looks like htop was showing the different "threads" of beagled and indexhelper ... I have asked him to confirm and am yet to hear back from him. But I think on this occassion, its only two instance and a total of 60 MB RSS (or 50 MB is you remove the SHARE) - pretty decent. - dBera -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Just notice, Kerry/Beagle is a Massive Memory Hog
Hi, On 10/13/07, D Bera <[EMAIL PROTECTED]> wrote: > > That is a total of 14 and I using over 300 Megs for beagle. (ouch) > > You are lucky that 14 instances are using only 300 Megs - thats about > "20" Megs a piece. Normally instances are about 30 Meg each. It's also not a very accurate representation of the amount of memory that's actually being used. If you're looking at the VSIZE column, well, it's a lie. :) And while the RSS column is closer, it still doesn't take into account memory that's shared between processes, which Mono apps use quite a bit. A tool like exmap or smap will give you a lot more accurate numbers of memory usage. ps or top are only good for very basic diagnostics, like if you have a (single instance of an) app using a ton of memory. I'm not denying that there's a bug here -- certainly why you have 14 instances of it running is puzzling -- but memory usage is often misunderstood on Linux. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagle (indexing)
Hi Joe, > > I can send you all data you need. Akregator now uses a db which has > > C++ (not C, so no P/Invoke) and python bindinds but no C# binding. > > How do the python bindings access the data? Do they reimplement the > database access, or use a library API underneath? If it's the latter, > it implies that there's a C API that could be used in C#. Its called metakit [1]. The python bindings uses "SCXX (Simplified CXX) ... is a lightweight C++ wrapper for dealing with PyObjects." Didn't look like any scope to P/Invoke when I first came across it. Could have changed since then. - dBera [1] http://www.equi4.com/metakit/ -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi, On 10/13/07, D Bera <[EMAIL PROTECTED]> wrote: > > First, there are always old log files floating around in the Log directory. > > Do we really need them ? Log files older than 14 days are removed automatically by Beagle, IIRC. > This is a good question. I am in favour of turning off the debug flag > in the "release" build (i.e. the installed one). Instead debugging can > be turned on when running uninstalled or by using some environment > variable. Though debugging helps and some hard to find bugs were found > because some users had the debugging enabled, the cost users pay is > too much. All the major distributions which ship Beagle turn the log level down by default; you have to pass --debug into the daemon to override it. I'm not a fan of turning it off for releases, and indeed in this case it probably wouldn't have helped. I'd be willing to bet that the vast majority of those 140 gigs are WARN or ERROR level messages, which are logged in all cases. Moreover, there is a level of degrees here. If a log file is 140 gigs, something is wrong. There is probably looping going on there, etc. I wouldn't expect an index helper log file to get anywhere near one gig in normal usage. (It would be restarted before it got anywhere near that.) So it's actually kind of a good thing, because they point the user to a problem that might only have otherwise been manifested in missed files, CPU pegging, etc. This way we can at least identify is the issue. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagle (indexing)
Hi, On 10/12/07, D Bera <[EMAIL PROTECTED]> wrote: > If you want to work on this, ping me back :) > I can send you all data you need. Akregator now uses a db which has > C++ (not C, so no P/Invoke) and python bindinds but no C# binding. How do the python bindings access the data? Do they reimplement the database access, or use a library API underneath? If it's the latter, it implies that there's a C API that could be used in C#. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Just notice, Kerry/Beagle is a Massive Memory Hog
> Right now with automatic index off, Which option is this ? Can you quote the full text ? > htop reports: > there are 8 incidences of: beagled: /usr/lib/beagle daemon.exe --replace --bg > > AND > there are 6 incidences of: beagle-helper: /usr/lib/beagle/IndexHelper.exe All for the same user ? I would love to say that it is impossible, at least more than one instance of beagled. Well, it is possible but should never happen. Which version of beagle are you using ? How big is your ~/.beagle/Log directory - and can that be shared/emailed-to-me ? > That is a total of 14 and I using over 300 Megs for beagle. (ouch) You are lucky that 14 instances are using only 300 Megs - thats about "20" Megs a piece. Normally instances are about 30 Meg each. - dBera -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Just notice, Kerry/Beagle is a Massive Memory Hog
Right now with automatic index off, htop reports: there are 8 incidences of: beagled: /usr/lib/beagle daemon.exe --replace --bg AND there are 6 incidences of: beagle-helper: /usr/lib/beagle/IndexHelper.exe That is a total of 14 and I using over 300 Megs for beagle. (ouch) Sorry, but this is WAY MUCH resources... please help with a solution Anyone ? TIA Richard ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi, > First, there are always old log files floating around in the Log directory. > Do we really need them ? This is a good question. I am in favour of turning off the debug flag in the "release" build (i.e. the installed one). Instead debugging can be turned on when running uninstalled or by using some environment variable. Though debugging helps and some hard to find bugs were found because some users had the debugging enabled, the cost users pay is too much. Besides, which production ready software has debugging turned on ;-) ? > Second, the current log file should be limited in size. I've found the > following Java code snippet on the Web: > http://searchdomino.techtarget.com/tip/0,289483,sid4_gci1110735,00.html > This could give an idea how to do it (not tested !). > > However, if it reaches a certain size, it should be deleted. And a new one > created. Or keep one or more (but limited in number) in compressed backups. > Similar what happens to the /var/log/messages on a Linux box. There are bz2 > compressed backups. I've just tried it with one of my beagle IndexHelper > log files: 8MB down to 300k, compress ratio > 20. I thought about this too and I am still debating if this is a good idea. Old log files are automatically deleted, so if a large log file appears then its mostly due to some looping bug (all the known ones are fixed but there might be new ones) in which case the single file will keep on growing and growing ... how much compression will help ?! The other approach, i.e. creating a new file and removing/compressing the old one - how will that work too in above situation when the user is caught in some looping bug ?! One view we can adopt is not treat the symptom but eliminate all sources of looping, but but ... we are trying our best and corner cases seem to pop up from time to time :( Personally, I prefer the first option (turn off debug by default with ways to turn it on while beagled is running). - dBera -- - Debajyoti Bera @ http://dtecht.blogspot.com beagle / KDE fan Mandriva / Inspiron-1100 user ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi Debajyoti, Debajyoti Bera wrote: > The reason of the large log file is probably a known issue (and already > fixed). But now this makes me thinking ... how to cap the log file size ?! > > - dBera > IMO, there are two points to consider: First, there are always old log files floating around in the Log directory. Do we really need them ? Second, the current log file should be limited in size. I've found the following Java code snippet on the Web: http://searchdomino.techtarget.com/tip/0,289483,sid4_gci1110735,00.html This could give an idea how to do it (not tested !). However, if it reaches a certain size, it should be deleted. And a new one created. Or keep one or more (but limited in number) in compressed backups. Similar what happens to the /var/log/messages on a Linux box. There are bz2 compressed backups. I've just tried it with one of my beagle IndexHelper log files: 8MB down to 300k, compress ratio > 20. Regards, Stephan. ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
