beagle memory
Hi! Found a second (third) issue with beagle 0.1 after running beagled for a while (2 hours or so) it uses 800M of memory? how come here is the line copied from top: 16390 seppl 15 0 797m 274m 3000 S 32.7 54.5 19:08.08 mono-beagled --debug /usr/lib/beagle/BeagleDaemon.exe --fg --deny-backend EvolutionDataServer . I started beagle without evolutiondataserver and imlog indexer -- see my previous mail. Gregor -- Dipl.-Ing. Gregor Hlawacek -- http://www.unileoben.ac.at/~hlawacek 8700 Leoben, Austria (o_ (o_ (o) (o -! //\ //\ //\ //\ V_/_ [ ]/_V_/_V_/_ penguin eating fish travelling being noisy GPG ID: FBCDF36E GPG Finger Print: 1AA0 DC5C 186F 6144 01F0 8FB2 7ABD D7FB FBCD F36E signature.asc Description: This is a digitally signed message part ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagle 0.1 troubles IMLog and evolution
On Mon, 2005-09-19 at 16:38 +0530, Vishy wrote: With what version of gtk-sharp2 have you compiled evolution-sharp? If you have updated gtk-sharp2 recently, you have to recompile evolution-sharp with the new version of gtk-sharp2. Vishy I am using gtk-sharp 2.5.91 I recompiled both gtk-sharp and evolution-sharp before recompiling beagle but only a slight improvement: After the crash the last line no reads: ** (beagled:25536): WARNING **: FIXME: wait for completion unimplemented Maybe you know what that means Gregor btw: I don't have to delete the index I just start without evolution backend and everything works (more or less). but I think this is normal DEBUG: Bailing out of HandleConnection -- shutdown requested DEBUG: (1) Waiting for 1 worker... DEBUG: waiting for server '/home/lawa/.beagle/socket' DEBUG: -calendar:///[EMAIL PROTECTED] DEBUG: +calendar:///[EMAIL PROTECTED] DEBUG: FileSystemWatcher: OnChangedEvent /home/lawa/.beagle INFO: Exiting DEBUG: Server '/home/lawa/.beagle/socket' shut down DEBUG: Leaving BeagleDaemon.Main DEBUG: No live ExceptionHandlingThreads! DEBUG: The daemon appears to have gone away. DEBUG: Shutting down helper. DEBUG: (1) Waiting for 1 worker... DEBUG: waiting for server '/home/lawa/.beagle/socket-helper' [EMAIL PROTECTED] ~ $ INFO: Exiting DEBUG: Server '/home/lawa/.beagle/socket-helper' shut down [EMAIL PROTECTED] ~ $ beagled --fg --deny-backend IMLog INFO: Starting Beagle Daemon (version 0.1.0) DEBUG: Command Line: /usr/lib/beagle/BeagleDaemon.exe --fg --deny-backend IMLog DEBUG: Starting main loop DEBUG: Starting messaging server DEBUG: Starting QueryDriver DEBUG: Found index helper at /usr/lib/beagle/beagled-index-helper inotify_init: Function not implemented Inotify not supported! You need a 2.6.13 kernel or later with CONFIG_INOTIFY enabled.WARN: Could not initialize inotify DEBUG: Starting FileSystemWatcher Backend DEBUG: Found 10 types in BeagleDaemonLib, Version=1.4.3.3, Culture=neutral DEBUG: Found 0 user-configured static queryables DEBUG: Adding root: /home/lawa INFO: Starting Evolution mail backend DEBUG: Starting mail crawl DEBUG: Loaded 314 records from /home/lawa/.beagle/Indexes/FileSystemIndex/FileAttributesStore.db in 0.006s DEBUG: FileSystemWatcher watching /home/lawa DEBUG: Will index mbox /home/lawa/.evolution/mail/local/Outbox DEBUG: Will index mbox /home/lawa/.evolution/mail/local/Inbox DEBUG: Will index mbox /home/lawa/.evolution/mail/local/Drafts DEBUG: Will index mbox /home/lawa/.evolution/mail/local/Sent DEBUG: Will index mbox /home/lawa/.evolution/mail/local/Alte Sendmails DEBUG: Done starting FileSystemQueryable INFO: Scanning Tomboy notes... INFO: Scanning addressbooks and calendars INFO: Scanned 2 notes in .11s DEBUG: Getting addressbook changes for file:///home/lawa/.evolution/addressbook/local/system DEBUG: Will index ... . DEBUG: Addressbook file:///home/lawa/.evolution/addressbook/local/system: 0 added, 0 changed, 0 removed DEBUG: Getting calendar changes for file:///home/lawa/.evolution/calendar/local/system Unhandled Exception: GLib.GException: Unknown error in [0x0008e] (at /var/tmp/portage/evolution-sharp-0.10.2/work/evolution-sharp-0.10.2/evolution/Cal.custom:122) Evolution.Cal:GetChanges (string,Evolution.CalComponent[],Evolution.CalComponent[],string[]) in 0x002a0 Beagle.Daemon.EvolutionDataServerQueryable.EvolutionDataServerQueryable:CalendarIndexSourceChanges (Evolution.Source) in 0x0003b (wrapper delegate-invoke) System.MulticastDelegate:invoke_void_Source (Evolution.Source) in 0x000f5 Beagle.Daemon.EvolutionDataServerQueryable.EdsSource:IndexSourceGroup (Evolution.SourceGroup) in 0x00072 Beagle.Daemon.EvolutionDataServerQueryable.EdsSource:Index () in 0x00345 Beagle.Daemon.EvolutionDataServerQueryable.EvolutionDataServerQueryable:Start () in 0x00016 Beagle.Daemon.Queryable:Start () in 0x000bb Beagle.Daemon.QueryDriver:Start () in 0x00159 Beagle.Daemon.BeagleDaemon:StartupProcess () in 0x00047 (wrapper delegate-invoke) System.MulticastDelegate:invoke_bool () in 0x0002a IdleProxy:Handler () in 0x00036 (wrapper native-to-managed) IdleProxy:Handler () in (unmanaged) 0xb7f61583 in 0x4 (wrapper managed-to-native) Gtk.Application:gtk_main () in 0x7 Gtk.Application:Run () in 0x00515 Beagle.Daemon.BeagleDaemon:Main (string[]) ** (beagled:25536): WARNING **: FIXME: wait for completion unimplemented On 19/09/05, Gregor Hlawacek [EMAIL PROTECTED] wrote: Hi! I updated from 0.0.12 to 0.1. Looks nice but still some troubles. I have troubles with the IMLog backend. When I use it beagle throws some error messages and doesn't to anything. Here is the output if I use only the IMLog Backend: INFO: Starting Gaim log backend DEBUG: Caught ResponseMessageException: Connection refused DEBUG: Launching helper process DEBUG: Starting messaging server WARN: Unable to set IO-priority for process to idle DEBUG: Helper Size: VmRSS=18.5 MB, size=1.04, 1.0% DEBUG:
Re: beagle memory
Hi! 2005/9/19, Gregor Hlawacek [EMAIL PROTECTED]: Found a second (third) issue with beagle 0.1 after running beagled for awhile (2 hours or so) it uses 800M of memory? @devs: Could this be because of shutdown issues? Whenever I shutdown beagle (via beagle-shutdown) *or* whenever beagle tries to shutdown one of its helpers (when they consume too much memory), I get Zombie processes. Beagle stays intact, but the number of its zombie children grows over time, and if there are such zombies I can't exit beagled without kill-9 it... Is there any additional info I can provide? I begin to love Beagle again, it's slowly getting really useful - but the memory consumption keeps me from daring to let it run while I'm away... Nico. ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagle memory
It might be useful to know if this is due to any specific backend ( the output of beagle-index-info while beagled is running reports active backends). Found a second (third) issue with beagle 0.1 after running beagled for a while (2 hours or so) it uses 800M of memory? @devs: Could this be because of shutdown issues? Whenever I shutdown beagle (via beagle-shutdown) *or* whenever beagle tries to shutdown one of its helpers (when they consume too much memory), I get Zombie processes. Beagle stays intact, but the number of its zombie children grows over time, and if there are such zombies I can't exit beagled without kill-9 it... Is there any additional info I can provide? I begin to love Beagle again, it's slowly getting really useful - but the memory consumption keeps me from daring to let it run while I'm away... Nico. ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagle memory
Nico Kaiser wrote: @devs: Could this be because of shutdown issues? Whenever I shutdown beagle (via beagle-shutdown) *or* whenever beagle tries to shutdown one of its helpers (when they consume too much memory), I get Zombie processes. Beagle stays intact, but the number of its zombie children grows over time, and if there are such zombies I can't exit beagled without kill-9 it... Is there any additional info I can provide? Which zombies are these? Daniel ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: e_book_load_uri: no factories available?
Hi, On Sat, 2005-09-17 at 12:59 +0200, Tomasz Torcz wrote: Current Beagle from CVS won't start: (beagled:2164): libebook-WARNING **: Can't find installed BookFactories Unhandled Exception: GLib.GException: e_book_load_uri: no factories available for uri `file:///home/zdzichu/.evolution/addressbook/local/system' What does mean? I have installed evolution-2.4.0, evolution-data-server-1.4, evolution-sharp-0.9. And I use Evolution as calendaring application. This exception shouldn't be fatal. I have no idea what it means, but I checked in a fix so that it won't block the startup of the daemon. If I had to guess, it would be that you have multiple versions of e-d-s on your system and Beagle is loading the wrong one. But either way, it appears to be an issue with libebook (the evolution addressbook library) and not Beagle directly. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle 0.1.0 - problem compiling
Hi, ** (/usr/lib/mono/1.0/mcs.exe:21547): WARNING **: The following assembly referen ced from /usr/lib/mono/gecko-sharp-2.0/gecko-sharp.dll could not be loaded: Assembly: gtk-sharp(assemblyref_index=2) Version:2.0.0.0 Public Key: 35e10195dab3c99f The assembly was not found in the Global Assembly Cache, a path listed in the MO NO_PATH environment variable, or in the location of the executing assembly (/usr /lib/mono/gecko-sharp-2.0). It looks like you don't have gtk-sharp 1.9.x or newer installed. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Some 0.1.0 questions
Hi, On Sun, 2005-09-18 at 12:25 +0200, Andreas Wasserman wrote: Q: I have a large archive of movie trailers, where Beagle could be very handy in handing me results quickly, but I rarely use beagle for this since it seems to produce weird results depending on what I search for. Q: Some search words (like The Rock) throw Best into loading mode not giving any results and making my cursor indicate it's loading until I give it a new search word that doesn't produce this behaviour. Best throws this exception but doesn't crash : Did you run previous versions of Beagle? You may want to delete your indexes (in ~/.beagle/Indexes) and have them rebuilt, since certain specifics of how files are indexed may have changed. Q: Window position for best, could it be saved from session to session? first thing I do every first launch after a startup is move the window to my prefered location. This is #304775. Just need someone to hack it up. :) Q: There is a feature request in bugzilla about Categories, any thoughts on this? For me it would be a godsend, now when Best shows results by date I usually get e-mails first and often end up having to scroll multiple pages to get to what I want, so if I find 110 E-Mails, 2 Video Files and 1 Audio file, having categories would even things out and show my hit on the very first page. I know I can search in specific categories but first of all it's a bunch of extra clicks I have to do before doing the actual search and second of all it doesn't cover categories, just types so I'm screwed there. The whole UI is going to get reworked here soon, including sorting by type, so hopefully this will be fixed before too long. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagle memory
Hi, On Mon, 2005-09-19 at 15:50 +0200, Nico Kaiser wrote: @devs: Could this be because of shutdown issues? Whenever I shutdown beagle (via beagle-shutdown) *or* whenever beagle tries to shutdown one of its helpers (when they consume too much memory), I get Zombie processes. Beagle stays intact, but the number of its zombie children grows over time, and if there are such zombies I can't exit beagled without kill-9 it... The zombie processes are a bug in mono. If you upgrade to mono 1.1.8.3 or 1.1.9 these should go away. (I also think we worked around a lot of them in 0.1.0 but I might be wrong about that.) That won't really fix the memory problem though for a singular instance. The index helper process can end up taking a large amount of memory, but it's a short-lived process and we closely monitor its memory usage and shut it down when that happens. That's because of the size and nature of people's data. If beagled grows large, like it has in Gregor's case, that's obviously a bug. And like D Bera said, it would be good if it can be narrowed down to a specific backend. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagle memory
Hi, On Mon, 2005-09-19 at 13:00 -0400, Joe Shaw wrote: Can you run beagled with the --debug-memory flag and then file a bug with the logs from ~/.beagle/Log attached? That might give us an idea of what is causing the memory usage to get out of hand. Also, like D Bera suggested, can you narrow it down to one specific backend? That would help quite a bit. I suggest trying the file and mail backends first. They're the most likely culprits. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Memory usage
Check out the discussion (currently going on) at http://mail.gnome.org/archives/dashboard-hackers/2005-September/msg00069.html ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Memory usage
Hi, On Mon, 2005-09-19 at 15:33 -0400, Gregory Pierce wrote: Beagle has been working great. I am very pleased with it overall; however, I am concerned about its memory usage. The gnome system monitor indicates that it is up to 1.1 GB of memory and that swap space (~1GB) is at 100% usage. Is this normal? I do have a great many files but can I expect the memory usage to decline after it finishes indexing all the files? Is this resident size (RSS) or virtual size (Vsize)? The latter is effectively a meaningless number. But if RSS size is anywhere near 1 gig or even, say, 100 megs, that's too big and there is probably a bug in there. If the main beagle process is that big (beagled), then it probably won't ever go down unless you restart the daemon. Sizes that big are a bug. If it's in the helper process (beagled-index-helper), then it should shut itself down and restart when memory usage gets high. This can happen with a particularly large, misbehaving file (often HTML or MS Word). Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle 0.1.0 crash
Hi, On Mon, 2005-09-19 at 22:14 +0100, Darren Davison wrote: Hi guys, congrats on the 0.1.0 release! Thanks! Now, I'm not yet running a 2.6.13 kernel, so have the wrong inotify version, but I'm not sure that's at fault here. My understanding is that inotify won't affect the indexer much when it's starting from scratch. I could be very wrong there, hence my post here and not bugzilla. I don't think that'd affect this. Unhandled Exception: System.ArgumentNullException: null key Parameter name: key in [0x000c7] System.Collections.Hashtable:Find (System.Object key) in [0x2] (at /tmp/portage/mono-1.1.9/work/mono-1.1.9/mcs/class/corlib/System.Collections/Hashtable.cs:449) System.Collections.Hashtable:Contains (System.Object key) in 0x003c9 Beagle.Util.Scheduler:Worker () in (wrapper delegate-invoke) System.MulticastDelegate:invoke_void () INFO: Kopete log backend worker thread done in 1.23s INFO: Gaim log backend worker thread done in 1.91s Someone else was seeing this too. The most helpful thing would be if you could track this down to a specific backend, by using --allow-backend as an argument to beagled. I'd suggest blowing away ~/.beagle each time so that you recreate a controlled environment each time. Thanks, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Memory usage
Hi, On Mon, 2005-09-19 at 17:55 -0400, Gregory Pierce wrote: I am not sure what to look for. I just typed beagled --debug-memory --allow-backend and before this I deleted the .beagle directory. Will this restart the indexing process again? And how can I tell whether the problem is in Files and Mail? Sorry, I should have been more clear. If you run beagled --list-backends, it will show you a list of backends that beagled starts up by default. By providing --allow-backend backend name you can limit it to one or more backends. So beagled --allow-backend Files would allow only the file backend. What I suggest doing is starting with just the files backend and then just the mail backend and see if you see the kind of memory spike you have been seeing. If it's in one and not the other (or in both or neither) then that helps narrow down the problem. Thanks, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Memory usage
On Mon, 2005-09-19 at 18:02 -0400, Joe Shaw wrote: Hi, On Mon, 2005-09-19 at 17:55 -0400, Gregory Pierce wrote: I am not sure what to look for. I just typed beagled --debug-memory --allow-backend and before this I deleted the .beagle directory. Will this restart the indexing process again? And how can I tell whether the problem is in Files and Mail? Sorry, I should have been more clear. If you run beagled --list-backends, it will show you a list of backends that beagled starts up by default. By providing --allow-backend backend name you can limit it to one or more backends. So beagled --allow-backend Files would allow only the file backend. What I suggest doing is starting with just the files backend and then just the mail backend and see if you see the kind of memory spike you have been seeing. If it's in one and not the other (or in both or neither) then that helps narrow down the problem. Thanks, Joe Thanks, that was clearer. I am now running beagled --allow-backend Files. We'll see if Files are the problem. Greg ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Memory usage
What I suggest doing is starting with just the files backend and then just the mail backend and see if you see the kind of memory spike you have been seeing. If it's in one and not the other (or in both or neither) then that helps narrow down the problem. Thanks, Joe Well, I did as you suggested (beagled --allow-backend Files) and mono-beagled is up to 601 MB of resident memory! So, as you suspected, it is the Files backend that is at least partly responsible. Is there any other information that I could provide that may help in tracking down what is causing this runaway memory usage? Greg ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
