Re: I have resurrected beagle
Hi, Wow! This is awesome. I am very happy Beagle is still solving problems for people -- your beagrep looks very interesting -- and thank you for updating the code to use MimeKit over GMime. It is definitely a big improvement. Keep up the great work! Joe On Tue, Dec 30, 2014 at 9:29 PM, Haojun Bao wrote: > Hi, > > I am using it to index my maildir mails. The gmime-sharp 2.6 is so broken, > it is completely unusable, so I have changed to use MimeKit. > > Please take a look at my fork at https://github.com/baohaojun/beagle if > you are still in love with this project. > > Also, I have wrote a beagrep which can grep 9G of android source code in > 0.3 second. I use a modified version of beagle first to find out which > files contained the words I want to search, then I grep only these files. > Please take a look at this project at https://github.com/baohaojun/beagrep > . I wrote an blog article about beagrep at > http://baohaojun.github.io/beagrep.html . > > ___ > dashboard-hackers mailing list > dashboard-hackers@gnome.org > https://mail.gnome.org/mailman/listinfo/dashboard-hackers > > ___ dashboard-hackers mailing list dashboard-hackers@gnome.org https://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Is beagle completely dead now?
Haha, that one was not me. :) Joe On Mon, Feb 14, 2011 at 5:10 PM, Joel Mandell wrote: > Allright, thanks for the pointer! > > I really like this codesnippet in Evolution.cs: > > " > > foreach (string shit in crap) > > folder_path = folder_path.Replace (shit, ""); > > > " > > :=) > > On Mon, Feb 14, 2011 at 10:50 PM, Joe Shaw wrote: >> >> Hi, >> >> On Mon, Feb 14, 2011 at 4:45 PM, Joel Mandell >> wrote: >> > Would love to eventually fix the Evolution filter in Util/Evolution.cs. >> > I >> > can send patches to dbera right? >> >> You can send patches to the list. I think I wrote the Evolution stuff >> back in the day, so I might be able to remember some of the details. >> :) >> >> At this point there's nobody really maintaining it, and the current >> Evo stuff is old and broken so as long as you've tested it and feel >> good about the code, go ahead and push it AFAIAC. >> >> Thanks, >> Joe >> >> > >> > peace! >> > -joel m aka dikatlon >> > >> > On Mon, Feb 14, 2011 at 3:25 PM, Lukas Lipka >> > wrote: >> >> >> >> I don't mean to sound nostalgic, but back then Beagle was one of the >> >> best and fun projects to hack on! >> >> >> >> L. >> >> >> >> On Mon, Feb 14, 2011 at 2:26 PM, Joe Shaw wrote: >> >> > Hi Adam, >> >> > >> >> > On Mon, Feb 14, 2011 at 6:36 AM, Adam Tauno Williams >> >> > wrote: >> >> >>> A major reason why I gave up on Beagle and >> >> >>> the whole Linux desktop itself was due to this attitude. I guess >> >> >>> the >> >> >>> developers of those apps are more thick skinned or resilient than I >> >> >>> was? I don't know. >> >> >> >> >> >> Time is also probably a factor, Beagle was AFAIK really the first >> >> >> desktop Mono application of any note. It was also ahead of its time >> >> >> as >> >> >> a concept [I recall no shortage of long rambling posts about how it >> >> >> was >> >> >> useless anyway]. >> >> > >> >> > Indeed. Writing a Mono application at the time was a... challenge. >> >> > Beagle surely had its own set of performance problems, and the tools >> >> > to profile and debug them were largely non-existent. We even wrote a >> >> > few of them (heap-buddy, which has only recently been superseded by a >> >> > new built-in profiler). I would have killed for a working debugger. >> >> > :) >> >> > >> >> > When Beagle was started, the concept was actually pretty clear to us. >> >> > We weren't looking to create a Spotlight for Linux (indeed, Beagle >> >> > was >> >> > first publicly demoed on the day Apple announced Spotlight) -- it was >> >> > really designed as a means to an end: Dashboard needed an index to >> >> > make intelligent queries against and get contextual clues. Beagle >> >> > really grew out of that need, and became a user-centric tool. >> >> > >> >> > Joe >> >> > ___ >> >> > dashboard-hackers mailing list >> >> > dashboard-hackers@gnome.org >> >> > http://mail.gnome.org/mailman/listinfo/dashboard-hackers >> >> > >> >> ___ >> >> dashboard-hackers mailing list >> >> dashboard-hackers@gnome.org >> >> http://mail.gnome.org/mailman/listinfo/dashboard-hackers >> > >> > >> > >> > -- >> > Web: >> > http://www.openzource.org >> > >> > Cellphone: >> > 0722-137374 >> > >> > >> > ___ >> > dashboard-hackers mailing list >> > dashboard-hackers@gnome.org >> > http://mail.gnome.org/mailman/listinfo/dashboard-hackers >> > >> > > > > > -- > Web: > http://www.openzource.org > > Cellphone: > 0722-137374 > > > ___ > dashboard-hackers mailing list > dashboard-hackers@gnome.org > http://mail.gnome.org/mailman/listinfo/dashboard-hackers > > ___ dashboard-hackers mailing list dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Is beagle completely dead now?
Hi, On Mon, Feb 14, 2011 at 4:45 PM, Joel Mandell wrote: > Would love to eventually fix the Evolution filter in Util/Evolution.cs. I > can send patches to dbera right? You can send patches to the list. I think I wrote the Evolution stuff back in the day, so I might be able to remember some of the details. :) At this point there's nobody really maintaining it, and the current Evo stuff is old and broken so as long as you've tested it and feel good about the code, go ahead and push it AFAIAC. Thanks, Joe > > peace! > -joel m aka dikatlon > > On Mon, Feb 14, 2011 at 3:25 PM, Lukas Lipka wrote: >> >> I don't mean to sound nostalgic, but back then Beagle was one of the >> best and fun projects to hack on! >> >> L. >> >> On Mon, Feb 14, 2011 at 2:26 PM, Joe Shaw wrote: >> > Hi Adam, >> > >> > On Mon, Feb 14, 2011 at 6:36 AM, Adam Tauno Williams >> > wrote: >> >>> A major reason why I gave up on Beagle and >> >>> the whole Linux desktop itself was due to this attitude. I guess the >> >>> developers of those apps are more thick skinned or resilient than I >> >>> was? I don't know. >> >> >> >> Time is also probably a factor, Beagle was AFAIK really the first >> >> desktop Mono application of any note. It was also ahead of its time as >> >> a concept [I recall no shortage of long rambling posts about how it was >> >> useless anyway]. >> > >> > Indeed. Writing a Mono application at the time was a... challenge. >> > Beagle surely had its own set of performance problems, and the tools >> > to profile and debug them were largely non-existent. We even wrote a >> > few of them (heap-buddy, which has only recently been superseded by a >> > new built-in profiler). I would have killed for a working debugger. >> > :) >> > >> > When Beagle was started, the concept was actually pretty clear to us. >> > We weren't looking to create a Spotlight for Linux (indeed, Beagle was >> > first publicly demoed on the day Apple announced Spotlight) -- it was >> > really designed as a means to an end: Dashboard needed an index to >> > make intelligent queries against and get contextual clues. Beagle >> > really grew out of that need, and became a user-centric tool. >> > >> > Joe >> > ___ >> > dashboard-hackers mailing list >> > dashboard-hackers@gnome.org >> > http://mail.gnome.org/mailman/listinfo/dashboard-hackers >> > >> ___ >> dashboard-hackers mailing list >> dashboard-hackers@gnome.org >> http://mail.gnome.org/mailman/listinfo/dashboard-hackers > > > > -- > Web: > http://www.openzource.org > > Cellphone: > 0722-137374 > > > ___ > dashboard-hackers mailing list > dashboard-hackers@gnome.org > http://mail.gnome.org/mailman/listinfo/dashboard-hackers > > ___ dashboard-hackers mailing list dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Is beagle completely dead now?
Hi Adam, On Mon, Feb 14, 2011 at 6:36 AM, Adam Tauno Williams wrote: >> A major reason why I gave up on Beagle and >> the whole Linux desktop itself was due to this attitude. I guess the >> developers of those apps are more thick skinned or resilient than I >> was? I don't know. > > Time is also probably a factor, Beagle was AFAIK really the first > desktop Mono application of any note. It was also ahead of its time as > a concept [I recall no shortage of long rambling posts about how it was > useless anyway]. Indeed. Writing a Mono application at the time was a... challenge. Beagle surely had its own set of performance problems, and the tools to profile and debug them were largely non-existent. We even wrote a few of them (heap-buddy, which has only recently been superseded by a new built-in profiler). I would have killed for a working debugger. :) When Beagle was started, the concept was actually pretty clear to us. We weren't looking to create a Spotlight for Linux (indeed, Beagle was first publicly demoed on the day Apple announced Spotlight) -- it was really designed as a means to an end: Dashboard needed an index to make intelligent queries against and get contextual clues. Beagle really grew out of that need, and became a user-centric tool. Joe ___ dashboard-hackers mailing list dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Is beagle completely dead now?
Hi, On Sun, Feb 13, 2011 at 4:44 PM, guido iodice wrote: > It is very sad that mono folks are so committed with applications like > f-spot or banshee, that have many good alternatives, or on mono-mac, > mono-iOS, mono-android mono-whatyouwant, while Beagle died. I can't really blame them. The GNOME landscape is (or at least, was) *so hostile* toward Mono apps that I am surprised that Banshee or F-Spot survived, frankly. A major reason why I gave up on Beagle and the whole Linux desktop itself was due to this attitude. I guess the developers of those apps are more thick skinned or resilient than I was? I don't know. On the other hand, Mono on iOS, Android, etc. have all been warmly welcomed by the communities that adopt them -- and they have much larger user bases to boot. Why wouldn't they target them? > In the past I appreciate beaglefs that could be the first step of a > next generation database-based filesystem. Now I use mono to cut my > photos. Thanks. It was a lot of fun to work on over the years. Joe ___ dashboard-hackers mailing list dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Use beagle to read (eh, grep) source code
Hi, Nice work! If you're comfortable hacking around in the Beagle codebase, you could probably make the tokenizer (called an analyzer in Lucene parlance) act more appropriately for code so that things like underscores aren't stripped out. Take a look at beagled/LuceneCommon.cs in the BeagleAnalyzer class for more info. Joe On Wed, May 19, 2010 at 11:12 AM, Haojun Bao wrote: > Hi, all > > Beagle put grep on steroid for me:-) Thanks a lot y'all beagle hackers! > > The idea is simple and practical, beagle-static-qeury first, then use > grep on the results. > > For e.g., to grep "ENGLISH_STOP_WORDS" in the beagle source code, I will > use beagle-static-query: > > beagle-static-query\ > --add-static-backend /src/beagle/.beagle\ > --backend none\ > --max-hits 10\ > 'ENGLISH STOP WORDS' > > (note how I figured out the `_' character should be removed when > beagling:-) > > Then I will only grep the original regexp target in the following files, > because beagle already decided only these files contain all the 3 words > of 'ENGLISH STOP WORDS': > > /src/beagle/beagled/ExtractContent.cs > /src/beagle/beagled/LuceneCommon.cs > /src/beagle/beagled/Lucene.Net/Analysis/Standard/StandardAnalyzer.cs > /src/beagle/beagled/Lucene.Net/Analysis/StopAnalyzer.cs > > /src/beagle/beagled/Snowball.Net/Lucene.Net/Analysis/Snowball/SnowballAnalyzer.cs > /src/beagle/NEWS > > This way, even with the ~2 gigabytes Andoid source code, you can usually grep > and get the results in a few seconds. Best of all, it works not only > with source code, but with any text files. > > If you are intested, the source code is at > > git://github.com/baohaojun/windows-config.git > > And there's a detailed README at > http://github.com/baohaojun/windows-config/raw/master/gcode/beagle/beagle-grep-readme.org > > > > > > > > ___ > dashboard-hackers mailing list > dashboard-hackers@gnome.org > http://mail.gnome.org/mailman/listinfo/dashboard-hackers > ___ dashboard-hackers mailing list dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle XML/Unix API
Hi, On Mon, Feb 22, 2010 at 8:57 AM, wrote: > Is there any documentation about the Beagle XML/Unix API, please ? I think > it's used by the web user interface. There's no documentation for it, sadly. Generally apps should use either the C# classes or one of the C-based language bindings. If you're looking to implement a client library yourself, building it on top of the C one is the easiest as it is an easily-bindable GObject-based API. If you can't go one of these routes for whatever reason, your best bet is to look at the C code itself. It's fairly simply broken into request and response classes, and I think the XML generation in it is pretty clean: http://git.gnome.org/browse/beagle/tree/libbeagle/beagle The socat tool might also be helpful to you, as you can wedge yourself in as a man-in-the-middle and watch the traffic going by. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle excepts [Was: anybody working/supporting beagle?]
Hi Gabriel, Sounds good to me. Joe On Mon, Jan 25, 2010 at 2:07 PM, Gabriel Burt wrote: > It exists in gac/Mono.Data.Sqlite/1.xxx/ but not the 2.xxx version. > I'm told we can explicitly link against the old one by specifying > -r:path/to/gac/Mono.Data.Sqlite/1.xxx/ I'm also told that the > 2.xxx version has been shipped since Mono 1.2.4 (which we already > require), so we should probably just upgrade to it. I have a patch to > do so that I'll push if nobody objects. > > Gabriel > > On Mon, Jan 25, 2010 at 6:28 AM, Joe Shaw wrote: >> Hi, >> >> Yep, that class does not exist in Mono.Data.Sqlite. >> >> A fix might be to change the "using" line to be >> Mono.Data.SqliteClient, as I believe that is the older code. >> Otherwise some sort will be needed to port to the newer >> Mono.Data.Sqlite API. >> >> Joe >> >> On Mon, Jan 25, 2010 at 8:53 AM, Adam Tauno Williams >> wrote: >>> On Mon, 2010-01-25 at 08:49 -0500, Adam Tauno Williams wrote: >>>> On Fri, 2010-01-22 at 10:59 -0500, Joe Shaw wrote: >>>> > Hi, >>>> > On Fri, Jan 22, 2010 at 6:19 AM, Adam Tauno Williams >>>> > wrote: >>>> > > BTW, I love Beagle - it is a massive improvement over 'managing files'. >>>> > > But, at least for me, it has stopped working. Since I last updated >>>> > > Mono >>>> > > it just churns out a lot of - >>>> > > --- >>>> > > 20100122 06:15:13.6246 10662 Beagle ERROR EX: at >>>> > > System.Reflection.MonoCMethod.Invoke (System.Object obj, BindingFlags >>>> > > invokeAttr, System.Reflection.Binder binder, System.Object[] >>>> > > parameters, >>>> > > System.Globalization.CultureInfo culture) [0x0] in >>> > > unknown>: >>>> > This doesn't look like the complete stack trace. Or, at least, there >>>> > isn't the exception message included. Can you provide that? >>>> > Hopefully it's something simple. >>>> Cleaned out .beagle/Logs, restarted Beagle, and attached logs. >>> >>> I'd guess the crux of the issue is all of the: >>> >>> 20100125 08:51:14.2123 08249 Beagle ERROR EX: >>> System.Reflection.TargetInvocationException: Exception has been thrown >>> by the target of an invocation. ---> System.TypeLoadException: Could not >>> load type 'Mono.Data.Sqlite.SqliteBusyException' from assembly >>> 'Mono.Data.Sqlite, Version=2.0.0.0, Culture=neutral, >>> PublicKeyToken=0738eb9f132ed756' >>> >>> >>> It appears to be having an SQLite issue. I've disabled all the backends >>> now except files. >>> >>> mono-data-sqlite-2.6.1-31.1.i586 is the package installed. >>> >>> awill...@linux-m3mt:~/.beagle/Log> rpm -ql >>> mono-data-sqlite-2.6.1-31.1.i586 >>> /usr/lib/mono/1.0/Mono.Data.Sqlite.dll >>> /usr/lib/mono/1.0/Mono.Data.SqliteClient.dll >>> /usr/lib/mono/2.0/Mono.Data.Sqlite.dll >>> /usr/lib/mono/2.0/Mono.Data.SqliteClient.dll >>> /usr/lib/mono/gac/Mono.Data.Sqlite >>> /usr/lib/mono/gac/Mono.Data.Sqlite/1.0.5000.0__0738eb9f132ed756 >>> /usr/lib/mono/gac/Mono.Data.Sqlite/1.0.5000.0__0738eb9f132ed756/Mono.Data.Sqlite.dll >>> /usr/lib/mono/gac/Mono.Data.Sqlite/1.0.5000.0__0738eb9f132ed756/Mono.Data.Sqlite.dll.mdb >>> /usr/lib/mono/gac/Mono.Data.Sqlite/2.0.0.0__0738eb9f132ed756 >>> /usr/lib/mono/gac/Mono.Data.Sqlite/2.0.0.0__0738eb9f132ed756/Mono.Data.Sqlite.dll >>> /usr/lib/mono/gac/Mono.Data.Sqlite/2.0.0.0__0738eb9f132ed756/Mono.Data.Sqlite.dll.mdb >>> /usr/lib/mono/gac/Mono.Data.SqliteClient >>> /usr/lib/mono/gac/Mono.Data.SqliteClient/1.0.5000.0__0738eb9f132ed756 >>> /usr/lib/mono/gac/Mono.Data.SqliteClient/1.0.5000.0__0738eb9f132ed756/Mono.Data.SqliteClient.dll >>> /usr/lib/mono/gac/Mono.Data.SqliteClient/1.0.5000.0__0738eb9f132ed756/Mono.Data.SqliteClient.dll.mdb >>> /usr/lib/mono/gac/Mono.Data.SqliteClient/2.0.0.0__0738eb9f132ed756 >>> /usr/lib/mono/gac/Mono.Data.SqliteClient/2.0.0.0__0738eb9f132ed756/Mono.Data.SqliteClient.dll >>> /usr/lib/mono/gac/Mono.Data.SqliteClient/2.0.0.0__0738eb9f132ed756/Mono.Data.SqliteClient.dll.mdb >>> >>> -- >>> openSUSE <http://www.opensuse.org/en/> >>> Linux for human beings who need to get things done. >>> >>> ___ >>> Dashboard-hackers mailing list >>> Dashboard-hackers@gnome.org >>> http://mail.gnome.org/mailman/listinfo/dashboard-hackers >>> >>> >> ___ >> Dashboard-hackers mailing list >> Dashboard-hackers@gnome.org >> http://mail.gnome.org/mailman/listinfo/dashboard-hackers >> > ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle excepts [Was: anybody working/supporting beagle?]
Hi, Yep, that class does not exist in Mono.Data.Sqlite. A fix might be to change the "using" line to be Mono.Data.SqliteClient, as I believe that is the older code. Otherwise some sort will be needed to port to the newer Mono.Data.Sqlite API. Joe On Mon, Jan 25, 2010 at 8:53 AM, Adam Tauno Williams wrote: > On Mon, 2010-01-25 at 08:49 -0500, Adam Tauno Williams wrote: >> On Fri, 2010-01-22 at 10:59 -0500, Joe Shaw wrote: >> > Hi, >> > On Fri, Jan 22, 2010 at 6:19 AM, Adam Tauno Williams >> > wrote: >> > > BTW, I love Beagle - it is a massive improvement over 'managing files'. >> > > But, at least for me, it has stopped working. Since I last updated Mono >> > > it just churns out a lot of - >> > > --- >> > > 20100122 06:15:13.6246 10662 Beagle ERROR EX: at >> > > System.Reflection.MonoCMethod.Invoke (System.Object obj, BindingFlags >> > > invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, >> > > System.Globalization.CultureInfo culture) [0x0] in > > > unknown>: >> > This doesn't look like the complete stack trace. Or, at least, there >> > isn't the exception message included. Can you provide that? >> > Hopefully it's something simple. >> Cleaned out .beagle/Logs, restarted Beagle, and attached logs. > > I'd guess the crux of the issue is all of the: > > 20100125 08:51:14.2123 08249 Beagle ERROR EX: > System.Reflection.TargetInvocationException: Exception has been thrown > by the target of an invocation. ---> System.TypeLoadException: Could not > load type 'Mono.Data.Sqlite.SqliteBusyException' from assembly > 'Mono.Data.Sqlite, Version=2.0.0.0, Culture=neutral, > PublicKeyToken=0738eb9f132ed756' > > > It appears to be having an SQLite issue. I've disabled all the backends > now except files. > > mono-data-sqlite-2.6.1-31.1.i586 is the package installed. > > awill...@linux-m3mt:~/.beagle/Log> rpm -ql > mono-data-sqlite-2.6.1-31.1.i586 > /usr/lib/mono/1.0/Mono.Data.Sqlite.dll > /usr/lib/mono/1.0/Mono.Data.SqliteClient.dll > /usr/lib/mono/2.0/Mono.Data.Sqlite.dll > /usr/lib/mono/2.0/Mono.Data.SqliteClient.dll > /usr/lib/mono/gac/Mono.Data.Sqlite > /usr/lib/mono/gac/Mono.Data.Sqlite/1.0.5000.0__0738eb9f132ed756 > /usr/lib/mono/gac/Mono.Data.Sqlite/1.0.5000.0__0738eb9f132ed756/Mono.Data.Sqlite.dll > /usr/lib/mono/gac/Mono.Data.Sqlite/1.0.5000.0__0738eb9f132ed756/Mono.Data.Sqlite.dll.mdb > /usr/lib/mono/gac/Mono.Data.Sqlite/2.0.0.0__0738eb9f132ed756 > /usr/lib/mono/gac/Mono.Data.Sqlite/2.0.0.0__0738eb9f132ed756/Mono.Data.Sqlite.dll > /usr/lib/mono/gac/Mono.Data.Sqlite/2.0.0.0__0738eb9f132ed756/Mono.Data.Sqlite.dll.mdb > /usr/lib/mono/gac/Mono.Data.SqliteClient > /usr/lib/mono/gac/Mono.Data.SqliteClient/1.0.5000.0__0738eb9f132ed756 > /usr/lib/mono/gac/Mono.Data.SqliteClient/1.0.5000.0__0738eb9f132ed756/Mono.Data.SqliteClient.dll > /usr/lib/mono/gac/Mono.Data.SqliteClient/1.0.5000.0__0738eb9f132ed756/Mono.Data.SqliteClient.dll.mdb > /usr/lib/mono/gac/Mono.Data.SqliteClient/2.0.0.0__0738eb9f132ed756 > /usr/lib/mono/gac/Mono.Data.SqliteClient/2.0.0.0__0738eb9f132ed756/Mono.Data.SqliteClient.dll > /usr/lib/mono/gac/Mono.Data.SqliteClient/2.0.0.0__0738eb9f132ed756/Mono.Data.SqliteClient.dll.mdb > > -- > openSUSE <http://www.opensuse.org/en/> > Linux for human beings who need to get things done. > > ___ > Dashboard-hackers mailing list > Dashboard-hackers@gnome.org > http://mail.gnome.org/mailman/listinfo/dashboard-hackers > > ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Help regarding integrating Beagle with the Nautilus File Manager
Hi, On Sun, Jan 24, 2010 at 4:39 PM, vatsal nidhi wrote: > I am working on a project which requires the integration of the Nautilus > File Manager with Beagle ,that is , i want to provide the functionality of > Beagle in the file manager . This has already been implemented in Nautilus: http://git.gnome.org/browse/nautilus/tree/libnautilus-private/nautilus-search-engine-beagle.c However, Nautilus must be built with Beagle support to use it. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: anybody working/supporting beagle?
Hi, On Fri, Jan 22, 2010 at 12:32 PM, Amish Shah wrote: > We will probably be writing a back-end to talk to postgres. Seems like the > current backend for html files doesn't analysis all the content of the html > files. Also, we want to add some more file attributes to the index > repository during indexing. Ok. There are two concepts at play here: a "backend", which is basically a data source. The file system is an example of a backend, and it seems like postgres would also be a backend. There is also a "filter", which is code that extracts data and metadata from an item from a backend. So there is an HTML filter which pulls the data out of it. Both backends and filters can add attributes to the document that is written to the index. I'm not sure what you mean when you say that HTML files don't analyze all the content. There is a limit of the number of tokens indexed. I forget what the number is, but it's something like 2,000 or 10,000 or 50,000. Something like that. If you think you're going to need more indexed than that, you'll want to bump that limit. > Eventually, we'd like to implement autosuggest. That'd be great! > Regarding Novell, is there a commercial license for Beagle present? Is it > required or it something one can buy for support :) Hopefully someone from Novell on the list can chime in on this. :) Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: anybody working/supporting beagle?
Hi, On Fri, Jan 22, 2010 at 6:19 AM, Adam Tauno Williams wrote: > BTW, I love Beagle - it is a massive improvement over 'managing files'. > But, at least for me, it has stopped working. Since I last updated Mono > it just churns out a lot of - > --- > 20100122 06:15:13.6246 10662 Beagle ERROR EX: at > System.Reflection.MonoCMethod.Invoke (System.Object obj, BindingFlags > invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, > System.Globalization.CultureInfo culture) [0x0] in unknown>:0 This doesn't look like the complete stack trace. Or, at least, there isn't the exception message included. Can you provide that? Hopefully it's something simple. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: anybody working/supporting beagle?
Hi, On Thu, Jan 21, 2010 at 7:13 PM, Amish Shah wrote: > If I have questions about how to do things, above and beyond the pretty good > documentation, am I likely to get answers on this forum? How responsive has > Novell been about bug fixes? I will certainly try to answer any questions you have, but if things require development, you're on your own for that. (Although, again, I can provide some guidance.) Novell has not been very responsive to bug fixes, although I am sure that is different if you are a customer of theirs. :) > Or am I better of with tracker which seems like its actively supported, even > though probably a bit less mature than beagle? I would say it depends on what you want to do, and how much time and effort you are willing to invest in development, if any. At this point, Tracker has a lot more development going on it. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: anybody working/supporting beagle?
Hi, Beagle isn't in active development. It is getting some occasional maintenance done by Novell. Joe On Thu, Jan 21, 2010 at 5:23 PM, Amish Shah wrote: > Hi, > > How active is this project? I'm doing a comparison between beagle and > tracker trying to decide which meta search engine to use. Beagle seems > great, but kind of inactive especially lately. > > thanks > Amish > > ___ > Dashboard-hackers mailing list > Dashboard-hackers@gnome.org > http://mail.gnome.org/mailman/listinfo/dashboard-hackers > > ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: About to declare 'unmaintained'
Hi, On Tue, Sep 22, 2009 at 11:39 AM, guido iodice wrote: > Can Novell leave unmantained a piece of its desktop? This is a suicide for > Suse. Exactly what Novell intends to do with Beagle is unclear, but I think "suicide" is a bit hyperbolic. From my time at Novell there was a love/hate relationship with Beagle among SUSE users. Some people loved it and wouldn't live without it, and other people despised it to the point where the very first thing they did after installing SUSE was remove Beagle. I think a lot of people would be pretty disappointed if they stopped shipping Beagle, but would they lose a lot of business? I'm not so sure. I don't have any inside knowledge anymore, but it is conceivable that SUSE would ship Tracker in a future release and replace a chunk of Beagle's functionality with that. > Well, I think there are some stupids men in Novell (did someone say > "Miguel"?), but I hope not so stupid. > Monotouch can't be Novell core business Well, Miguel is a friend of mine, so I don't really appreciate that. Novell's a big company, and Mono is a totally separate entity from their desktop business. If Novell wants to put resources toward Beagle, it's going to come from the SUSE organization and not Miguel's Mono organization. I agree with you, though, in that I am not really sure what Novell's aims are with Monotouch, but that's orthogonal to this. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: About to declare 'unmaintained'
Hi, I just wanted to reiterate what Bera said. Unfortunately it's been a long time coming, and I had the revelation fairly recently that our meager maintenance efforts were simply not going to keep up with the positive forward progress of other projects upon which we depend, like Evolution. Adam asked about bounties. There doesn't appear to be any official bounties system, although some are posted informally here: http://live.gnome.org/Bounties But there doesn't appear to be a comprehensive system similar to Elance. A couple people mentioned whether Novell or Canonical (Ubuntu) would fund Beagle development. I used to work at Novell, and I had the great fortune of working on Beagle pretty much from the start. For a couple of years there they paid two full time developers to work on the project: Jon Trowbridge and myself. When Jon left the company, it was just me -- although there was occasional part-time help, like Dan Winship's excellent work on the search UI. Since I left Novell nearly two years ago, there has been none. I think it's safe to say that Novell no longer has any dedication to the project. I don't mean that as a dig -- having worked on Ximian and SUSE distributions you have to make strategic and tactical decisions where to put your resources, since you can't hack full time on everything. It appears clear that desktop search hasn't panned out as they thought and that experimental projects like Dashboard, Association Browser, etc. aren't feasible. As for Canonical and Ubuntu, a number of releases ago that community decided to go with Tracker instead of Beagle, I believe in part due to a major backlash against Mono following the Microsoft/Novell patent agreement. Although I think Beagle is still for the moment ahead of Tracker in terms of core user functionality, Tracker has a vibrant development community backed by open source companies whereas Beagle's is completely stagnant and bordering on nonexistent. If I were an impartial party trying to decide in which to invest development resources, Beagle is simply a tougher case to make. Having said all that, please don't let me stop anyone from making those overtures. Nothing would make me happier than to see the old dog get a new lease on life. :) As Roger said, maybe it can do so with a reimagined focus. Like I said, I've gotten to hack on Beagle from the beginning, which has been five years (!) now. I am very proud of the work I've done personally, and that we've done as a community. I have my regrets too, both technical and non-technical. But I've also moved on with my life and I hack on other stuff now, and I don't personally have much interest in returning to full-time Beagle development. I take great comfort in the fact that we created the first user-centric search system on Linux, and that it is open source software. Thanks, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: [beagle] Remove duplicate 'beagle-part-property.h' from include_HEADERS
Hi, On Sun, Jul 26, 2009 at 1:46 PM, Arun Raghavan wrote: > commit 95b10dd1dc58068fa09d38fa6bac3763afc9c8f9 > Author: Arun Raghavan > Date: Sun Jul 26 23:10:44 2009 +0530 > > Remove duplicate 'beagle-part-property.h' from include_HEADERS > > This was causing breakage with recent versions of 'install'. Thanks to > Matthias Clasen (mcla...@redhat.com) for the patch on bug 588994. For the future, you can mark the author of the patch by using the --author command-line flag to "git commit". This will make it more clear that you're the committer and not the author in git history. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: (resend)patch to beaglefs
Hi Guido, On Mon, Jul 6, 2009 at 4:44 AM, guido iodice wrote: > Dears, I'm a Beagle fan: > http://guiodic.wordpress.com/category/gnulinux/guide-pratiche-gnulinux/beagle-guide-pratiche-gnulinux-gnulinux/ > > I discovered that *beaglefs* has a bug because new beagle syntax. > > Sorry, I'm unable to use 'diff' and 'patch' properly. > > Changement is simple, in hit.c: > > beagle_query_add_text (query, "type:File OR type:IMLog"); /* CHANGHE */ > /* beagle_query_add_text (query, "type:IMLog"); */ /* DELETE OR > COMMENT THIS LINE*/ Looks pretty straightforward. Would you mind filing a bug at http://bugzilla.gnome.org and attaching a patch? I'm not able to check it in at the moment, and that will ensure that it doesn't get lost. Thanks, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: ODT/ODP/ODS
Hi, On Wed, Jul 15, 2009 at 7:50 AM, Adam Tauno Williams wrote: > Does Beagle (0.3.8-46) index the content of OpenOffice files? [It sure > seems like I remember it indexing StarOffice files...] But double > checking on my system and it only seems to be indexing based on the > names of OpenOffice files. In fact, looking at my search results, it > doesn't appear to be indexing anything but XML and text (source mostly) > files. > > awill...@linux-m3mt:~> beagle-info --list-filters | grep -i open > FilterOpenOffice - Version 0 (/usr/lib/beagle/Filters/Filters.dll) > - MimeType: application/vnd.oasis.opendocument.text > - MimeType: application/vnd.oasis.opendocument.text-template > - MimeType: application/vnd.oasis.opendocument.spreadsheet > - MimeType: application/vnd.oasis.opendocument.spreadsheet-template > - MimeType: application/vnd.oasis.opendocument.presentation > - MimeType: application/vnd.oasis.opendocument.presentation-template > - MimeType: application/vnd.oasis.opendocument.graphics > - MimeType: application/vnd.oasis.opendocument.graphics-template It should be able to, yes. Try running beagle-extract-content on one of the ODF files to see how it is identified. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: (proposal) BeagleFS GUI
Hi, On Fri, Jul 10, 2009 at 5:49 AM, guido iodice wrote: > I wrote a simple GUI for BeagleFS (yes, I love it). > > You can find it on my blog: > http://guiodic.wordpress.com/2009/07/10/beaglefs-gui-uninterfaccia-grafica-per-beaglefs/ > (my blog is in Italian, but you can find links to source and binaries > easily). Nice! > Well, I would like to integrate it in Beagle-search and propose the > patch to you, but I'm not able to find a Monodevelop project. > Does it exist? There isn't a monodevelop project for Beagle. Beagle predated Monodevelop, and we've never taken the effort to switch it over. This would be a wonderful project for someone to take on. Right now, as I'm sure you've seen, beaglefs is packaged and distributed separately from Beagle itself. Since beaglefs itself is all C and your UI is in C#, you'll need to do some hacking to get it to build all together. I suggest cloning the git repository, making a local branch and doing the hacking there, and then submitting a patch when you're done. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Make Beagle accesible from everywhere
Hi, On Tue, Jun 2, 2009 at 8:25 AM, j r wrote: > I obtained a different answer depending on the URL call, and if I invoke the > server from 10.5.36.50:4000 I receive an invalid XML string. For example: > > FROM HTTP://10.5.36.50:4000 (Invalid xml string) > > 391 // This number seems to be a random one This looks like chunked transfer encoding to me. This is part of the HTTP 1.1 spec in which the server sends the number of bytes in a chunk, followed by the chunk itself. You might want to check the headers if you can (with something like "curl -v") to see if this is indeed the case. If the client says that it's HTTP 1.1 it should know how to deal with chunked transfer encoding. > 1 > � This, however, looks odd. I'm not sure what that straggler byte is, and that could be why the XML document is invalid, although it looks like you also are receiving it from localhost below. > AND FROM HTTP://localhost:4000 (Right XML string) > > > http://www.w3.org/2001/XMLSchema-instance"; > xmlns:xsd="http://www.w3.org/2001/XMLSchema";> > > 0.3.9 > > > > > > > IsIndexing="false" /> > IsIndexing="true" /> > ProgressPercent="-1" IsIndexing="false" /> > ProgressPercent="-1" IsIndexing="false" /> > > true > > � Why one request is receiving it that way and the other isn't, I have no idea. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
build system (was Re: gitignore foo)
Hi, On Thu, May 14, 2009 at 2:14 PM, Arun Raghavan wrote: > I don't really see any major disadvantage to either approach. A very > minor plus for the git.mk is that you don't need to update .gitignore > even those few rare times that you might otherwise have to (since > it'll just "introspect" it from the build system). Speaking of the build system, I've been working off and on into trying to better organize our source code, with the eventual hope that we'd be able to use MonoDevelop with the tree. I would also like to convert the backends and filters to use Mono.Addins and stop maintaining our own plugin system. The git branch can be found here: http://github.com/joeshaw/beagle/tree/build-cleanup It's only semi-working at the moment. It builds most of the code up through the daemon, index helper and backends, but it doesn't load any of the backends yet, doesn't load the index helper (because the wrapper scripts aren't ready), and generally doesn't work. :) Filters and clients aren't built at all yet. The organization of the tree might be too broad now, but I'm trying to feel my way around at the moment. If anyone wants to take a peek and give feedback (or pick up the next task!) I'm all for it. The build system is largely stolen from Banshee's. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: gitignore foo
Hi, On Thu, May 14, 2009 at 1:51 PM, Arun Raghavan wrote: > Hello folks, > I've committed a couple of changes to the autofoo files to allow > automagic generation of .gitignore files. There was some information > about this on the gnome-infrastructure mailing list a while ago. Looks good, although I never understood why checking .gitignore files into git was a bad idea. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle git migration issue
Hey, On Fri, Apr 24, 2009 at 12:21 AM, Arun Raghavan wrote: > The git repo should be up again. Please give it a spin and let me > know. I'll try to build everything once in a while, and diff the 2 > trees to make doubly sure. I am updating my Ubuntu to Jaunty so I can't test it at the moment, but thanks for noticing this and getting it taken care of. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Firefox indexer not working: files in ToIndex never removed
Hi, On Tue, Apr 21, 2009 at 3:24 AM, Frederik Himpe wrote: > Hi, I'm using Mandriva 2009.1 with Beagle 0.3.9 running on mono 2.2. I > have enabled the Beagle Indexer 1.1 add-on in Firefox 3.0.8, but it does > not seem to be working: I had a huge amount of unindexed web pages which > were never removed from ~/.beagle/ToIndex. > > The only error I could find in the logs is this backtrace: > > 20090420 22:18:48.7264 12883 Beagle ERROR EX: Caught exception while > instantiating IndexingService backend > 20090420 22:18:48.7264 12883 Beagle ERROR EX: > System.Reflection.TargetInvocationException: Exception has been thrown by the > target of an invocation. ---> System.UnauthorizedAccessException: Access to > the path > "/home/frederik/.beagle/Indexes/IndexingServiceIndex/Locks/lucene-87e2d2d7eab46d37e08cb23db1c76b0c-write.lock" > is denied. This definitely looks like the problem. Can you take a look at those files and directories and see if there are any ownership or modal problems that could prevent your user from being able to write or delete files from there? Something owned as root, perhaps? Worst case, you could delete the ~/.beagle/Indexes/IndexingServiceIndex directory entirely. That would fix the permissions problem, but you would lose any previously-indexed web pages. (And unlike other backends which could just recrawl the data, these will be permanently lost unless you navigate back to that web page.) Thanks, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: File category disappears
Hi, On Thu, Apr 16, 2009 at 8:14 AM, D Bera wrote: > This is quite strange! It appears as though the Files backend is not > running. And there is no notification message for it either! > > I don't know if gentoo disables debug building by default; if not, > then my best bet is to run "beagled --debug" and observe the log at > ~/.beagle/Log/current-Beagle Wasn't there code added fairly recently to completely disable backends if they encounter many errors? This might explain this, and hopefully would suggest clues in the log files. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle
Hi, On Tue, Mar 31, 2009 at 10:48 AM, veeraraghavan ravi wrote: > I tried to build the beagle assembly in windows via VS.NET. But failed. There are several Beagle assemblies. Util.dll is one, BeagleClient.dll is another, BeagleDaemon.exe, etc. As I've said, there is a lot of Linux specific code you have to fix or remove or replace, so it's probably better to take it a small piece at a time. > Actually, I need a component for extracting the metadata and the text > content from any type of files ( Filter in Beagle). > > Do you know any API's (C#.NET) which can be used in windows.. No, the Filter API is a Beagle-specific one; there aren't any other .Net assemblies which provide that API. There are a bunch of projects which extract textual content from structured files, but they all have their own APIs -- that is one of the things the Filter API was designed to abstract away. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle
Hi, On Tue, Mar 31, 2009 at 10:19 AM, veeraraghavan ravi wrote: > I have downloaded the latest source code (beagle-0.3.9) from the web site. > > Can I built the assembly(beagle.dll etc..) by opening the C# files in a new > project via VS.NET in windows. I've never tried -- I'm not a Windows user -- but you can certainly give it a shot. Start with the Util directory, it's first in the dependency chain. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle
Hi, On Tue, Mar 31, 2009 at 7:25 AM, veeraraghavan ravi wrote: > Can I run the Beagle components in windows environment. Not without some work, no. Today Beagle is pretty Linux-specific, and some work would need to be done to port it to Windows. I think it's probably very worthwhile work, however. > Where can I get the following components > 1. Beagle.Daemon In the source tree from SVN, trunk/beagle/beagled. > 2. Beagle.Util trunk/beagle/Util > Do I need Mono environment to run these components. Because of the Linuxisms, yes, but if it were ported it shouldn't. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: how to erase the origin data?
Hi, On Sat, Mar 21, 2009 at 8:43 AM, waterloo wrote: > now I have changed my directories structure and changed names to utf8. > but I find that beagle always remember original directories with beagled > --fg --debug. > It says : Delaying add of ... until FSQ comes across it . > but that directory doesnot exist. I believe that is the Nautilus metadata backend which prints that message. If you don't care about indexing Nautilus emblems, notes, etc. you can disable it on the beagled command-line with "--disable-backend NautilusMetadata". I am not 100% sure that is the name of it, so you should run "beagled --list-backends" to make sure I have the name right. But it'll be something like that. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Parsing an XML file in an archive
Hi, On Wed, Mar 18, 2009 at 5:04 PM, wrote: > I want to create a filter for Beagle! (Actually, several filters for > different formats that are built > similar….) I have red the documentation, but I cannot wrap my head around > it. I hope some- > one where will take the time to get me started! If you haven't already, definitely the first place to look at is the filter tutorial on the wiki: http://beagle-project.org/Filter_Tutorial That will hopefully give you an overview on the structure of the Filter code, how to register it with Beagle, and how to test it. > The format is simple: zipped files (with extensions other than .zip, but they > are still just zips) > Only one file of interest: meta.xml that contains strings that can be mapped > to dc values. Definitely take a look at the OpenOffice filter. OpenOffice files follow this exact model: a zip file (with a different extension) containing a bunch of XML files inside of it. The code is not the easiest to follow, but it's a decent starting point: http://svn.gnome.org/viewvc/beagle/trunk/beagle/Filters/FilterOpenOffice.cs?view=markup Look at the core overridden Filter methods for a start: DoOpen(), DoPullProperties(), DoPull(), and DoClose(). If I were writing this code today I might use XPath instead of walking every node in the document, and if I had C# 3.0 support I might even use Linq-to-XML on it. We're not officially supporting C# 3.0 yet, although since it's fully supported in Mono 2.2 and 2.4 is coming out soon, there's no reason why we couldn't make that jump. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle: Looking for maintainer
Hi, On Mon, Jan 26, 2009 at 6:13 PM, D Bera wrote: > Beagle project is looking for a maintainer. Yeah, I just want to reiterate that it'd be great if someone could step up. I also don't have the time to dedicate to Beagle what I would like, but I can definitely help mentor people if they want to get involved. > I know time is always a concern for open source developers ... if only > we could "#define DAY 1 hours"! But seriously ... there is not a > lot of "maintaining" to do. The current situation is - everything > mostly works as it should and whatever does not work, would not work > without serious design changes. So you can mark them "WONTFIX" :-) Of course, there are lots of things to hack on too. :) Philip Van Hoof wrote an EPlugin to make indexing of Evolution mail a lot saner than the current hack of poking around in ~/.evolution. He's blogged about his work here: http://pvanhoof.be/blog/index.php/2009/01/15/the-evolution-dbus-metadata-api and a wiki page here: http://live.gnome.org/Evolution/Metadata That's a great way to learn the internals of Beagle and really put your fingerprint on it. Your humble maintainer got his first start in the Beagle codebase by writing (and then re-writing, and re-writing again) the Evolution backends. The file system backend could use a rewrite since the current design is not optimal... I've written a bit about it in the past and can fill people in on details if they're interested. It'd be nice to get it building and running under MonoDevelop if people are interested in that. Making the code more modular so that it's portable is a worthy task. A lot of the KDE backends are becoming out-of-date with the new KDE 4 apps coming out. UI tweaks are always welcome. Lots and lots of things to do. Thanks, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: compiling beagle
Hi, On Thu, Nov 6, 2008 at 4:30 PM, Darren Govoni <[EMAIL PROTECTED]> wrote: > I pulled the beagle source code down and want to compile it. The > INSTALL refers to ./configure script which isn't there. Nor are there > csproj files I could use with Mono. What is the best way to rebuild > beagled or the other projects? I'm assuming you checked it out of SVN. If so, substitute ./autogen.sh for ./configure. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: search restrict to a single directory
Hi, On Wed, Oct 1, 2008 at 12:52 PM, Bernhard Kleine <[EMAIL PROTECTED]> wrote: > strange: "beagle-query inuri:FMRF --keywords snail" returns I think you just want: beagle-query inuri:FMRF snail That will return all documents with "snail" in it inside directories named FMRF. Both "inuri:FMRF" and "snail" are search terms. The "inuri" just instructs Beagle that that search term is to be treated specially. There are others like "startdate", "filetype", etc. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Nemo like display using Beagle?
Hi, On Tue, Sep 9, 2008 at 1:16 AM, Mikkel Kamstrup Erlandsen <[EMAIL PROTECTED]> wrote: > If you where adventurous you could grab a hold of the Beagle index and > crawl that directly. Traversing a Lucene index is quite fast... Would > be an act of great evil and I am sure the wrath of the Beagle lords > would come swift, but just a thought :-) Like any good data-siloed application, internal data structures subject to change blah blah. :) You can feel free to walk the index, but you probably won't get quite the data you want. We're storing only UIDs in the index -- not file paths or URIs -- and you'd have to resolve those against the sqlite database which contains the mapping. Not as quick as just iterating over the index, I'm afraid. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagled-helper maxing out CPU core
Hi, On Tue, Aug 12, 2008 at 4:25 PM, Sandy Armstrong <[EMAIL PROTECTED]> wrote: > The only open file I saw that looked suspicious was > ~/.gnome2/f-spot/photos.db. But all I get when I run beagle-extract-content > on that is this (which runs very quickly, of course): It's much more likely to have been this: > lr-x-- 1 sandy users 64 2008-08-12 13:19 17 -> /tmp/tmp53d12170.a2w What's the result of beagle-extract-content on that? Thanks, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagled-helper maxing out CPU core
Hi, On Tue, Aug 12, 2008 at 1:01 PM, Sandy Armstrong <[EMAIL PROTECTED]> wrote: > I notice that I constantly have a beagled-helper process taking up 99% of > the resources on one of my cores. I've been having some issues with my > system hanging and thought maybe this was responsible (haven't found > anything conclusive yet). Is this amount of CPU usage normal? Definitely not for that amount of time. > And saw this resultant output in ~/.beagle/Log/current-IndexHelper: > > 20080812 09:55:23.7245 04698 IndexH WARN: Handling signal 12 (SIGUSR2) > 20080812 09:55:23.7614 04698 IndexH WARN: Filtering status (2h54m13s ago): > no document is currently being filtered. So, I think the error message here is a little bit misleading. I think in this case it means not that no document is being filtered but that it's trying to determine the MIME type for the document, and it's that code (possibly inside xdgmime) that is spinning. You might be able to better track down the problematic file by looking at /proc/fd/4698 and look for files that are open that it's likely crawling. You could then run beagle-extract-content on those files and see if you can easily reproduce the CPU issue. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: patch to port beagle to gmime-2.4
Hey Jeff, On Sat, Jul 19, 2008 at 10:18 PM, Jeffrey Stedfast <[EMAIL PROTECTED]> wrote: > This is a preliminary patch, GMime-2.4 isn't yet released and the API > isn't written in stone yet; I've written the patch to get feedback from > you guys on the API changes I've made. The changes look fine to me, and pretty straightforward. I don't have any particular comments on them. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Google protocol buffers
Hey, On Fri, Jul 11, 2008 at 2:39 PM, Brad Taylor <[EMAIL PROTECTED]> wrote: > Now I haven't done that much research into Google's protocol buffers, > but from first blush, this seems similar in spirit to Thrift[1], which > is a lightweight, cross-platform binary serialization method, and works > with C# (among many other languages). Maybe this warrants a look as > well? Yeah, good point, it's definitely worth taking a look at this as well. Unfortunately it also doesn't support C# out of the box, though. Thrift also seems like it may be more focused on defining service definitions -- you can define things like method calls with it -- which is a little more like D-Bus. Beagle at this point only really cares about serializing data, so some of the power of Thrift is diminished. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Google protocol buffers
Hi, In case you didn't see it, Google released their "protocol buffers" as open source the other day: http://code.google.com/p/protobuf/ http://code.google.com/apis/protocolbuffers/docs/overview.html One of the big weaknesses in Beagle is that the messaging system between beagled and beagled-index-helper and between beagled and clients is a slow, bloated XML format. It seems like protobufs are an ideal replacement for this. Google's implementation outputs stubs for C++, Java, and Python -- no C# -- but there is plenty of demand for a C# generator and at least one person says he's working on it: http://groups.google.com/group/protobuf/browse_thread/thread/957a0aee416e8b44/2791c6776a8fcff8?lnk=raot#2791c6776a8fcff8 If you look at the Java generator: http://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/compiler/java/ you'll see that the files are almost entirely simple print statements. As Java and C# are very similar, it should be trivial to hack up a quick C# generator. There would be some additional work on the Beagle side to send and parse these (as the XmlSerializer in C# does a lot of the work for us now), but I think the win would be tremendous. It would be interesting to compare the speed differences between the protobuf and XML, and I'd also be curious to see how it stacks up against D-Bus. If it's a good fit, large chunks of libbeagle could be replaced to use the C++-generated protobufs and maybe a native Python implementation would be fairly straightforward as well. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Hal script to index removable storage medium ?
Hi, On Thu, Jul 10, 2008 at 9:03 AM, D Bera <[EMAIL PROTECTED]> wrote: > I am not much familiar with HAL scripts and how the nice popups > "Open the Audio CD in Amarok" shows up when I insert an audio cd. Essentially what happens is that when you attach a device, HAL notices through the hotplug mechanisms and instantiates a HAL device object which represents the hardware. Some standard properties are added based on the bus, type of device, etc., and then scripts are run to set additional properties or set policies. Once the device is added in HAL, it emits D-Bus signals on the system bus. The main ones that Beagle would care about would be DeviceAdded and DeviceRemoved, and possibly also NewCapability and LostCapability. The popup dialogs you see in GNOME or KDE are desktop-specific policy pieces. On GNOME, this is handled by gnome-volume-manager. It sees the added and removed signals, and performs some action (ie, policy) based on configuration. It's a good piece of code to use as an example for this because Beagle will care about a lot of the same things. I forget if GVM itself emits signals on the session bus when file systems are mounted... it seems like it would have... If it doesn't, I suspect they'd take a patch to add it. > I am thinking something along the same lines: > > (1) When the medium is inserted, if it of the "right kind", then show > the option "Index and search this medium". It would probably make sense to put this in GVM, or else you would probably have two separate (but related) dialogs popping up. Then other services like Tracker could use it. The downside is that you'd also have to implement similar functionality a second time (for KDE, or for GNOME if you did KDE first). Another option is to put this in the Beagle UI -- I can imagine a notification window popping up asking if the user wants to index the volume. Plus side to this is that works across all desktops. > (3) If the medium is writable then create a .beagleindex directory in > the medium's root directory; if the directory already exists then skip > this step > (4) If the medium is not writable, inform the user and ask for a > separate location to store the index I think the determination has to be more nuanced than writable/non-writable. You probably don't want to be storing indexes on flash drives. HAL's properties should make it a little easier to figure out which is which, but a blunt way would be to assume that anything > 8 GB is a hard drive and anything smaller is flash. > (5) If the indexdirectory already contains RemovableIndex.xml (which > is a sign that a previous index already exists), then call > "beagle-removable-index --mount..." followed by > "beagle-removable-index --removable..."; otherwise call > "beagle-build-index --removable.." and then run > "beagle-removable-index --mount..." > (6) Running beagled will automatically add this index to its search > list and pick up changes as beagle-build-index finds new files You could do these, but it almost seems like it would make more sense to move all of this functionality into beagled itself, since (at least in theory) it'll always be running and can catch these signals. > (7) (is this doable with HAL ?) When unmounting, > "beagle-removable-index --unmount..." needs to be called - the index > will be dropped from beagled's list and all open files in the > removable medium will be closed There are definitely mount/unmount signals, but I don't recall if they come from HAL or from GVM/KDE-equivalent. There are hooks in the kernel (called uevents) which tell userspace when an event happens. When Robert and I worked on Project Utopia the first thing we did was add uevents for mount/unmount and got HAL to listen to them. So you should get these signals even if someone unmounts a device by hand. On the other hand, there's a possibility that the power goes out and the signal is never received, so I'm not sure you can depend on them reliably. Hope this helps, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Speaking of Google backends
Hey, I just came across this today: http://code.google.com/apis/ajaxsearch/documentation/#fonje It might be interesting to bring back the Google search backend! Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: need help with testing: GMail live search (available in svn trunk)
Hi,
On Thu, Apr 10, 2008 at 10:11 AM, Debajyoti Bera <[EMAIL PROTECTED]> wrote:
> The "domain name" in
>
> https://mail.google.com/a/joeshaw.org/#search/[EMAIL PROTECTED]
> would be "joeshaw.org" or "https://mail.google.com/a/joeshaw.org/";. What I am
> asking is that is there is a notion of "domain name" for Google Apps that any
> google app user is supposed to know. And that I can safely add it to the end
> of "https://mail.google.com/a/"; to create the URL ("safely" - based on your
> experience).
Yeah, it would be "joeshaw.org". The URLs are predictable for google
apps... it's always https://mail.google.com/a/example.com instead of
https://mail.google.com/mail for standard Gmail.
Joe
___
Dashboard-hackers mailing list
Dashboard-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: need help with testing: GMail live search (available in svn trunk)
Hey, On Wed, Apr 9, 2008 at 9:45 PM, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > Now that you point out that gmail and google app uses different URLs, some > configuration seems required. I guess the "a" points to "app". Does not > matter though, since it will be tough to cover all cases. > > Instead how about a config option GMailSearchOpenURL with > default "https://mail.google.com/mail/#search/%s"; and can be replaced by > whatever the user wants. I think you could even go one simpler and ask (a) if it's a google apps account and (b) if so, what is the domain name. > > Yeah, it's pretty slow. It seems like we could cache the headers for > > certain IDs though? So at least largely overlapping services would be > > a bit quicker. > > Could you elaborate on this one ? Which IDs to cache ? Is this something > specific to Google Apps ? There are log messages to the effect of "Downloading headers for message ID 21". What I was suggesting was that if those IDs are stable, we could cache the headers locally so that we didn't have to download them again. I don't know anything about xemail-net, so I don't know if that is a possibility. > xemail-net anyway needs improvement. It is buggy and I didnot like its IMAP > implementation. I was surprised it worked for searching. Unfortunately it > seems to be unmaintained :-/ (Joe, no jokes please :) My lips are sealed. ;) Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: need help with testing: GMail live search (available in svn trunk)
Hi, On Mon, Apr 7, 2008 at 8:20 PM, D Bera <[EMAIL PROTECTED]> wrote: > > Hey folks, > > I wanted to see how a live GMail search "backend" will work so > here is one. > > It just queries GMail IMAP server directly for searching. I need some help > > I checked this in. For instructions, see > http://beagle-project.org/Live_GMail_Search Very nice work! I've enabled it and have been playing around with it. I use a Google Apps account, and was pleasantly surprised that my username and password just worked for the IMAP interface. However, the search results don't. I am getting URLs like: https://mail.google.com/mail/#search/[EMAIL PROTECTED] but for my apps account, they should be: https://mail.google.com/a/joeshaw.org/#search/[EMAIL PROTECTED] So I guess some configuration for Google apps accounts would be nice? > There were only some minor changes from the test dlls. Use > --enable-googlebackends during configure to build this. Its disabled > by default since its not too exciting (to me) and is not that fast :-( > (I mean querying over IMAP and then getting the headers of the matched > results is pretty slow). Yeah, it's pretty slow. It seems like we could cache the headers for certain IDs though? So at least largely overlapping services would be a bit quicker. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: gobject api help
Hi, On Thu, Apr 3, 2008 at 1:34 PM, D Bera <[EMAIL PROTECTED]> wrote: > For those who deal with gobject API, when you pass a string (char* or > const char*) to an API function, whose responsibility is it to free > the string ? What is the usual practice ? Is there any way to > distinguish methods that take a string and "own" it and methods that > make a copy of the string so the caller can safely free it or reassign > it to something else ? The general policy is that strings are always copied when passed as an argument, and they should always be declared "const char *". That way neither side needs to know how the memory of the argument was allocated. As to your distinguishing question, the answer is no, but it shouldn't really come up anyhow. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle on encrypted partitions yeilds horrible system performance
Hi, On Wed, Mar 26, 2008 at 11:12 AM, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > > I am hitting kind of a nasty performance problem, my current test setup is > > a two disk mdraid RAID0 setup with lvm ontop of a dmcrypt, all partitions > > beagle touches are ext4. Now every time beagle 0.3.4 indexes a folder, the > > entire system becomes near non responsive, typing yeilds detection of > > multiple key presses and kcryptd and beagle-helper are combined using 100% > > CPU (in about a 80/20 split with beagle being the 20%). > > Hmm... and the "encrypted partition" is not a red-herring ? I mean, could it > be ext4 ? Could it be extended attribute in ext4 ? Could it be just some > undetected bug in beagle ? Ahh ... ok - "kcryptd and beagle-helper are > combined using 100% CPU (in about a 80/20 split with beagle being the 20%)" - > so there is something to do with kcryptd. Beagle does do a tremendous amount of IO, so I don't think it's unreasonable that reading a large chunk of data off the disk and writing large chunks to the index is CPU-bound when we're talking about an encrypted home directory. I would be interested in knowing if other similarly IO-heavy operations (like "find /") are also CPU bound. Does setting BEAGLE_STORAGE_DIR to something on a different (non-encrypted) home directory change things? There may be no real solution to this, short of having Beagle's scheduler throttle itself more extremely, which will slow down indexing. Load average is already taken into account, but maybe it should be given more weight. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Summer of Code 2008
Hi, On Sat, Mar 1, 2008 at 7:06 AM, Lukas Lipka <[EMAIL PROTECTED]> wrote: > It's that time of the year again when Google's SOC program opens. For > more information see: > > http://code.google.com/soc/2008/ > > Is Beagle going to participate again this year? I would like us to, but I am personally unable to commit to being a mentor this year. The main thing I've learned as a mentor over the past two years is that you must have a significant chunk of time to devote to it, or else the program doesn't work. If you (or anyone else) is interested and can commit to being a mentor, I'd be happy to throw our names into the ring again this year. I'll be happy to do the administrative work behind it. Thanks, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle presence on Wikipedia
Hi, On Tue, Feb 26, 2008 at 4:31 PM, D Bera <[EMAIL PROTECTED]> wrote: > > > I read that section as Spotlight indexing the metadata itself stored > > > on the file system, which Beagle doesn't really do. Beagle does use > > > that file system metadata for its own bookkeeping, though. > > > > /me reads up on what all Spotlight does > > > > Ah, I see, so Spotlight also indexes the filesystem metadata. I took > > the phrase "file metadata" from the wikipedia article in the wrong > > context -- content attributes rather than filesystem metadata. > > Umm... what exactly is "filesystem metadata" other than name, size, > last modified time etc. ? I took that to mean extended attributes, basically. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle presence on Wikipedia
Hi, On Tue, Feb 26, 2008 at 1:33 PM, Nirbheek Chauhan <[EMAIL PROTECTED]> wrote: > Actually, I think he was referring to the fact that MacOSX's Spotlight > and Vista's Instant Search were mentioned, but Beagle which predates > both of them wasn't. > > "Apple Computer's Mac OS X operating system supports cataloguing and > searching for file metadata through a feature known as Spotlight, as > of version 10.4. Microsoft worked in the development of similar > functionality with the Instant Search system in Windows Vista, as well > as being present in SharePoint Server. Linux implements file metadata > using extended file attributes." I read that section as Spotlight indexing the metadata itself stored on the file system, which Beagle doesn't really do. Beagle does use that file system metadata for its own bookkeeping, though. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Getting started with beagle
Hi,
On Thu, Feb 14, 2008 at 12:03 AM, Debajyoti Bera <[EMAIL PROTECTED]> wrote:
> > An architectural decision to be made, do we want to actually index the
> > data off of every webservice, or just offer 'transparent' backends to
> > query the existing query API's for each service. I'm more for a local
>
> A transparent "proxy" backend to query using webservice API (in beagle lingo,
> a "QueryDriver") is fine for some kind of data but ideally a real backend
> that fetches the data and indexes it ("backend") would be the best option.
Yeah, I agree. If the data is reasonably small such that we can pull
it down and index it, and it doesn't violate the service's terms of
service, we should do that. Searches will be quicker in most cases,
they work while offline, etc.
And the most important reason is that local desktop searches don't
leak out onto the internet. This is the large reason why we have
"query domains" in Beagle; you don't necessarily want your searches
for "top secret plans to defeat Google and Facebook" to also go out
and hit Google's and Facebook's servers. There's a privacy issue
here. It's also one that's not addressed currently in the user
interfaces, and it would be a prerequisite to any such backend going
in.
> An out-of-process script will work but it is really not that complicated to
> do
> this in process. All you have to do is create an IndexableGenerator and feed
> indexables as asked in GetNextIndexable. Depending on how fast the data can
> be accessed from the webservice, either download some 30/40 "indexables" from
> the webservice in HasNextIndexable or use a separate thread to download them
> and put in a shared queue from which GetNextIndexable will get them.
Yeah, and we can add additional controls in the scheduler if that is needed.
> If you do it out of process, make sure you dont choke the internet by
> downloading all 10K emails in one go i.e. you can't ignore some kind of
> scheduling.
Agreed. You already have a scheduling system inside Beagle. You're
better off using it than writing your own. (Sometimes you can't avoid
it, like the Thunderbird backend running inside TBird itself.)
Joe
___
Dashboard-hackers mailing list
Dashboard-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Getting started with beagle
Hi, On Feb 13, 2008 1:07 PM, D Bera <[EMAIL PROTECTED]> wrote: > > On Feb 13, 2008 11:43 AM, Dirk Uys <[EMAIL PROTECTED]> wrote: > > > Other that the TODO list, is there any filters/backends in need of > > > attension? > > I have a fancy one :-) > Use one of the many C# POP/IMAP libraries to build a gmail backend. > Conceptually simple, just fetch a bunch of emails (10/15/something > low) at a time and feed them to beagle. Checkout the other backends on > how to hook into the scheduler so that the gmail backend is not > continuously fetching and indexing emails. Ah, yes! I knew there was something I was forgetting. Integration into wider web services would be nice. As a relatively recent Gmail user myself, I would be personally grateful for such a backend. ;) Google Docs, Facebook, Remember the Milk, etc. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Getting started with beagle
Hi, On Feb 13, 2008 11:43 AM, Dirk Uys <[EMAIL PROTECTED]> wrote: > Other that the TODO list, is there any filters/backends in need of attension? Yikes, the TODO list is pretty out of date these days. :) dBera recently posted to the list about two backends that need some work: Liferea (an RSS reader for GNOME) and Akregator (an RSS reader for KDE). Both went from storing their feed data as XML to (different) database formats. You can find some more info in bugzilla or in the mailing list archives. In particular, the Liferea guy seems pretty open to the idea. It's never sexy, but we always need better test harnesses. There is some code in Bludgeon that can be reused, and I'd love to see a harness where we can build a directory structure, index it, and run some automated queries against it. Bigger tasks are that the file system backend should probably be rewritten. I've written about that in the past, but it's definitely not a "first project" task to take on. Those are kind of the "big things" -- there are always bugs and feature requests in bugzilla. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Getting started with beagle
Hi Dirk, On Feb 10, 2008 4:35 PM, Dirk Uys <[EMAIL PROTECTED]> wrote: > I've been keeping a close eye on the beagle project for the last few > months or so and I want to play around with the project. > Is there a nice IDE to use? I know about monodevelop, but I don't know > if that's the best route to go? Can someone please give some advice as > where to start and what to use. Personally, I use emacs for editing source and make to compile it. I tend to run Beagle out of the source tree by hand. But I am an old fart. I know that some other people do use Monodevelop to do source hacking, but I think they still use make to build it. Because Beagle predates Monodevelop our source tree and library installation doesn't match up naturally, and so you have to do some work to get it to do all the autocompletion, etc. There's no reason why it *has* to, though, and I'm open to changing it. If you've played around with Monodevelop and like it quite a bit, a good way to get started would be to set it up so that you can do development in it. That'll give you an idea of the different libraries, how they work together, and how the source code is organized. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle Properties
Hi, On Feb 10, 2008 10:49 AM, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > That sounds overwhelming ! There are only a few that are within our reach ... > yelp, nautilus (and possibly brassero) in gnome cvs and kerry in kde svn. > There are more to which we dont have direct access. Sure, but we could provide patches. In general it's probably a pretty good exercise anyway to find out who the consumers of our APIs are so that we can find out how good they are, where they could use improvement, etc. > Isnt there a standard way of making string changes in a way to automatically > make applications aware of it ? Can these changes qualify as breaking binary > incompatibility; then we can increase library versions for both libbeagle1 > and beagle-0.0. We could do this, yeah. We could bump the API versions of the C# assemblies and the so number of the libbeagle shared libraries. This is probably a good idea. And not to sound like a broken record, but this stuff should probably go on a branch (or create a 0.3.x branch and continue on trunk) so that we don't find ourselves in another rut. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle Properties
Hi, On Jan 12, 2008 5:31 PM, Lukas Lipka <[EMAIL PROTECTED]> wrote: > Voila, my ongoing effort is almost finished. > > http://beagle-project.org/Properties_Hack_Week > > There is still a small amount of stuff that needs to be finished before > we can fire off this event. Stay tuned. Changing these properties is pretty dangerous, because we will effectively be changing a string API. That means that apps that use Beagle will still compile, but they will silently break when their old string mappings don't line up to the new ones. So I think it is important for us to take the initiative ourselves to fix the applications and add-on backends and filters that use Beagle. I think it would be helpful to collect a list of these on the wiki page and have it be a core part of this work. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: help with writing a python client
Hi, On Jan 30, 2008 1:16 PM, D Bera <[EMAIL PROTECTED]> wrote: > > >>> resp = client.send_request(query) > > >>> resp > > > (BeagleSearchTermResponse at 0x81ee600)> Interesting. Like dBera mentioned query requests have to be sent asynchronously. I thought that trying to start a query synchronously would throw an exception or an error response or something along those lines. I seem to vaguely remember code to that effect in the server somewhere. :) Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: [Xesam] Metadata Storage Daemon
Hi, On Jan 11, 2008 8:16 AM, Jamie McCracken <[EMAIL PROTECTED]> wrote: > Why do you need the timestamping? Is that just for beagle to query whats > changed? (beagle should really use notifications from the daemon for it > although if its not running it might miss them so I assume timestamping > is there for backup?) The most common case would be to listen to notifications. However, I think it's a mistake to assume that both systems will be running at any given point in time such that it's not a lossy channel. The ability for Beagle to ask "what's changed since I last shut down?" helps prevent this, and means that distributions and users don't have to worry about sequencing the startup of the daemons. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: suggest: make searches efficient
Hi, On Jan 11, 2008 3:50 AM, Enrico Minack <[EMAIL PROTECTED]> wrote: > > > ++ > > > || > > > | [Icon Docs] [Icon EMails] [Icon Images]| > > > | (3 hits) (20 hits) (124 hits) | > > > || > > > ++ > > I think shortcuts like these are a good idea, but not at the expense > > of showing any items at all. You would effectively be adding an extra > > click for every search, and the tradeoff would be sometimes not > > needing to scroll. > > > > So basically what I'm saying is that I like the idea, but we shouldn't > > *require* clickthrough to get at the results. > right, I agree. > > What if those icons above are just added to the current beagle-search UI > at the top of the results. It gives a quick overview, but all results > are shown below as they are now. By clicking on a icon, the result list > jumps down to the respective group. The user does not need to scroll, > and scrolling is more effort than one click ;-) Yes, exactly what I had in mind. :) Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: suggest: make searches efficient
Hi, On Jan 9, 2008 9:44 AM, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > Beagle-search does not show a flat list of results, and I dont see how to > implement a sidebar with all those extra links (buttons ?) without making it > look cluttered. Flatness might not be the worst thing in the world if you have rather simple one-click filtering of certain types. In general, I am not sure how much people like the grouping of results by type. Certainly the UI for paging them is a little awkward. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: suggest: make searches efficient
Hi, On Jan 8, 2008 3:40 AM, Enrico Minack <[EMAIL PROTECTED]> wrote: > > So I ask you for suggestions, how to make searches efficient ? > though Beagle already arranges results into groups of types (Documents, > EMails, Images, ...), it already displays the results inside these > groups, so that the user may have to scroll down to find the actual > interesting group. What if the first thing being shown are the groups / > types of results, along with the number of hits: > ++ > || > | [Icon Docs] [Icon EMails] [Icon Images]| > | (3 hits) (20 hits) (124 hits) | > || > ++ > and after selecting a group only the results of this one are shown > below. So the first thing you see is in which section of your desktop > you have hits. So you would not need to specify the type in the query > but can select from the matching ones without being overwhelmed by the > hits. I think shortcuts like these are a good idea, but not at the expense of showing any items at all. You would effectively be adding an extra click for every search, and the tradeoff would be sometimes not needing to scroll. So basically what I'm saying is that I like the idea, but we shouldn't *require* clickthrough to get at the results. > Another way is give information where the hits match the query, e.g. if > in the subject or the title, the author field or the Text. This is > similar to the facated view pointed to by Mikkel, recently. Yeah, I like this idea. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Problems with beagle-0.3.1 with .htm- files
Hi, On Jan 4, 2008 5:23 PM, D Bera <[EMAIL PROTECTED]> wrote: > > 20080104 09:49:03.9370 17841 IndexH DEBUG: No filter for > > file:///opt/zeitschriften/ct/html/05/19/220/art.htm > > (/opt/zeitschriften/ct/html/05/19/220/art.htm) > > [application/x-mozilla-bookmarks] > > This is a problem in recognizing mimetypes of the files. Beagle uses > freedesktop.org spec shared-mime-info and implementation xdgmime to > determine the type of a file. In this case, shared-mime-info diagnosed > those files are mozilla-bookmark files. Yeah, this isn't a Beagle-specific problem. If you google for "application/x-mozilla-bookmarks" you'll find a bunch of similar problems. > Mozilla-bookmark files have a > slightly different structure, so beagle does not index them. :( Really? Aren't they just a specialized type of HTML? Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: ANNOUNCE: Beagle 0.3.2
Hi,
On 12/28/07, Arun Raghavan <[EMAIL PROTECTED]> wrote:
> The query changes can be put off for the next release. Basically,
> while parsing queries, we might encounter OR clauses (which are fine)
> and AND clauses (which, I guess, are not) both of which might even be
> nested (definitely not supported). Joe and I spoke about it while I
> was first implementing the parser and decided to worry about this
> later.
Right. Beagle doesn't support nesting of AND, only OR. I don't think
there's any concrete reason why it couldn't support nesting of AND
within OR, but it's a huge change to the underlying query code.
There's a lot of work done to merge the logical outcomes of searching
two Lucene indexes; I don't know how expensive or difficult it would
be to support nested AND.
> Right now, I'm parsing Xesam queries and translating them to text
> searches when I should be using QueryParts.
Yeah. Any programmatic use of the APIs beyond simple text searches
should use QueryParts rather than doing things like
AddText("mimetype:text/plain"). Parsing the text is slower and more
error prone if the behavior changes.
Adding general convenience APIs so you can do things like "Give me
property 'mimetype' equal to 'text/plain'" would be nice; I suspect
that people fall back on the text because it's easier than the
alternative.
> I think we might need to
> extend QueryParts and the mapping to Lucene Queries to support this,
> though I'm not sure if this is the correct approach.
You mean nested ANDs? There's a lot more work than just QueryParts. ;)
Joe
___
Dashboard-hackers mailing list
Dashboard-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: insufficient? beagle-0.3.1.tar.bz2/gz on ftp.gnome.org
Hey, On 12/19/07, D Bera <[EMAIL PROTECTED]> wrote: > Eeerie! Never mind - I will pull this up myself. I only have one > problem - can't upload to ftp.gnome.org - my request for gnome shell > account isnt ready yet (its been more than a week now, so its > probabyly about time). Could you make a space in beagle-project.org to > keep the tarball ? Also, gnome ftp has all these md5sum, news and > changelog - are they separately generated and submitted or they are > automatically created ? You should already be able to use the files/ directory to create whatever directories and drop whatever you need there. For the GNOME stuff, it's all done automatically through a script named "install-module". So it repackages as a bz2, does md5sums, and pushes out to mirrors. I can do that when I get back, if you don't get a shell account first. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: insufficient? beagle-0.3.1.tar.bz2/gz on ftp.gnome.org
Hi, On 12/19/07, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > > svn version beagle-0.3.1 and tar.gz on ftp are differed. > > tar.gz without Util/AvahiBrowser.cs (not work configure --enable-avahi; > > make). Archive.cs not contained too. > > Yes, there was an error in the Makefile which didnt package the above two > files. Sorry for the trouble. Please download those two files from svn and > add them to the extracted tarball. We have a handful of bug fixes as well, maybe doing a 0.3.2 release soon would be a good idea. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagle r4293 - in trunk/beagle: BeagleClient beagled search search/Pages
Hey Lukas, On 12/15/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > Implement suggestions in an unobtrusive way using FuzzyTermEnum. Suggestions > are only generated upon request. How expensive is the search for suggestions? If they're inexpensive, we might want to consider returning a SuggestionsResponse for every Query instead of or in addition to explicitly requesting them. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagle problem
Hi, On 12/13/07, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > > I've made some other test, and I've seen that beagle-search does find > > files in the "applications" static index, but fails in the > > "documentation" static index. beagle-query and nautilus work fine. > > Oh ... beagle-search explicitly excludes documentation index while searching. > The documentation index is used by yelp (gnome help browser). Apparently > documentation results in beagle-search was confusing users and so it was > blacklisted from general search. I'm now of the mind that we should probably re-enable documentation as a separate category in the search. The UI can handle it reasonably well, whereas Best definitely could not. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: API changes in libbeagle application
Hi, On 12/4/07, D Bera <[EMAIL PROTECTED]> wrote: > Just a reminder that, when you set the source/hitttype/mimetype of a > query as explained below, note the WHITESPACE _before_ "type:AAA" etc. > i.e. beagle_query_add_text (query, " type:AAA") instead of > beagle_query_add_text (query, "type:AAA"). All that is happening is we > are adding another query term "type:AAA" at the end of the actual > query text. This is kind of a lame API detail. Maybe we can make beagle_query_add_text() smarter to handle being called multiple times. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Advisory: beagle-0.3.0 crashes at start
Hi, On 12/4/07, Stephan Hegel <[EMAIL PROTECTED]> wrote: > Hi Debajyoti, > > Debajyoti Bera wrote: > > ... If there are no config files in ~/.beagle/config, deleting ~/.beagle > > will not cause any additional loss of data. > Is it safe to delete everything in ~/.beagle except the config subdir ? > In other words: is the format of the config files the same ? For the purposes of the textcache crash, you really only need to delete the ~/.beagle/TextCache directory. You just won't get snippets for search results until documents are reindexed. However, I think the index format changed for 0.3.0, so everything will be reindexed anyway, right? Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Install fails
Hi, On 12/3/07, D Bera <[EMAIL PROTECTED]> wrote: > How about I use System.Environment.MachineName instead of > Mono.Unix.UnixEnvironment.MachineName ? Seems to work with long names > too... (I tried with a 24-char name). Yeah, let's do that. S.E.MachineName just calls gethostname() in an internal call in the runtime. M.U.UE.MachineName calls the same thing, but does it using P/Invoke and apparently doesn't do it correctly. So even if it worked, the results would be exactly the same. Thanks, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Install fails
Hi again, > > get_machinename() crasher ... where have I seen this before ? > > > > Maybe this ? > > https://bugzilla.novell.com/show_bug.cgi?id=MONO82460 > > Looks like it, yeah. Unfortunately it's a Mono bug, and it seems to > be deep enough in Mono XmlSerializer magic that we can't work around > it. Oh wait, maybe not. I missed a stack frame: at Mono.Unix.UnixEnvironment.get_MachineName () [0xb] in /amd/nfs/hippocampus/disk/ptn057/projects/lcontrib/src/mono-1.2.5.2/mcs/class/Mono.Posix/Mono.Unix/UnixEnvironment.cs:55 at Beagle.Util.NetworkingConfig..ctor () <0x0001d> at (wrapper runtime-invoke) Beagle.Util.ConfigSection.runtime_invoke_void (object,intptr,intptr,intptr) <0x> Can we work around this somehow in the Beagle.Util.NetworkingConfig class constructor? Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Install fails
Hi, On 12/3/07, D Bera <[EMAIL PROTECTED]> wrote: > > at (wrapper managed-to-native) System.String.InternalAllocateStr (int) > > <0x4> > > at (wrapper managed-to-native) System.String.InternalAllocateStr (int) > > <0x> > > at System.Text.StringBuilder.InternalEnsureCapacity (int) [0x000b1] in > > /amd/nfs/hippocampus/disk/ptn057/projects/lcontrib/src/mono-1.2.5.2/mcs/class/corlib/System.Text/StringBuilder.cs:681 > > at System.Text.StringBuilder.set_Capacity (int) [0x00017] in > > /amd/nfs/hippocampus/disk/ptn057/projects/lcontrib/src/mono-1.2.5.2/mcs/class/corlib/System.Text/StringBuilder.cs:142 > > at Mono.Unix.UnixEnvironment.get_MachineName () [0xb] in > > /amd/nfs/hippocampus/disk/ptn057/projects/lcontrib/src/mono-1.2.5.2/mcs/class/Mono.Posix/Mono.Unix/UnixEnvironment.cs:55 > > get_machinename() crasher ... where have I seen this before ? > > Maybe this ? > https://bugzilla.novell.com/show_bug.cgi?id=MONO82460 Looks like it, yeah. Unfortunately it's a Mono bug, and it seems to be deep enough in Mono XmlSerializer magic that we can't work around it. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Install fails (was Re: ANNOUNCE: Beagle 0.3.0 and Beagle-Xesam 0.1)
Hi, Just to add to what dBera said: On 12/3/07, Henry S. Thompson <[EMAIL PROTECTED]> wrote: > produces a segfault with the following trace: > > at (wrapper managed-to-native) System.String.InternalAllocateStr > (int) <0x4> > at (wrapper managed-to-native) System.String.InternalAllocateStr > (int) <0x> > at System.Text.StringBuilder.InternalEnsureCapacity (int) [0x000b1] > in > /amd/nfs/hippocampus/disk/ptn057/projects/lcontrib/src/mono-1.2.5.2/mcs/class/corlib/System.Text/StringBuilder.cs:681 Could you post the entire output of this crash? Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Need help: New screencasts
Hey gang, While updating the Beagle web page for the 0.3.0 release yesterday, I went and re-watched the screencasts that Nat did a long time ago and boy are they dated. So, we need someone to create some new screencasts to really highlight beagle. Preferably they would be in Flash format so they can be viewed right from the browser. The old demos are here: http://nat.org/demos and they would be a good starting point. Any takers? Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
ANNOUNCE: Beagle 0.3.0 and Beagle-Xesam 0.1
yPart for searching explicitly for URIs. * Support for excluding KMail folders. * Support for indexing Evolution notes. * Refactor the indexing of IM conversations. * New TeX filter. * Structure snippets rather than returning them as HTML strings. This gives application authors considerably more control over how to display them. * Scan the first 4k of files to determine their MIME type in addition to looking at the file extension, and choose the better match. * PDFs are now included when indexing system documentation. * Improvements to indexing files inside archives. Now all extracted files are handled in the index helper process, removing a costly round trip. * Redirect crash logs from Mono into the Beagle log files. They were previously lost and users had to reproduce problems with --fg. * Add many more keywords to the query language; run beagle-query --keywords to see a full list. * Lots and lots of indexing and searching bugfixes. Too many to count. * Translations: - Arabic (Djihed Afifi) - Basque (Mikel Paskual, Inaki Larranaga Murgoitio) - Brazillian Portuguese (Igor Pires Soares, Raul Pereira, Leonardo Fontenelle, Og Maciel) - British English(David Lodge) - Catalan(Jordi Mas) - Czech (Jakub Freidl) - Dzongkha (Pema Geyleg) - Finnish(Ilkka Tuohela) - French (Guillaume Ayoub, Stéphane Raimbault, Christophe Bilard, Claude Paroz) - Galician (Ignacio Casal Quinteiro) - Hungarian (Gabor Kelemen) - Japanese (Takeshi Aihana) - Latvian(Raivis Dejus) - Macedonian (Arangel Angov, Jovan Naumovski) - Norwegian bokmål (Kjartan Maraas) - Occitan(Yannig Marchegay) - Polish (GNOME PL Team) - Portuguese (Filipe Gomes) - Simplified Chinese (Funda Wang) - Spanish(Roberto Majadas, Jorge Gonzalez) - Swedish(Daniel Nylander) Contributors to this release: Debajyoti Bera, Joe Shaw, Lukas Lipka, Pierre Östlund, Tao Fei, Arun Raghavan, Nirbheek Chauhan, Kevin Kubasik, Alexis Christoforides, Kyle Ambroff, Fredrik Hedberg, Larry Ewing, Pat Double, Matteo Gottardi, Jose Carlos Garcia Sogo, Joseph Benavidez, Enrico Minack, JP Rosevear, Jason Kivlighn, and Stephane Loeuillet. Full set of changes: http://svn.gnome.org/viewvc/beagle/tags/BEAGLE_0_3_0/beagle/ChangeLog?view=markup http://svn.gnome.org/viewvc/beagle/tags/BEAGLE_0_3_0/libbeagle/ChangeLog?view=markup http://svn.gnome.org/viewvc/beagle/tags/BEAGLE_XESAM_0_1/ChangeLog?view=markup Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: telling beagle the client to open a file indexed.
Hi, On 11/21/07, Hugo Melo <[EMAIL PROTECTED]> wrote: > On Nov 21, 2007 5:42 PM, D Bera <[EMAIL PROTECTED]> wrote: > > > > You can change the source and add application/lotus.notes as a mail > > mimetype - that will work but requires you to change the source. > > > > Since beagle-search does not know how to handle non-evo and non-tbird > > email files, I dont see any other way for beagle-search to open your > > email results using notes. > Why fixme:client does not change the default handler always it is set? The short answer: there is no standard way to handle emails. Every client is different, which is why we have to hardcode them some of the time. The longer explanation: Evolution, for instance, does not accept file URIs for individual email files and cannot open them. It can only open emails that are "managed" by Evolution, and it has its own URI scheme for accessing emails whether they are local mbox or remote IMAP messages. Thunderbird has a similar setup. Kmail, on the other hand, can open individual files. But I am not sure how it opens IMAP messages. And I think it doesn't support mbox. I'm not a Kmail user so I don't know for sure. > If the problem is the parameter to pass to the client, beagle-search > could to require the field uri setting the correct parameter, > otherwise it ignores fixme:client and calls the system default > handler, as it do now. This seems like a reasonable solution. We'll happily take a patch for that. Otherwise, a bug in Bugzilla would be best to make sure that it isn't forgotten. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Dashboard?
Hi Nils, Welcome back! Nice to see you again. :) On 11/21/07, Nils Erik Svangård <[EMAIL PROTECTED]> wrote: > * libdashboard - Or as its better known, a C library that generates > valid, parseable clues. This is not only critical since most plugin > authors aren't going to want to spend the time validating that mono > can deserialize all their XML. But because we can generate bindings > for most every other language once we have them in C. I actually think this is the wrong thing to do. This is something we explicitly avoided the first time around with Dashboard, and was a large reason why we were able to instrument as many applications as we did. The problem with creating a library and making people use it is that it introduces a new library dependency. That's extra work for the maintainers of the software, developers, and packagers of the software, and for something as immature and experimental as Dashboard, they simply won't do it. So the way we did it originally was to include a fully self-contained .c file which people would #include into their project, and all the functionality could be included as a simple patch. Ok, that was a long setup, but I think it might even be easier today: with inotify, we don't even really need a programmatic IPC mechanism -- we can just drop XML files into a watched directory and the Dashboard can pick them up automatically. For projects which already use D-Bus, that's a viable alternative as well (but not the sole one -- no adding D-Bus to projects which don't already use it, because, again, added dependencies). > * Real Bug/Performance Testing/Fixing - This is a huge one, Dashboard > is still pre-alpha, and mostly proof of concept. While the code base > could be brought up to production level, it needs tons of cleanup. The amount of Dashboard code there is sufficiently small enough that it can either be used as a starting point, as a good chunk of reference code to look at, or redone completely. I would strongly encourage unit and automated regression tests for this sort of thing; it's something we haven't been particularly good about in Beagle and it has clearly hurt us. > Is there more stuff that have to be done? Well, there isn't a UI for it, so that's a big amount of work. :) It's unclear to me exactly how to present results most usefully to users. The decaying method we had before was a fairly simplistic way of doing it. Some machine learning would probably be pretty useful here. > Is anybody working on dashboard? Sadly, I don't think so. There were some patches sent to the list, and I believe some were committed, but nobody has really taken ownership of the project, which is what it really needs at this point. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle indexing when running on batteries
Hi, On 11/19/07, Miguel de Icaza <[EMAIL PROTECTED]> wrote: > > > 2) If the Beagle already entered the "powersaving" mode successfully, > > > is there any way to reduce the number of wakeups-per-second? As I > > > understand, if Beagle is using /proc filesystem to get AC status, it > > > needs some kind of polling. Is that the reason of wakeups? > > > > As I mentioned, this is a Mono bug. You're correct, it does poll > > /proc, but not 10 times a second. > > The polling is not related to /proc, at least the Mono side of things. > The polling is related to thread sleeping in WaitForXXX APIs that need > to be Thread.Aborted() at some point. Yes, sorry, I didn't mean to conflate the two. The "it" in my original paragraph was referring to Beagle, not Mono. These are two totally separate and unrelated things. So, to recap: (1) Mono has a bug related to Beagle's use of Monitor.Wait() that causes 10 wakeups per second. (2) Beagle pokes the AC adapter files in /proc every once in a while. I think every 5 or 10 seconds, although I don't know offhand. It'd be nice if this were ported over to HAL, which talks to the ACPI daemon directly, I believe. (1) and (2) have nothing to do with each other. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle indexing when running on batteries
Hi, On 11/19/07, Andrey Melentyev <[EMAIL PROTECTED]> wrote: > But actually I'm mostly worried by the number of wakeups-from-idle per > second that beagled generates when AC is unplugged. PowerTOP ( > http://www.lesswatts.org/projects/powertop/ ) utility says that > beagled produces: >1,6% ( 10,0) beagled : futex_wait (hrtimer_wakeup) > > 10 wakeups-from-idle per second. It is not much, but it reduces the > time that my laptop can work from battery a little bit. This is a Mono bug, it has little to do with Beagle itself. It was supposed to be fixed for the 1.2.5 release, but it's been put off. Unfortunately it seems to be something we can't do anything about. I don't have a bugzilla number offhand, but I know that the Mono developers are aware of the problem. I've CCed Miguel to poke him about it again. > So I have two questions: > 1) How can I check if Beagle got the fact that system works on battery There should be something in the logs in ~/.beagle/Log/current-Beagle to that effect. > 2) If the Beagle already entered the "powersaving" mode successfully, > is there any way to reduce the number of wakeups-per-second? As I > understand, if Beagle is using /proc filesystem to get AC status, it > needs some kind of polling. Is that the reason of wakeups? As I mentioned, this is a Mono bug. You're correct, it does poll /proc, but not 10 times a second. Thanks, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Opera Memory Fix
Hi, On 11/16/07, Kevin Kubasik <[EMAIL PROTECTED]> wrote: > The real issue is the reflection/serialization stuff, some of this > might be mono issues, some might be on us, but there are a ton of > strings hanging around in the XML serialization class libraries. There > are over 8,000 System.MonoType instances that pretty much hang around > (apperently they are not collectable, and abundantly used in > reflection) . I'll keep the list posted if I discover anything we can > do to reduce it. Indeed, Mono type instances can't be collected. This was a big reason why we moved to requiring plugins (like backends and filters) to be registered inside AssemblyInfo. We were previously walking all the types in all of the assemblies we had loaded. Needless to say, when we fixed this our memory usage dropped in half. Anyway, these are all related to XmlSerializer. The right thing to do here really is to remove all the message passing stuff and replace it with managed D-Bus. Just throw away almost all of the old code. I think the end result will be (a) more memory friendly, (b) probably faster, and (c) quite possibly simpler code. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Opera Memory Fix
Hi, On 11/16/07, Kevin Kubasik <[EMAIL PROTECTED]> wrote: > p.s. Also, our in-memory hashtable of Guid's for the FSQ is immense, > it would be trivial to change the LuceneNameResolver to use a sqlite > db to store that data on-disk and out of memory, freeing a few megs. I > don't know a ton about the FSQ in general, and almost nothing about > the uid mapping, if this class is super-dynamic then maybe its a bad > call, but it appears that its a big build at startup, then static. > Could be a big memory saver. This is why I keep saying that we need to rewrite the FSQ. That mapping is how we map the internal GUID URIs to the external, real but changeable file URIs. The FSQ essentially creates a parallel directory tree in memory so that paths and so forth can be quickly reconstructed. It might be possible to try to load those on-demand, but I suspect performance will suffer quite a bit since they would have to be pulled from the sqlite database. It might be worth playing around with, but I think the larger fix is necessary. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Accuratly mesure Beagle's Memory Usage
Hi, On 11/16/07, Kevin Kubasik <[EMAIL PROTECTED]> wrote: > Hey, I'm doing some quick number crunching and was looking for a good > way to get accurate numbers on Beagles memory footprint, I know just > ps or top isn't very accurate, are the numbers reported by mono > internally reliable (what we print when run in debug memory mode)? If you're looking for our managed memory usage, heap-buddy (a summarizing profiler) and heap-shot (a snapshotting profiler) are the best to use. They both give very accurate accounting for our managed uses. If you're looking for overall system usage, I suggest using smaps and/or exmap to take a look at how much memory is actually being used (as opposed to which pages are shared by other processes, etc.). *Never* use VMSize as an indicator, it is mostly worthless because it includes the memory mapped into the process (even if it is never used). If I mmap() in a 2 gigabyte file, for example, my VMSize will be over 2 GB, but that doesn't mean that any of it will actually be in memory. RSS is memory that is "in use" but it doesn't take into account memory shared between processes. For example, if you were running Beagle and Tomboy, a large amount of the class libraries between the two would likely be shared. RSS - shared is a reasonable approximation of the memory that is allocated by that process. Exmap installs a kernel module which is able to track the sharing of memory at the page level. Lastly, there are some environment variables you can pass to Mono to get, for example, detailed garbage collection statistics. Search the Mono wiki for that info. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle-Project.org is Down
Hi, On 11/15/07, Kevin Kubasik <[EMAIL PROTECTED]> wrote: > beagle-project.org is down, anyone who knows someone who knows > possible fixs please share! The machine that hosts beagle-project.org changed IP addresses, and we weren't given any warning about this ahead of time. We're trying to get the DNS updated, but we don't have direct control over the domain. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Two ideas - up for adoption
Hi, On 11/1/07, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > (1) GMail indexing: Figure out (reverse-enginneer ?) how gmail-desktop-search > for linux indexes GMail emails. There is no public API and the usual gmail > apis on the web are not search friendly. It should be an easy step of dumping > the internet traffic when a search is performed followed by a harder step of > finding out what the dump means. (*) All of the above assuming, Google > doesn't download the emails using POP/IMAP and then index them. This might be a useful piece of code: http://johnvey.com/features/gmailapi/ It's already in C#. Unfortunately it's licensed under the GPL, but maybe the author would be willing to relicense for us. Also, Gmail has recently added support for IMAP, so there's always the ability to index those emails through Evolution, Thunderbird, etc. Obviously not an ideal solution, but it should work. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
State of the Pooch
Hi, It's that time again. Time for a "State of the Pooch" email to let the community know how we're doing with Beagle and where we're going. Previous addresses are here: http://mail.gnome.org/archives/dashboard-hackers/2006-November/msg00064.html http://mail.gnome.org/archives/dashboard-hackers/2005-May/msg00011.html A lot of the stuff in the previous SotP, roughly a year ago, still applies in some way today. * dBera is now co-maintainer I'm happy to announce that Debajyoti Bera, who has easily written more code for Beagle in the last year than anyone else, has become a co-maintainer of the project. This is great news because he has solid knowledge of the codebase and is the first non-Novell maintainer of the code. dBera will still be mostly coding, and he will have equally final say about patches, technical direction, etc. with me. He may also do releases from time to time. :) * The never-ending quest for 0.3.0 Work continues in trying to make a great 0.3.0 release, and in the meantime we're pushing out 0.2.x maintenance releases. I'd love it if people could be regularly running from SVN trunk so that we can stress test a lot of the features that I'll mention below and get a 0.3.0 release out there that the less adventurous users out there can enjoy. * Networked searches Thanks to work from Lukas Lipka and Fredrik Hedberg, we've (finally!) merged the network search code from Alexis and Kyle's Summer of Code projects from last year into the codebase. The Beagle daemon now provides a backend which can query other Beagle instances. There is some preliminary support for Avahi and autodiscovery of other Beagle daemons on the network, but that's currently disabled while some stability bugs are worked out. There's still a lot of work to be done here in terms of how we access non-file resources on remote machines, security concerns, etc., so this code should be considered experimental for now. You can turn it on by toggling the networked setting in one of the configuration tools. * Web user interface dBera and Nirbheek Chauhan have been working on a Web interface to Beagle. In addition to search results, index information, daemon status, and the ability to shut down the daemon are all possible through this UI. The Web UI relies on the network infrastructure. It's not meant to be a replacement to beagle-search, but it is nice in that it is easily skinable, will be easy to view email in the browser, etc. http://beagle-project.org/Beagle_Webinterface * New configuration system dBera has been working on a new configuration system to handle two shortcomings in the current system: (1) Allowing a system-wide configuration file, so sysadmins can apply policy to all users and (2) allow plugins (like filters and backends) to store and retreive their own configuration options. The configuration manager loads the global config file (in /etc/beagle/config/) and the local one, merging the two. This also fixes the current problem where all settings were saved in the user's config file, not just the ones that are changed from the default. * Xesam support Arun Raghavan has written an adapter to Beagle which implements the Xesam freedesktop.org search spec, and the reference tools run against it. Exactly how this will be integrated into the code is unclear at this point, however. As of right now, there are no fully fledged search tools which use the Xesam API, so we're not ready to commit to the APIs natively. Also, integrating D-Bus back into Beagle is a worthy goal, but will require quite a bit of work. * Firefox extension More great Summer of Code work, the new Firefox extension has been merged into the source tree. In addition to indexing web pages as you fiew them, you can now index web pages, links, and images on demand. The settings UI is greatly improved as well. http://dtecht.blogspot.com/2007/08/hey-firefox-beagle-this-now.html * Thunderbird extension Another SoC project, we decided to take a different approach from the previous Thunderbird work and the Evolution backend, and add support for Thunderbird through an extension. This extension is responsible for sending emails to the running Beagle daemon for indexing. While you have to be running Thunderbird for this to work, it's fast and much, much friendlier on the system resources. * Experimental RDF branch This is an experimental branch which will export an RDF service that clients can query. This is something that has been planned from the beginning in Beagle, but we've never gotten around to it until now. As data is indexed, an RDF store will be created alongside the text index, and more complex relationships between the data can be examined. * Lo
Re: How does beagle get stopped?
Hi, On 10/22/07, Max Wiehle <[EMAIL PROTECTED]> wrote: > I took a look at the code and that is exactly what happens. But "beagle > --fg" does not seem to return. I use it in a script that waits for > beagled to return to stop the repository afterwards. This works fine > with beagle-shutdown. However it does not seem to work when logging > out /shutting down. Hmm, then I guess the process isn't really exiting. It's just hanging. You might want to try sending SIGQUIT to the process. This will cause it to print out a stack trace of every running thread. That might give an indication of where in the code it is hanging. I can't think of any good reason why it would be hanging for an extended period of time. Normally some delay is normal as beagled signals beagled-helper to shut down, and it finishes indexing its current batch of data, but this shouldn't be too long. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle as a Web site indexer.
Hi, On 10/15/07, Omri Schwarz <[EMAIL PROTECTED]> wrote: > I am writing Yet Another web frontend for Subversion, > (it's a fork of Insurrection) and I need a search engine > for it, one that will index by file and revision, and present > the results in a form presentable in HTML. > > Has anyone done anything along those lines yet? While possible, I don't really think that Beagle is the right tool for the job here. It certainly isn't suitable out of the box, and will probably require a good amount of hacking. You might be interested in something like Solr, which is an Apache project to build a web search server on top of Lucene: http://lucene.apache.org/solr/ Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: We have a Google Ad
Hey, Yeah, Google's Open Source office (the ones that bring you (and us) the Google Summer of Code program) were kind enough to offer us some free AdWords credit. The campaign has been running since late June, and we've had over 300,000 ad impressions. Considerably fewer clicks, though. ;) Joe On 10/3/07, Kevin Kubasik <[EMAIL PROTECTED]> wrote: > Apparently we are just that cool... ;) > > > -- > Cheers, > Kevin Kubasik > http://kubasik.net/blog > > ___ > Dashboard-hackers mailing list > Dashboard-hackers@gnome.org > http://mail.gnome.org/mailman/listinfo/dashboard-hackers > > > ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: GSoC Weekly Report
Hi, On 10/16/07, D Bera <[EMAIL PROTECTED]> wrote: > A followup question, I didnot find any API documentation of > Mono.Data.Sqlite :( #mono was also sleeping when I asked the question > there. My understanding is that both M.D.SqliteClient and M.D.Sqlite follow the general ADO.Net API patterns and that the latter is more or less a drop-in replacement for the former. A few things may need to be tweaked, but in general just changing the "using" statements at the top of each source file should be all that's needed. > If M.D.Sqlite does not have a way to return rows on demand, I > am against the migration. In the worst case, we can ship with a > modified copy of M.D.Sqlite but I am not sure what will that buy us. You've always been able to get rows on demand via ADO.Net, it's just a matter of the implementation underneath. The old one (not modified by us) would load all of them into memory. I'm not sure how the new one performs memory-wise. If the Mono guys don't have any idea, the right thing to do here would be to create a large test database (or use an existing TextCache or FAStore db) and do a "SELECT *" using the 3 implementations and walk the results, using heap-buddy and/or heap-shot to analyze their memory usage. > In the same breath, what is the benefit of M.D.Sqlite over > M.D.SqliteClient for beagle ? I figured out there are some ADO.Net > advantages but other than that ... ? It's maintained for one, which our modified one essentially isn't. It has the backing of the Mono team. The code is much cleaner and easier to understand, largely because it doesn't have two separate codepaths (one for v2 and one for v3). I am sure the Mono guys have other good reasons too. :) Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: How does beagle get stopped?
Hi, On 10/17/07, Max Wiehle <[EMAIL PROTECTED]> wrote: > I am currently working on beagle++ and for us it's quite crucial to shut > down beagled cleanly and then shut down the rdf repository as well. I > did write a shell script that does just that if beagle is terminated, > shut down with beagle-shutdown or a TERM or INT signal is send to the > script. However this does not seem to work when logging out of a gnome > session or shutting down the computer. > So how does this normally work. What stops beagle when i halt my system? The beagle daemon tries to connect to the X server when it's started and will start its shutdown process if the X connection is severed (because the user logged out, for instance). The code which does this is in beagled/BeagleDaemon.cs. Look for XssInit() and XIOErrorHandler. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: System.InvalidOperationException: Invalid connection string
Hi, On 10/17/07, D Bera <[EMAIL PROTECTED]> wrote: > Well that was just a guess. Joe would know better. We'll have to test those characters. Could very well be the colon, but I have no idea offhand. We already protect against forward slashes and commas. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Beagle-query and dates
Hi, On 10/17/07, DIMonS <[EMAIL PROTECTED]> wrote: > I was hoping to script beagle into checking multiple search terms on a daily > basis outputting to a file so users can see results in a single file at > start work. > > No problems except the daily part. Misread the Searching_Data page and > have realised that I cannot use the date: keyword to limit hits from > "yesterday" only. > > Does anyone have any ideas on how I can overcome this hitch in my master > plan? It would be good to add support for date operators for things like "yesterday", "2 weeks ago", etc. But we don't have that yet. I suggest filing a bug in bugzilla or sending a patch. ;) Beagle does support date queries, though. So if you knew the date, you could programmatically construct the extra query parameter, which would be something like: date:20071017 Check out the wiki for more info: http://beagle-project.org/Searching_Data Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: System.InvalidOperationException: Invalid connection string
Hi, On 10/17/07, Brian J. Murrell <[EMAIL PROTECTED]> wrote: > Nope. They all report as being "SQLite 3.x database", such as: > > $ file ~/.beagle/Indexes/EvolutionMailIndex/SummaryTracker-*.db > ... > /home/brian/.beagle/Indexes/EvolutionMailIndex/[EMAIL PROTECTED]: > SQLite 3.x database > ... Can you attach a list of all of the *.db files in that directory? I suspect there is a character in there that is causing sqlite some trouble, and it's probably related to your folder renaming, yes. Thanks, Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: GSoC Weekly Report
Hi, On 10/16/07, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > > > What to do with our local changes to Mono.Data.SqliteClient ? I always > > > get confused with them. Dont even know what are those changes and why are > > > they there :-/ (it has something to with threading and locking) ? > > > > The work done locally was mainly for memory usage reasons. IIRC, the > > upstream bindings pull all of the results into memory at once, whereas > > our locally modified ones do so only as needed. I don't think > > threading/locking was ever an issue -- you might be confusing it with > > the fact that we couldn't use early sqlite 3.x versions because of > > broken policy in the library to that effect. > > Probably you are right. I still had to verify ... > beagle:/source=mind?query=sqlite+beagle+lock > returned nothing :-D > but google returned, > http://lists.ximian.com/pipermail/mono-devel-list/2005-November/015977.html > which mentions "Lock" ... yay! My faith in my memory is restored ;-) Indeed you're right, but those changes did get merged upstream. So the memory usage I believe is the only outstanding reason. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi, On 10/14/07, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > Ya. Its possibly because of a lower inotify max_watches limit. It used to > print too many warning messages in the past, even without --debug (besides > causing other problems). > > BTW, this problem was fixed in the next bugfix release, 0.2.18. And you should > up the max_watches limit; the value in /proc/sys/fs/inotify/max_user_watches > should be significantly larger that the number of directories in your home > directory. (OpenSUSE 10.3 possibly does that ... not sure). Yeah, we made inotify (and a higher max_user_watches) a requirement in 0.2.18. openSUSE 10.3 has the number of watches increased to 65535 in /etc/sysctl.conf. Other distros should follow suit. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi, On 10/13/07, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > I was referring to this part: > // FIXME: We always turn on full debugging output! We are > //still debugging this code, after all... > //arg_debug ? LogLevel.Debug : LogLevel.Warn, > LogLevel.Debug, > ... > > which basically ignores the --debug. I think it is high time we respect the > users decision by default. > > Did you mean all distributions (I know about OpenSUSE and Fedora, maybe Ubuntu > also does) know about this and patch the code with a higher loglevel ? Yes, I know that openSUSE, Fedora, and Ubuntu all do this. Packages from the openSUSE Build Service (which I also maintain w/ the stock openSUSE Beagle package) don't have this patch, though, and that is intentional. A lot of the people who are running the build service package were asked to upgrade to it from whatever shipped with 10.2 to ensure that bugs were fixed. This was the easiest way to do that. > I don't think all distributions know about this; even if they do, what purpose > does it serve us. Only users who install from source will ever see large > logfiles and thereby figure out there is some problem. There's a tradeoff here, to be sure, and my assumption was that users who build from source are more likely to be savvy and more able and willing to help us with debugging if they ran into a problem. It's no small task to build from source and the majority of people won't do it. > What I was suggesting was a way around this: respect the --debug flag and > allow turning on the debug during runtime (so that if a user is suspicious of > something, debug can be turned on and log file inspected). That way > distributions wont have to patch it and we can still ask users for debugging > information even if they do. Sure, and we have this today with most of the Beagle instances out there, since most users just use packages. They can run with --debug, which turns up the debug level, and they can send the process SIGUSR1 to raise debugging to the equivalent to --debug at runtime. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: GSoC Weekly Report
Hi, On 10/13/07, Debajyoti Bera <[EMAIL PROTECTED]> wrote: > What to do with our local changes to Mono.Data.SqliteClient ? I always get > confused with them. Dont even know what are those changes and why are they > there :-/ (it has something to with threading and locking) ? The work done locally was mainly for memory usage reasons. IIRC, the upstream bindings pull all of the results into memory at once, whereas our locally modified ones do so only as needed. I don't think threading/locking was ever an issue -- you might be confusing it with the fact that we couldn't use early sqlite 3.x versions because of broken policy in the library to that effect. I'm not sure what the memory side effects of the newer upstream bindings are. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Just notice, Kerry/Beagle is a Massive Memory Hog
Hi, On 10/13/07, D Bera <[EMAIL PROTECTED]> wrote: > > That is a total of 14 and I using over 300 Megs for beagle. (ouch) > > You are lucky that 14 instances are using only 300 Megs - thats about > "20" Megs a piece. Normally instances are about 30 Meg each. It's also not a very accurate representation of the amount of memory that's actually being used. If you're looking at the VSIZE column, well, it's a lie. :) And while the RSS column is closer, it still doesn't take into account memory that's shared between processes, which Mono apps use quite a bit. A tool like exmap or smap will give you a lot more accurate numbers of memory usage. ps or top are only good for very basic diagnostics, like if you have a (single instance of an) app using a ton of memory. I'm not denying that there's a bug here -- certainly why you have 14 instances of it running is puzzling -- but memory usage is often misunderstood on Linux. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: Giant logfile
Hi, On 10/13/07, D Bera <[EMAIL PROTECTED]> wrote: > > First, there are always old log files floating around in the Log directory. > > Do we really need them ? Log files older than 14 days are removed automatically by Beagle, IIRC. > This is a good question. I am in favour of turning off the debug flag > in the "release" build (i.e. the installed one). Instead debugging can > be turned on when running uninstalled or by using some environment > variable. Though debugging helps and some hard to find bugs were found > because some users had the debugging enabled, the cost users pay is > too much. All the major distributions which ship Beagle turn the log level down by default; you have to pass --debug into the daemon to override it. I'm not a fan of turning it off for releases, and indeed in this case it probably wouldn't have helped. I'd be willing to bet that the vast majority of those 140 gigs are WARN or ERROR level messages, which are logged in all cases. Moreover, there is a level of degrees here. If a log file is 140 gigs, something is wrong. There is probably looping going on there, etc. I wouldn't expect an index helper log file to get anywhere near one gig in normal usage. (It would be restarted before it got anywhere near that.) So it's actually kind of a good thing, because they point the user to a problem that might only have otherwise been manifested in missed files, CPU pegging, etc. This way we can at least identify is the issue. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
Re: beagle (indexing)
Hi, On 10/12/07, D Bera <[EMAIL PROTECTED]> wrote: > If you want to work on this, ping me back :) > I can send you all data you need. Akregator now uses a db which has > C++ (not C, so no P/Invoke) and python bindinds but no C# binding. How do the python bindings access the data? Do they reimplement the database access, or use a library API underneath? If it's the latter, it implies that there's a C API that could be used in C#. Joe ___ Dashboard-hackers mailing list Dashboard-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/dashboard-hackers
