[Tracker] Support in Tracker for ultra-new Evolution installs that use SQLite for the summary format
This patch makes ultra-new Evolution installs work again with Tracker. There's one problem and that is that the query will only find E-mails in the INBOX folder. You can easily find the Query and figure out what the problem is: The design that Carlos made assumes that for each folder there's a "summary" file. In the new Evolution cache format there's just one "folders.db" for each account. I could do a generated UNION select after first doing "select * from folders" on folders.db and then generating a query that includes all folders. I just have not done this for now and instead I'm just using INBOX and I'm neglecting the other folders. This is NOT the same as the proposal that I am doing at (a). This is instead a ad-hoc solution for the new situation (Evolution using SQLite for the summaries). I find this solution rather nasty, to be honest. (a) http://live.gnome.org/Evolution/Metadata For Carlos: I have also fixed a serious problem in evolution-pop.c, which is by the way unaffected by Evolution's changes (and works, if you just apply the patch that I included in this larger patch). The POP support's get_message_metadata was not returning metadata. This was crashing my tracker-indexer (as seemingly my compiler was putting "return 0x2" where the return was omitted, and the memory I have at 0x2 didn't dereference TrackerModuleMetadata's members very well). Please review and/or rework the patch. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be Index: src/tracker-indexer/modules/evolution-pop.c === --- src/tracker-indexer/modules/evolution-pop.c (revision 2701) +++ src/tracker-indexer/modules/evolution-pop.c (working copy) @@ -360,6 +360,8 @@ } g_list_free (list); + + return metadata; } static TrackerModuleMetadata * Index: src/tracker-indexer/modules/evolution.c === --- src/tracker-indexer/modules/evolution.c (revision 2701) +++ src/tracker-indexer/modules/evolution.c (working copy) @@ -70,7 +70,8 @@ strchr (basename, '.') == NULL) { type = MAIL_STORAGE_LOCAL; } else if (g_str_has_prefix (path, imap_dir) && - strcmp (basename, "summary") == 0) { + (strcmp (basename, "summary") == 0 || + strcmp (basename, "folders.db") == 0)) { type = MAIL_STORAGE_IMAP; } Index: src/tracker-indexer/modules/Makefile.am === --- src/tracker-indexer/modules/Makefile.am (revision 2701) +++ src/tracker-indexer/modules/Makefile.am (working copy) @@ -16,7 +16,8 @@ $(GIO_CFLAGS) \ $(GLIB2_CFLAGS) \ $(GCONF_CFLAGS) \ - $(GMIME_CFLAGS) + $(GMIME_CFLAGS) \ + $(SQLITE3_CFLAGS) indexer_modules_LTLIBRARIES = \ libtracker-module-applications.la\ @@ -67,6 +68,7 @@ $(GMODULE_LIBS) \ $(GLIB2_LIBS) \ $(GCONF_LIBS) \ - $(GMIME_LIBS) + $(GMIME_LIBS) \ + $(SQLITE3_LIBS) endif Index: src/tracker-indexer/modules/evolution-imap.c === --- src/tracker-indexer/modules/evolution-imap.c (revision 2701) +++ src/tracker-indexer/modules/evolution-imap.c (working copy) @@ -128,7 +128,18 @@ g_free (file->imap_dir); g_free (file->cur_message_uid); -fclose (file->summary); + if (file->db) { + sqlite3_close (file->db); + } + + if (file->stmt) { + sqlite3_finalize (file->stmt); + } + + if (file->summary) { + fclose (file->summary); + } + close (file->fd); G_OBJECT_CLASS (tracker_evolution_imap_file_parent_class)->finalize (object); @@ -494,25 +505,62 @@ NULL); path = g_file_get_path (tracker_module_file_get_file (file)); -self->fd = tracker_file_open (path, TRUE); -g_free (path); -if (self->fd == -1) { -return; -} + self->db = NULL; + self->summary = NULL; + self->stmt = NULL; -self->summary = fdopen (self->fd, "r"); -self->n_messages = read_summary_header (self->summary); -self->cur_message = 1; + if (g_str_has_suffix (path, ".db")) { + /* New SQLite based format */ + sqlite3_stmt *stmt; + gint result = SQLITE_OK; -if (self->n_messages > 0) { -/* save current message uid */ -read_summary (self->summary, - SUMMARY_TYPE_STRING, &self->cur_message_uid, /* message uid */ - -1); -} + sqlite3_open (path, &self->db); -ensure_imap_accounts (); + sqlite3_prepare_v2 (self->db, "select saved_count from folders where folder_name = 'INBOX'", +-1, &stmt, NULL); + + result = sqlite3_step (stmt); + self->n_messages = sqlite3_column_int (stmt, 0); + self->cur_message = 1; + + if (self->n_mes
Re: [Tracker] Support in Tracker for ultra-new Evolution installs that use SQLite for the summary format
hi philip, short comment (the tablet is not the best device to review patches): can we call this module "evolution224" and keep also the old module? we could choose between one or other with a flag in compilation time (the distributions will choose the correct for them) regards, Ivan On 12/11/08, Philip Van Hoof wrote: > This patch makes ultra-new Evolution installs work again with Tracker. > > There's one problem and that is that the query will only find E-mails in > the INBOX folder. You can easily find the Query and figure out what the > problem is: > > The design that Carlos made assumes that for each folder there's a > "summary" file. In the new Evolution cache format there's just one > "folders.db" for each account. > > I could do a generated UNION select after first doing "select * from > folders" on folders.db and then generating a query that includes all > folders. I just have not done this for now and instead I'm just using > INBOX and I'm neglecting the other folders. > > This is NOT the same as the proposal that I am doing at (a). This is > instead a ad-hoc solution for the new situation (Evolution using SQLite > for the summaries). I find this solution rather nasty, to be honest. > > (a) http://live.gnome.org/Evolution/Metadata > > For Carlos: I have also fixed a serious problem in evolution-pop.c, > which is by the way unaffected by Evolution's changes (and works, if you > just apply the patch that I included in this larger patch). The POP > support's get_message_metadata was not returning metadata. > > This was crashing my tracker-indexer (as seemingly my compiler was > putting "return 0x2" where the return was omitted, and the memory I have > at 0x2 didn't dereference TrackerModuleMetadata's members very well). > > Please review and/or rework the patch. > > -- > Philip Van Hoof, freelance software developer > home: me at pvanhoof dot be > gnome: pvanhoof at gnome dot org > http://pvanhoof.be/blog > http://codeminded.be > ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] Support in Tracker for ultra-new Evolution installs that use SQLite for the summary format
On Thu, 2008-12-11 at 23:58 +0200, Ivan Frade wrote: > hi philip, > > short comment (the tablet is not the best device to review patches): > can we call this module "evolution224" and keep also the old module? > we could choose between one or other with a flag in compilation time > (the distributions will choose the correct for them) With this patch the code will detect the format and automatically choose the right way to parse both. > regards, > > Ivan > > On 12/11/08, Philip Van Hoof wrote: > > This patch makes ultra-new Evolution installs work again with Tracker. > > > > There's one problem and that is that the query will only find E-mails in > > the INBOX folder. You can easily find the Query and figure out what the > > problem is: > > > > The design that Carlos made assumes that for each folder there's a > > "summary" file. In the new Evolution cache format there's just one > > "folders.db" for each account. > > > > I could do a generated UNION select after first doing "select * from > > folders" on folders.db and then generating a query that includes all > > folders. I just have not done this for now and instead I'm just using > > INBOX and I'm neglecting the other folders. > > > > This is NOT the same as the proposal that I am doing at (a). This is > > instead a ad-hoc solution for the new situation (Evolution using SQLite > > for the summaries). I find this solution rather nasty, to be honest. > > > > (a) http://live.gnome.org/Evolution/Metadata > > > > For Carlos: I have also fixed a serious problem in evolution-pop.c, > > which is by the way unaffected by Evolution's changes (and works, if you > > just apply the patch that I included in this larger patch). The POP > > support's get_message_metadata was not returning metadata. > > > > This was crashing my tracker-indexer (as seemingly my compiler was > > putting "return 0x2" where the return was omitted, and the memory I have > > at 0x2 didn't dereference TrackerModuleMetadata's members very well). > > > > Please review and/or rework the patch. > > > > -- > > Philip Van Hoof, freelance software developer > > home: me at pvanhoof dot be > > gnome: pvanhoof at gnome dot org > > http://pvanhoof.be/blog > > http://codeminded.be > > > -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] Support in Tracker for ultra-new Evolution installs that use SQLite for the summary format
Hi, philip How about to open folders.db in readonly mode? I just check sqlite's API. sqlite3_open_v2 support a para with readonly flag On 2008ǯ12·î12Æü 00:34, Philip Van Hoof wrote: This patch makes ultra-new Evolution installs work again with Tracker. There's one problem and that is that the query will only find E-mails in the INBOX folder. You can easily find the Query and figure out what the problem is: The design that Carlos made assumes that for each folder there's a "summary" file. In the new Evolution cache format there's just one "folders.db" for each account. I could do a generated UNION select after first doing "select * from folders" on folders.db and then generating a query that includes all folders. I just have not done this for now and instead I'm just using INBOX and I'm neglecting the other folders. This is NOT the same as the proposal that I am doing at (a). This is instead a ad-hoc solution for the new situation (Evolution using SQLite for the summaries). I find this solution rather nasty, to be honest. (a) http://live.gnome.org/Evolution/Metadata For Carlos: I have also fixed a serious problem in evolution-pop.c, which is by the way unaffected by Evolution's changes (and works, if you just apply the patch that I included in this larger patch). The POP support's get_message_metadata was not returning metadata. This was crashing my tracker-indexer (as seemingly my compiler was putting "return 0x2" where the return was omitted, and the memory I have at 0x2 didn't dereference TrackerModuleMetadata's members very well). Please review and/or rework the patch. ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] Support in Tracker for ultra-new Evolution installs that use SQLite for the summary format
On Fri, 2008-12-12 at 01:59 +0100, Philip Van Hoof wrote: > On Thu, 2008-12-11 at 23:58 +0200, Ivan Frade wrote: > > hi philip, > > > > short comment (the tablet is not the best device to review patches): > > can we call this module "evolution224" and keep also the old module? > > we could choose between one or other with a flag in compilation time > > (the distributions will choose the correct for them) > > With this patch the code will detect the format and automatically choose > the right way to parse both. > This version of the patch adds support for IMAP subfolders and fixes the fact that in the last patch the "email://" URI was generated wrong (it still contained the "folders.db" string-piece). > > regards, > > > > Ivan > > > > On 12/11/08, Philip Van Hoof wrote: > > > This patch makes ultra-new Evolution installs work again with Tracker. > > > > > > There's one problem and that is that the query will only find E-mails in > > > the INBOX folder. You can easily find the Query and figure out what the > > > problem is: > > > > > > The design that Carlos made assumes that for each folder there's a > > > "summary" file. In the new Evolution cache format there's just one > > > "folders.db" for each account. > > > > > > I could do a generated UNION select after first doing "select * from > > > folders" on folders.db and then generating a query that includes all > > > folders. I just have not done this for now and instead I'm just using > > > INBOX and I'm neglecting the other folders. > > > > > > This is NOT the same as the proposal that I am doing at (a). This is > > > instead a ad-hoc solution for the new situation (Evolution using SQLite > > > for the summaries). I find this solution rather nasty, to be honest. > > > > > > (a) http://live.gnome.org/Evolution/Metadata > > > > > > For Carlos: I have also fixed a serious problem in evolution-pop.c, > > > which is by the way unaffected by Evolution's changes (and works, if you > > > just apply the patch that I included in this larger patch). The POP > > > support's get_message_metadata was not returning metadata. > > > > > > This was crashing my tracker-indexer (as seemingly my compiler was > > > putting "return 0x2" where the return was omitted, and the memory I have > > > at 0x2 didn't dereference TrackerModuleMetadata's members very well). > > > > > > Please review and/or rework the patch. > > > > > > -- > > > Philip Van Hoof, freelance software developer > > > home: me at pvanhoof dot be > > > gnome: pvanhoof at gnome dot org > > > http://pvanhoof.be/blog > > > http://codeminded.be > > > > > -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be Index: src/tracker-indexer/modules/evolution-pop.c === --- src/tracker-indexer/modules/evolution-pop.c (revision 2701) +++ src/tracker-indexer/modules/evolution-pop.c (working copy) @@ -360,6 +360,8 @@ } g_list_free (list); + + return metadata; } static TrackerModuleMetadata * Index: src/tracker-indexer/modules/evolution.c === --- src/tracker-indexer/modules/evolution.c (revision 2701) +++ src/tracker-indexer/modules/evolution.c (working copy) @@ -70,7 +70,9 @@ strchr (basename, '.') == NULL) { type = MAIL_STORAGE_LOCAL; } else if (g_str_has_prefix (path, imap_dir) && - strcmp (basename, "summary") == 0) { + (strcmp (basename, "summary") == 0 || + strcmp (basename, "folders.db") == 0) || + strcmp (basename, "cmeta") == 0) { type = MAIL_STORAGE_IMAP; } Index: src/tracker-indexer/modules/Makefile.am === --- src/tracker-indexer/modules/Makefile.am (revision 2701) +++ src/tracker-indexer/modules/Makefile.am (working copy) @@ -16,7 +16,8 @@ $(GIO_CFLAGS) \ $(GLIB2_CFLAGS) \ $(GCONF_CFLAGS) \ - $(GMIME_CFLAGS) + $(GMIME_CFLAGS) \ + $(SQLITE3_CFLAGS) indexer_modules_LTLIBRARIES = \ libtracker-module-applications.la\ @@ -67,6 +68,7 @@ $(GMODULE_LIBS) \ $(GLIB2_LIBS) \ $(GCONF_LIBS) \ - $(GMIME_LIBS) + $(GMIME_LIBS) \ + $(SQLITE3_LIBS) endif Index: src/tracker-indexer/modules/evolution-imap.c === --- src/tracker-indexer/modules/evolution-imap.c (revision 2701) +++ src/tracker-indexer/modules/evolution-imap.c (working copy) @@ -128,7 +128,18 @@ g_free (file->imap_dir); g_free (file->cur_message_uid); -fclose (file->summary); + if (file->db) { + sqlite3_close (file->db); + } + + if (file->stmt) { + sqlite3_finalize (file->stmt); + } + + if (file->summary) { + fclose (file->summary); + } + close (file->fd); G_OBJECT_CLASS (tracker_evolution_imap_file_parent_class)->finalize (object); @@ -482,10 +
Re: [Tracker] Support in Tracker for ultra-new Evolution installs that use SQLite for the summary format
I just posted a new version of the patch which now opens the DB in read-only mode. Thanks for the suggestion On Fri, 2008-12-12 at 15:12 +0800, Jerry Tan wrote: > Hi, philip > > How about to open folders.db in readonly mode? > > I just check sqlite's API. > sqlite3_open_v2 support a para with readonly flag > > > > On 2008ǯ12·î12Æü 00:34, Philip Van Hoof wrote: > > This patch makes ultra-new Evolution installs work again with Tracker. > > > > There's one problem and that is that the query will only find E-mails in > > the INBOX folder. You can easily find the Query and figure out what the > > problem is: > > > > The design that Carlos made assumes that for each folder there's a > > "summary" file. In the new Evolution cache format there's just one > > "folders.db" for each account. > > > > I could do a generated UNION select after first doing "select * from > > folders" on folders.db and then generating a query that includes all > > folders. I just have not done this for now and instead I'm just using > > INBOX and I'm neglecting the other folders. > > > > This is NOT the same as the proposal that I am doing at (a). This is > > instead a ad-hoc solution for the new situation (Evolution using SQLite > > for the summaries). I find this solution rather nasty, to be honest. > > > > (a) http://live.gnome.org/Evolution/Metadata > > > > For Carlos: I have also fixed a serious problem in evolution-pop.c, > > which is by the way unaffected by Evolution's changes (and works, if you > > just apply the patch that I included in this larger patch). The POP > > support's get_message_metadata was not returning metadata. > > > > This was crashing my tracker-indexer (as seemingly my compiler was > > putting "return 0x2" where the return was omitted, and the memory I have > > at 0x2 didn't dereference TrackerModuleMetadata's members very well). > > > > Please review and/or rework the patch. > > > > > > > > > > > > > > ___ > > tracker-list mailing list > > tracker-list@gnome.org > > http://mail.gnome.org/mailman/listinfo/tracker-list > -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] Support in Tracker for ultra-new Evolution installs that use SQLite for the summary format
On Thu, 2008-12-11 at 17:34 +0100, Philip Van Hoof wrote: > This patch makes ultra-new Evolution installs work again with Tracker. > > There's one problem and that is that the query will only find E-mails in > the INBOX folder. You can easily find the Query and figure out what the > problem is: > > The design that Carlos made assumes that for each folder there's a > "summary" file. In the new Evolution cache format there's just one > "folders.db" for each account. > > I could do a generated UNION select after first doing "select * from > folders" on folders.db and then generating a query that includes all > folders. I just have not done this for now and instead I'm just using > INBOX and I'm neglecting the other folders. > > This is NOT the same as the proposal that I am doing at (a). This is > instead a ad-hoc solution for the new situation (Evolution using SQLite > for the summaries). I find this solution rather nasty, to be honest. > > (a) http://live.gnome.org/Evolution/Metadata > > For Carlos: I have also fixed a serious problem in evolution-pop.c, > which is by the way unaffected by Evolution's changes (and works, if you > just apply the patch that I included in this larger patch). The POP > support's get_message_metadata was not returning metadata. Ugh, right. Tracker now compiles with a variety of compiler flags, so this and other errors have been spotted and fixed in trunk > > This was crashing my tracker-indexer (as seemingly my compiler was > putting "return 0x2" where the return was omitted, and the memory I have > at 0x2 didn't dereference TrackerModuleMetadata's members very well). > > Please review and/or rework the patch. > I think I agree with Ivan here, it could be better done as a new TrackerModuleFile implementation, the code to deal with DBs and summaries look quite unrelated (perhaps some functions should be moved to evolution-common.[ch]), and it would help maintenance. In src/tracker-indexer/modules/evolution.c you can see the code that returns the appropriate TrackerModuleFile for the currently scanned mail store, so you could hook there the implementation for newer evolution versions. Regards, Carlos ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list