[Evolution-hackers] help debugging a disk-space leak ...
Hi there, I have (perhaps) an unusal setup - I get mail to an IMAP inbox, and then filter much of it off to local storage, and move the rest manually as I need to. I have around 1000 mostly short, text only messages in my IMAP inbox; and yet: du -m /home/michael/.local/share/evolution/mail/groupwise/mme...@prv1-3.novell.com/folders/INBOX/cur | tail -n 1 2562 /home/michael/.local/share/evolution/mail/groupwise/mme...@prv1-3.novell.com/folders/INBOX/cur 2.5 Gb is a tad large for that. Interestingly, the side-panel says 760 unread, 876 total, whereas the right-click-properties says: Unread messages 760 Total messages 2711 and of course, I've got synchronise for off-line use set. So ... how should I go about debugging that ? I assume that blowing away the entire folders directory there, and just re-building it from the server is the easy/cheating hack-around the problem but ... Thoughts appreciated. Thanks, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Memory leaks
On Mon, 2010-08-23 at 16:32 +0200, Milan Crha wrote: On Mon, 2010-08-23 at 13:10 +0100, David Woodhouse wrote: The fixes I mentioned are sitting in my working tree, and I'll commit and push them today. This is awesome work - thanks ! :-) If they're referenced by a global variable, surely valgrind would know that they're reachable and would not complain about them? I do not know how it works precisely. It should work nicely as David says. I just got a thought, what if some of those possibly lost come from GSlice? Then G_SLICE=always-malloc should help valgrind with it. At least a bit. Right - except for this, which is a disaster area :-) I had forgotten that it was critical to turn this on (which turns off the glib slab allocator) before valgrinding myself ;-) Hmm - I wonder how best to deal with that. I wonder if the glib guys would accept a patch that incorporated /usr/include/valgrind.h's RUNNING_ON_VALGRIND magic ;-) [ amazingly fun etc. ]. Thanks, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] evolution-kolab: Lifting the veil
Hi Christian, You write-up sounds really exciting :-) On Thu, 2010-07-08 at 10:28 +0200, Christian Hilberg wrote: We will try to map as much of Kolab's functionality as possible onto Evolution whithout changing Evolution itself (other than providing a plugin, that is). Especially, we will not touch core Evolution or E-D-S. It would be good to know what we are missing though - if there are new and interesting things that can be added to the front-end. * targeting Linux for client and server *solely*. No specific attempts will be made to be portable to other platforms (like *BSD, Windows, ...). That is quite normal; Fridrich loves to port stuff to Windows ;-) and as you say using the right platform libs should make that easier. * targeting Evo/EDS 2.30 only (2.30.2 presently, we should be able to follow patchlevels). It is a customer requirement that the plugin be used with current stable versions of GNOME and we will not be able to stand API churns or move to GTK+-3 on our development platforms for several reasons, mostly due to development deadlines. Unfortunate, but understandable :-) * providing DEB packages for Debian/GNU Linux only. RPM packages should not be too hard to generate from our debs (or by other means), but we will not be able to provide them. Ah - each distro will package it their own way. Our project will be GPLv2 (or a comparable FLOSS license which will assure at least GPLv2's freedom level ;-). There should not be any licensing issues here. If there are doubts, please feel free to ask for clarification. Soo - personally, I would love to see the code live inside e-d-s and/or Evolution itself: that should make it easier to maintain longer term, adapt for re-factoring, get translators involved etc. Licensing wise, the e-d-s code is currently LGPLv2, and Evolution is LGPLv2 or v3 (at your choice). I would recommend sticking with that, or going LGPLv2+, rather than having a plain GPLv2. Our customer, for whom we will develop this solution, is also very specific about the project being hosted publicly and accessible by everyone. Wonderful :-) If Evolution staff will be willing to host our project sources within Evo git repo, we'll happily transfer our stuff there as soon as we have a first preview ready. Perhaps something to dicuss on #evolution (irc.gimp.net) - but it'd be great to have you working in the same git repo IMHO [ not that it's my decision of course - I defer to Matthew/Chen etc. ;-] ATB, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] alarm-notify ...
Hi guys,
I was reading calendar/gui/alarm-dialog/alarm-notify.c - trying (of
course) to work out why alarm delivery is apparently not working at all.
I just committed some code enabling dbus threading - without which you
can quite happily use dbus from multiple threads, only it will crash and
burn very horribly. That seemed to fix a nasty race I was getting
whereby no alarms showed up at all.
It might also help to undo:
static void
message_push (Message *msg)
{
/* This used be pushed through the thread pool. This fix is made to
work-around
the crashers in dbus due to threading. The threading is not completely
removed as
its better to have alarm daemon running in a thread rather than
blocking main thread.
This is the reason the creation of thread pool is commented out */
msg-func (msg);
}
This sort of thing.
I could also not see where the Repeat functionality (customize an
alarm, and select Repeat the alarm is setup - surely not on the e-d-s
side ? - do we still have it ?
It was also a surprise to me to see the default setting for create an
alarm 15 mins before each appointment to be off - is that a
feature ? :-) it was unexpected.
Finally - it seems some migration code went awry somewhere here, and
busted the use systray setting for the alarm daemon:
https://bugzilla.gnome.org/show_bug.cgi?id=616658
HTH,
Michael.
--
michael.me...@novell.com , Pseudo Engineer, itinerant idiot
___
evolution-hackers mailing list
evolution-hackers@gnome.org
To change your list options or unsubscribe, visit ...
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] edbus branch review ...
On Sat, 2010-02-27 at 09:37 +, Ross Burton wrote: On Fri, 2010-02-26 at 22:31 +, Michael Meeks wrote: Thoughts / flameage ? :-) My thoughts are woohoo and great work. If I'm feeling brave on Sunday I'll start a clean build of eds and evo-edbus and dogfood it. Sooo ... what with the code freezes, and the 'real' interactivity bug fix being in dbus itself: http://cgit.freedesktop.org/dbus/dbus/commit/?h=dbus-1.2id=ae95a0ff9651a774cd983436d7e656c62492659c I suggest that we defer the GDBus integration until we have a GDBus IDL compiler, and the code has been included into glib. HTH, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] looong blockup in the UI ...
Hi guys, I just got an, oh - several second blocking of the UI of Evolution inside: Thread 1 (Thread 0xb5c1a740 (LWP 775)): #0 0xe424 in __kernel_vsyscall () #1 0xb6d56aef in fsync () from /lib/libpthread.so.0 #2 0xb5fc0cbc in write_to_temp_file (err=value optimized out, dest_file=value optimized out, length=value optimized out, contents=value optimized out) at gfileutils.c:978 #3 IA__g_file_set_contents (err=value optimized out, dest_file=value optimized out, length=value optimized out, contents=value optimized out) at gfileutils.c:1082 #4 0xb76c83e4 in shell_view_save_state (shell_view=value optimized out) at e-shell-view.c:235 #5 0xb76c8451 in shell_view_state_timeout_cb (shell_view=0x846cad0 [EMailShellView]) at e-shell-view.c:249 #6 0xb5fd1b06 in g_timeout_dispatch (source=0xa63b1dd0, callback=0, user_data=0x846cad0) at gmain.c:3396 #7 0xb5fd1448 in g_main_dispatch (context=value optimized out) at gmain.c:1960 #8 IA__g_main_context_dispatch (context=value optimized out) at gmain.c:2513 #9 0xb5fd4bf3 in g_main_context_iterate (context=0x8074808, block=1, dispatch=1, self=0x807bf88) at gmain.c:2591 #10 0xb5fd50ba in IA__g_main_loop_run (loop=0x94b2380) at gmain.c:2799 #11 0xb675cc69 in IA__gtk_main () at gtkmain.c:1218 #12 0x0804ab37 in main (argc=134960160, argv=0x1) at main.c:611 This of course is down to the geniuses who decided that we need an 'fsync' in glib's g_file_set_contents (we don't). Does anyone object if instead of g_file_set_contents we just use fopen, fwrite, fclose ? any API method that does fsync is just a disaster-area for latency on a loaded ext3 system; and it is -totally- un-necessary to have some heavy-lifting transactional guarentees for an utter corner-case, for a non-critical file anyway :-) Thanks, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] camel-folder-summary locking ...
On Fri, 2010-02-26 at 22:20 +, Michael Meeks wrote:
I just hit a nasty in camel-folder-summary; suggested patch attached,
seemingly a simple problem of a re-enterancy hazard in the same thread
with a ghashtable:
Ok; this was fixed by Chen's reversion of freeing the uid and NULL'ing
it (recently). However reading the code some more, I think we can do a
lot better, and more cleanly.
I attach the patch; discussed with Chen Srini; I just committed to
gnome-2-28 and master.
HTH,
Michael.
--
michael.me...@novell.com , Pseudo Engineer, itinerant idiot
From c5a86b33ef7d10d25b227221efec8106d8c82ed9 Mon Sep 17 00:00:00 2001
From: Michael Meeks michael.me...@novell.com
Date: Mon, 1 Mar 2010 11:23:18 +
Subject: [PATCH] Fix unpleasant issue with (remove_item) more elegantly. Instead of
taking an untaking the ref-lock thousands of times, and trying to
free messages inside the loop (while cobbering their uid to encourage
message_info_free not to do an unsave hash table removal, inside the
g_hash_table_foreach_remove loop).
Instead - we now just build a list very quickly of the messages we
want to drop. As such, the hash table iteration will be ~instant,
and we do not need to take / drop the lock each iteration, simplifying
and perhaps accelerating the code. We then free the messages later
before releasing the summary lock.
---
camel/camel-folder-summary.c | 25 ++---
1 files changed, 14 insertions(+), 11 deletions(-)
diff --git a/camel/camel-folder-summary.c b/camel/camel-folder-summary.c
index 85265ac..3218a88 100644
--- a/camel/camel-folder-summary.c
+++ b/camel/camel-folder-summary.c
@@ -778,21 +778,13 @@ cfs_count_dirty (CamelFolderSummary *s)
/* FIXME[disk-summary] I should have a better LRU algorithm */
static gboolean
-remove_item (gchar *key, CamelMessageInfoBase *info, CamelFolderSummary *s)
+remove_item (gchar *key, CamelMessageInfoBase *info, GSList **to_free_list)
{
d(printf(%d(%d)\t, info-refcount, info-dirty)); /* camel_message_info_dump (info); */
- CAMEL_SUMMARY_LOCK(info-summary, ref_lock);
if (info-refcount == 1 !info-dirty !(info-flags CAMEL_MESSAGE_FOLDER_FLAGGED)) {
- CAMEL_SUMMARY_UNLOCK(info-summary, ref_lock);
- /* Hackit so that hashtable isn;t corrupted. */
- /* FIXME: These uid strings are not yet freed. We should get this done soon. */
- camel_pstring_free (info-uid);
- info-uid = NULL;
- /* Noone seems to need it. Why not free it then. */
- camel_message_info_free (info);
+ *to_free_list = g_slist_prepend (*to_free_list, info);
return TRUE;
}
- CAMEL_SUMMARY_UNLOCK(info-summary, ref_lock);
return FALSE;
}
@@ -807,6 +799,7 @@ remove_cache (CamelSession *session, CamelSessionThreadMsg *msg)
struct _folder_summary_free_msg *m = (struct _folder_summary_free_msg *)msg;
CamelFolderSummary *s = m-summary;
CamelException ex;
+ GSList *to_free_list = NULL, *l;
CAMEL_DB_RELEASE_SQLITE_MEMORY;
camel_exception_init (ex);
@@ -819,7 +812,17 @@ remove_cache (CamelSession *session, CamelSessionThreadMsg *msg)
dd(printf(removing cache for %s %d %p\n, s-folder ? s-folder-full_name : s-summary_path, g_hash_table_size (s-loaded_infos), (gpointer) s-loaded_infos));
/* FIXME[disk-summary] hack. fix it */
CAMEL_SUMMARY_LOCK (s, summary_lock);
- g_hash_table_foreach_remove (s-loaded_infos, (GHRFunc) remove_item, s);
+
+ CAMEL_SUMMARY_LOCK(s, ref_lock);
+ g_hash_table_foreach_remove (s-loaded_infos, (GHRFunc) remove_item, to_free_list);
+ CAMEL_SUMMARY_UNLOCK(s, ref_lock);
+
+ /* Deferred freeing as _free function will try to remove
+ entries from the hash_table in foreach_remove otherwise */
+ for (l = to_free_list; l; l = l-next)
+ camel_message_info_free (l-data);
+ g_slist_free (to_free_list);
+
CAMEL_SUMMARY_UNLOCK (s, summary_lock);
dd(printf(done .. now %d\n,g_hash_table_size (s-loaded_infos)));
--
1.6.0.2
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] camel-folder-summary locking ...
Hi guys,
I just hit a nasty in camel-folder-summary; suggested patch attached,
seemingly a simple problem of a re-enterancy hazard in the same thread
with a ghashtable:
Thread 121 (Thread 0xa7f37b70 (LWP 23631)):
#0 0xb68c13ef in g_hash_table_resize (hash_table=value optimized out) at
ghash.c:424
#1 g_hash_table_maybe_resize (hash_table=value optimized out) at ghash.c:457
#2 g_hash_table_remove_internal (hash_table=value optimized out) at
ghash.c:982
#3 0xb7c7fdba in message_info_free (s=0x8e7e438, info=0x90f89f0) at
camel-folder-summary.c:3497
#4 0xb7c7fe75 in camel_message_info_free (o=0x90f89f0) at
camel-folder-summary.c:4518
#5 0xb7c7ffce in remove_item (key=0x9142618 217647, info=0x90f89f0,
s=0x8e7e438) at camel-folder-summary.c:788
#6 0xb68c0b2c in g_hash_table_foreach_remove_or_steal (hash_table=0x8f7d0c0 =
{...}, func=0xb7c7ff40 remove_item, user_data=0x8e7e438, notify=1)
at ghash.c:1109
#7 0xb7c85356 in remove_cache (session=0x80ce680, msg=0xb093250) at
camel-folder-summary.c:818
#8 0xb7ca0162 in session_thread_proxy (msg=0xb093250, session=0x80ce680) at
camel-session.c:590
#9 0xb68fb036 in g_thread_pool_thread_proxy (data=0x8aa0cb8) at
gthreadpool.c:265
---Type return to continue, or q return to quit---
#10 0xb68f99ef in g_thread_create_proxy (data=0xb0c1ee8) at gthread.c:635
#11 0xb764c725 in start_thread () from /lib/libpthread.so.0
#12 0xb680626e in clone () from /lib/libc.so.6
Having said that, the code looks a bit nervous (judging by the
comments) in this area. Review much appreciated etc. Reading through the
code there though, I too began to get a bit nervous trying to keep track
of what locks are held.
eg.
camel_folder_summary_migrate_infos
+ doesn't seem to take the summary_lock - why ?
likewise 'message_info_free'
+ called from camel_message_info_free
+ doesn't seem to take the lock ... - why ?
if it is just assumed then:
+ most callers of camel_message_info_free *have* that lock,
+ except eg. camel_folder_summary_remove ...
+ or eg. camel_folder_summary_remove_index_fast
+ or summary_header_save, or message_info_load,
+ or msg_update_preview,
+ or camel_folder_summary_migrate_infos
Soo ... if you read the GType / GObject code - which is similarly
twisted / complicated - then it uses name mangling to denote what lock
is held (personally, I rather like that). Thus you can see by reading
it:
do_foo()
{
take_lock();
do_foo_T();
release_lock();
}
is safe, and
do_foo_T() { do_baa(); }
is essentially not :-) (the '_T' suffix needs propagating). ORBit2 does
the same where things get sticky.
Failing that, the linux kernel seems in places to use comments of
the form:
* The files-file_lock should be held on entry, and will be held on exit.
Clearly, something to reduce the barrier to entry here such that the
code is more readable would be good ;-)
HTH,
Michael.
--
michael.me...@novell.com , Pseudo Engineer, itinerant idiot
diff --git a/camel/camel-folder-summary.c b/camel/camel-folder-summary.c
index 85265ac..787f229 100644
--- a/camel/camel-folder-summary.c
+++ b/camel/camel-folder-summary.c
@@ -778,18 +778,15 @@ cfs_count_dirty (CamelFolderSummary *s)
/* FIXME[disk-summary] I should have a better LRU algorithm */
static gboolean
-remove_item (gchar *key, CamelMessageInfoBase *info, CamelFolderSummary *s)
+remove_item (gchar *key, CamelMessageInfoBase *info, GSList **to_free_list)
{
d(printf(%d(%d)\t, info-refcount, info-dirty)); /* camel_message_info_dump (info); */
CAMEL_SUMMARY_LOCK(info-summary, ref_lock);
if (info-refcount == 1 !info-dirty !(info-flags CAMEL_MESSAGE_FOLDER_FLAGGED)) {
CAMEL_SUMMARY_UNLOCK(info-summary, ref_lock);
- /* Hackit so that hashtable isn;t corrupted. */
- /* FIXME: These uid strings are not yet freed. We should get this done soon. */
- camel_pstring_free (info-uid);
- info-uid = NULL;
- /* Noone seems to need it. Why not free it then. */
- camel_message_info_free (info);
+
+ /* free the entry later */
+ *to_free_list = g_slist_prepend (*to_free_list, info);
return TRUE;
}
CAMEL_SUMMARY_UNLOCK(info-summary, ref_lock);
@@ -807,6 +804,7 @@ remove_cache (CamelSession *session, CamelSessionThreadMsg *msg)
struct _folder_summary_free_msg *m = (struct _folder_summary_free_msg *)msg;
CamelFolderSummary *s = m-summary;
CamelException ex;
+ GSList *to_free, *l;
CAMEL_DB_RELEASE_SQLITE_MEMORY;
camel_exception_init (ex);
@@ -819,7 +817,14 @@ remove_cache (CamelSession *session, CamelSessionThreadMsg *msg)
dd(printf(removing cache for %s %d %p\n, s-folder ? s-folder-full_name : s-summary_path, g_hash_table_size (s-loaded_infos), (gpointer)
[Evolution-hackers] edbus branch review ...
Hi guys, I have a new fun branch (to review for merging) mmeeks-gdbus-import So I am wickedly piling up other misc. fixes in there (so they don't mask other issues) - but I have the edbus code imported, and running - and, indeed, it seems to block rather less nastily than the dbus-glib variants[1]: which is all good. It is essentially the work from origin/treitter-client-gdbus but adding the 'refresh' method, and of course porting it all to the internal copy of 'libedbus.so'. This should let us stick with glib-2.28 (what I'm using), and yet get the benefits of moving closer to what gdbus will be in the end (I hope) - as well as the interactivity wins. I incorporate the fairly simple perl script I used to do the substitution in the first instance. Why am I telling you this ? - I'd love some review of course; I've been playing with calendar addressbook here for a bit; but would greatly appreciate some more input. Particularly since I didn't write the original gdbus port. I'd also -really- like to get this in soonish; Thoughts / flameage ? :-) Thanks, Michael. [1] - which seems to like blocking the mainloop any time any other thread does a synchronous call (not optimal). -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot This is a version of David Zeuthan's gdbus code, intended for inclusion in glib. Until it is included, it helps us to have a copy to work with included in Evolution. Of course, some regexps are required to rename g_dbus - e_dbus etc. to avoid potential symbol conflicts. The code is taken from: git://anongit.freedesktop.org/~david/gdbus-standalone commit 7457cdc8863f08163c48de455e8cd7469b0fe876 With a number of fixes layered on top. Also included is a renamed copy of gvariant from glib commit 1433655e23634446b990d1cf782b22ad6430496a perl script to in-place re-write code using g_dbus: #!/usr/bin/perl -pi.bak s/G_VARIANT/E_VARIANT/g; s/G_TYPE_VARIANT/E_TYPE_VARIANT/g; s/GVariant/EVariant/g; s/g_variant/e_variant/g; s/g_bit_/e_bit_/g; s/g_bus_/e_bus_/g; s/GDBus/EDBus/g; s/G_DBUS_/E_DBUS_/g; s/G_TYPE_DBUS/E_TYPE_DBUS/g; s/g_dbus_/e_dbus_/g; ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Moving from the single mbox file format for the local folders
On Wed, 2009-12-16 at 01:16 +0530, Chenthill wrote: One advantage which I see with #1 is that its a standard way. One thing about both approaches, is that they will consume more space; eg. on my 'Sent' folder with 21k messages - on average (on ext3) we will chew ~2k of space for each of these; which is ~40Mb - around 2%. For my cvs commits mail archive, perhaps the worst case, of 350Mb, (22334 mails) - this would also be around 43Mb - at ~12%. That's not as bad as I was worried about; though of course there is some overhead in terms of inodes and directory entries to worry about that will crank up the overall size - but it doesn't seem horrible even on ext3. Of course - ext4 / btrfs will do a much better job here too; so - less to worry about in future. HTH, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto
Hi Matthew, On Fri, 2009-11-27 at 11:58 -0500, Matthew Barnes wrote: In fact the mail-to-eds effort is part of what's motivating all this. Ah ! - ok :-) sounds good. To make the discussion more concrete, these are my current plans for Camel's extreme makeover. The final API will hopefully feel like a natural extension of GIO. So - I'm well up for that :-) reducing the incidence of wheel re-invention, reducing LOC, increasing familiarity and reducing the skill-set required to hack on the code are all excellent work; I'm really eager to see the good results. Evolution will then be free to call Camel's async methods from the main thread. I'll still have to deal with queuing up operations in order on the Evolution side, but everything will be cancellable (and actually cancel when you tell it to, especially when the network is flaky) and we'll hopefully have far fewer issues with frozen UIs. Does that sound like a more down-to-earth approach? Yes - very sensible; of course, I love step-wise re-factoring, so we don't end up with these massive branches that take forever to merge; ie. merging just the GObjectification, then merging the GIO-ification [ or whatever ]: I guess that could conflict with keeping Camel API stable, but is that even a useful goal ? The only thing that slightly concerns me, is that I am not certain (knowing ~nothing about it) that the Camel API matches well what the Evolution mailer uses, and/or what works in terms of IPC. When I last did the analysis of a good place to cut here, the line went through some of the Evo mailer code, as well as camel. If the idea is to use GObject introspection to provide a useful usable D-BUS interface, I rather suspect that the result will be an over-granular, and round-trippy disaster :-) nevermind exposing lifecycle counting over the wire (which is a total disaster) [ or if not that, then the GObject API will be somewhat unpleasantly chunky to use in-process (if that is a goal) ]. As such, presumably it might be interesting to use the mail / d-bus interfaces to provide the initial bulk of the asynchronicity, to clean up the Evolution UI process. Then, in future - if people want to use the backend API itself in an async fashion from other pieces of code, that can be done as needed (?) It just seems to me that since this is D-BUS and we have to implement these awful in-lined object adaptors, we can at least use the DBusMessage encapsulation to throw all the parameters to a slow operation into it's own thread, before unpacking all them all. That would contrast with the method of implementing all slow D-BUS methods using async calls that then spawn threads - which would require unpacking all arguments, and then carefully re-packing them into various async closures. But I speculate, as always :-) HTH, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto
Hi Matthew, On Wed, 2009-11-18 at 14:07 -0500, Matthew Barnes wrote: With work on Bonobo removal wrapping up, I've finally started taking a closer look at Camel (Evolution's mail storage and networking library) Ah - another life-time of cleaning up, and polishing code: the goal sounds really nice. This is a distant future goal and will have to happen gradually, but I would like Camel to shift to a single-threaded design where all file and network operations directly use or are derived from GIO's asynchronous file and stream APIs. Hmm; you really propose to remove all threading from camel's implementation ? or just from it's API ? a full removal might be problematic. While clearly threading, if done in an un-constrained way, has it's own peculiar problems - there are a couple of obvious advantages: First debugging - while both async and threaded programming loose determinism due to event re-ordering - the debugger at least understands threads - and can hopefully show you your state in a reasonably follow-able way - you can step through slow/blocking calls with 'next', 'finish' etc.; I worry about a world packed with highly granular asynchronous state-machines, all chained together - with no good way to see what is happening, what state everything is in, and (of course) why the app is serenely inactive in it's mainloop suddenly ;-) Secondly - of course, a fully async mode potentially would loose any (possible) benefit of parallelism that in theory threads can provide - in this world of dual-core laptop CPUs, and even hyper-threaded Atoms. OTOH - I'm sure you know what you're doing ;-) ATB, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Status of the DBus port, future plans
On Tue, 2009-01-13 at 17:36 +, Chenthill wrote: On Tue, 2009-01-13 at 14:50 +, Ross Burton wrote: At present it contains a minimal port of the addressbook part of eds-dbus to a fairly current (~1 week old) EDS tree. This mostly works and after a little cleanup should be ready for more testing. Nice. So, the gist of this rambling message is this: to merge the DBus port in the current state I'd need to add a context argument to all of the methods in e-cal-backend. This will break API and I'll obviously be fixing the backends which are shipped as part of EDS at the same time. The good news is that the e-cal-backend-sync helper class effectively shields users of that from the change, so this affects less backends than you'd expect. Sounds rather good. Any comments? As long as you don't break the addressbook API, or ABI I'm happy enough ;-) Those are the only bits OO.o uses, and by ABI I mean the behaviour of libebook-1.2.so.9 and it's name / version [ configure.in LIBEBOOK_CURRENT etc. ]. HTH, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] camel-folder-summary.c - 64bit-ness ...
Hi guys,
I was just trying to reproduce some migration performance tests with my
mbox and summary data rsync'd from a 32bit machine to a 64bit machine.
Surprisingly this appears to crash immediately. Looking at the
camel-file-utils.c code I was surprised to see simultaneously an
apparent concern for network byte ordering:
camel_file_util_encode_fixed_int32 (FILE *out, gint32 value)
{
guint32 save;
save = g_htonl (value);
if (fwrite (save, sizeof (save), 1, out) != 1)
return -1;
return 0;
}
and also things like:
CFU_ENCODE_T(time_t)
that appear to generate data based on the sizeof the platform's time_t
- on my 64bit machine time_t is 8 bytes, on 32bit it is only 4.
Presumably this summary code is made obsolete by the new SQLite summary
code - and modulo some data as to what architecture a file was written
by it's perhaps less than obvious how to fix this. Also - why we're not
using fgetc_unlocked in these tight loops I don't know.
I guess I need an old evo. version to re-build all my summaries for
64bit now; or am I barking up the wrong tree ?
Thanks,
Michael.
--
michael.me...@novell.com , Pseudo Engineer, itinerant idiot
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] [Tracker] [Evolution] Beagle and Tracker, letting Evolution feed those beasts RDF triples instead
Hi Mikkel, On Sat, 2008-12-13 at 00:18 +0100, Mikkel Kamstrup Erlandsen wrote: Is it that big a problem? I mean if you store 100,000 uris of avg. length 50 chars you will have a file about 5mb... One needs only keep an absolute minimum amount of metadata around. Well, true - it's not that much data I suppose - but without an agreed lifecycle mechanism it grows without bound as you delete mail, and since we have to do a lookup prune for every new UUID we generate [ or arguably emit ], there are potential size performance issues prolly worth addressing (in some simple way) in the design. HTH, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] [Evolution] Beagle and Tracker, letting Evolution feed those beasts RDF triples instead
Hi Philip, On Wed, 2008-12-10 at 12:49 +0100, Philip Van Hoof wrote: What does the lifecycle for the data in that Unset store look like ? I think the LifeCycle is best described by this document: http://live.gnome.org/MetadataOnRemovableDevices It specifies a metadata cache format for removable devices in Turtle format. Not read that before; I just read it - and, as you say here is how things are removed: For your information when reading the document: The removal of a resource as a special notation using blank resources , and the removal of a predicate (of a field of a resource) uses the notation pfx:predicate . Sure - so, that is fine - it's a representational detail of how removals are stored. My concern is not that we can't represent removals well - but that the life-cycle of that removal information is undefined. Say eg. we install beagle, and tracker - but we never run beagle. Then we have two parties that have registered an interest in changes. If we run beagle only every year or so - we need to know all mails that were deleted since a year ago. Unfortunately, perhaps we never run it again. Does that mean we endlessly accumulate in some monster journal a huge list of 'UnSets' ? For a cache it's important to know the modified timestamp so that you know whether your copy of the metadata is most recent, or the cache is about the resource is most recent. Sure - I buy the timestamp thing; that's all great. - When a resource got deleted then the RDF store wants to know about this as soon as possible. Asynchronously (like if the RDF store, being a subscriber, joins the subscription after the deletion took place) this also counts: as soon as possible. Preferably immediately after the subscription. Sure - so my problem is the life-cycle of the store of deletion information: how long do we grow that list for, if people eg. turn off the search client after finding it chews more resource than they had hoped on their small machine :-) With IMAP there's a trick that you can do: you can assume that a hole in the UIDSET meant that some sort of deleting occurred. Sounds interesting. I think, anyway, that it would make sense for Evolution to start doing two things in the CamelDB: Agreed. * Log all deletions (just the UID should suffice), if the service reuses UIDs then upon effective reuse of the UID, this log's UID deletion should be removed from the log. Else you loose the E-mail at whoever depends on this log for knowing about effective deletions. So there is at least some bound to the growth of the deleted UUID log ;-) which is the size / likelyhood of re-use in the UUID space. It's hard to think of solutions that are that satisfying; but - perhaps something like cropping the deletion log-size at a percentage of stored mail size, with some log overflow type message to flag that; or having some arbitrary size bound on it, or more carefully disabling logging when search services are disabled, or ... having only a single client, or warning the user that they should run their search service some more, or perhaps even coupling the indexing piece more closely to the mailer itself somehow. HTH, Michael. -- [EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Getting rid of shipped libdb
On Mon, 2008-10-06 at 18:35 +0100, Rob Bradford wrote: Okay. Have you got these details? It would be good to see which of those still apply, etc.. Sure - the original rational here (AFAIR) is quite simple. If you share the same .evolution across multiple machines, and the version of libstupid is different: bad luck, you loose your contacts. Basically all database-y things seem to love back-compatibility ( if you're lucky ), but the idea of forward compatibility seems to completely bypass them; the concept that the functionality is good enough already, and doesn't require further file-format breakage is apparently not present ;- AFAIR the addressbook usages of libdb for the local addressbook were annoying enough in previous instances that we moved to having an internal version. What I'd love to see instead, is a one-shot migration to a simple plain text, authoritative file with the contacts and then (perhaps) optionally a binary cache I guess. But for the volume of data there, presumably slurping it and grokking it with some small / simple piece of code would be rather more efficient. Ultimately contact searches are in response to user-input, so we have loads of time to do work there. HTH, Michael. -- [EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Getting rid of shipped libdb
Hi Rob, On Tue, 2008-10-07 at 16:49 +0100, Rob Bradford wrote: Hhh. But. The use case you outlined directly above about where this goes wrong also applies here: Oh. You ran e-d-s on a machine with a version that migrates it to Some Other Format (tm). You then add some contacts which go into the new format. Then you go back to your old machine. *Boing*. Same problem, your new contacts are missing :-( Yes true - but at least, this is a once-and-for-all fix ;-) and of course there is no reason to say that we can't sync them to the old db format too for a while. I would expect migrating from one version of GNOME and then back again is probably pretty problematic anyway...ultimately I think you need to draw the line at some point. True - but having a problem at -every- version point, and across ~every distribution such that I used SUSE and now I can't switch back to FooBaked Linux ! is not good ;-) Somewhat orthogonally: Also, I wonder on the performance impact of the flat-file approach wrt, modification / deletion when dealing with ~1k contacts. Fast enough I guess; I have ~3k real-life contacts, and that is ~1.5Mb of addressbook.db [ which seems pretty much a flat vcard file when you read it ;-]. It seems my pmap is struggling to work ATM, but it'd be interesting to know how much of addressbook.db is 'hot' in e-d-s anyway. HTH, Michael. -- [EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] [opensuse-gnome] Evolution 2.22 for Factory
On Sat, 2008-09-20 at 00:47 -0500, Hans Petter Jansson wrote: Wouldn't it be possible to use a different directory, e.g. mail/local-index/folders.db? That would avoid both problems. That seems like a great idea to me; at least - if the 'hot' data set of evolution is normally just the summary information, then keeping that in a single directory [ ie. mangle the path names into file names ] would make it rather more likely to be contiguous on disk too which might be nice: ~/.evolution/mail/summaries/local-Inbox-suse-kernel.db # etc. ;-) I still think relocating folders.db is a better option, since it would work for everyone without necessitating an upgrade. Sounds best to me, if we can do it that is. Regards, Michael. -- [EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Evolution: Taking forward...
On Sat, 2008-07-12 at 13:56 +0300, Felipe Contreras wrote: Could you consider talking with the samba guys to make libmapi LGPLv3? That would make things much easier, right? Yep - would make life easier; OTOH I've spoken to them in the past along these lines - but though they were initially interested some years ago, sadly I think they are less interested today. OTOH - it's always worth asking again I guess. HTH, Michael. -- [EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Evolution: Taking forward...
Hi Paul, On Fri, 2008-07-11 at 15:30 +0200, Paul Bolle wrote: * Move Evolution licensing to LGPL v2 and LGPL v3 to let us re-use the code more easily around the platform. Did you mean LGPLv2 _or_ LGPLv3 here? Yes; it's dual licensed - which gives people rather a choice of licenses, GPLv2, GPLv3, LGPLv2, LGPLv3 etc. Anyway, I am not familiar with libmapi, but it does indeed seem to be (what is commonly referred to as) a library that is licensed under the GPLv3. According to the FSF you can't release a project under the LGPLv2 or LGPLv3 if you want to use a library under GPLv3. You'll have to convert your LGPLv2 or LGPLv3 code to the GPLv3 [1]. Sure - so, IANAL etc. however I don't see a problem with a LGPLv3 / GPLv3 for the samba piece - I imagine, linking that in as a plugin makes Evo, with the current structure as a whole GPLv3; but without it potentially LGPLv2/GPLv2/LGPLv3 :-) AFAICS that gives some flexibility; but of course it's possible there are yet further un-anticipated problems ? :-) In the longer term, I'd prefer to see the samba piece isolated into a separate process. HTH, Michael. -- [EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] massive e-d-s memory leak persuit ...
Hi dudie, So - I started to look at the e-d-s memory explosion situation quickly, took a nice dump from gdb, ran strings on it and the heap has a ton of strings around the place (as you would expect) - [ currently running at only ~60Mb strings /tmp/eds-heap | sort | uniq -c | sort -n gives me: 1666 -CONTACT-UID 1666 -NAME 1736 ION-DEST-NAME 1894 OLUTION-BOOK-URI 2100 -EMAIL 2184 ION-DEST-EMAIL 2318 OLUTION-FILE-AS 2506 OLUTION-LIST 2992 ION-LIST 3058 comp 3321 OLUTION-DEST-EMAIL 3329 OLUTION-DEST-CONTACT-UID 3993 OLUTION-DEST-NAME 4534 pwise://[EMAIL PROTECTED]/;Novell GroupWise Address Book 5343 BEGIN:VCARD 5372 ION-DEST-EMAIL 5504 END:VCARD 5505 VERSION:3.0 6786 ION-DEST-NAME 8606 para 12739 ION-DEST-CONTACT-UID 13642 OLUTION-DEST-CONTACT-UID 18082 OLUTION-DEST-NAME 19252 OLUTION-DEST-EMAIL 21991 prop 32508 ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION; 40253 pA, Where the first column is the count ... 32508 copies of that ATTENDEE string seems a little excessive, as do the (apparently mangled?) OLUTION-DEST-... strings. Does that provide any insight wrt. code to audit for this huge leak ? apparently it afflicts everything from SLED10-SP1 onwards. Also - in general to reduce the (high) e-d-s memory usage, should we be using GQuarks for some of these field names as we store them ? Thanks, Michael. -- [EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Synching Evolution/GNOME version
Hi Matthew, On Thu, 2007-10-04 at 10:00 +0530, Srinivasa Ragavan wrote: And while we're at it, can we please drop the meaningless -1.2 suffix from the library names (e.g. libedataserver-1.2.so)? As far as I can tell this is just an artifact from an age before the EDS sonames were properly versioned. Cue, screaming ... please read: http://lxr.go-oo.org/source/dba/connectivity/source/drivers/evoab2/EApi.cxx#046 and reconsider ;-) be aware that it takes weeks to months to get that change up-stream, and months to get a new version of OO.o out, and all the while OO.o will ~silently fail to work with people's new 'clean' renamed library e-d-s :-) AFAIR the name versioning was originally intended to ensure you could compile develop multiple versions of evo. on the same system. I don't think those application would be happy to do this. There are a few apps that use the .so directly without pkgconfig (iirc OpenOffice.org). Quite - we build our OO.o integration (which since it is up-stream has to run on ~all existing systems - cf. the ISV problem ...), with internal headers (to unwind the ABI breakage), and explicit dlopening hooking out of symbols etc. Unless there is a clear nod from the stake holders of those projects, I wouldn't be favor of doing this. Thanks ! :-) Of course, as long as the dlopens still work, I'm fine with renaming the core library (to suit people's personal hygiene issues) as long as we have a compat symlink in perpetuity ;-) Thanks, Michael. -- [EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] More e-d-s ABI breakage ?
Hi Ross,
On Fri, 2006-08-11 at 17:32 +0100, Ross Burton wrote:
The change was required for the 770 as ferrying the photos over the bus
Sure - I like the core change, but not the API/ABI detail :-)
If it's required it might be possible to revert that change and
introduce an alternative. As you say, a photo2 property that returns
a new type should be sufficient, I'll have a look later.
Sure, although of course it may be more complex than that; this was a
1st glance review.
I have Grand Plans to write a replacement for EVCard and EContact
though. I've been adding new API to e-vcard to make it more usable, but
that is just complicating the API for no gain. Extending EContact is a
pain as all of the data types are public structs with no padding.
Sounds lovely. From an OO.o perspective we want a thread-safe 'cursor /
iterator' API ;-) as in:
for (line = do_query_get_1st; line; line = line-getnext()) {
...
}
but I guess we handle all our async stuff via threads.
EContact also manages to hit that sweet spot between
complicated-but-powerful and easy-but-limited, so it's
complicated-and-limited at once.
Heh ;-) I'm sure there are (no doubt) tons of problems with the API -
however, a drip, drip of incompatible breakage is no solution IMHO.
Clearly building your nice new API in parallel, deprecating the old API,
and [ in a few versions time ] doing a big bang switch to the new (by
now perfect, tested, GObject based, extensible etc. ;-) API seems a
better approach. Hopefully we can write an 'evoab3' connector to it
then.
All the best,
Michael.
--
[EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Something screwy when Evolution Shell invokes bonobo_activation_active_server_unregister()
On Mon, 2006-06-19 at 00:18 +0300, Tor Lillqvist wrote: I think I have a good guess now: The problem is that all sockets by default are inherited by child processes in Windows. (Like file descriptors in Unix.) Ah - we had some wonderful b-a-s bugs for the few sockets we didn't CLOEXEC in the past. Of course, we still allowed b-a-s clients to inherit stdout/in/err (at least, until the g_spawn rework (?)) - but that is sometimes useful. HTH, Michael. -- [EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] Gender ...
So, At the risk of offending the gender-confused; it would be -extremely- useful to have a Gender boolean in the evolution addresbook [ of course, perhaps there is I'm just missing it but I dug into the code ]. The reason is OO.o will vary it's salutation on gender; ie. Dear Mrs. Foo Dear Mr. Foo etc. - now one can argue whether this is broken etc. but there it is ;-) the current ergonomics are rather built around this - and, you can see that such a boolean internationalizes rather nicely. So - the question is: can we have it ? currently the OO.o mail merge is rather feeble without it. Thanks, Michael. -- [EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] e-d-s pokeage ...
Hi there, So - I was pleased to see that the libebook .so remains unchanged from evo 2.4 to 2.6 - and I just did a little review to try to ensure that this indeed reflects an unchanged ABI ;-) It seems that is the case - which is great thanks - I need only to update a comment in OO.o ;-) however, as with all reviews I happened on some other things: libebook: + e-name-western.h: urgh not good practice + do we really want to export this structure ? surely an opaque type accessors is ~always better ? + e-name-western-tables.h: + looks broken to me: * if you include this header it will instantiate this large set of strings in each module / shlib. * all such lists of strings should prolly be defined 'const' so we put them in a shared linker section, to save memory. * I imagine the header should have 'extern const char *' and just the symbol names + why is this installed anyway ? are there better ways to expose this ? libedataserver/e-data-server-module.c + you add: - module-library = g_module_open (module-path, 0); + module-library = g_module_open (module-path, G_MODULE_BIND_LAZY); in moving to 2.6 - which is some optimization; of course - it would be rather better to use: (G_MODULE_BIND_LAZY | G_MODULE_BIND_LOCAL) instead. What that does is ensure that the plugin is not included in the search path for symbols for other plugins - ie. subsequent plugins as they link don't have to search this plugin for symbols. Assuming your plugins don't depend on each other to provide symbols [ pretty broken IMHO ] this not only accelerates linking, but prolly helps avoid various potential tangled linking problems. HTH thanks for not breaking the ABI again :-) Regards, Michael. -- [EMAIL PROTECTED] , Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
