Re: [Tracker] trackerd memory efficiency
On Fri, Jan 23, 2009 at 2:52 PM, Philip Van Hoof wrote: > On Fri, 2009-01-23 at 14:41 +0200, Tshepang Lekhonkhobe wrote: >> On Fri, Jan 23, 2009 at 2:36 PM, Carlos Garnacho wrote: >> > On vie, 2009-01-23 at 14:08 +0200, Tshepang Lekhonkhobe wrote: >> >> Hi, >> >> >> >> README champions the daemon as memory efficient (3-30MB RAM) and I'd >> >> like to send a patch that removes the whole paragraph since that's no >> >> longer the case. >> >> >> > >> > Could you back up such statement with valgrind logs? If that's truly not >> > the case, patches to make that paragraph true would be more appreciated, >> > really >> >> No, but am just looking at top that tells me more of 35MB currently. > > Note that top usually reports VmSize. VmSize is all of the memory being > used by the process. This is of course an irrelevant figure as that > includes the shared memory too (memory shared with other processes, like > libraries being mapped into memory already). > > What you want to look at it usually VmRss. That's a far more interesting > number to know. > > You can get the VmRss from the kernel by reading the /proc entries. > > Valgrind will only report what malloc() & co have allocated. Also that > ain't a correct number as a kernel wont always immediately allocate the > entire requested size. Instead it creates pages as the memory that was > requested is being written to. > > So if you do malloc(10M), then you won't be using 10M until you start > writing to all of the bytes of the 10M. > > The extractor of tracker will for example request a rather large > allocation this way, but we assume the kernel to only really alloc as > pages are requested (to avoid having to excessively do realloc instead). > > So just looking at top is definitely not a good way to measure. What I meant was the RES column in top, just to be clear. I anyways ran $ valgrind --leak-check=full --show-reachable=yes --leak-resolution=high -v /usr/libexec/trackerd and go the following output, and I don't know what it says: ==6792== LEAK SUMMARY: ==6792==definitely lost: 36 bytes in 1 blocks. ==6792==indirectly lost: 120 bytes in 10 blocks. ==6792== possibly lost: 43,872 bytes in 84 blocks. ==6792==still reachable: 6,253,647 bytes in 26,607 blocks. ==6792== suppressed: 4,496 bytes in 110 blocks. --6792-- memcheck: sanity checks: 29089 cheap, 220 expensive --6792-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use --6792-- memcheck: auxmaps_L1: 0 searches, 0 cmps, ratio 0:10 --6792-- memcheck: auxmaps_L2: 0 searches, 0 nodes --6792-- memcheck: SMs: n_issued = 856 (13696k, 13M) --6792-- memcheck: SMs: n_deissued= 7 (112k, 0M) --6792-- memcheck: SMs: max_noaccess = 65535 (1048560k, 1023M) --6792-- memcheck: SMs: max_undefined = 35 (560k, 0M) --6792-- memcheck: SMs: max_defined = 1348 (21568k, 21M) --6792-- memcheck: SMs: max_non_DSM = 849 (13584k, 13M) --6792-- memcheck: max sec V bit nodes:1581 (80k, 0M) --6792-- memcheck: set_sec_vbits8 calls: 51138 (new: 1581, updates: 49557) --6792-- memcheck: max shadow mem size: 13968k, 13M --6792-- translate:fast SP updates identified: 48,718 ( 88.1%) --6792-- translate: generic_known SP updates identified: 4,920 ( 8.8%) --6792-- translate: generic_unknown SP updates identified: 1,647 ( 2.9%) --6792-- tt/tc: 32,955,230 tt lookups requiring 50,735,061 probes --6792-- tt/tc: 32,955,230 fast-cache updates, 10 flushes --6792-- transtab: new35,700 (767,467 -> 11,066,732; ratio 144:10) [0 scs] --6792-- transtab: dumped 0 (0 -> ??) --6792-- transtab: discarded 355 (7,022 -> ??) --6792-- scheduler: 2,835,614,942 jumps (bb entries). --6792-- scheduler: 29,089/43,763,705 major/minor sched events. --6792--sanity: 29090 cheap, 220 expensive checks. --6792--exectx: 12,289 lists, 11,820 contexts (avg 0 per list) --6792--exectx: 9,597,286 searches, 9,605,875 full compares (1,000 per 1000) --6792--exectx: 0 cmp2, 189 cmp4, 30,892,554 cmpAll --6792-- errormgr: 1,500 supplist searches, 190,814 comparisons during search --6792-- errormgr: 83 errlist searches, 199 comparisons during search -- my place on the web: floss-and-misc.blogspot.com ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] trackerd memory efficiency
On Fri, 2009-01-23 at 14:41 +0200, Tshepang Lekhonkhobe wrote: > On Fri, Jan 23, 2009 at 2:36 PM, Carlos Garnacho wrote: > > On vie, 2009-01-23 at 14:08 +0200, Tshepang Lekhonkhobe wrote: > >> Hi, > >> > >> README champions the daemon as memory efficient (3-30MB RAM) and I'd > >> like to send a patch that removes the whole paragraph since that's no > >> longer the case. > >> > > > > Could you back up such statement with valgrind logs? If that's truly not > > the case, patches to make that paragraph true would be more appreciated, > > really > > No, but am just looking at top that tells me more of 35MB currently. Note that top usually reports VmSize. VmSize is all of the memory being used by the process. This is of course an irrelevant figure as that includes the shared memory too (memory shared with other processes, like libraries being mapped into memory already). What you want to look at it usually VmRss. That's a far more interesting number to know. You can get the VmRss from the kernel by reading the /proc entries. Valgrind will only report what malloc() & co have allocated. Also that ain't a correct number as a kernel wont always immediately allocate the entire requested size. Instead it creates pages as the memory that was requested is being written to. So if you do malloc(10M), then you won't be using 10M until you start writing to all of the bytes of the 10M. The extractor of tracker will for example request a rather large allocation this way, but we assume the kernel to only really alloc as pages are requested (to avoid having to excessively do realloc instead). So just looking at top is definitely not a good way to measure. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] NEWS and TODO patches
On Thu, Jan 22, 2009 at 6:25 PM, Tshepang Lekhonkhobe
wrote:
> Hi,
>
> After applying the TODO diff, will you also update that file, the
> section on removable media. Thanks...
will you rather apply this patch instead
--
my place on the web:
floss-and-misc.blogspot.com
Index: docs/manpages/trackerd.1
===
--- docs/manpages/trackerd.1 (revision 2825)
+++ docs/manpages/trackerd.1 (working copy)
@@ -1,4 +1,4 @@
-.TH trackerd 1 "July 2007" GNU "User Commands"
+.TH trackerd 1 "Jan 2008" GNU "User Commands"
.SH NAME
trackerd \- indexer daemon for tracker search tool
@@ -97,6 +97,7 @@ needs a
to be running within your current session.
.SH SEE ALSO
+.BR tracker-applet (1),
.BR tracker-search-tool (1),
.BR tracker-search (1),
.BR tracker-tag (1),
Index: README
===
--- README (revision 2825)
+++ README (working copy)
@@ -13,9 +13,10 @@ Table of Contents
6.1 Usage
6.2 Setting Inotify Watch Limit
6.3 Tracker files
-7 Tracker & Nautilus Search
-8 Tracker & Deskbar applet
-9 Tracker tools
+7 Integration with other tools
+7.1 Nautilus search
+7.2 Deskbar applet
+8 Tracker tools
1 Introduction
@@ -42,11 +43,12 @@ Table of Contents
Supported first class objects include:
* Files, Documents, Music, Images, Videos, Applications, Emails,
-Conversations, History, Playlists
+Conversations, Playlists
Planned support:
- * Appointments, Contacts, Projects, Tasks, Bookmarks, Notes
+ * Appointments, Contacts, Projects, Tasks, Bookmarks, Notes,
+Firefox Web History
All discussion related to tracker happens on the Tracker
mailing list
@@ -343,8 +345,10 @@ Table of Contents
session data.
-7 Tracker & Nautilus Search
+7 Integration with other tools
+7.1 Nautilus Search
+
Once you have installed Tracker and have some indexed contents,
you should now compile Nautilus (ver 2.13.4 or higher) which
should auto-detect that Tracker is installed and automatically
@@ -358,17 +362,26 @@ Table of Contents
/usr/bin/trackerd). For non-GNOME installations, see the
desktop docs for how to achieve similar.
+7.2 Deskbar applet
-8 Tracker & Deskbar applet
+ Tracker is also integrated in GNOME's deskbar applet.
+
+ Here are the compile options on how to get it built:
- Tracker is also integrated in GNOME's deskbar applet. See
- Compile Options above on how to get it built.
+--enable-deskbar-applet=(auto, handler, module) : enables
+Deskbar-applet support; 'auto' should do since it automatically
+chooses whether to install the 'handler' (for Deskbar-applet
+>=2.16) or the 'module' (for Deskbar-applet >=2.19)
+
+--with-deskbar-applet-dir=(directory where Deskbar-applet
+should find tracker-handler; this should be automatically
+detected, perhaps in
+/usr/lib/deskbar-applet/{handlers,modules-2.20-compatible})
-9 Tracker Tools
+8 Tracker Tools
The Tracker distribution comes with a number of useful utilities which
are listed at the bottom of trackerd man page, "man trackerd". It's
recommended that you use the "--help" command line switch
- (e.g., "tracker-services --help") for a more up-to-date usage info.
-
+ (e.g., "tracker-services --help") for more up-to-date usage info.
Index: TODO
===
--- TODO (revision 2825)
+++ TODO (working copy)
@@ -1,28 +1,32 @@
General:
-
* XESAM integration
-* Indexer backend independence (not just rely on SQLite)
* Automatic language detection
Application support:
-
* Balsa mboxes
* Brasero integration
* Epiphany integration (bookmarks and history)
* Rhythmbox integration
* Tomboy notes indexing
+* Modest, a mobile email client (much work already exists in svn trunk)
Search:
-
* Search filtering (e.g. search within specific folders)
* Search results that update in real time
* Search within archives
-* "exact phrase" search
-* Wildcard search
+Internal:
+* Crawler to move to Indexer
+* FTS support (replacement of QDBM, the current index db)
+ -> enables "exact phrase" searches
+ -> enables wildcard searches
+ -> enables prefix searches (pre*)
+* Denormalised tables
+* Better libtracker support for Maemo platform applications?
-Tracker SoC project ideas:
+2007 Tracker SoC project ideas:
+
1. HAL integration into Tracker
* We need to be able to handle files in removable media
Index: NEWS
===
--- NEWS (revision 2825)
+++ NEWS (working copy)
@@ -12,15 +12,18 @@ Tracker 0.6.90
GLib's GIO usage
* In-house file monitoring dropped.
* Addition of unit tests.
- * TST (GUI search) migrated away from the deprecated gnome-vfs2
+ * TST (GUI search) migrated away from the deprecated gnome-vfs2.
Miscellaneous
- * Adde
Re: [Tracker] trackerd memory efficiency
On Fri, Jan 23, 2009 at 2:36 PM, Carlos Garnacho wrote: > On vie, 2009-01-23 at 14:08 +0200, Tshepang Lekhonkhobe wrote: >> Hi, >> >> README champions the daemon as memory efficient (3-30MB RAM) and I'd >> like to send a patch that removes the whole paragraph since that's no >> longer the case. >> > > Could you back up such statement with valgrind logs? If that's truly not > the case, patches to make that paragraph true would be more appreciated, > really No, but am just looking at top that tells me more of 35MB currently. -- my place on the web: floss-and-misc.blogspot.com ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] trackerd memory efficiency
On vie, 2009-01-23 at 14:08 +0200, Tshepang Lekhonkhobe wrote: > Hi, > > README champions the daemon as memory efficient (3-30MB RAM) and I'd > like to send a patch that removes the whole paragraph since that's no > longer the case. > Could you back up such statement with valgrind logs? If that's truly not the case, patches to make that paragraph true would be more appreciated, really ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
[Tracker] trackerd memory efficiency
Hi, README champions the daemon as memory efficient (3-30MB RAM) and I'd like to send a patch that removes the whole paragraph since that's no longer the case. -- my place on the web: floss-and-misc.blogspot.com ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] status of GIO regarding TST
Ivan Frade wrote: Hi, El vie, 23-01-2009 a las 13:02 +0200, ext Tshepang Lekhonkhobe escribió: On Fri, Jan 23, 2009 at 1:01 PM, Cosimo Cecchi wrote: On Fri, 2009-01-23 at 12:56 +0200, Tshepang Lekhonkhobe wrote: Hi, Using lsof revealed that TST still uses gnome-vfs. Last I checked we moved to GIO so me wonders what going on? here's the relevant output: $ tracker-s 14844wena mem REG8,3 57328 606773 /usr/lib/libgvfscommon.so.0.0.0 GVfs is not gnome-vfs, it's a set of extensions for GIO, supporting other backends than local files. Sorry, that was a bit careless. I meant to attach the following piece: tracker-s 14844wena mem REG8,3 368128 423102 /usr/lib/libgnomevfs-2.so.0.2200.0 I did a quick check into tracker-search-tool code, and we are not using gnome_vfs functions anywhere. Indeed we dont have a gnome-vfs dependency in the configure.ac... I guess we are linking with libgnomeui and that library links to libgnomevfs and other useless components (for us) like libbonoboui. I tend to agree with Ivan here. To see if we link with a library, you should really use ldd: $ ldd /usr/bin/tracker-search-tool |grep -i gio libgio-2.0.so.0 => /usr/lib/libgio-2.0.so.0 (0x7f75abaa) $ ldd /usr/bin/tracker-search-tool |grep -i gnomevfs libgnomevfs-2.so.0 => /usr/lib/libgnomevfs-2.so.0 (0x7f79b6366000) To see if we use any GnomeVFS or other library functions, you can use nm or objdump: $ nm /usr/bin/tracker-search-tool |grep -i g_file_new_for_path U g_file_new_for_pat $ nm /usr/bin/tracker-search-tool |grep -i gnome_vfs $ -- Regards, Martyn ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] about WebHistory
Hi Tshepang, El vie, 23-01-2009 a las 13:14 +0200, ext Tshepang Lekhonkhobe escribió: > Hi, > > What's up with Firefox webhistory support? Does not work here. > WebHistory should be reimplemented using the indexer-modules API. Regards, Ivan ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] Tracker and SQLite
Tshepang Lekhonkhobe wrote: On Fri, Jan 23, 2009 at 12:12 PM, Martyn Russell wrote: Tshepang Lekhonkhobe wrote: On Thu, Jan 22, 2009 at 10:50 PM, Martyn Russell wrote: If we wanted to write another backend, we would have a lot less work than before of course, but the connection management would need writing. This part is quite tricky for SQLite. We can write a more generic interface here if we need to and if we find it makes sense depending on future requirements. Is there things specific to Tracker that SQLite is incapable of providing that would necessitate such a move in future? That is, do you currently to hack around such inadequacies, or is SQLite (with its FTS) up to the task? There are no hacks for this. It is simply that we have multiple databases and they are not all stored in the same place. That coupled with the fact that some databases have to be attached to other databases to be usable, means we have complex management code depending on what part of Tracker needs database access and for what use (reading/writing). If this was all done using one database in PostgreSQL, for example, then this management of the databases changes somewhat. So I suppose SQLite is limited in that sense (Me not too familiar with db tech, so excuse me for poking too hard)? Well, the same limitation is also a strength in other areas. This is something that you have to evaluate of course when you choose this model. You could instead just use one SQLite database instead of many if you wanted to. What of networked Tracker? I just browsed through SQLite docs and that makes me think there'll have to be a move to a client/server db in such a case since SQLite wouldn't be suitable if multiple clients connected to it, especially if those clients had write access? This is true. But we don't have multiple clients connecting to the database itself, just the daemon which talks to the database. This is needed because the daemon knows about "state" and can control things from there too. When I say state, this includes a myriad of things, battery state, removable media state, paused state, etc, etc. -- Regards, Martyn ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
[Tracker] about WebHistory
Hi, What's up with Firefox webhistory support? Does not work here. -- my place on the web: floss-and-misc.blogspot.com ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] status of GIO regarding TST
Hi, El vie, 23-01-2009 a las 13:02 +0200, ext Tshepang Lekhonkhobe escribió: > On Fri, Jan 23, 2009 at 1:01 PM, Cosimo Cecchi wrote: > > On Fri, 2009-01-23 at 12:56 +0200, Tshepang Lekhonkhobe wrote: > >> Hi, > >> > >> Using lsof revealed that TST still uses gnome-vfs. Last I checked we > >> moved to GIO so me wonders what going on? > >> > >> here's the relevant output: > >> > >> $ tracker-s 14844wena mem REG8,3 57328 > >> 606773 /usr/lib/libgvfscommon.so.0.0.0 > > > > GVfs is not gnome-vfs, it's a set of extensions for GIO, supporting > > other backends than local files. > > Sorry, that was a bit careless. I meant to attach the following piece: > tracker-s 14844wena mem REG8,3 368128 > 423102 /usr/lib/libgnomevfs-2.so.0.2200.0 I did a quick check into tracker-search-tool code, and we are not using gnome_vfs functions anywhere. Indeed we dont have a gnome-vfs dependency in the configure.ac... I guess we are linking with libgnomeui and that library links to libgnomevfs and other useless components (for us) like libbonoboui. Regards, Ivan ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
[Tracker] about required GLib
Hi, Shouldn't we get rid of configure.ac variable, GLIB_WITH_UNIT_TESTING, since minimum required version (2.26) has that capability? -- my place on the web: floss-and-misc.blogspot.com ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] status of GIO regarding TST
On Fri, Jan 23, 2009 at 1:01 PM, Cosimo Cecchi wrote: > On Fri, 2009-01-23 at 12:56 +0200, Tshepang Lekhonkhobe wrote: >> Hi, >> >> Using lsof revealed that TST still uses gnome-vfs. Last I checked we >> moved to GIO so me wonders what going on? >> >> here's the relevant output: >> >> $ tracker-s 14844wena mem REG8,3 57328 >> 606773 /usr/lib/libgvfscommon.so.0.0.0 > > GVfs is not gnome-vfs, it's a set of extensions for GIO, supporting > other backends than local files. Sorry, that was a bit careless. I meant to attach the following piece: tracker-s 14844wena mem REG8,3 368128 423102 /usr/lib/libgnomevfs-2.so.0.2200.0 > > Cheers, > > Cosimo > > -- my place on the web: floss-and-misc.blogspot.com ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] status of GIO regarding TST
On Fri, 2009-01-23 at 12:56 +0200, Tshepang Lekhonkhobe wrote: > Hi, > > Using lsof revealed that TST still uses gnome-vfs. Last I checked we > moved to GIO so me wonders what going on? > > here's the relevant output: > > $ tracker-s 14844wena mem REG8,3 57328 > 606773 /usr/lib/libgvfscommon.so.0.0.0 GVfs is not gnome-vfs, it's a set of extensions for GIO, supporting other backends than local files. Cheers, Cosimo ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
[Tracker] status of GIO regarding TST
Hi, Using lsof revealed that TST still uses gnome-vfs. Last I checked we moved to GIO so me wonders what going on? here's the relevant output: $ tracker-s 14844wena mem REG8,3 57328 606773 /usr/lib/libgvfscommon.so.0.0.0 -- my place on the web: floss-and-misc.blogspot.com ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] Tracker and SQLite
On Fri, Jan 23, 2009 at 12:12 PM, Martyn Russell wrote: > Tshepang Lekhonkhobe wrote: >> >> On Thu, Jan 22, 2009 at 10:50 PM, Martyn Russell >> wrote: >>> >>> If we wanted to write another backend, we would have a lot less work than >>> before of course, but the connection management would need writing. This >>> part is quite tricky for SQLite. We can write a more generic interface >>> here >>> if we need to and if we find it makes sense depending on future >>> requirements. >> >> Is there things specific to Tracker that SQLite is incapable of >> providing that would necessitate such a move in future? That is, do >> you currently to hack around such inadequacies, or is SQLite (with its >> FTS) up to the task? > > There are no hacks for this. It is simply that we have multiple databases > and they are not all stored in the same place. That coupled with the fact > that some databases have to be attached to other databases to be usable, > means we have complex management code depending on what part of Tracker > needs database access and for what use (reading/writing). > > If this was all done using one database in PostgreSQL, for example, then > this management of the databases changes somewhat. So I suppose SQLite is limited in that sense (Me not too familiar with db tech, so excuse me for poking too hard)? What of networked Tracker? I just browsed through SQLite docs and that makes me think there'll have to be a move to a client/server db in such a case since SQLite wouldn't be suitable if multiple clients connected to it, especially if those clients had write access? -- my place on the web: floss-and-misc.blogspot.com ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
Re: [Tracker] Tracker and SQLite
Tshepang Lekhonkhobe wrote: On Thu, Jan 22, 2009 at 10:50 PM, Martyn Russell wrote: If we wanted to write another backend, we would have a lot less work than before of course, but the connection management would need writing. This part is quite tricky for SQLite. We can write a more generic interface here if we need to and if we find it makes sense depending on future requirements. Is there things specific to Tracker that SQLite is incapable of providing that would necessitate such a move in future? That is, do you currently to hack around such inadequacies, or is SQLite (with its FTS) up to the task? There are no hacks for this. It is simply that we have multiple databases and they are not all stored in the same place. That coupled with the fact that some databases have to be attached to other databases to be usable, means we have complex management code depending on what part of Tracker needs database access and for what use (reading/writing). If this was all done using one database in PostgreSQL, for example, then this management of the databases changes somewhat. -- Regards, Martyn ___ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
