Hi there, With this patch in mind we started working on an API that allows multiple inserts per DBus call (and yet get multiple errors back, so the queries don't all belong in the same transaction set).
This experimental work is taking place in the branch multi-insert. Today GNOME's git is down, so I have pushed the latest work on this branch to gitorious. The branch on gitorious: http://meego.gitorious.org/tracker/tracker/commits/multi-insert Note that later this week or today, when GNOME's git comes back for SSH access, I'll push that branch to GNOME's git too. And we might of course also merge it all to master. Here are some test results: pvanh...@lors:~/repos/gnome/tracker/master$ git push gitorious multi-insert Counting objects: 2459, done. Delta compression using up to 2 threads. Compressing objects: 100% (1741/1741), done. Writing objects: 100% (2211/2211), 378.62 KiB, done. Total 2211 (delta 1776), reused 595 (delta 466) => Syncing Gitorious... [OK] To g...@gitorious.org:tracker/tracker.git * [new branch] multi-insert -> multi-insert pvanh...@lors:~/repos/gnome/tracker/master$ cd tests/functional-tests/ pvanh...@lors:~/repos/gnome/tracker/master/tests/functional-tests$ ./update-array-performance-test First run (first update then array) Array: 0.103675, Update: 0.139094 Reversing run (first array then update) Array: 0.290607, Update: 0.161749 pvanh...@lors:~/repos/gnome/tracker/master/tests/functional-tests$ ./update-array-performance-test First run (first update then array) Array: 0.105920, Update: 0.137554 Reversing run (first array then update) Array: 0.118785, Update: 0.130630 pvanh...@lors:~/repos/gnome/tracker/master/tests/functional-tests$ ./update-array-performance-test First run (first update then array) Array: 0.108501, Update: 0.136524 Reversing run (first array then update) Array: 0.117308, Update: 0.151192 pvanh...@lors:~/repos/gnome/tracker/master/tests/functional-tests$ ./update-array-performance-test First run (first update then array) Array: 0.104705, Update: 0.138569 Reversing run (first array then update) Array: 0.108777, Update: 0.134969 pvanh...@lors:~/repos/gnome/tracker/master/tests/functional-tests$ ./update-array-performance-test First run (first update then array) Array: 0.105046, Update: 0.155692 Reversing run (first array then update) Array: 0.114671, Update: 0.132269 pvanh...@lors:~/repos/gnome/tracker/master/tests/functional-tests$ ./update-array-performance-test First run (first update then array) Array: 0.107001, Update: 0.139992 Reversing run (first array then update) Array: 0.118955, Update: 0.133169 pvanh...@lors:~/repos/gnome/tracker/master/tests/functional-tests$ ./update-array-performance-test First run (first update then array) Array: 0.106915, Update: 0.140673 Reversing run (first array then update) Array: 0.192792, Update: 0.136646 pvanh...@lors:~/repos/gnome/tracker/master/tests/functional-tests$ cat update-array-performance-test.c /* * Copyright (C) 2010, Codeminded BVBA <abust...@gnome.org> * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public * License as published by the Free Software Foundation; either * version 2 of the License, or (at your option) any later version. * * This library is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * General Public License for more details. * * You should have received a copy of the GNU General Public * License along with this library; if not, write to the * Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, * Boston, MA 02110-1301, USA. * * Copied from ../tracker-steroids/tracker-test.c */ #include <stdlib.h> #include <string.h> #include <tracker-bus.h> #include <tracker-sparql.h> typedef struct { GMainLoop *main_loop; const gchar *query; guint len, cur; } AsyncData; static TrackerSparqlConnection *connection; #define MSIZE 90 #define TEST_STR "Brrr0092323" static const gchar *queries[90] = { "INSERT { _:a0 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:a9 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:a11 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b0 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b9 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b11 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c0 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c9 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c12 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d0 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d9 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d12 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e0 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e9 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e11 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f0 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f9 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f11 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b1 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b8 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b13 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c1 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c8 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c13 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d1 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d8 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d14 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e1 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e8 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e14 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f1 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f8 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f15 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b2 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b7 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b15 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c2 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c7 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c15 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d2 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d7 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d16 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e2 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e7 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e16 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f2 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f7 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f17 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b3 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b6 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b16 a nmo:Message; nie:title '" TEST_STR "'}", "INSERT { _:c3 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c6 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c18 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d3 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d6 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d19 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e3 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e6 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e20 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f3 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f6 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f21 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b4 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b5 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:b22 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c4 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c5 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c23 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d4 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d5 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d24 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e4 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e5 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e24 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f4 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f5 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f25 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c5 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c2 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:c26 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d5 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d2 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:d28 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e5 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e2 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:e29 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f5 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f2 a nmo:Message; nie:title '" TEST_STR "' }", "INSERT { _:f33 a nmo:Message; nie:title '" TEST_STR "' }"}; static void async_update_array_callback (GObject *source_object, GAsyncResult *result, gpointer user_data) { AsyncData *data = user_data; GPtrArray *errors; errors = tracker_sparql_connection_update_array_finish (connection, result); g_ptr_array_unref (errors); g_main_loop_quit (data->main_loop); } static void test_tracker_sparql_update_array_async () { GMainLoop *main_loop; AsyncData *data; main_loop = g_main_loop_new (NULL, FALSE); data = g_slice_new (AsyncData); data->main_loop = main_loop; /* Cast here is because vala doesn't make const-char-** possible :( */ tracker_sparql_connection_update_array_async (connection, (char**) queries, MSIZE, 0, NULL, async_update_array_callback, data); g_main_loop_run (main_loop); g_slice_free (AsyncData, data); g_main_loop_unref (main_loop); } static void async_update_callback (GObject *source_object, GAsyncResult *result, gpointer user_data) { AsyncData *data = user_data; GError *error = NULL; data->cur++; tracker_sparql_connection_update_finish (connection, result, &error); if (error) g_error_free (error); if (data->cur == data->len) g_main_loop_quit (data->main_loop); } static void test_tracker_sparql_update_async () { guint i; GMainLoop *main_loop; AsyncData *data; main_loop = g_main_loop_new (NULL, FALSE); data = g_slice_new (AsyncData); data->len = MSIZE; data->main_loop = main_loop; data->cur = 0; for (i = 0; i < data->len; i++) { tracker_sparql_connection_update_async (connection, queries[i], 0, NULL, async_update_callback, data); } g_main_loop_run (main_loop); g_slice_free (AsyncData, data); g_main_loop_unref (main_loop); } gint main (gint argc, gchar **argv) { GTimer *array_t, *update_t; g_type_init (); /* do not require prior installation */ g_setenv ("TRACKER_SPARQL_MODULE_PATH", "../../src/libtracker-bus/.libs", TRUE); connection = tracker_sparql_connection_get (NULL, NULL); g_print ("First run (first update then array)\n"); tracker_sparql_connection_update (connection, "DELETE { ?r a rdfs:Resource } WHERE { ?r nie:title '" TEST_STR "' }", 0, NULL, NULL); update_t = g_timer_new (); test_tracker_sparql_update_async (); g_timer_stop (update_t); tracker_sparql_connection_update (connection, "DELETE { ?r a rdfs:Resource } WHERE { ?r nie:title '" TEST_STR "' }", 0, NULL, NULL); array_t = g_timer_new (); test_tracker_sparql_update_array_async (); g_timer_stop (array_t); tracker_sparql_connection_update (connection, "DELETE { ?r a rdfs:Resource } WHERE { ?r nie:title '" TEST_STR "' }", 0, NULL, NULL); g_print ("Array: %f, Update: %f\n", g_timer_elapsed (array_t, NULL), g_timer_elapsed (update_t, NULL)); g_print ("Reversing run (first array then update)\n"); g_timer_destroy (array_t); g_timer_destroy (update_t); array_t = g_timer_new (); test_tracker_sparql_update_array_async (); g_timer_stop (array_t); tracker_sparql_connection_update (connection, "DELETE { ?r a rdfs:Resource } WHERE { ?r nie:title '" TEST_STR "' }", 0, NULL, NULL); update_t = g_timer_new (); test_tracker_sparql_update_async (); g_timer_stop (update_t); tracker_sparql_connection_update (connection, "DELETE { ?r a rdfs:Resource } WHERE { ?r nie:title '" TEST_STR "' }", 0, NULL, NULL); g_print ("Array: %f, Update: %f\n", g_timer_elapsed (array_t, NULL), g_timer_elapsed (update_t, NULL)); g_timer_destroy (array_t); g_timer_destroy (update_t); g_object_unref (connection); return 0; } pvanh...@lors:~/repos/gnome/tracker/master/tests/functional-tests$ On Thu, 2010-09-09 at 17:04 +0800, Chen, Zhenqiang wrote: > >> I will try d-bus-1.3 and latest tracker. > > > >Opps, yea, I meant 1.3.1, not 3.7. Also, note, 1.3.0 is buggy so you > >will need 1.3.1. This should avoid quite some memory copies > >when indexing. > > > > I tried tracker git code (master, last update Sep 7). But test results show > it is ~15% slower than tracker-0.9.16. > And I tried d-bus-1.3.1. It did not help in my case. > > dbus is one of the bottlenecks. Tests show 1000 continuous INSERTs will block > the d-bus in my system. > > To reduce the dbus overhead, I tried to group the UPDATE for files of a dir > into one update. Tests show it is ~2X faster. Here are the logs: > (Notes: among the 10693 files, 10654 files are photos.) > > tracker git master: > Finished mining in seconds:232.713859, total directories:354, total > files:10693 > > tracker-0.9.16 > Finished mining in seconds:200.689239, total directories:354, total > files:10693 > > tracker-0.9.16 with group update (testing code segment is attached.) > Finished mining in seconds:92.813985, total directories:354, total files:10693 > > What do you think about the idea "grouping the updates for files of one dir > into one update"? > > Thanks! > -Zhenqiang > > _______________________________________________ > tracker-list mailing list > tracker-list@gnome.org > http://mail.gnome.org/mailman/listinfo/tracker-list -- Philip Van Hoof freelance software developer Codeminded BVBA - http://codeminded.be -- Philip Van Hoof freelance software developer Codeminded BVBA - http://codeminded.be _______________________________________________ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list
