On Thu, 2010-08-12 at 14:54 -0400, Jamie McCracken wrote: > your proposal sounds fine - what are you complaining about? > > Only one thing stands out - direct access. You will surely need IPC to > signal changes made by a direct access user as well as to receive them
Direct-access is always read-only (only SELECT): They can't make any changes that would cause a signal (aka. why SQLite's WAL is sufficient MVCC-ish for our use-case). Updates go over either traditional D-Bus or D-Bus with FD-passing (the Steroids D-Bus object). Those can of course cause the signal. Cheers, Philip > On Thu, 2010-08-12 at 20:43 +0200, Philip Van Hoof wrote: > > Comon guys, > > > > I know I'm a natural born pessimist, and I know I shouldn't be. But > > still, there must be *something* wrong about this proposal?! > > > > Nobody is commenting at all? You know, the idea of posting it here is to > > get some discussion going "before" I implement it ;-) > > > > Ping, everybody! > > > > Cheers, > > > > Philip > > > > > > On Thu, 2010-08-12 at 15:03 +0200, Philip Van Hoof wrote: > > > A new class signal for Tracker > > > > > > Today's situation > > > > > > Today we have a simple signal system that causes quite a bit of > > > overhead which we over time tried to reduce. The overhead comes from: > > > A. Having to store the URIs of the resources involved in a > > > changeset in tracker-store's memory; > > > B. Having to store the predicates involved in a changeset in > > > tracker-store's memory (although far less severe than #1); > > > C. Having to UTF-8 validate the strings when we emit them over > > > D-Bus (D-Bus does this implicitly); > > > D. D-Bus's own copying and handling of string data; > > > E. Heavy traffic on D-Bus; > > > F. Context switching between tracker-store and dbus-daemon; > > > G. We have to wait with turning on the D-Bus objects until after > > > we have the latest ontology. So after journal replay. And we > > > need to reset the situation after a backup restore. Complex! > > > Besides this overhead there are problems the consumers have too. I'll > > > make a list in the next section. > > > > > > Problems of today's signal > > > 1. Aforementioned overhead: consumes a lot of D-Bus traffic. This > > > is caused by sending over URLs for the subjects and the > > > predicates; > > > 2. Doesn't make it possible, in case of a delete of <a>, to know > > > <b> in <a> nfo:isLogicalPartOf <b>, as <a> is removed at the > > > point of signal emission; > > > 3. Round trips to know the literals create more D-Bus traffic; > > > 4. Transactional changes can't be reliably identified with > > > SubjectsAdded, SubjectsChanged and SubjectsRemoved being > > > separate signals; > > > 5. A lot of D-Bus objects, instead of letting clients use D-Bus's > > > filtering system. > > > > > > The drive for a solution > > > > > > Jürg Billeter and me brainstormed a bit about all these problems. Last > > > few months while optimizing tracker-store's INSERT performance and > > > memory utilization, we brainstormed a lot about how we could reduce > > > the overhead. I believe we have a good idea of the current situation, > > > its internal problems and our current solution (hey of course, we > > > implemented it :p). > > > > > > We also gained know how about most of the problems consumers have from > > > the maintainer of libqttracker, Petteri Iridian Kiiskinen. Thanks > > > Iridian! > > > > > > Today I believe that we must abandon the old ship, redo the signal > > > system, break the API. Break it all. Get over it, heal our wounds. > > > Even if that means taking the stress away from all sorts of people > > > who've been using the old signal system, offering massages, giving out > > > sauna coupons. You know, the usual stuff that we won't do for real. > > > Although I'm sure that at a next code-camp in Helsinki we'll have a > > > good sauna to burn all our own stress away. > > > > > > Anyway ... *shrug* > > > > > > A proposed solution > > > > > > Part one: Direct access > > > With direct-access we will reduce the round-trip cost of a query from > > > a consumer who wants a literal object involved in a changeset: it'll > > > be executed directly on meta.db; you wont use libsqlite's API yourself > > > but libtracker-sparql. However, libtracker-sparql is for direct-access > > > a layer on top of aforementioned libsqlite. The so-called "round-trip" > > > won't even involve IPC: by utilizing the TrackerSparqlCursor API, > > > you'll end up doing sqlite3_step() in your own process, directly on > > > meta.db. > > > > > > For the consumers of the signal, this removes 3. > > > > > > Part two: Sending IDs > > > A while ago we introduced the SPARQL function tracker:id(). The > > > tracker:id() function gives you a unique number that Tracker's RDF > > > store internally. It's not RDF, RDF uses subject URL strings. We just > > > convert this internally for performance reasons, and with tracker:id() > > > you can access that. > > > > > > Each resource, each class and each predicate (latter two are resources > > > like any other) have such an unique internal ID. > > > > > > Given that Tracker's class signal system isn't RDF anyway, we decided > > > not to give you subject URL strings in it anymore. Instead, we'll give > > > you these integer IDs. > > > > > > This for us removes A, B, C, D and E. For the consumers of the signal, > > > this removes 1. Whoohoo! > > > > > > Part three: Combine SubjectsAdded and SubjectsChanged, and put > > > SubjectsRemoved in the same signal > > > So we give you two arrays: Inserts and Deletes. > > > > > > For consumers of the signal, this removes 4. > > > > > > Part five: Add the class name to the signal > > > This allows you to use a string filter on your signal subscription in > > > D-Bus. > > > > > > For us this removes G. For consumers of the signal, this removes 5. > > > > > > Part six: Pass the object-id for resource objects > > > You'll get a third number in the Inserts and Deletes arrays: > > > object-id. We wont send you object literals, although for integral > > > objects we're still discussing this. But for resource objects we can > > > without much extra cost give you the object-id. > > > > > > For consumers of the signal, this removes 2. Whoohoo (this was a hard > > > one)! > > > > > > Part seven: SPARQL IN, tracker:id() and tracker:subject() > > > We recently added support for SPARQL IN, we already have tracker:id() > > > and we'll implement tracker:subject(). > > > > > > This makes things like this possible: > > > > > > SELECT ?t { ?r nie:title ?t . > > > FILTER (tracker:id(?r) IN (800, 801, 802, 807)) } > > > > > > Where 800, 801, 802 and 807 will be the IDs that you receive in the > > > class signal. > > > > > > The tracker:subject() SPARQL function will allow you to make a very > > > fast version of this: > > > > > > SELECT ?s { ?s a rdfs:Resource . > > > FILTER (tracker:id(?s) IN (800)) } > > > > > > So it would be something like ... (not sure that you can omit { } in > > > SPARQL, though): > > > > > > SELECT tracker:subject (800) > > > > > > For consumers this removes most of the burden introduced by IDs. > > > Consumers are also advised to keep a local Map<tracker:id(), subject> > > > to avoid a lot of SPARQL queries. Although with direct-access it might > > > be just fine. > > > > > > Part eight: What is left? > > > > > > What is left is context switching between tracker-store and > > > dbus-daemon, F. But that's our problem. We'll reduce them by grouping > > > transactions and signals together. It's mostly a problem on ARM > > > hardware, but yeah that's a major and important target platform for > > > us. We're on it, we will care about this! > > > > > > Let's take a look! > > > > > > <node name="/org/freedesktop/Tracker1/Resources"> > > > <interface name="org.freedesktop.Tracker1.Resources.Class"> > > > <signal name="class-signal"> > > > <arg type="s" name="class-name" /> > > > <arg type="a(iii)" name="inserts" /> > > > <arg type="a(iii)" name="deletes" /> > > > </signal> > > > </interface> > > > </node> > > > > > > Or in short: sa(iii)a(iii). Here's a bit of pseudo code how it'll look > > > clientside: > > > > > > void m_callback (cursor) { > > > while (cursor.next()) { > > > // With direct-access are these c.next()s, sqlite_step() calls > > > print ("title: %s", cursor.get_string ()); > > > } > > > } > > > > > > void on_signal (class_name, deleted, inserted) { > > > string in_qry = "", qry; > > > bool first = true; > > > > > > foreach (insert in inserted) { > > > if (insert.subject_id is_in (my_resources)) { > > > if (!first) { in_qry += ", "; } > > > in_qry += insert.subject_id > > > first = false; > > > } > > > } > > > > > > qry = string.printf ("SELECT ?titles { ?r nie:title ?titles . > > > FILTER (tracker:id(?r) IN (%s)) }", in_qry); > > > > > > connection.query_async (qry, m_callback); > > > } > > > > > > > > > Cheers! :-) > > > > > > Philip > > > > > > > > > -- > > > > > > > > > Philip Van Hoof > > > phi...@codeminded.be > > > freelance software developer > > > Codeminded BVBA - http://codeminded.be > > > _______________________________________________ > > > tracker-list mailing list > > > tracker-list@gnome.org > > > http://mail.gnome.org/mailman/listinfo/tracker-list > > > > -- > > > > > > Philip Van Hoof > > freelance software developer > > Codeminded BVBA - http://codeminded.be > > > > _______________________________________________ > > tracker-list mailing list > > tracker-list@gnome.org > > http://mail.gnome.org/mailman/listinfo/tracker-list > > > -- Philip Van Hoof freelance software developer Codeminded BVBA - http://codeminded.be _______________________________________________ tracker-list mailing list tracker-list@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list