Re: [Zeitgeist] [Bug 660307] Re: zeitgeist fails to run if its database structure is not complete
However noble it may be I don't think we stand a realistic chance of implementing a stable repair function if the DB corrupts at an undefined point in the upgrade process. There are just *way* too many variables if we have fx. 4 different DB schemes that can all intermix and corrupt in different ways. I'd rather we simply made the N - N+1 transition more reliable. A few ideas: a) Implementing a validation step after an upgrade has completed and taking appropriate measures if it fails (fx. by using c) below) b) Back up the DB before starting the upgrade. I recommend that we back up to a gzipped ttl file because that can be used in a streaming (aka appending) way c) Implement recover from backup (see point point b)) These are all relatively simple measures that can be properly unit tested and are much more limited in the ways they can fail ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
Re: [Zeitgeist] [Bug 660307] Re: zeitgeist fails to run if its database structure is not complete
Oh, and one point in I forgot to add in my previous comment - I don't want to validate the DB on startup. That's just way too expensive - and whas in fact one of the primary points when I implemented DB versioning. One thing we could do to easily, and almost freely, detect when we are killed during an upgrade would be to set the core schema version to -1 just before we start the upgrade, and then set it to the correct version if the upgrade completes correctly. Then if we start up and see DB schema version -1 we know that we have to recover from backup. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 602211] Re: Monitoring Create/Move/Copy Files events
I think it easy for clients to determine subjects that have been deleted by calling exists on the uri. Of course we can detect delete events from nautilus too and thusly change the current_uri for it. But again to the solution for solving moved files. The solution proposed by Mikkel (modified by me) could make things messy. Once a file is moved we have to query on both subject_current_uri and subject_uri. right? So lets say I have a file that has been moved from /home/seif/foo to /home/seif/bar. This means I query for all events where subject_uri = /home/seif/foo so I get all results until the point it was moved, right? And if I ask for /home/seif/bar I get all events with subject_uri = /home/seif/foo as well as all subject_uri = /home/seif/bar ... ? -- Monitoring Create/Move/Copy Files events https://bugs.launchpad.net/bugs/602211 You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. Status in Unity: Triaged Status in Unity Files Place: Triaged Status in Zeitgeist Framework: Confirmed Status in Zeitgeist Datahub: Confirmed Bug description: An issue we are facing at the moment is that ppl lose track of there files in a timeline if the file was moved around or renamed. I would propose using taskview or patch nautilus to actually grab those events and either: 1) Modify the uris in the uris table 2) Create a new table with| new_id | old_uri_id | event | to map uris to their actual ids and the event that allowed the change, this would allow us to track a history of renaming or moving a file. It will look a bit like the following: 9 | 9 | 48124 # CREATE EVENT 12 | 9 | 48126 # MOVE EVENT In other words the last row means uri 12 was moved from uri 9 with event 48126 UPDATE: 3) Create a changable_uri table that is a map of the uri table. it gets updated upon moved and rename. We then add new resulttype that allow you to ask for either pureSubject or adaptedSubject. depending on which one is chosen we then use the according table in the join of the find_events_query :) ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] Translation template import - zeitgeist in Zeitgeist Framework 0.1
Hello Zeitgeist Framework Team, On 2010-11-15 14:45z (2 minutes ago), you uploaded a translation template for zeitgeist in Zeitgeist Framework 0.1 in Launchpad. The template has now been imported successfully. Thank you, The Launchpad team ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] Translation template import - zeitgeist in Zeitgeist Framework 0.3
Hello Zeitgeist Framework Team, On 2010-11-15 14:45z (5 minutes ago), you uploaded a translation template for zeitgeist in Zeitgeist Framework 0.3 in Launchpad. The template has now been imported successfully. Thank you, The Launchpad team ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] Translation template import - zeitgeist in Zeitgeist Framework 0.5
Hello Zeitgeist Framework Team, On 2010-11-15 14:45z (7 minutes ago), you uploaded a translation template for zeitgeist in Zeitgeist Framework 0.5 in Launchpad. The template has now been imported successfully. Thank you, The Launchpad team ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Merge] lp:~thekorn/zeitgeist/db_schema_3 into lp:zeitgeist
Markus Korn has proposed merging lp:~thekorn/zeitgeist/db_schema_3 into lp:zeitgeist. Requested reviews: Zeitgeist Framework Team (zeitgeist) Related bugs: #673394 Queries for subj_uri and subj_origin are using no index https://bugs.launchpad.net/bugs/673394 #673452 Using the subj_origin column of event_view is slower than it should be https://bugs.launchpad.net/bugs/673452 This branch introduces a new db schema (version 3) and bundles a few performance related fixes, [0] compares the performance of this branch lp:zeitgeist (each query is run ~10 times, the plot shows the average). Changes in detail: * renamed event_view.subj_origin to event_view.subj_origin_uri and event_view.subj_origin_id to event_view.subj_origin, this makes subj_origin point to an id in the uri table, and subj_origin_uri the resolved value. Also we are not using the redundant SELECT id FROM uri WHERE uri.id=event.subj_origin query anymore. (LP: #673452) * removed event_view.subj_uri_id and added event.subj_id, both values are in fact the same, but we again sting on another SELECT query in the event_view. * Optimization in FindEvent queries: we are not ordering and grouping by resolved values anymore, we are doing it on integer ids, which is much faster (LP: #673394) [0] https://dl.dropbox.com/u/174479/compare_queries.svg -- https://code.launchpad.net/~thekorn/zeitgeist/db_schema_3/+merge/40877 Your team Zeitgeist Framework Team is requested to review the proposed merge of lp:~thekorn/zeitgeist/db_schema_3 into lp:zeitgeist. === modified file '_zeitgeist/engine/__init__.py' --- _zeitgeist/engine/__init__.py 2010-10-19 13:54:12 + +++ _zeitgeist/engine/__init__.py 2010-11-15 15:11:50 + @@ -55,7 +55,7 @@ # Required version of DB schema CORE_SCHEMA=core - CORE_SCHEMA_VERSION = 2 + CORE_SCHEMA_VERSION = 3 USER_EXTENSION_PATH = os.path.join(DATA_PATH, extensions) === modified file '_zeitgeist/engine/main.py' --- _zeitgeist/engine/main.py 2010-11-12 17:39:28 + +++ _zeitgeist/engine/main.py 2010-11-15 15:11:50 + @@ -148,8 +148,9 @@ def _get_subject_from_row(self, row): subject = Subject() - for field in (uri, origin, text, storage): + for field in (uri, text, storage): setattr(subject, field, row[subj_ + field]) + setattr(subject, origin, row[subj_origin_uri]) for field in (interpretation, manifestation, mimetype): setattr(subject, field, getattr(self, _ + field).value(row[subj_ + field])) @@ -353,10 +354,13 @@ sql += ( ORDER BY timestamp DESC, ORDER BY timestamp ASC, - GROUP BY subj_uri ORDER BY timestamp DESC, - GROUP BY subj_uri ORDER BY timestamp ASC, - GROUP BY subj_uri ORDER BY COUNT(subj_uri) DESC, timestamp DESC, - GROUP BY subj_uri ORDER BY COUNT(subj_uri) ASC, timestamp ASC, + # thekorn: please note, event.subj_id == uri.id, as in + # the subj_id points directly to an entry in the uri table, + # so we are in fact grouping by subj_uris here + GROUP BY subj_id ORDER BY timestamp DESC, + GROUP BY subj_id ORDER BY timestamp ASC, + GROUP BY subj_id ORDER BY COUNT(subj_id) DESC, timestamp DESC, + GROUP BY subj_id ORDER BY COUNT(subj_id) ASC, timestamp ASC, GROUP BY actor ORDER BY COUNT(actor) DESC, timestamp DESC, GROUP BY actor ORDER BY COUNT(actor) ASC, timestamp ASC, GROUP BY actor ORDER BY timestamp DESC, === modified file '_zeitgeist/engine/sql.py' --- _zeitgeist/engine/sql.py 2010-10-25 20:26:03 + +++ _zeitgeist/engine/sql.py 2010-11-15 15:11:50 + @@ -369,14 +369,12 @@ AS payload, (SELECT value FROM uri WHERE uri.id=event.subj_id) AS subj_uri, -(SELECT id FROM uri WHERE uri.id=event.subj_id) - AS subj_uri_id, +event.subj_id, -- #this directly points to an uri event.subj_interpretation, event.subj_manifestation, +event.subj_origin, (SELECT value FROM uri WHERE uri.id=event.subj_origin) - AS subj_origin, -(SELECT id FROM uri WHERE uri.id=event.subj_origin) - AS subj_origin_id, + AS subj_origin_uri, event.subj_mimetype, (SELECT value FROM text WHERE text.id = event.subj_text) AS subj_text, @@ -528,14 +526,17 @@ def add_text_condition(self, column, value, like=False, negation=False, cache=None): if like: assert column in (subj_uri, subj_origin, actor, subj_mimetype), \ -prefix search on the %r column is not supported by zeitgeist - if column in (subj_uri, subj_origin): -view_column = %s_id %column +prefix search on the %r column is not supported by zeitgeist %column + if column == subj_uri: +# subj_id directly points to the id of an uri entry +view_column = subj_id else: view_column = column optimized_glob, value = self.optimize_glob(id, TABLE_MAP.get(column, column), value) sql = %s %sIN (%s) %(view_column, self.NOT if negation else , optimized_glob) else: + if column == subj_origin: +column = subj_origin_uri sql = %s %s= ? %(column, ! if
Re: [Zeitgeist] [Bug 602211] Re: Monitoring Create/Move/Copy Files events
+1 to Mikkel on using it for deletes too, having two different systems doesn't make sense. Also, I'd keep uri and current_uri as separate things (current_uri being a new parameter in event, that's why it's a nice array...), else this is going to be a real mess. -- Monitoring Create/Move/Copy Files events https://bugs.launchpad.net/bugs/602211 You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. Status in Unity: Triaged Status in Unity Files Place: Triaged Status in Zeitgeist Framework: Confirmed Status in Zeitgeist Datahub: Confirmed Bug description: An issue we are facing at the moment is that ppl lose track of there files in a timeline if the file was moved around or renamed. I would propose using taskview or patch nautilus to actually grab those events and either: 1) Modify the uris in the uris table 2) Create a new table with| new_id | old_uri_id | event | to map uris to their actual ids and the event that allowed the change, this would allow us to track a history of renaming or moving a file. It will look a bit like the following: 9 | 9 | 48124 # CREATE EVENT 12 | 9 | 48126 # MOVE EVENT In other words the last row means uri 12 was moved from uri 9 with event 48126 UPDATE: 3) Create a changable_uri table that is a map of the uri table. it gets updated upon moved and rename. We then add new resulttype that allow you to ask for either pureSubject or adaptedSubject. depending on which one is chosen we then use the according table in the join of the find_events_query :) ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 602211] Re: Monitoring Create/Move/Copy Files events
I still didn't understand why would the current_uri be part of event, isn't it by definition a property of the subject? That way you don't need to update dozens of events, just one subject entry... -- Monitoring Create/Move/Copy Files events https://bugs.launchpad.net/bugs/602211 You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. Status in Unity: Triaged Status in Unity Files Place: Triaged Status in Zeitgeist Framework: Confirmed Status in Zeitgeist Datahub: Confirmed Bug description: An issue we are facing at the moment is that ppl lose track of there files in a timeline if the file was moved around or renamed. I would propose using taskview or patch nautilus to actually grab those events and either: 1) Modify the uris in the uris table 2) Create a new table with| new_id | old_uri_id | event | to map uris to their actual ids and the event that allowed the change, this would allow us to track a history of renaming or moving a file. It will look a bit like the following: 9 | 9 | 48124 # CREATE EVENT 12 | 9 | 48126 # MOVE EVENT In other words the last row means uri 12 was moved from uri 9 with event 48126 UPDATE: 3) Create a changable_uri table that is a map of the uri table. it gets updated upon moved and rename. We then add new resulttype that allow you to ask for either pureSubject or adaptedSubject. depending on which one is chosen we then use the according table in the join of the find_events_query :) ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
