Bug#775990: akonadi-backend-sqlite: Akonadi reports deadlocks
Hi David, I've just uploaded Akonadi-1.13.0-6 with fix of reproducible deadlock. I think deadlock is not depending on backend so I hope there will be some relief to you. Could you confirm if 1.13.0-6 any better please? Also I've been testing different backends and I'm not too happy to see database errors on PSQL and SQLITE. SQLITE3 always leaves .db-shm and .db-wal files behind and the only backend that seems to work OK is MYSQL... -- Cheers, Dmitry Smirnov. --- Problems are not stop signs, they are guidelines. -- Robert H. Schuller signature.asc Description: This is a digitally signed message part.
Bug#775990: akonadi-backend-sqlite: Akonadi reports deadlocks
On 17/07/15 21:12, David Goodenough wrote: So this looks like the QSQLITE3 one. I have no idea where to find the other one. I think the other one is built-in. You can change Driver=QSQLITE3 to Driver=QSQLITE (and corresponding section) in your ~/.config/akonadi/akonadiserverrc. Please remember to make a backup if you are going to try. ;) I also dread the idea of loosing all my folders when moving from one DB to another. I have never found a good HOWTO on the subject. Basically you won't loose your folders as akonadi will re-scan Maildir folders and re-download contents of IMAP accounts. It may take a while though... What you should be careful about is to update your filter rules that will be mixed up regarding destination folders where messages are copied or moved. It make sense to update rules in offline mode before allowing akonadi do download anything. Also you will need to update destination folders in POP3 accounts and folders under Advanced tab in identities. That's pretty much it. Yes, they seem to use both names for it, but either way they are moving away from SQL to a key-value store, so all these problems with go away!!! Cool. KV store is probably easier to use so probably many bugs will disappear. -- Regards, Dmitry Smirnov signature.asc Description: OpenPGP digital signature
Bug#775990: akonadi-backend-sqlite: Akonadi reports deadlocks
Hi David, First of all, there are two Akonadi SQLite backends: QSQLITE and QSQLITE3. Which one are you using? Can you try another one please? On 16/07/15 01:22, David Goodenough wrote: There is a script that can be run (pg_upgradecluster from memory) which does the upgrade, but the postinst for the debs does not use it. I guess there may be times when you do not want it, and originally it did not always work (I have been using pg for a long time), but I have never had a problem with it other than having to run in it manually. So what's the problem then? Besides PostgreSQL is the only RDBMS offering data checksums... Personally I will not have MySql on my machine, I simply do not trust it. Bad experience or abstract theory? Just curious... Its rather like the difference between Access and MS-SQL, one is the full function, the other is a simplified version designed to make life quick and easy. Again this is my email system, I want something reliable and which follows all the SQL rules (MySql and its variants do not). I know what you mean but it doesn't make much sense. PSQL is not necessary more reliable and it seems that (just like many other applications) Akonadi is fine with MySQL functionality (remember, MySQL is a recommended backend). I see no evidence that Akonadi needs backend with all the SQL rules and if you consistently follow that logic you should eliminate SQLite as well since the latter supports even more limited SQL dialect. Besides MySQL have great tools like MySQL Workbench and others... There was talk of an Akonadi2, You suppose you meant this: https://community.kde.org/KDE_PIM/Akonadi_Next -- Regards, Dmitry Smirnov signature.asc Description: OpenPGP digital signature
Bug#775990: akonadi-backend-sqlite: Akonadi reports deadlocks
On Wednesday 15 July 2015 22:13:49 Dmitry Smirnov wrote: Akonadi reports deadlocks. I tried to reproduce this problem and sure it looks scary but do we have a real problem beyond this warning? In my testing this warning followed by retrying transaction and everything works just fine so IMHO it appears to be harmless. Besides deadlock disappear with upgrading to sqlite3/3.8.10.2-1 from testing. I'm lowering severity to normal for now. I am running sid, so I have 3.8.10.2-1 and I am still getting these deadlocks. What gives you the idea that they shoud have gone away? David -- To UNSUBSCRIBE, email to debian-qt-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/1720478.REN4CxZ3x9@stargate
Bug#775990: akonadi-backend-sqlite: Akonadi reports deadlocks
Akonadi reports deadlocks. I tried to reproduce this problem and sure it looks scary but do we have a real problem beyond this warning? In my testing this warning followed by retrying transaction and everything works just fine so IMHO it appears to be harmless. Besides deadlock disappear with upgrading to sqlite3/3.8.10.2-1 from testing. I'm lowering severity to normal for now. -- Cheers, Dmitry Smirnov GPG key : 4096R/53968D1B --- Each generation imagines itself to be more intelligent than the one that went before it, and wiser than the one that comes after it. -- George Orwell, Review of A Coat of Many Colours: Occasional Essays by Herbert Read, Poetry Quarterly (Winter 1945) signature.asc Description: This is a digitally signed message part.
Bug#775990: akonadi-backend-sqlite: Akonadi reports deadlocks
On Thursday 16 July 2015 00:19:59 Dmitry Smirnov wrote: I wonder if upstream having plans to decommission akonadi as it seems to be our only hope unless they manage to magically fix it... Apparently some work is underway with Akonadi Next https://community.kde.org/KDE_PIM/Akonadi_Next -- GPG: 0x138E41915C7EFED6 signature.asc Description: This is a digitally signed message part.
Bug#775990: akonadi-backend-sqlite: Akonadi reports deadlocks
Hi David, On 15/07/15 23:34, David Goodenough wrote: My prefered DBM is Postgresql, but it has real problems when updating in that it does not automatically upgrade DBs on update, at least the way that Debian packages it. Interesting. Something tells me that it must be an upstream problem but if this is a packaging issue then perhaps we can fix it -- we just need more details. Also sometimes, especially on sid you get an update where you can not even use the supplied upgrade scripts and you have to do it manually. Having that underlying my email system is not an acceptable risk. Indeed. Personally I will not have MySql on my machine, I simply do not trust it. Bad experience or abstract theory? Just curious... Since 1.13.0-4 you should be able to use MariaDB or any other MySQL flavour if they are any better... Lots of other systems use Sqlite without problems, [...] I would also prefer SQLite if only akonadi could use it reliably... All in all this is leading me to an inexorable conclusion, I have will have (after many many years) to give up on kmail and move to something else. My problem is what to move to. I really would like to remain within the KDE world as it will then be better integrated, and I need both IMAP and POP - there appears to be little option. I'm with you. I am addicted to KMail and many of its features. IMHO there is no replacement to it... I wonder if upstream having plans to decommission akonadi as it seems to be our only hope unless they manage to magically fix it... -- Regards, Dmitry Smirnov signature.asc Description: OpenPGP digital signature
Bug#775990: akonadi-backend-sqlite: Akonadi reports deadlocks
On 15/07/15 22:38, David Goodenough wrote: I am running sid, so I have 3.8.10.2-1 and I am still getting these deadlocks. What gives you the idea that they shoud have gone away? Experimentation: I fed ~500 MiB maildir folder to akonadi/sqlite and deadlocks disappeared with upgrade of sqlite3/libsqlite. It is quite possible that I did not test long enough however I'm trying to determine whether it is a real error or merely a warning? Can you confirm? By the way I think I've seen some quiet deadlocks (when akonadi doesn't do anything while kmail waits for akonadi) even on backend-mysql. I think akonadi _is_ a problem but I'm just not sure if deadlocks on sqlite are really that bad, comparing to the whole thing. In any case upstream admits that there are problems with concurrent access on sqlite backend: https://techbase.kde.org/Projects/PIM/Akonadi#Why_not_use_sqlite.3F So as bad as it sounds sqlite just seems to be a poorly supported backend... :( -- Regards, Dmitry Smirnov signature.asc Description: OpenPGP digital signature
Bug#775990: akonadi-backend-sqlite: Akonadi reports deadlocks
On Wednesday 15 July 2015 23:16:08 Dmitry Smirnov wrote: On 15/07/15 22:38, David Goodenough wrote: I am running sid, so I have 3.8.10.2-1 and I am still getting these deadlocks. What gives you the idea that they shoud have gone away? Experimentation: I fed ~500 MiB maildir folder to akonadi/sqlite and deadlocks disappeared with upgrade of sqlite3/libsqlite. It is quite possible that I did not test long enough however I'm trying to determine whether it is a real error or merely a warning? Can you confirm? I suspect that doing something essentially sequential does not show deadlocks. I hit this all the time because I have LOTS of mail filters and they seem to happen while processing those - I guess it spawns processes at least for the spam filter. By the way I think I've seen some quiet deadlocks (when akonadi doesn't do anything while kmail waits for akonadi) even on backend-mysql. I think akonadi _is_ a problem but I'm just not sure if deadlocks on sqlite are really that bad, comparing to the whole thing. In any case upstream admits that there are problems with concurrent access on sqlite backend: https://techbase.kde.org/Projects/PIM/Akonadi#Why_not_use_sqlite.3F So as bad as it sounds sqlite just seems to be a poorly supported backend... :( My prefered DBM is Postgresql, but it has real problems when updating in that it does not automatically upgrade DBs on update, at least the way that Debian packages it. Also sometimes, especially on sid you get an update where you can not even use the supplied upgrade scripts and you have to do it manually. Having that underlying my email system is not an acceptable risk. Personally I will not have MySql on my machine, I simply do not trust it. Lots of other systems use Sqlite without problems, and I have to say that my suspicion is that the problems with akonadi are simply becuase it was implemented by people who did not really understand how to use an SQL DB. For instance, a deadlock is a retryable error, you hit it, rollback your transaction and try again. In development this can lead to problems if you do not plan the order in which you access tables, but this is a basic principle of all SQL DBs. So these errors should never appear as errors, they should just trigger an internal retry. Sqlite has had sufficient locking to get around this problem for a while now, and this suggests tha the ordering of table access has simply not been planned properly. All in all this is leading me to an inexorable conclusion, I have will have (after many many years) to give up on kmail and move to something else. My problem is what to move to. I really would like to remain within the KDE world as it will then be better integrated, and I need both IMAP and POP - there appears to be little option. David -- To UNSUBSCRIBE, email to debian-qt-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/15300265.4JRVSBbMy9@stargate
Bug#775990: akonadi-backend-sqlite: Akonadi reports deadlocks
On Thursday 16 July 2015 00:19:59 Dmitry Smirnov wrote: Hi David, On 15/07/15 23:34, David Goodenough wrote: My prefered DBM is Postgresql, but it has real problems when updating in that it does not automatically upgrade DBs on update, at least the way that Debian packages it. Interesting. Something tells me that it must be an upstream problem but if this is a packaging issue then perhaps we can fix it -- we just need more details. There is a script that can be run (pg_upgradecluster from memory) which does the upgrade, but the postinst for the debs does not use it. I guess there may be times when you do not want it, and originally it did not always work (I have been using pg for a long time), but I have never had a problem with it other than having to run in it manually. Also sometimes, especially on sid you get an update where you can not even use the supplied upgrade scripts and you have to do it manually. Having that underlying my email system is not an acceptable risk. Indeed. Personally I will not have MySql on my machine, I simply do not trust it. Bad experience or abstract theory? Just curious... Its rather like the difference between Access and MS-SQL, one is the full function, the other is a simplified version designed to make life quick and easy. Again this is my email system, I want something reliable and which follows all the SQL rules (MySql and its variants do not). Since 1.13.0-4 you should be able to use MariaDB or any other MySQL flavour if they are any better... Lots of other systems use Sqlite without problems, [...] I would also prefer SQLite if only akonadi could use it reliably... All in all this is leading me to an inexorable conclusion, I have will have (after many many years) to give up on kmail and move to something else. My problem is what to move to. I really would like to remain within the KDE world as it will then be better integrated, and I need both IMAP and POP - there appears to be little option. I'm with you. I am addicted to KMail and many of its features. IMHO there is no replacement to it... I wonder if upstream having plans to decommission akonadi as it seems to be our only hope unless they manage to magically fix it... There was talk of an Akonadi2, but I am not sure what happens to its DB usage. The other odd thing is that if I read its architecture page on kde.org correctly Baloo (the Nepomuk replacement) uses SqLite. I do not use Baloo consciously, but it lurks in the bowels of KDE and seems to work. David -- To UNSUBSCRIBE, email to debian-qt-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/62490499.xqnbJykhQP@stargate
Bug#775990: [akonadi-backend-sqlite] Akonadi reports deadlocks
tag 775990 -patch +moreinfo thanks hey, the mentioned patch in http://osdir.com/ml/kde-commits/2014-07/msg02228.html is already part of current akonadi version in unstable. In the ~/.local/share/akonadi/akonadiserver.error there are repeated deadlock errors reported. Can you give us more details? Regards, sandro -- To UNSUBSCRIBE, email to debian-qt-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/6189483.QFarGjfOvV@tabin.local
Bug#775990: [akonadi-backend-sqlite] Akonadi reports deadlocks
Package: akonadi-backend-sqlite Version: 1.13.0-2 Severity: normal --- Please enter the report below this line. --- In the ~/.local/share/akonadi/akonadiserver.error there are repeated deadlock errors reported. In http://osdir.com/ml/kde-commits/2014-07/msg02228.html what looks to be a very simple fix for this problem can be found. --- System information. --- Architecture: amd64 Kernel: Linux 3.16.0-4-amd64 Debian Release: 8.0 500 unstablehttp.debian.net 500 stable dl.google.com --- Package information. --- Depends (Version) | Installed =-+-= libc6 (= 2.14) | libgcc1 (= 1:4.1.1) | libqt4-sql (= 4:4.8.0) | libqtcore4 (= 4:4.8.0) | libsqlite3-0 (= 3.5.9) | libstdc++6 (= 4.1.1) | Recommends (Version) | Installed =-+-=== akonadi-server| 1.13.0-2 Package's Suggests field is empty. -- To UNSUBSCRIBE, email to debian-qt-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/2853289.NkdaTaOiX5@stargate
