Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wed, May 19, 2010 at 05:57:06PM -0300, Nicolas Alvarez wrote: Allan Sandfeld Jensen wrote: Yes, the header contains the http expire date there is no other way to get it. Though I would have thought it would keep the statistics in memory instead of rereading it on every cycle. Are you sure that this isn't just on startup? In what memory? In KDE 4.3, kio_http_cache_cleaner is not a daemon. It's started to clean the cache, apparently by kio_http, and it quits when it's done doing that. That seems to be exactly what changed in 4.4: it's now a permanent daemon. Perhaps it should cache the state of the cache ... Nick -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100520230215.ga21...@leverton.org
Re: KDE 4.4.3 upgrade eats 141 MB of /home
Allan Sandfeld Jensen wrote: On Saturday 15 May 2010, Nicolas Alvarez wrote: After some stracing: It doesn't just stat the files. It first stats, then actually opens the files and *reads* the first few bytes of all of them (where the HTTP cache information is stored). Yes, the header contains the http expire date there is no other way to get it. Though I would have thought it would keep the statistics in memory instead of rereading it on every cycle. Are you sure that this isn't just on startup? In what memory? In KDE 4.3, kio_http_cache_cleaner is not a daemon. It's started to clean the cache, apparently by kio_http, and it quits when it's done doing that. That seems to be exactly what changed in 4.4: it's now a permanent daemon. -- Nicolas (I read mailing lists through Gmane. Please don't Cc me on replies; it makes me get one message on my newsreader and another on email.) -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/ht1jb3$mv...@dough.gmane.org
Re: KDE 4.4.3 upgrade eats 141 MB of /home
Am Mittwoch 12 Mai 2010 schrieb Mike Kasick: On Wed, May 12, 2010 at 04:51:23PM +0200, Alejandro Exojo wrote: I think that disk usage is an issue in most software, not just KDE apps. For example, look at ~/.thumbnails, and you will see that is probably filled with many thumbnails of images that you deleted long ago or that you saw just once. Same thing for apps that store some data about all files that you open, without limit and without expiration. I purge .thumbnails occasionally for this reason. I also wasn't happy to see my /var/tmp/kdecache-* grow to 400 MBish either, so I regularly purge that now on apt-get upgrades. Most recently I saw .xsession-errors grow to ~600 MB in just under a few days, so I've /dev/nulled that too except for when I need it. I would run a cron job like this: mar...@shambhala:~ du -sh .thumbnails 434M.thumbnails mar...@shambhala:~ find ~/.thumbnails -atime +30 -type f | wc -l 10030 mar...@shambhala:~ find ~/.thumbnails -atime +30 -type f -delete mar...@shambhala:~ du -sh .thumbnails 30M .thumbnails This way you have the most recently accessed thumbnails at hand while still reducing the size of the directory by a big margin. Note: This will not work with noatime ;-). There you could go after mtime or ctime, but this might delete thumbnails created long before even so they are still being accessed regularily. I use relatime here. Of course, the user should be not be required to create a cronjob for that, but I have a lot of those already, so i do not mind for now. I use the handy fcron for it: mar...@shambhala:~ fcrontab -l 17:04:44 listing martin's fcrontab # Backup der KDE-Konfiguration @ 2d /usr/bin/rdiff-backup ~/.kde/ ~/Backup/KDE @ 2m /usr/bin/rdiff-backup --remove-older-than 2M --force ~/Backup/KDE # Backup der Iceweasel-Konfiguration @ 2m /usr/bin/rdiff-backup --remove-older-than 2M --force ~/Backup/Iceweasel @ 2d /usr/bin/rdiff-backup ~/.mozilla/ ~/Backup/Iceweasel # Spam wegräumen @ 2d /usr/bin/find ~/.crm114/reaver_cache -type f -and -mtime +1 -delete # Thumbnails aufräumen, 17.5.2010 @ 10d find ~/.thumbnails -atime +30 -type f -delete # Statistiken @ 5d /home/martin/bin/machine-update -m @ 5d (w ; cat /sys/power/tuxonice/debug_info) | mail -s TuxOnIce stats mar...@localhost @ 1m uprecords | mail -s Uptime records mar...@localhost So yeah, it's a fairly epidemic problem. I probably notice more than most because I do daily remote-incremental backups over a DSL line, so an incremental backup of a few hundred MB definitely gets noticed. Sometimes it's OK, sometimes it's not. Deactivating all debug output in kdebugdialog has helped here a lot: mar...@shambhala:~ ls -lh .xsession-errors -rw--- 1 martin martin 3,9M 17. Mai 16:31 .xsession-errors Just check the checkmark option at the bottom of the window. In case of debugging a problem, you can activate it again. I think debugging should be off by default. -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 signature.asc Description: This is a digitally signed message part.
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Saturday 15 May 2010, Nicolas Alvarez wrote: Allan Sandfeld Jensen wrote: On Thursday 13 May 2010, Nicolas Alvarez wrote: As a KDE4.3 user with 1TB of disk, I'm way more annoyed at kio_http_cache_cleaner starting regularly and eating my computer's I/O than the HTTP cache growing big. I can easily notice when the the cache cleaner is running from the HD light, the HD noise, and the general system lag. There *has* to be a better way to clean an HTTP cache than to stat every single file in the cache to then delete a tiny fraction of them. Someone told me 4.4 improved in this area, but I don't know specifics, and remain skeptical. If stat updates the files' atime, you may get a lot better performance by simply mounting the relevant partition with relatime or noatime (which for some reason is still not default). After some stracing: It doesn't just stat the files. It first stats, then actually opens the files and *reads* the first few bytes of all of them (where the HTTP cache information is stored). Yes, the header contains the http expire date there is no other way to get it. Though I would have thought it would keep the statistics in memory instead of rereading it on every cycle. Are you sure that this isn't just on startup? `Allan -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201005152041.21376.deb...@carewolf.com
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Thursday 13 May 2010, Nicolas Alvarez wrote: As a KDE4.3 user with 1TB of disk, I'm way more annoyed at kio_http_cache_cleaner starting regularly and eating my computer's I/O than the HTTP cache growing big. I can easily notice when the the cache cleaner is running from the HD light, the HD noise, and the general system lag. There *has* to be a better way to clean an HTTP cache than to stat every single file in the cache to then delete a tiny fraction of them. Someone told me 4.4 improved in this area, but I don't know specifics, and remain skeptical. If stat updates the files' atime, you may get a lot better performance by simply mounting the relevant partition with relatime or noatime (which for some reason is still not default). Mvh Allan -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201005141837.20338.deb...@carewolf.com
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Friday 14 May 2010 11:37:19 Allan Sandfeld Jensen wrote: On Thursday 13 May 2010, Nicolas Alvarez wrote: There *has* to be a better way to clean an HTTP cache than to stat every single file in the cache to then delete a tiny fraction of them. Someone told me 4.4 improved in this area, but I don't know specifics, and remain skeptical. If stat updates the files' atime, you may get a lot better performance by simply mounting the relevant partition with relatime or noatime (which for some reason is still not default). noatime breaks a few applications. I recommend against it. I've not seen similar issues with relatime, although it is theoretically possible. I mount using relatime, normally. -- Boyd Stephen Smith Jr. ,= ,-_-. =. b...@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/\_/ signature.asc Description: This is a digitally signed message part.
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Friday 14 May 2010, Boyd Stephen Smith Jr. wrote: On Friday 14 May 2010 11:37:19 Allan Sandfeld Jensen wrote: On Thursday 13 May 2010, Nicolas Alvarez wrote: There *has* to be a better way to clean an HTTP cache than to stat every single file in the cache to then delete a tiny fraction of them. Someone told me 4.4 improved in this area, but I don't know specifics, and remain skeptical. If stat updates the files' atime, you may get a lot better performance by simply mounting the relevant partition with relatime or noatime (which for some reason is still not default). noatime breaks a few applications. I recommend against it. I've not seen similar issues with relatime, although it is theoretically possible. I mount using relatime, normally. It breaks mutt, relatime was invented to fix the mutt issue. A lot of netbooks and machines with only SSDs use noatime now, if noatime broke anything serious at any point, those issues will have been solved by now, SSDs have been out for some time and they need noatime. Also possibly if you don't like the mount flags: chattr -R +A /var/tmp Also add chattr -R +A $KDEHOME The later is good for the config files. `Allan -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201005141955.46390.deb...@carewolf.com
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Friday 14 May 2010 12:55:45 Allan Sandfeld Jensen wrote: On Friday 14 May 2010, Boyd Stephen Smith Jr. wrote: noatime breaks a few applications. I recommend against it. I've not seen similar issues with relatime, although it is theoretically possible. I mount using relatime, normally. It breaks mutt, relatime was invented to fix the mutt issue. A lot of netbooks and machines with only SSDs use noatime now, if noatime broke anything serious at any point, those issues will have been solved by now, SSDs have been out for some time and they need noatime. SSDs marketed for HD replacement do not need noatime. They have re-write limits such that, if written to at maximum speed for the lifetime of a comparable HD, less than the number of reserved cells would fail and no data would be lost. They are roughly 1.5 orders of magnitude more durable than thumbdrives -- they have to be to stand a MS Windows page file or a Linux swap partition/file. noatime can increase performance, it does so by breaking the expectations of at least mutt (possibly others). atime updates are a normal expectation, since they've been included in the documentation of dozens of UNIX system calls since before standardization started on POSIX. relatime is a much better alternative and is roughly equivalent to the OS deciding to combine atime updates, which is explicitly allowed by the standards. -- Boyd Stephen Smith Jr. ,= ,-_-. =. b...@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/\_/ signature.asc Description: This is a digitally signed message part.
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wed, May 12, 2010 at 17:36, Christoph Burgmer christoph.burg...@stud.uni-karlsruhe.de wrote: 4.4 was supposed to have major improvements over 4.3. 4.2 was supposed to fix major issues of 4.1 and 4.0. Also called, I believe, the first stable version of KDE 4. Every minor release has been significantly better than its predecessor. 4.4 might not be perfect, but it's the best KDE version since 3.5.10 and it's the only one which has a realistic chance of ending up in Squeeze. Richard -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/aanlktikdvaco7hfpdg1mxiyeqnfz9yb7pqzb9srpp...@mail.gmail.com
Re: KDE 4.4.3 upgrade eats 141 MB of /home
Allan Sandfeld Jensen wrote: On Thursday 13 May 2010, Nicolas Alvarez wrote: As a KDE4.3 user with 1TB of disk, I'm way more annoyed at kio_http_cache_cleaner starting regularly and eating my computer's I/O than the HTTP cache growing big. I can easily notice when the the cache cleaner is running from the HD light, the HD noise, and the general system lag. There *has* to be a better way to clean an HTTP cache than to stat every single file in the cache to then delete a tiny fraction of them. Someone told me 4.4 improved in this area, but I don't know specifics, and remain skeptical. If stat updates the files' atime, you may get a lot better performance by simply mounting the relevant partition with relatime or noatime (which for some reason is still not default). After some stracing: It doesn't just stat the files. It first stats, then actually opens the files and *reads* the first few bytes of all of them (where the HTTP cache information is stored). -- Nicolas (I read mailing lists through Gmane. Please don't Cc me on replies; it makes me get one message on my newsreader and another on email.) -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/hslb13$ib...@dough.gmane.org
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wednesday, 2010-05-12, Mike Kasick wrote: On Wed, May 12, 2010 at 10:41:35AM +0200, Kevin Krammer wrote: Since you are writing a bit down that you think it is caused by kres-migrator, where did you get it from (here it seems to be part of the kdepim-runtime package). Yes, kres-migrator is part of kdepim-runtime. I do have that package installed, as it seems to be indirectly depended upon by kde-minimal. It's kdepim itself, and its application dependencies (kaddresbook, kalarm, kmail knode, knotes, kontact, korganizer, etc.) I don't have installed. Maybe the kdepim-runtime dependency itself is a bug--can't say. It depends. One possible way would be to check every single application (and potentially their plugins) for PIM related functionality. I guess it is just easier to depend on runtime. These processes are run on- demand only anyway, so in the worst case they consume disk space if nothing is actually using PIM functionality. kres-migrator is called when an application accesses the KResource framework, e.g. some app accessing the old addressbook API. Not using KDEPIM apps does not necessarily mean non of your other applications access PIM data. Looks like the culprit here is libkabc. There's a Default Addressboook created by the library, that's presumably empty. I'm not sure what's loading libkabc in the first place. I do know that I didn't even have kabc database files (.kde/share/apps/kabc/std.vcf*) until upgrading to 4.4. Maybe it's an explicit part of the migration? Or I suppose one of the panel widgets I'm using might depend on it now, but I don't believe that's the case. Ah, right. Probably a KRunner plugin for actions on contacts. Looked into this a bit. The InnoDB documentation itself is a little lacking on describing its particular architecture, but there's an InnoDB tuning tutorial [1] that's rather helpful. These files serve as InnoDB's REDO logs. They serve two purposes. First, committed transactions are written to the REDO logs sequentially, so that table updates (with possible random seeks) can be done in a write-back mode at leisure. Second, REDO logs serve as a durability mesaure. Each time the database is restarted, the REDO logs are replayed to ensure that recent transactions have been properly commited--say if either the database is kill -9ed or there's other table corruption. They may also be used in recovery, whereby if table corruption is found and old tables can be reloaded from backup, then the REDO logs can be replayed to bring the tables up to date. You can also forward REDO logs to standby (fail-over) servers to ensure their database tables are up to date. The REDO logs themselves contain row updates from insert/update statements. So for a given row length, the REDO logs contain the last LOG_SIZE/ROW_LENGTH transactions. They're not used in selects or other non-mutating accesses. REDO log size is not an issue of correctness. A small log size might result in decreased performance by forcing a burst of inserts/updates to be committed to table before completing a transaction. A larger log size may also be of benefit in data recovery if database corruption is found, and a recent enough table backup is maintained so that the REDO log still contains all non-backed up transactions. Ah, good research, thank you! Let's try to quantify this a bit. I'm not exactly sure what kind of database workloads Akonadi is targetting, but for PIM applications we're looking at managing (1) contacts, (2) calendar entries, (3) TODO tasks, (4) notes-to-self, etc. It seems to me that each of these things results in: Just for the context, this is for KDE 4.4 4.5 potentially adds (5) emails - Table row length on order of 1 kB. I think the preconfigured threshold for database cached parts is 4KB, though there can be several such parts per item (depends on the data type). - Total number of rows 10,000 (how many people do you know?) - Largely read-only data sets, grows over period of years. - A working set (actively updated rows) 1,000 per day. Probably 100. Right, again in in the 4.4 context. With emails these can easily be surpassed, especially on update rate (mails come in, get marked read, moved, deleted, etc) The part that bothers me is that the Akonadi folks are basically aware of the situation, and feel justified in claiming [2] that 100+ MB of disk is reasonable. Franlky, if you ask even an arm-chair DBA if using InnoDB with these parameters are appropriate for per-user PIM management, they'll look at you like your crazy--which is, from what I can tell, the underlying reason for so much of the dislike with KDE 4.4. My take is that there is quite some room for optimizations through input by people with good knowledge of database systems. As far as I know the developers had help from some MySQL expert on the initial configuration, but since MySQL evolves over time the chosen
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wednesday, 2010-05-12, Mike Kasick wrote: On Wed, May 12, 2010 at 10:48:43AM +0200, Richard Hartmann wrote: - Disable kres-migrator. Or at least add a debconf option to kdepim-runtime presenting the option of running kres-migrator by default or disabling it. Did you do any research as to what the longer-term implications of foregoing this might be? I am not saying there are any, I honestly don't know. There shouldn't be any long-term implications as long as the old KResource code still exists. I'm assuming it's being deprecated as of KDE 4.4, but I'd be surprised if it's removed in KDE 4.5, especially since delaying migration is an official recommendation. The KResource API is public API of KDE4, it can at best removed at the first KDE5 version (which is probably still several years away). Some of its plugins are public API as well (those from kdepimlibs) so they stay too. Problematic are plugins from kdepim, they can be replaced by respective Akonadi base resource implementations. Once the DB mess is sorted out (KDE 4.5?), one can manually migrate the data by running kres-migrator. I actually did this accidentally. The migrator _might_ also run if we remove the configuration bit, but I've not tried. Right, I think it will run if it does not find its configuration, thus not knowing whether it has already run. Could lead to duplicated resources on Akonadi side though (if it has been run before). Cheers, Kevin signature.asc Description: This is a digitally signed message part.
Re: KDE 4.4.3 upgrade eats 141 MB of /home
Mike Kasick wrote: On Wed, May 12, 2010 at 04:51:23PM +0200, Alejandro Exojo wrote: I think that disk usage is an issue in most software, not just KDE apps. For example, look at ~/.thumbnails, and you will see that is probably filled with many thumbnails of images that you deleted long ago or that you saw just once. Same thing for apps that store some data about all files that you open, without limit and without expiration. I purge .thumbnails occasionally for this reason. I also wasn't happy to see my /var/tmp/kdecache-* grow to 400 MBish either, so I regularly purge that now on apt-get upgrades. Most recently I saw .xsession-errors grow to ~600 MB in just under a few days, so I've /dev/nulled that too except for when I need it. As a KDE4.3 user with 1TB of disk, I'm way more annoyed at kio_http_cache_cleaner starting regularly and eating my computer's I/O than the HTTP cache growing big. I can easily notice when the the cache cleaner is running from the HD light, the HD noise, and the general system lag. There *has* to be a better way to clean an HTTP cache than to stat every single file in the cache to then delete a tiny fraction of them. Someone told me 4.4 improved in this area, but I don't know specifics, and remain skeptical. -- Nicolas (I read mailing lists through Gmane. Please don't Cc me on replies; it makes me get one message on my newsreader and another on email.) -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/hshkqb$8o...@dough.gmane.org
Re: KDE 4.4.3 upgrade eats 141 MB of /home
As a KDE4.3 user with 1TB of disk, I'm way more annoyed at kio_http_cache_cleaner starting regularly and eating my computer's I/O than the HTTP cache growing big. I can easily notice when the the cache cleaner is running from the HD light, the HD noise, and the general system lag. There *has* to be a better way to clean an HTTP cache than to stat every single file in the cache to then delete a tiny fraction of them. Someone told me 4.4 improved in this area, but I don't know specifics, and remain skeptical. Officially this problem has been solved. The only difference I see though is that killing the process will instantly create a new one. So for me it got worse, rather then better. One would think there was another way, after all other browsers seem to get by without that much IO. I guess it's just an unpopular area of KDE so nobody really wants to sit down and fix it. Never had any issues with 3.5, not sure what was changed afterwards. -Christoph -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201005132216.14037.christoph.burg...@stud.uni-karlsruhe.de
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wednesday, 2010-05-12, Mike Kasick wrote: KDE 4 uses Akonadi as its PIM storage backend. Akonadi, at least in unstable, only supports MySQL (and PostgreSQL) as backends. And the default configuration of Akonadi is to run a per-user instance of MySQL with InnoDB tables with two 64 MB transaction logs. Apparently such has been the case for a while, and anyone who has been using KDE PIM applications has probably noticed such. But I _don't_ use KDE PIM! I don't even have kdepim installed. Since you are writing a bit down that you think it is caused by kres-migrator, where did you get it from (here it seems to be part of the kdepim-runtime package). The only reason why I even have akonadi-server and mysql-server-core installed is because kde-minimal ultimately depends on them, and I'd rather not micromanage my KDE packages. In any event, Akonadi is a known problem, and many folks are unhappy about kdepim 4.4's inclusion in squeeze, etc. It's not worth going into that here. Point is, up until now, folks who didn't use KDE PIM applications never had to worry about akonadi/mysql/141 MB. But apparently in KDE 4.4, kres-migrator automatically migrates the old KResource mechanism to an Akonadi-backed data store. kres-migrator is called when an application accesses the KResource framework, e.g. some app accessing the old addressbook API. Not using KDEPIM apps does not necessarily mean non of your other applications access PIM data. Thus burdening every KDE user, including brand new user accounts, with akonadi/mysql/141 MB less disk space. I guess one possible improvement would be to have /etc/skel include a kres- migratorrc with migration being disabled. There's a couple solutions to this problem. - Long term: Don't use MySQL as the default Akonadi backend, which is currently infeasible. Actually it is currently only one of two feasable solutions. The other one being Postgres. Virtuoso's SQL interface is not up to the task yet and SQLite. while most working already, has still some problems (last I heard about it). - Reduce the default size of the InnoDB transaction logs in the Akonadi per-user MySQL configuration. I'm not familiar enough with InnoDB tuning to offer an appropriate value, but I would suggest 4 MB to be the upper limit of a reasonable per-user log size, it terms of what I'm comfortable spending towards unproductive disk usage. Presumably if 64 MB is the default, it's intended for databases with somewhat significant transaction volumes. I expect that in average usage, Akonadi sees very, very few transactions relatively speaking. If one can get away with a 1 MB transaction log or less, even better! Anyone know what kind of data is stored in these logs? Does the necessary size depend on the number of accesses, number of operations per transaction or amount of data in transactions? Cheers, Kevin signature.asc Description: This is a digitally signed message part.
Re: KDE 4.4.3 upgrade eats 141 MB of /home
El Miércoles, 12 de Mayo de 2010, Mike Kasick escribió: Apologies for my rant. I just can't comprehend how such disk usage is even remotely acceptable for PIM usage patterns. Thanks a lot for doing this research. I've noticed that Akonadi takes a lot of space in ~/.local and I was concerned, but I didn't know what was causing this. I think that disk usage is an issue in most software, not just KDE apps. For example, look at ~/.thumbnails, and you will see that is probably filled with many thumbnails of images that you deleted long ago or that you saw just once. Same thing for apps that store some data about all files that you open, without limit and without expiration. -- Alex (a.k.a. suy) | GPG ID 0x0B8B0BC2 http://barnacity.net/ | http://disperso.net -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201005121651.24377@badopi.org
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wed, May 12, 2010 at 10:48:43AM +0200, Richard Hartmann wrote: - Long term: Don't use MySQL as the default Akonadi backend, which is currently infeasible. It is my understanding that KDE 4.5 will let you choose different backends. Recent threads on this list suggest that 4.5 will have a number of improvements over 4.4. It's unfortunate that the release timings don't line up well, because we're going to end up having to support 4.4 in squeeze for a long time to come. noah signature.asc Description: Digital signature
Re: KDE 4.4.3 upgrade eats 141 MB of /home
Am Mittwoch, 12. Mai 2010 schrieb Alejandro Exojo: I think that disk usage is an issue in most software, not just KDE apps. For example, look at ~/.thumbnails, and you will see that is probably filled with many thumbnails of images that you deleted long ago or that you saw just once. Same thing for apps that store some data about all files that you open, without limit and without expiration. You can tell Gwenview to remove .thumbnails on exit. -Christoph -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201005121709.19874.christoph.burg...@stud.uni-karlsruhe.de
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wednesday 12 May 2010 10:14:52 Noah Meyerhans wrote: On Wed, May 12, 2010 at 10:48:43AM +0200, Richard Hartmann wrote: - Long term: Don't use MySQL as the default Akonadi backend, which is currently infeasible. It is my understanding that KDE 4.5 will let you choose different backends. Recent threads on this list suggest that 4.5 will have a number of improvements over 4.4. It's unfortunate that the release timings don't line up well, because we're going to end up having to support 4.4 in squeeze for a long time to come. Options: 1. Conspire to keep the RC-bug count high enough on other packages for long enough that KDE 4.5 gets into testing. 2. Convince upstream to to release KDE SC 4.4½.0 before the freeze, or roll these new features into another KDE SC 4.4.x release before the freeze 3. Where possible, get the feature into Debian by working on the dependencies outside of KDE-proper. I think I'm going to work on 3. Virtuoso, Akonadi, and Strigi (among others) may be appropriate for a upgrade before either the freeze or the KDE SC 4.5.0 release. The KDE-integration parts might be missing, but as long as new versions don't cripple KDE (AFAIK, their only consumer in Debian) new upstream versions and bug fixes can be accepted until the freeze. I'm against delaying a Debian freeze/release until after some particular upstream release. If we routinely did that, whether for bug fixes or features we'd never make a release. If it is an RC-bug, the fix can be backported. -- Boyd Stephen Smith Jr. ,= ,-_-. =. b...@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/\_/ signature.asc Description: This is a digitally signed message part.
Re: KDE 4.4.3 upgrade eats 141 MB of /home
Am Mittwoch, 12. Mai 2010 schrieb Noah Meyerhans: On Wed, May 12, 2010 at 10:48:43AM +0200, Richard Hartmann wrote: - Long term: Don't use MySQL as the default Akonadi backend, which is currently infeasible. It is my understanding that KDE 4.5 will let you choose different backends. Recent threads on this list suggest that 4.5 will have a number of improvements over 4.4. It's unfortunate that the release timings don't line up well, because we're going to end up having to support 4.4 in squeeze for a long time to come. 4.4 was supposed to have major improvements over 4.3. 4.2 was supposed to fix major issues of 4.1 and 4.0. Also called, I believe, the first stable version of KDE 4. I'd say none of the current versions can be called stable. Every new feature seems to need at least two point versions to become stable. With currently every new version having major new features I don't see a release coming soon that could honestly be called stable. 4.5 will have similar improvements and new issues over 4.4 as that had over 4.3 as that had over 4.2 ... Come to think of it, less expectations raised would've maybe led to less disappointment. -Christoph -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201005121736.43081.christoph.burg...@stud.uni-karlsruhe.de
Re: KDE 4.4.3 upgrade eats 141 MB of /home
El Miércoles, 12 de Mayo de 2010, Christoph Burgmer escribió: Am Mittwoch, 12. Mai 2010 schrieb Alejandro Exojo: I think that disk usage is an issue in most software, not just KDE apps. For example, look at ~/.thumbnails, and you will see that is probably filled with many thumbnails of images that you deleted long ago or that you saw just once. Same thing for apps that store some data about all files that you open, without limit and without expiration. You can tell Gwenview to remove .thumbnails on exit. The point of .thumbnails to be created, I thought it was to have the scaled images already generated. If I did this, what's the point of having this directory? -- Alex (a.k.a. suy) | GPG ID 0x0B8B0BC2 http://barnacity.net/ | http://disperso.net -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201005121828.35294@badopi.org
Re: KDE 4.4.3 upgrade eats 141 MB of /home
[Alejandro Exojo - Mittwoch 12 Mai 2010 18:28:35] El Miércoles, 12 de Mayo de 2010, Christoph Burgmer escribió: Am Mittwoch, 12. Mai 2010 schrieb Alejandro Exojo: I think that disk usage is an issue in most software, not just KDE apps. For example, look at ~/.thumbnails, and you will see that is probably filled with many thumbnails of images that you deleted long ago or that you saw just once. Same thing for apps that store some data about all files that you open, without limit and without expiration. You can tell Gwenview to remove .thumbnails on exit. The point of .thumbnails to be created, I thought it was to have the scaled images already generated. If I did this, what's the point of having this directory? If you want to have these images there in case they are needed, keep the folder, if disk space is more important than loading speed, remove it regularily. Regards -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201005121845.25780.schwarz...@gmail.com
Re: KDE 4.4.3 upgrade eats 141 MB of /home
Hello, On trečiadienis 12 Gegužė 2010 18:48:04 Boyd Stephen Smith Jr. wrote: Options: 1. Conspire to keep the RC-bug count high enough on other packages for long enough that KDE 4.5 gets into testing. 2. Convince upstream to to release KDE SC 4.4½.0 before the freeze, or roll these new features into another KDE SC 4.4.x release before the freeze 3. Where possible, get the feature into Debian by working on the dependencies outside of KDE-proper. I think I'm going to work on 3. Virtuoso, Akonadi, and Strigi (among others) may be appropriate for a upgrade before either the freeze or the KDE SC 4.5.0 release. The KDE-integration parts might be missing, but as long as new versions don't cripple KDE (AFAIK, their only consumer in Debian) new upstream versions and bug fixes can be accepted until the freeze. Except that kdesupport applications are typically not the problem. It is KDE integration where majority of bugs are. -- Modestas Vainius modes...@vainius.eu signature.asc Description: This is a digitally signed message part.
Re: KDE 4.4.3 upgrade eats 141 MB of /home
Am Mittwoch, 12. Mai 2010 schrieb Alejandro Exojo: El Miércoles, 12 de Mayo de 2010, Christoph Burgmer escribió: Am Mittwoch, 12. Mai 2010 schrieb Alejandro Exojo: I think that disk usage is an issue in most software, not just KDE apps. For example, look at ~/.thumbnails, and you will see that is probably filled with many thumbnails of images that you deleted long ago or that you saw just once. Same thing for apps that store some data about all files that you open, without limit and without expiration. You can tell Gwenview to remove .thumbnails on exit. The point of .thumbnails to be created, I thought it was to have the scaled images already generated. If I did this, what's the point of having this directory? You would at least have them available while using the current Gwenview instance. Not sure how Dolphin interacts with that. -Christoph -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201005121835.54871.christoph.burg...@stud.uni-karlsruhe.de
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wednesday 12 May 2010 11:47:08 Modestas Vainius wrote: On trečiadienis 12 Gegužė 2010 18:48:04 Boyd Stephen Smith Jr. wrote: Virtuoso, Akonadi, and Strigi (among others) may be appropriate for a upgrade before either the freeze or the KDE SC 4.5.0 release. The KDE-integration parts might be missing, but as long as new versions don't cripple KDE (AFAIK, their only consumer in Debian) new upstream versions and bug fixes can be accepted until the freeze. Except that kdesupport applications are typically not the problem. It is KDE integration where majority of bugs are. I propose we drop KDE from stable for the Squeeze release, then. ;P Seriously though, if the RC-bug list keeps falling, there's no time to wait on KDE SC 4.5. Let's try and get a high-quality KDE SC 4.4 ready for Squeeze. -- Boyd Stephen Smith Jr. ,= ,-_-. =. b...@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/\_/ signature.asc Description: This is a digitally signed message part.
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wed, May 12, 2010 at 10:41:35AM +0200, Kevin Krammer wrote: Since you are writing a bit down that you think it is caused by kres-migrator, where did you get it from (here it seems to be part of the kdepim-runtime package). Yes, kres-migrator is part of kdepim-runtime. I do have that package installed, as it seems to be indirectly depended upon by kde-minimal. It's kdepim itself, and its application dependencies (kaddresbook, kalarm, kmail knode, knotes, kontact, korganizer, etc.) I don't have installed. Maybe the kdepim-runtime dependency itself is a bug--can't say. kres-migrator is called when an application accesses the KResource framework, e.g. some app accessing the old addressbook API. Not using KDEPIM apps does not necessarily mean non of your other applications access PIM data. Looks like the culprit here is libkabc. There's a Default Addressboook created by the library, that's presumably empty. I'm not sure what's loading libkabc in the first place. I do know that I didn't even have kabc database files (.kde/share/apps/kabc/std.vcf*) until upgrading to 4.4. Maybe it's an explicit part of the migration? Or I suppose one of the panel widgets I'm using might depend on it now, but I don't believe that's the case. Anyone know what kind of data is stored in these logs? Looked into this a bit. The InnoDB documentation itself is a little lacking on describing its particular architecture, but there's an InnoDB tuning tutorial [1] that's rather helpful. These files serve as InnoDB's REDO logs. They serve two purposes. First, committed transactions are written to the REDO logs sequentially, so that table updates (with possible random seeks) can be done in a write-back mode at leisure. Second, REDO logs serve as a durability mesaure. Each time the database is restarted, the REDO logs are replayed to ensure that recent transactions have been properly commited--say if either the database is kill -9ed or there's other table corruption. They may also be used in recovery, whereby if table corruption is found and old tables can be reloaded from backup, then the REDO logs can be replayed to bring the tables up to date. You can also forward REDO logs to standby (fail-over) servers to ensure their database tables are up to date. The REDO logs themselves contain row updates from insert/update statements. So for a given row length, the REDO logs contain the last LOG_SIZE/ROW_LENGTH transactions. They're not used in selects or other non-mutating accesses. REDO log size is not an issue of correctness. A small log size might result in decreased performance by forcing a burst of inserts/updates to be committed to table before completing a transaction. A larger log size may also be of benefit in data recovery if database corruption is found, and a recent enough table backup is maintained so that the REDO log still contains all non-backed up transactions. Let's try to quantify this a bit. I'm not exactly sure what kind of database workloads Akonadi is targetting, but for PIM applications we're looking at managing (1) contacts, (2) calendar entries, (3) TODO tasks, (4) notes-to-self, etc. It seems to me that each of these things results in: - Table row length on order of 1 kB. - Total number of rows 10,000 (how many people do you know?) - Largely read-only data sets, grows over period of years. - A working set (actively updated rows) 1,000 per day. Probably 100. Thus, I would conclude that tables rarely grow larger than 10 MB (1 kB * 10,000). The number of inserts/updates per table shouldn't exceed 100 kB-1 MB per day. We're also unlikely to see bursty updates anytime information is manually provided, since it has to be typed in. Bursty updates would happen on device synchronization, at which point you might see 100 kB-1MB of table updates in a few-second window. This means that we should be able to record 1-10 days of update history with 1 MB transaction logs. And that's under a heavy PIM load. We also know that InnoDB's leisure write pace is 64 pages per second. Each page is 16 kB. If we're pessimistic, and say that row updates are randomly distributed enough such that there's only a single updated-row per page, then it would take up to 1.5s-15.6s to leisurly flush 100 kB-1MB of table updates. So it really only makes sense to increase the REDO log beyond 1 MB for performance purposes if we expect 1,000 of random-row (1 MB) updates to occur more frequently than once every 15 seconds. That doesn't really seem plausible with these kinds of workloads. In the even it _does_ happen, then it just takes a little longer to finish the sync. So that's my argument for 1 MB REDO logs. Let's look at the other defaults for a bit: innodb_log_buffer_size=1M -- Should be large enough to assemble a transaction in memory. 1 MB fits 1000 1kB-single-row transactions. Fine. innodb_buffer_pool_size=80M -- Page cache size, including cached read pages. Seems fairly large and
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wed, May 12, 2010 at 04:51:23PM +0200, Alejandro Exojo wrote: I think that disk usage is an issue in most software, not just KDE apps. For example, look at ~/.thumbnails, and you will see that is probably filled with many thumbnails of images that you deleted long ago or that you saw just once. Same thing for apps that store some data about all files that you open, without limit and without expiration. I purge .thumbnails occasionally for this reason. I also wasn't happy to see my /var/tmp/kdecache-* grow to 400 MBish either, so I regularly purge that now on apt-get upgrades. Most recently I saw .xsession-errors grow to ~600 MB in just under a few days, so I've /dev/nulled that too except for when I need it. So yeah, it's a fairly epidemic problem. I probably notice more than most because I do daily remote-incremental backups over a DSL line, so an incremental backup of a few hundred MB definitely gets noticed. Sometimes it's OK, sometimes it's not. -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100512174327.gb17...@club.cc.cmu.edu
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wed, May 12, 2010 at 10:48:43AM +0200, Richard Hartmann wrote: - Disable kres-migrator. Or at least add a debconf option to kdepim-runtime presenting the option of running kres-migrator by default or disabling it. Did you do any research as to what the longer-term implications of foregoing this might be? I am not saying there are any, I honestly don't know. There shouldn't be any long-term implications as long as the old KResource code still exists. I'm assuming it's being deprecated as of KDE 4.4, but I'd be surprised if it's removed in KDE 4.5, especially since delaying migration is an official recommendation. Once the DB mess is sorted out (KDE 4.5?), one can manually migrate the data by running kres-migrator. I actually did this accidentally. The migrator _might_ also run if we remove the configuration bit, but I've not tried. - Add a warning dialog during kdepim-runtime upgrade stating that kres-migrator will run by default and potentially consume a large amount of per-user disk space, unless migration is manually disabled by the above command. Personally, I would like to see such a warning, but I am not sure if that is acceptable by Debian's standards. apt-listchanges exists for a reason. It extends beyond just a change in package behavior, it's a non-reversible and potentially aggravating alteration to user data. I've seen similar warnings before, most recently to rebuild OpenOffice.org .rdb files due to possible corruption. - Reduce the default size of the InnoDB transaction logs in the Akonadi per-user MySQL configuration. Seems like the best bet to me. If need be, the file size could be adapted automagically. But that functionality would be in 4.5 at the earliest unless it's patched downstream. Forgot to mention before, apparently there's an Innodb_log_waits counter that increments when you have write bursts that exceed the REDO log size. It can be used for feedback to increase the log size if needed. Perhaps something to suggest manually doing in NEWS. Also, what about /root? Should have same problem, if one runs a KDE desktop as root, which I do not. -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100512174253.ga17...@club.cc.cmu.edu
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wed, May 12, 2010 at 04:51:23PM +0200, Alejandro Exojo wrote: I think that disk usage is an issue in most software, not just KDE apps. For example, look at ~/.thumbnails, and you will see that is probably filled with many thumbnails of images that you deleted long ago or that you saw just once. Same thing for apps that store some data about all files that you open, without limit and without expiration. I purge .thumbnails occasionally for this reason. I also wasn't happy to see my /var/tmp/kdecache-* grow to 400 MBish either, so I regularly purge that now on apt-get upgrades. Most recently I saw .xsession-errors grow to ~600 MB in just under a few days, so I've /dev/nulled that too except for when I need it. So yeah, it's a fairly epidemic problem. I probably notice more than most because I do daily remote-incremental backups over a DSL line, so an incremental backup of a few hundred MB definitely gets noticed. Sometimes it's OK, sometimes it's not. -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100512173310.gc17...@club.cc.cmu.edu
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wed, May 12, 2010 at 10:48:43AM +0200, Richard Hartmann wrote: - Disable kres-migrator. Or at least add a debconf option to kdepim-runtime presenting the option of running kres-migrator by default or disabling it. Did you do any research as to what the longer-term implications of foregoing this might be? I am not saying there are any, I honestly don't know. There shouldn't be any long-term implications as long as the old KResource code still exists. I'm assuming it's being deprecated as of KDE 4.4, but I'd be surprised if it's removed in KDE 4.5, especially since delaying migration is an official recommendation. Once the DB mess is sorted out (KDE 4.5?), one can manually migrate the data by running kres-migrator. I actually did this accidentally. The migrator _might_ also run if we remove the configuration bit, but I've not tried. - Add a warning dialog during kdepim-runtime upgrade stating that kres-migrator will run by default and potentially consume a large amount of per-user disk space, unless migration is manually disabled by the above command. Personally, I would like to see such a warning, but I am not sure if that is acceptable by Debian's standards. apt-listchanges exists for a reason. It extends beyond just a change in package behavior, it's a non-reversible and potentially aggravating alteration to user data. I've seen similar warnings before, most recently to rebuild OpenOffice.org .rdb files due to possible corruption. - Reduce the default size of the InnoDB transaction logs in the Akonadi per-user MySQL configuration. Seems like the best bet to me. If need be, the file size could be adapted automagically. But that functionality would be in 4.5 at the earliest unless it's patched downstream. Forgot to mention before, apparently there's an Innodb_log_waits counter that increments when you have write bursts that exceed the REDO log size. It can be used for feedback to increase the log size if needed. Perhaps something to suggest manually doing in NEWS. Also, what about /root? Should have same problem, if one runs a KDE desktop as root, which I do not. -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100512172757.gb17...@club.cc.cmu.edu
Re: KDE 4.4.3 upgrade eats 141 MB of /home
El Miércoles, 12 de Mayo de 2010, Frederik Schwarzer escribió: If you want to have these images there in case they are needed, keep the folder, if disk space is more important than loading speed, remove it regularily. Of course, that's what I do. But a plain user should not be asked to regularly remove it, should he? It probably should be handled like the trash folder, which IIRC has a size/time limit. -- Alex (a.k.a. suy) | GPG ID 0x0B8B0BC2 http://barnacity.net/ | http://disperso.net -- To UNSUBSCRIBE, email to debian-kde-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201005130022.21523@badopi.org
Re: KDE 4.4.3 upgrade eats 141 MB of /home
On Wednesday 12 May 2010 17:22:21 Alejandro Exojo wrote: El Miércoles, 12 de Mayo de 2010, Frederik Schwarzer escribió: If you want to have these images there in case they are needed, keep the folder, if disk space is more important than loading speed, remove it regularily. Of course, that's what I do. But a plain user should not be asked to regularly remove it, should he? It probably should be handled like the trash folder, which IIRC has a size/time limit. I think sweeper (?) from kdeutils is supposed to handle this, right? -- Boyd Stephen Smith Jr. ,= ,-_-. =. b...@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/\_/ signature.asc Description: This is a digitally signed message part.
