Re: [gentoo-performance] Horrible performance
On Wednesday 25 August 2010 21:38:10 Alex Schuster wrote: J. Roeleveld writes: On Wednesday 25 August 2010 03:32:40 Alex Schuster wrote: I have an AMD Athlon(tm) Dual Core Processor 4850e CPU, on-board Radeon HD 3200 graphics, 4GB of memory, an 1.5 TB drive. Lots of LVM volumes, all encrypted, except for /usr/src and portage stuff. The system is ~amd64, and I have -march=k8-sse3 in my CFLAGS. Current kernel is 2.6.34-tuxonice, but I also tried others. I'm running KDE4 with desktop effects enabled, X itself takes about 30-40% of CPU time according to top. After system startup and login into KDE, 3.5G of RAM are occupied. This increases after a while, and I need swap space. Nothing to worry about I think. Encrypted filesystems can cause additional with activity, but I would expect that to remain the same over a long period. And I just moved my PORTAGE_TMPDIR to an unencrypted partition. Can LVM create noticeable overhead? I also resized my logical volumes a couple of times, could this lead to some LVM fragmentation? Theoretically, LVM will create an additional overhead. But I am extensibly using LVM on all my machines and haven't noticed any significant performance drops. LVM-fragmentation is a definite possibility. To defragment it, have a look at the following: http://bisqwit.iki.fi/source/lvm2defrag.html http://www.linuxquestions.org/questions/linux-software-2/how-do-i-lvm2-defrag- or-move-based-on-logical-volumes-689335/ I played with the first one on an older machine once and it does work quite nicely. However, how is the write and read performance on those disks? Here's the output of hdparm -t for all drives, 4 times. /dev/sda: (SATA system drive) Timing buffered disk reads: 118 MB in 3.08 seconds = 38.37 MB/sec Timing buffered disk reads: 194 MB in 3.11 seconds = 62.47 MB/sec Timing buffered disk reads: 322 MB in 3.01 seconds = 106.82 MB/sec Timing buffered disk reads: 244 MB in 3.00 seconds = 81.21 MB/sec /dev/sdb: (PATA master) Timing buffered disk reads: 114 MB in 3.02 seconds = 37.70 MB/sec Timing buffered disk reads: 114 MB in 3.00 seconds = 37.97 MB/sec Timing buffered disk reads: 116 MB in 3.05 seconds = 38.06 MB/sec Timing buffered disk reads: 116 MB in 3.05 seconds = 38.07 MB/sec /dev/sdc: (PATA slave) Timing buffered disk reads: 164 MB in 3.03 seconds = 54.21 MB/sec Timing buffered disk reads: 166 MB in 3.02 seconds = 55.04 MB/sec Timing buffered disk reads: 166 MB in 3.01 seconds = 55.10 MB/sec Timing buffered disk reads: 158 MB in 3.01 seconds = 52.41 MB/sec /dev/sdd: (SATA backup drive) Timing buffered disk reads: 314 MB in 3.00 seconds = 104.55 MB/sec Timing buffered disk reads: 312 MB in 3.01 seconds = 103.67 MB/sec Timing buffered disk reads: 308 MB in 3.01 seconds = 102.34 MB/sec Timing buffered disk reads: 314 MB in 3.00 seconds = 104.60 MB/sec The system drive throughput varies a lot, depending on other I/O. Those look ok to me, except that I would expect SATA-drives to be faster then PATA drives. You're running KDE4, guess you went for the default and use mysql for app- office/akonadi-server. I switched to using sqlite for this due to issues getting it to work with mysql. I think this might help there? So I only have to set the sqlite use flag and remove the mysql use flag for akonadi-server? I'm doing this now. And this also gives an example for what is going on here: And unset mysql. There is one issue that needs to be resolved manually with getting it to work with sqlite. See: http://forums.gentoo.org/viewtopic-t-834883-view- next.html?sid=3ae77f5bfba5e006e8745eedb4b6cfc4 Here is the bit that will solve the problem: -- $ cd ~/.local/share/akonadi $ sqlite3 akonadi.db sqlite INSERT INTO ResourceTable (name, isVirtual) VALUES ('akonadi_search_resource', 1); sqlite .exit -- I have just logged into KDE. I did not log out since yesterday, and when started a VM with vmplayer, the system swapped like crazy, I could not use it for minutes. After this, the panel did not react any more, and the desktop background did not redraw, so I logged out and in again. The VM started fine now. Well, could be faster, but maybe it was okay. Then I started answering your mail, and tried to reemerge akonadi-server, but I had a type, so portage took a long search for akomadi-server. meanwhile the dektop became quite unresponsive, load went high, and I made a screenshot [*]. If you look at the top right, gkrellm shows this above 'Proc'. The first increase at the left was after I started emerge, the 2nd at the right was after I pressed the PrtSc key. VMWare allows virtual machines to use more memory then is actually available. Also, there are settings in VMWare (possibly enabled by default) that cause the memory to be duplicated onto disk. This can cause issues like you are seeing. Performance does not feel too bad at
Re: [gentoo-performance] Horrible performance
On Thursday 26 August 2010 01:06:59 Alex Schuster wrote: J. Roeleveld writes: Can you post the result of: ps axu? This will give an indication which processes are running and using a lot of memory. The fifth colums gives the memory, right? Should this add up to the total of the 'used' column in free -m? Because it does not: wo...@weird ~ $ free -m total used free sharedbuffers cached Mem: 3452 3117334 0 39279 -/+ buffers/cache: 2798653 Swap: 4094 1472 2622 wo...@weird ~ $ total=0; for rss in $( ps aux | grep -v USER | awk '{print $6}' ); do (( total += rss )); done; echo $(( total / 1024 )) 1984 So, I get a sum of around 2 G with ps, while free -m shows 4.5 G. Whoops? Am I missing something here, or does it look like lots of RAM is not being freed? Wonko Don't forget the buffers/cache. The 2.5G you're missing is what is used in the buffers/cache line. At least, that is how I see it. Also, I generally look at the percentages used for the memory to find the memory-hogs. And your tv-viewer (java) is using quite a bit as well. Maybe someone with more experience with tv-viewer apps can take a look and maybe give a few pointers? -- Joost
Re: [gentoo-performance] Horrible performance
On Thursday 26 August 2010 02:11:44 Alex Schuster wrote: I think I just found the problem! I decided to give the open source ATI drivers a try again, and replaced 'fglrx' by 'radeon' in my xorg.conf. And all is fast now. Wow. I had already forgotten how fast a desktop should be. I'm not yet sure all is fine. On the first start, kwin crashed and all windows were on the first desktop, this never happened before. But it's working on the second login. I have no direct rendering, Xorg.0.log shows (EE) RADEON(0): [dri] RADEONDRIGetVersion failed (libdri too old). I had this before with xorg-server-1.7.7, now I'm emerging 1.8.2. Stay tuned. Wonko Ok, looks like your video-driver was using up too much resources. Either with all the desktop-effects or with something else. I can't help further with ATI-issues as I don't use them myself. I do know that I need to change more then just the driver in my xorg.conf when using nvidie-drivers. Am wondering if your crashes occured because of a similar issue. Eg. the dri module is only for the ATI-driver and not for the open-source one? -- Joost
Re: [gentoo-performance] Horrible performance
Update: I think I just found the problem! I decided to give the open source ATI drivers a try again, and replaced 'fglrx' by 'radeon' in my xorg.conf. And all is fast now. Wow. I had already forgotten how fast a desktop should be. I'm not yet sure all is fine. On the first start, kwin crashed and all windows were on the first desktop, this never happened before. But it's working on the second login. I have no direct rendering, Xorg.0.log shows (EE) RADEON(0): [dri] RADEONDRIGetVersion failed (libdri too old). I had this before with xorg-server-1.7.7, now I'm emerging 1.8.2. Stay tuned. I first got no dri with xorg-server 1.8, too, and then weird effects started happening, like the shift and ctrl keys not working. When tings crashed, I got a block screen and still could move the mouse, but that was all, even after Alt-SysRQ-R (which usually helps in these cases) I could not switch to a text terminal. But after a reboot all seemed perfect, but when I was writing a followe-up mail, X crashed. And I got several more crashes after working 5 minutes in KDE. I downgraded back to xorg-server 1.7.7-r1, and now dri is working, too. Looking good! KDE is running for over an hour now, I'm not using any swap at all, amarok is emerging in the background, and mplayer is playing smoothly. I experience little rendering problems in chromium (images appear over text) and quake3 feels a little sluggish, but I can live with this. Thanks for anyone who helped! Wonko
Re: [gentoo-performance] Horrible performance
On Thursday 26 August 2010 11:34:23 Alex Schuster wrote: Update: I first got no dri with xorg-server 1.8, too, and then weird effects started happening, like the shift and ctrl keys not working. When tings crashed, I got a block screen and still could move the mouse, but that was all, even after Alt-SysRQ-R (which usually helps in these cases) I could not switch to a text terminal. But after a reboot all seemed perfect, but when I was writing a followe-up mail, X crashed. And I got several more crashes after working 5 minutes in KDE. I downgraded back to xorg-server 1.7.7-r1, and now dri is working, too. Looking good! KDE is running for over an hour now, I'm not using any swap at all, amarok is emerging in the background, and mplayer is playing smoothly. I experience little rendering problems in chromium (images appear over text) and quake3 feels a little sluggish, but I can live with this. Thanks for anyone who helped! Wonko Glad you got it working properly now :) Enjoy the smoother desktop now -- Joost
Re: [gentoo-performance] Horrible performance
Joost Roeleveld writes: On Thursday 26 August 2010 02:11:44 Alex Schuster wrote: I think I just found the problem! I decided to give the open source ATI drivers a try again, and replaced 'fglrx' by 'radeon' in my xorg.conf. And all is fast now. Wow. I had already forgotten how fast a desktop should be. I'm not yet sure all is fine. On the first start, kwin crashed and all windows were on the first desktop, this never happened before. But it's working on the second login. I have no direct rendering, Xorg.0.log shows (EE) RADEON(0): [dri] RADEONDRIGetVersion failed (libdri too old). I had this before with xorg-server-1.7.7, now I'm emerging 1.8.2. Stay tuned. Wonko Ok, looks like your video-driver was using up too much resources. Either with all the desktop-effects or with something else. I often had desktop effects turned off. I can't help further with ATI-issues as I don't use them myself. I do know that I need to change more then just the driver in my xorg.conf when using nvidie-drivers. Am wondering if your crashes occured because of a similar issue. Eg. the dri module is only for the ATI-driver and not for the open-source one? I did not configure modules like dri in my xorg.conf, this seems tzo work automatically. With ati-drivers, I got errors in Xorg.0.log about dri and dri2 missing, apparently those come with the fglrx driver. With radeon, I get messages that dr and dri2 are loaded. So I only had to replace 'fglrx' by 'radeon' inthe device section. And to reboot. Or something. At least it did not work until I did that. I had loaded the drm kernel module by hand, so this was not the problem. Whatever, I'm glad the radeon driver is runnign fine now, for the first time after several tries. Wonko
Re: [gentoo-performance] Horrible performance
Joost Roeleveld writes: On Wednesday 25 August 2010 21:38:10 Alex Schuster wrote: And I just moved my PORTAGE_TMPDIR to an unencrypted partition. Can LVM create noticeable overhead? I also resized my logical volumes a couple of times, could this lead to some LVM fragmentation? Theoretically, LVM will create an additional overhead. But I am extensibly using LVM on all my machines and haven't noticed any significant performance drops. That's also what I heard. LVM-fragmentation is a definite possibility. To defragment it, have a look at the following: http://bisqwit.iki.fi/source/lvm2defrag.html Cool! http://www.linuxquestions.org/questions/linux-software-2/how-do-i-lvm2- defrag- or-move-based-on-logical-volumes-689335/ I played with the first one on an older machine once and it does work quite nicely. Any idea how to check how bad the fragmentation actually is? BTW: wo...@weird ~ $ mount | wc -l 48 I use LVM for about everything now, it makes things so much easier. First, I had two volume groups on my system drive, one for the system, placed at the front where the drive is supposed to be faster, and one for data. But I don't do this any more, it cuts down flexibility, and is probably not worth the effort. However, how is the write and read performance on those disks? Here's the output of hdparm -t for all drives, 4 times. /dev/sda: (SATA system drive) Timing buffered disk reads: 118 MB in 3.08 seconds = 38.37 MB/sec Timing buffered disk reads: 194 MB in 3.11 seconds = 62.47 MB/sec Timing buffered disk reads: 322 MB in 3.01 seconds = 106.82 MB/sec Timing buffered disk reads: 244 MB in 3.00 seconds = 81.21 MB/sec /dev/sdb: (PATA master) Timing buffered disk reads: 114 MB in 3.02 seconds = 37.70 MB/sec Timing buffered disk reads: 114 MB in 3.00 seconds = 37.97 MB/sec Timing buffered disk reads: 116 MB in 3.05 seconds = 38.06 MB/sec Timing buffered disk reads: 116 MB in 3.05 seconds = 38.07 MB/sec /dev/sdc: (PATA slave) Timing buffered disk reads: 164 MB in 3.03 seconds = 54.21 MB/sec Timing buffered disk reads: 166 MB in 3.02 seconds = 55.04 MB/sec Timing buffered disk reads: 166 MB in 3.01 seconds = 55.10 MB/sec Timing buffered disk reads: 158 MB in 3.01 seconds = 52.41 MB/sec /dev/sdd: (SATA backup drive) Timing buffered disk reads: 314 MB in 3.00 seconds = 104.55 MB/sec Timing buffered disk reads: 312 MB in 3.01 seconds = 103.67 MB/sec Timing buffered disk reads: 308 MB in 3.01 seconds = 102.34 MB/sec Timing buffered disk reads: 314 MB in 3.00 seconds = 104.60 MB/sec The system drive throughput varies a lot, depending on other I/O. Those look ok to me, except that I would expect SATA-drives to be faster then PATA drives. Well, they are, except for the sometimes busy system drive. And by now I get similar results as for the 2nd SATA drive, throughput is between 90 and 110 MB/sec. You're running KDE4, guess you went for the default and use mysql for app- office/akonadi-server. I switched to using sqlite for this due to issues getting it to work with mysql. I think this might help there? So I only have to set the sqlite use flag and remove the mysql use flag for akonadi-server? I'm doing this now. And this also gives an example for what is going on here: And unset mysql. There is one issue that needs to be resolved manually with getting it to work with sqlite. See: http://forums.gentoo.org/viewtopic-t-834883-view- next.html?sid=3ae77f5bfba5e006e8745eedb4b6cfc4 Here is the bit that will solve the problem: -- $ cd ~/.local/share/akonadi $ sqlite3 akonadi.db sqlite INSERT INTO ResourceTable (name, isVirtual) VALUES ('akonadi_search_resource', 1); sqlite .exit I did nothing about this by now, but I probabyl will soon. Thanks for the tip! I also had trouble with akonadi in the past, and it still gives warnings/errors at every startup, but at least it seems to work now. Disk I/O is, in my experience, a very common cause for freeze-ups. Can you test with unencrypted disks to see if the issue occurs then as well? Yes, I can do this. It's some work, but I tried so much, why not this. I have some free space, and already have written a backup script that automatically creates LVM snapshots, decrypts them, and backs it up, so I can do this from the running system. Ok, am interested to see if running unencrypted actually has benefits here. Me too, but as things are quite faster now already, the priority for this task is much lower than it was yesterday :) Can you post the result of: ps axu? This will give an indication which processes are running and using a lot of memory. First, here is free -m: total used free sharedbuffers cached Mem: 3452 3225226 0 54
Re: [gentoo-performance] Horrible performance
On Thursday 26 August 2010 17:48:44 Alex Schuster wrote: Joost Roeleveld writes: LVM-fragmentation is a definite possibility. To defragment it, have a look at the following: http://bisqwit.iki.fi/source/lvm2defrag.html Cool! Do test it first and check what it wants to do. Basically, it moves all the blocks around untill you have them all in the sequence you want them. There are some limitations, but it worked when I tested it. Btw, I provide no warranty what-so-ever, especially as I did not write any part of it :) http://www.linuxquestions.org/questions/linux-software-2/how-do-i-lvm2- defrag- or-move-based-on-logical-volumes-689335/ I played with the first one on an older machine once and it does work quite nicely. Any idea how to check how bad the fragmentation actually is? Yes: desktop ~ # lvs -o lv_name,seg_pe_ranges LV PE Ranges distfiles /dev/sda4:37632-39423 home /dev/sda4:9472-12031 home /dev/sda4:39424-40703 home /dev/sda4:42240-44799 opt/dev/sda4:8192-9471 opt/dev/sda4:40704-42239 packages /dev/sda4:45312-47359 portage/dev/sda4:3072-3327 swap /dev/sda4:0-3071 tmp/dev/sda4:3328-4607 usr/dev/sda4:6400-8191 usr/dev/sda4:81664-81919 usr/dev/sda4:44800-45311 var/dev/sda4:4608-5119 vartmp /dev/sda4:5120-6399 vartmp /dev/sda4:60160-61183 virtualbox /dev/sda4:12032-32511 virtualbox /dev/sda4:76544-81663 virtualbox /dev/sda4:58112-60159 virtualbox /dev/sda4:81920-83310 virtualbox /dev/sda4:47360-58111 virtualbox /dev/sda4:32512-33680 home, vartmp and virtualbox are fragmented into different sections. Alternatively, the first part of that script actually generates a text-file which you then need to edit into the next text-file. The first one actually shows you how the different parts are laid out. BTW: wo...@weird ~ $ mount | wc -l 48 desktop ~ # lvs | wc -l 12 server ~ # lvs | wc -l 99 (Ok, this one runs virtual machines with Xen, but online-resizing works and xen-virtual machines get notified of the new size) I use LVM for about everything now, it makes things so much easier. First, I had two volume groups on my system drive, one for the system, placed at the front where the drive is supposed to be faster, and one for data. But I don't do this any more, it cuts down flexibility, and is probably not worth the effort. I don't think it's worth the effort as well. You can still move the LVs around physically using the lvm-defrag tool. It's very verbose by nature as it doesn't do any changes untill you tell it to. And you're the one starting the final script. That tool basically generates a script that calls pvmove a couple of times. And the script even contains comments describing what it is doing for each step. It does expect the user to determine which LVs end up where. Those look ok to me, except that I would expect SATA-drives to be faster then PATA drives. Well, they are, except for the sometimes busy system drive. And by now I get similar results as for the 2nd SATA drive, throughput is between 90 and 110 MB/sec. Btw, I tend to use hdparm -Tt device to do the testing: desktop ~ # hdparm -Tt /dev/sda /dev/sda: Timing cached reads: 3456 MB in 2.00 seconds = 1728.51 MB/sec Timing buffered disk reads: 276 MB in 3.01 seconds = 91.75 MB/sec And unset mysql. There is one issue that needs to be resolved manually with getting it to work with sqlite. See: http://forums.gentoo.org/viewtopic-t-834883-view- next.html?sid=3ae77f5bfba5e006e8745eedb4b6cfc4 Here is the bit that will solve the problem: -- $ cd ~/.local/share/akonadi $ sqlite3 akonadi.db sqlite INSERT INTO ResourceTable (name, isVirtual) VALUES ('akonadi_search_resource', 1); sqlite .exit I did nothing about this by now, but I probabyl will soon. Thanks for the tip! I also had trouble with akonadi in the past, and it still gives warnings/errors at every startup, but at least it seems to work now. It doesn't complain for me anymore since I switched to sqlite. Main reason for that: I don't like MySQL and prefer not to run a full database on my desktop anyway. And configuring it to use the Database on the server goes too far. Disk I/O is, in my experience, a very common cause for freeze-ups. Can you test with unencrypted disks to see if the issue occurs then as well? Yes, I can do this. It's some work, but I tried so much, why not this. I have some free space, and already have written a backup script that automatically creates LVM snapshots, decrypts them, and backs it up, so I can do this from the running system. Ok, am interested to see if running unencrypted actually has benefits here. Me too, but as things are quite faster now already, the priority for this task is much lower than it was yesterday :) Ofcourse, depending on
Re: [gentoo-performance] Horrible performance
Hi Alex, On Wednesday 25 August 2010 03:32:40 Alex Schuster wrote: Me again. I have an AMD Athlon(tm) Dual Core Processor 4850e CPU, on-board Radeon HD 3200 graphics, 4GB of memory, an 1.5 TB drive. Lots of LVM volumes, all encrypted, except for /usr/src and portage stuff. The system is ~amd64, and I have -march=k8-sse3 in my CFLAGS. Current kernel is 2.6.34-tuxonice, but I also tried others. I'm running KDE4 with desktop effects enabled, X itself takes about 30-40% of CPU time according to top. After system startup and login into KDE, 3.5G of RAM are occupied. This increases after a while, and I need swap space. Nothing to worry about I think. Encrypted filesystems can cause additional with activity, but I would expect that to remain the same over a long period. However, how is the write and read performance on those disks? You're running KDE4, guess you went for the default and use mysql for app- office/akonadi-server. I switched to using sqlite for this due to issues getting it to work with mysql. I think this might help there? Performance does not feel too bad at first. But after a while, I cannot even play videos during emerges. The playback stutters, sometimes I have pauses for several seconds. As long as there is no swap space occpied, it's not so bad I think. Maybe I have a probelm with disk I/O, and things get much worse when swapping occurs. When I look at iotop, I see various programs like chromium and various KDE applications appear. I guess that's normal, but should not be noticeable. Hey, there were times when I created a 2G tmpfs for /var/tmp/portage, with only 3G on my 32bit system. BTW, I lowered my swappiness to 10. This helps a little I think, because the swapping occurs later, the system is more responsive. Do you also encrypt swap? Disk I/O is, in my experience, a very common cause for freeze-ups. Can you test with unencrypted disks to see if the issue occurs then as well? And I have similar problems when copying data between some old PATA drives. When I copy stuff and do a mkfs on another partition, mplayer sometimes stutters and hangs for ten seconds. No joy. Working with KDE sucks, switching dektops sometimes takes ages, and even now I am typing faster than kmail can display the characters. That's with am emerge of chromium running, with PORTAGE_NICENESS=10 and using ionice -c 3. Load is around 8, but sometimes gets even higher. And then, load suddenly drops back to lower values, as if somthing was blocking. Some applications swapping, maybe. Very possibly, maybe an idea to check which applications are hogging the memory. If it is the swapping of the system, then this will be caused by the most memory-hungry processes. Can you post the result of: ps axu? This will give an indication which processes are running and using a lot of memory. Now I am out of ideas. I really hope someone here has one. I cannot work with this system any more when emerges are going on. Had similar issues with a desktop machine myself, managed to kill some features that I wasn't using and this solved most of the problems. I put my kernel config, make.conf, dmesg and such stuff to http://www.wonkology.org/gentoo/ in case someone wants to have a look at it. Any help is GREATLY appreciated. Lets see where checking for IO-speeds and memory-usage of your apps take us :) -- Joost
Re: [gentoo-performance] Horrible performance
On Tue, Aug 24, 2010 at 9:32 PM, Alex Schuster wo...@wonkology.org wrote: Now I am out of ideas. I really hope someone here has one. I cannot work with this system any more when emerges are going on. I put my kernel config, make.conf, dmesg and such stuff to http://www.wonkology.org/gentoo/ in case someone wants to have a look at it. Any help is GREATLY appreciated. Just a thought: why -ggdb in your CFLAGS? If you have =gcc-4.2, try: CFLAGS=-march=native -O2 -pipe Then you should re-emerge gcc itself (twice?) and then world: # emerge --oneshot binutils gcc virtual/libc # emerge -e world Best of luck. -- Mansour Moufid
Re: [gentoo-performance] Horrible performance
Dnia 2010-08-25, o godz. 03:32:40 Alex Schuster wo...@wonkology.org napisał(a): Me again. I wrote here about my problems with mplayer stuttering during emerges. Then I wrote that the problem went away when I installed Gentoo again, moving from i686 to x86_64. But the problems are back, and worse than ever. This is driving me crazy. CRAZY! I have an AMD Athlon(tm) Dual Core Processor 4850e CPU, on-board Radeon HD 3200 graphics, 4GB of memory, an 1.5 TB drive. Lots of LVM volumes, all encrypted, except for /usr/src and portage stuff. The system is ~amd64, and I have -march=k8-sse3 in my CFLAGS. Current kernel is 2.6.34-tuxonice, but I also tried others. I'm running KDE4 with desktop effects enabled, X itself takes about 30-40% of CPU time according to top. After system startup and login into KDE, 3.5G of RAM are occupied. This increases after a while, and I need swap space. Nothing to worry about I think. Performance does not feel too bad at first. But after a while, I cannot even play videos during emerges. The playback stutters, sometimes I have pauses for several seconds. As long as there is no swap space occpied, it's not so bad I think. Maybe I have a probelm with disk I/O, and things get much worse when swapping occurs. When I look at iotop, I see various programs like chromium and various KDE applications appear. I guess that's normal, but should not be noticeable. Hey, there were times when I created a 2G tmpfs for /var/tmp/portage, with only 3G on my 32bit system. BTW, I lowered my swappiness to 10. This helps a little I think, because the swapping occurs later, the system is more responsive. And it feels like things get worse and worse, it's not like there was a specific point when I thought it's slow again. Like there were some degragation going on - fragmentation, bitrot, I don't know. It's just how it feels to me. I am debugging this for some days now. I tried different kernels, from 2.6.29 to 2.6.35, including the kernel I had running after the switch to 64bit, when I thought all was fine. No change. But all kernels were configured nearly identical, so I booted a GRML live-cd and used this kernel .config as a template. Does not feel better. When I thought the problem was gone, I had installed the system on my 2nd 1.5 TB drive. Meanwhile I copied the partitions back to the 1st drive, so I suspected a difference in the drives. I use the 2nd drive for backups (using rdiff-backup), with similar partitions, so I only have to exchange the LVM volume group names of the two drives in order to run my system from the 2nd one. I tried this, but it did not help. And I have similar problems when copying data between some old PATA drives. When I copy stuff and do a mkfs on another partition, mplayer sometimes stutters and hangs for ten seconds. No joy. Working with KDE sucks, switching dektops sometimes takes ages, and even now I am typing faster than kmail can display the characters. That's with am emerge of chromium running, with PORTAGE_NICENESS=10 and using ionice -c 3. Load is around 8, but sometimes gets even higher. And then, load suddenly drops back to lower values, as if somthing was blocking. Some applications swapping, maybe. Now I am out of ideas. I really hope someone here has one. I cannot work with this system any more when emerges are going on. I put my kernel config, make.conf, dmesg and such stuff to http://www.wonkology.org/gentoo/ in case someone wants to have a look at it. Any help is GREATLY appreciated. Wonko Checkout configuration of your video driver - X should not take more than 5% of cpu when iddle (with some minor effects). Perhaps kde is not using OpenGL? Do you have OpenGL enabled? Amount of ram you have should be sufficent to compile everything without access to swap space. -- Kacper Kopczyński
Re: [gentoo-performance] Horrible performance
Mansour Moufid writes: On Tue, Aug 24, 2010 at 9:32 PM, Alex Schuster wo...@wonkology.org wrote: Just a thought: why -ggdb in your CFLAGS? If you have =gcc-4.2, try: CFLAGS=-march=native -O2 -pipe I just added this some days ago in order to give debug information for a a bug in strigi I had reported. I forgot to take it out, but so far only few packages were compiled with this setting. But thanks for mentioning this, I removed the debug setting now. But should it matter? Optimization still happens. And when these interrupts hapen, the CPU is not at 100%. I do not use march=native because I sometimes use another host with distcc. Wonko
Re: [gentoo-performance] Horrible performance
Kacper Kopczyński writes: Checkout configuration of your video driver - X should not take more than 5% of cpu when iddle (with some minor effects). Perhaps kde is not using OpenGL? Do you have OpenGL enabled? I'm using ati-drivers-10.7 with xorg-server-1.7.7-r1. No special settings in xorg.conf. I never had much success with the open source radeon drivers. First they did not run at all, I either did not get X started, or I only got a blank screen. One month ago then I had success with xf86- video-ati-6.13.1, but X crashed instantly when I moved the mouse cursor onto the KDE panel. But now I have a new KDe version, so I wil give them a try soon. Wow, at the moment X takes 95%. When I disable desktop effects, it's down to 5%. When enable desktop effects again, X uses 60%. Now I went into systemsettings - desktop effects - all effects, and played around with the individual settings. When I disable the blur effect (two lines after transparency), X usage is at less than 20%. Weird, I do not even see any difference this effect makes. Cool, much less X usage by disabling something I did not even notice :) Is 15-25% X usage still too much? I have several plasmoids running, so I'm not surprised this eats some performance. BTW, at [*] there are some screenshots of my desktop, which did not change much since I did the screenshots. The system already feels better now. But this cannot be the only problem, I sometimes (especially during large emerges) turn desktop effects off, and when I have too much load, they are turned off automatically. Amount of ram you have should be sufficent to compile everything without access to swap space. Damn right. I used to have less memory, and ran very memory-intensive applications, and the system was much more responsive than now. Wonko [*] http://www.wonkology.org/comp/desktop/2010-06-19/
Re: [gentoo-performance] Horrible performance
On Wed, Aug 25, 2010 at 7:09 AM, Alex Schuster wo...@wonkology.org wrote: I just added this some days ago in order to give debug information for a a bug in strigi I had reported. I forgot to take it out, but so far only few packages were compiled with this setting. But thanks for mentioning this, I removed the debug setting now. But should it matter? Optimization still happens. And when these interrupts hapen, the CPU is not at 100%. I do not use march=native because I sometimes use another host with distcc. Oh, I thought that might have been behind the memory issue... But actually it sounds like you don't have direct rendering? Check with glxinfo (from x11-apps/mesa-progs): $ glxinfo | grep -i direct If not then check out this thread in the forums for the correct kernel configuration: http://forums.gentoo.org/viewtopic-t-840613-highlight-radeon+3200.html If that works you can also try building the KDE libraries with -Os to help with memory. I don't think you should be using swap at all. -- Mansour Moufid
Re: [gentoo-performance] Horrible performance
J. Roeleveld writes: On Wednesday 25 August 2010 03:32:40 Alex Schuster wrote: I have an AMD Athlon(tm) Dual Core Processor 4850e CPU, on-board Radeon HD 3200 graphics, 4GB of memory, an 1.5 TB drive. Lots of LVM volumes, all encrypted, except for /usr/src and portage stuff. The system is ~amd64, and I have -march=k8-sse3 in my CFLAGS. Current kernel is 2.6.34-tuxonice, but I also tried others. I'm running KDE4 with desktop effects enabled, X itself takes about 30-40% of CPU time according to top. After system startup and login into KDE, 3.5G of RAM are occupied. This increases after a while, and I need swap space. Nothing to worry about I think. Encrypted filesystems can cause additional with activity, but I would expect that to remain the same over a long period. And I just moved my PORTAGE_TMPDIR to an unencrypted partition. Can LVM create noticeable overhead? I also resized my logical volumes a couple of times, could this lead to some LVM fragmentation? However, how is the write and read performance on those disks? Here's the output of hdparm -t for all drives, 4 times. /dev/sda: (SATA system drive) Timing buffered disk reads: 118 MB in 3.08 seconds = 38.37 MB/sec Timing buffered disk reads: 194 MB in 3.11 seconds = 62.47 MB/sec Timing buffered disk reads: 322 MB in 3.01 seconds = 106.82 MB/sec Timing buffered disk reads: 244 MB in 3.00 seconds = 81.21 MB/sec /dev/sdb: (PATA master) Timing buffered disk reads: 114 MB in 3.02 seconds = 37.70 MB/sec Timing buffered disk reads: 114 MB in 3.00 seconds = 37.97 MB/sec Timing buffered disk reads: 116 MB in 3.05 seconds = 38.06 MB/sec Timing buffered disk reads: 116 MB in 3.05 seconds = 38.07 MB/sec /dev/sdc: (PATA slave) Timing buffered disk reads: 164 MB in 3.03 seconds = 54.21 MB/sec Timing buffered disk reads: 166 MB in 3.02 seconds = 55.04 MB/sec Timing buffered disk reads: 166 MB in 3.01 seconds = 55.10 MB/sec Timing buffered disk reads: 158 MB in 3.01 seconds = 52.41 MB/sec /dev/sdd: (SATA backup drive) Timing buffered disk reads: 314 MB in 3.00 seconds = 104.55 MB/sec Timing buffered disk reads: 312 MB in 3.01 seconds = 103.67 MB/sec Timing buffered disk reads: 308 MB in 3.01 seconds = 102.34 MB/sec Timing buffered disk reads: 314 MB in 3.00 seconds = 104.60 MB/sec The system drive throughput varies a lot, depending on other I/O. I You're running KDE4, guess you went for the default and use mysql for app- office/akonadi-server. I switched to using sqlite for this due to issues getting it to work with mysql. I think this might help there? So I only have to set the sqlite use flag and remove the mysql use flag for akonadi-server? I'm doing this now. And this also gives an example for what is going on here: I have just logged into KDE. I did not log out since yesterday, and when started a VM with vmplayer, the system swapped like crazy, I could not use it for minutes. After this, the panel did not react any more, and the desktop background did not redraw, so I logged out and in again. The VM started fine now. Well, could be faster, but maybe it was okay. Then I started answering your mail, and tried to reemerge akonadi-server, but I had a type, so portage took a long search for akomadi-server. meanwhile the dektop became quite unresponsive, load went high, and I made a screenshot [*]. If you look at the top right, gkrellm shows this above 'Proc'. The first increase at the left was after I started emerge, the 2nd at the right was after I pressed the PrtSc key. Performance does not feel too bad at first. But after a while, I cannot even play videos during emerges. The playback stutters, sometimes I have pauses for several seconds. As long as there is no swap space occpied, it's not so bad I think. Maybe I have a probelm with disk I/O, and things get much worse when swapping occurs. When I look at iotop, I see various programs like chromium and various KDE applications appear. I guess that's normal, but should not be noticeable. Hey, there were times when I created a 2G tmpfs for /var/tmp/portage, with only 3G on my 32bit system. BTW, I lowered my swappiness to 10. This helps a little I think, because the swapping occurs later, the system is more responsive. Do you also encrypt swap? Yes. Disk I/O is, in my experience, a very common cause for freeze-ups. Can you test with unencrypted disks to see if the issue occurs then as well? Yes, I can do this. It's some work, but I tried so much, why not this. I have some free space, and already have written a backup script that automatically creates LVM snapshots, decrypts them, and backs it up, so I can do this from the running system. And I have similar problems when copying data between some old PATA drives. When I copy stuff and do a mkfs on another partition, mplayer sometimes stutters and hangs for ten seconds. No joy. Working with KDE sucks,
Re: [gentoo-performance] Horrible performance
Mansour Moufid writes: But actually it sounds like you don't have direct rendering? Check with glxinfo (from x11-apps/mesa-progs): $ glxinfo | grep -i direct Thanks, but opengl is running fine. Well, it took me quite a while until I got it working, but finally there was a version of ati-drivers that worked, and from then on it kept working fine. If that works you can also try building the KDE libraries with -Os to help with memory. I don't think you should be using swap at all. Me too. So -Os would only be a workaround to make the probelm less bad. I hope I find a better solution. Wonko
Re: [gentoo-performance] Horrible performance
I think I just found the problem! I decided to give the open source ATI drivers a try again, and replaced 'fglrx' by 'radeon' in my xorg.conf. And all is fast now. Wow. I had already forgotten how fast a desktop should be. I'm not yet sure all is fine. On the first start, kwin crashed and all windows were on the first desktop, this never happened before. But it's working on the second login. I have no direct rendering, Xorg.0.log shows (EE) RADEON(0): [dri] RADEONDRIGetVersion failed (libdri too old). I had this before with xorg-server-1.7.7, now I'm emerging 1.8.2. Stay tuned. Wonko
[gentoo-performance] Horrible performance
Me again. I wrote here about my problems with mplayer stuttering during emerges. Then I wrote that the problem went away when I installed Gentoo again, moving from i686 to x86_64. But the problems are back, and worse than ever. This is driving me crazy. CRAZY! I have an AMD Athlon(tm) Dual Core Processor 4850e CPU, on-board Radeon HD 3200 graphics, 4GB of memory, an 1.5 TB drive. Lots of LVM volumes, all encrypted, except for /usr/src and portage stuff. The system is ~amd64, and I have -march=k8-sse3 in my CFLAGS. Current kernel is 2.6.34-tuxonice, but I also tried others. I'm running KDE4 with desktop effects enabled, X itself takes about 30-40% of CPU time according to top. After system startup and login into KDE, 3.5G of RAM are occupied. This increases after a while, and I need swap space. Nothing to worry about I think. Performance does not feel too bad at first. But after a while, I cannot even play videos during emerges. The playback stutters, sometimes I have pauses for several seconds. As long as there is no swap space occpied, it's not so bad I think. Maybe I have a probelm with disk I/O, and things get much worse when swapping occurs. When I look at iotop, I see various programs like chromium and various KDE applications appear. I guess that's normal, but should not be noticeable. Hey, there were times when I created a 2G tmpfs for /var/tmp/portage, with only 3G on my 32bit system. BTW, I lowered my swappiness to 10. This helps a little I think, because the swapping occurs later, the system is more responsive. And it feels like things get worse and worse, it's not like there was a specific point when I thought it's slow again. Like there were some degragation going on - fragmentation, bitrot, I don't know. It's just how it feels to me. I am debugging this for some days now. I tried different kernels, from 2.6.29 to 2.6.35, including the kernel I had running after the switch to 64bit, when I thought all was fine. No change. But all kernels were configured nearly identical, so I booted a GRML live-cd and used this kernel .config as a template. Does not feel better. When I thought the problem was gone, I had installed the system on my 2nd 1.5 TB drive. Meanwhile I copied the partitions back to the 1st drive, so I suspected a difference in the drives. I use the 2nd drive for backups (using rdiff-backup), with similar partitions, so I only have to exchange the LVM volume group names of the two drives in order to run my system from the 2nd one. I tried this, but it did not help. And I have similar problems when copying data between some old PATA drives. When I copy stuff and do a mkfs on another partition, mplayer sometimes stutters and hangs for ten seconds. No joy. Working with KDE sucks, switching dektops sometimes takes ages, and even now I am typing faster than kmail can display the characters. That's with am emerge of chromium running, with PORTAGE_NICENESS=10 and using ionice -c 3. Load is around 8, but sometimes gets even higher. And then, load suddenly drops back to lower values, as if somthing was blocking. Some applications swapping, maybe. Now I am out of ideas. I really hope someone here has one. I cannot work with this system any more when emerges are going on. I put my kernel config, make.conf, dmesg and such stuff to http://www.wonkology.org/gentoo/ in case someone wants to have a look at it. Any help is GREATLY appreciated. Wonko