Re: [gentoo-performance] Goodbye
On Thursday 26 August 2010 17:26:04 Alex Schuster wrote: > J. Roeleveld writes: > > I actually didn't find out about this list untill I was looking for a > > different list on the Gentoo site. > > > > Most of the activity is on the gentoo-user list, but that tends to get > > too busy at times. > > > > Also from me, goodbye to this list, it was nice while it lasted :) > > At least I will have the eternal high-score according to > http://archives.gentoo.org/stats/gentoo-performance-per-year.xml . > Not couting those empty cheating messages from imail...@capstonecomm.com. Yes, with me in second place > Wonko, winner of gentoo-performance@lists.gentoo.org Joost, runner-up of gentoo-performance@lists.gentoo.org
Re: [gentoo-performance] Horrible performance
On Thursday 26 August 2010 17:48:44 Alex Schuster wrote:
> Joost Roeleveld writes:
> > LVM-fragmentation is a definite possibility.
> > To defragment it, have a look at the following:
> >
> > http://bisqwit.iki.fi/source/lvm2defrag.html
>
> Cool!
Do test it first and check what it wants to do.
Basically, it moves all the blocks around untill you have them all in the
sequence you want them.
There are some limitations, but it worked when I tested it.
Btw, I provide no warranty what-so-ever, especially as I did not write any
part of it :)
> > http://www.linuxquestions.org/questions/linux-software-2/how-do-i-lvm2-
> > defrag- or-move-based-on-logical-volumes-689335/
> >
> > I played with the first one on an older machine once and it does work
> > quite nicely.
>
> Any idea how to check how bad the fragmentation actually is?
Yes:
desktop ~ # lvs -o lv_name,seg_pe_ranges
LV PE Ranges
distfiles /dev/sda4:37632-39423
home /dev/sda4:9472-12031
home /dev/sda4:39424-40703
home /dev/sda4:42240-44799
opt/dev/sda4:8192-9471
opt/dev/sda4:40704-42239
packages /dev/sda4:45312-47359
portage/dev/sda4:3072-3327
swap /dev/sda4:0-3071
tmp/dev/sda4:3328-4607
usr/dev/sda4:6400-8191
usr/dev/sda4:81664-81919
usr/dev/sda4:44800-45311
var/dev/sda4:4608-5119
vartmp /dev/sda4:5120-6399
vartmp /dev/sda4:60160-61183
virtualbox /dev/sda4:12032-32511
virtualbox /dev/sda4:76544-81663
virtualbox /dev/sda4:58112-60159
virtualbox /dev/sda4:81920-83310
virtualbox /dev/sda4:47360-58111
virtualbox /dev/sda4:32512-33680
home, vartmp and virtualbox are fragmented into different sections.
Alternatively, the first part of that script actually generates a text-file
which you then need to edit into the next text-file.
The first one actually shows you how the different parts are laid out.
> BTW:
> wo...@weird ~ $ mount | wc -l
> 48
desktop ~ # lvs | wc -l
12
server ~ # lvs | wc -l
99
(Ok, this one runs virtual machines with Xen, but online-resizing works and
xen-virtual machines get notified of the new size)
> I use LVM for about everything now, it makes things so much easier. First,
> I had two volume groups on my system drive, one for the system, placed at
> the front where the drive is supposed to be faster, and one for data. But
> I don't do this any more, it cuts down flexibility, and is probably not
> worth the effort.
I don't think it's worth the effort as well. You can still move the LVs around
physically using the "lvm-defrag" tool.
It's very verbose by nature as it doesn't do any changes untill you tell it
to. And you're the one starting the final script.
That tool basically generates a script that calls "pvmove" a couple of times.
And the script even contains comments describing what it is doing for each
step.
It does expect the user to determine which LVs end up where.
> > Those look ok to me, except that I would expect SATA-drives to be
> > faster then PATA drives.
>
> Well, they are, except for the sometimes busy system drive. And by now I
> get similar results as for the 2nd SATA drive, throughput is between 90
> and 110 MB/sec.
Btw, I tend to use "hdparm -Tt " to do the testing:
desktop ~ # hdparm -Tt /dev/sda
/dev/sda:
Timing cached reads: 3456 MB in 2.00 seconds = 1728.51 MB/sec
Timing buffered disk reads: 276 MB in 3.01 seconds = 91.75 MB/sec
> > And unset mysql.
> > There is one issue that needs to be resolved manually with getting it
> > to work with sqlite.
> > See:
> > http://forums.gentoo.org/viewtopic-t-834883-view-
> > next.html?sid=3ae77f5bfba5e006e8745eedb4b6cfc4
> >
> > Here is the bit that will solve the problem:
> > --
> > $ cd ~/.local/share/akonadi
> > $ sqlite3 akonadi.db
> > sqlite> INSERT INTO ResourceTable (name, isVirtual) VALUES
> > ('akonadi_search_resource', 1);
> > sqlite> .exit
>
> I did nothing about this by now, but I probabyl will soon. Thanks for the
> tip!
> I also had trouble with akonadi in the past, and it still gives
> warnings/errors at every startup, but at least it seems to work now.
It doesn't complain for me anymore since I switched to sqlite.
Main reason for that: I don't like MySQL and prefer not to run a full database
on my desktop anyway. And configuring it to use the Database on the server goes
too far.
> > > > Disk I/O is, in my experience, a very common cause for
> > > > "freeze-ups". Can you test with unencrypted disks to see if the
> > > > issue occurs then as well?
> > >
> > > Yes, I can do this. It's some work, but I tried so much, why not
> > > this. I have some free space, and already have written a backup
> > > script that automatically creates LVM snapshots, decrypts them, and
> > > backs it up, so I can do this from the running system.
> >
> > Ok, am interested to see if running unencrypted actually has benefits
> > here.
>
> Me too, but as things are quite fas
Re: [gentoo-performance] Horrible performance
Joost Roeleveld writes:
> On Wednesday 25 August 2010 21:38:10 Alex Schuster wrote:
> > And I just moved my PORTAGE_TMPDIR to an unencrypted partition.
> > Can LVM create noticeable overhead? I also resized my logical volumes
> > a couple of times, could this lead to some LVM fragmentation?
>
> Theoretically, LVM will create an additional overhead.
> But I am extensibly using LVM on all my machines and haven't noticed
> any significant performance drops.
That's also what I heard.
> LVM-fragmentation is a definite possibility.
> To defragment it, have a look at the following:
>
> http://bisqwit.iki.fi/source/lvm2defrag.html
Cool!
> http://www.linuxquestions.org/questions/linux-software-2/how-do-i-lvm2-
> defrag- or-move-based-on-logical-volumes-689335/
>
> I played with the first one on an older machine once and it does work
> quite nicely.
Any idea how to check how bad the fragmentation actually is?
BTW:
wo...@weird ~ $ mount | wc -l
48
I use LVM for about everything now, it makes things so much easier. First,
I had two volume groups on my system drive, one for the system, placed at
the front where the drive is supposed to be faster, and one for data. But
I don't do this any more, it cuts down flexibility, and is probably not
worth the effort.
> > > However, how is the write and read performance on those disks?
> >
> > Here's the output of hdparm -t for all drives, 4 times.
> >
> > /dev/sda: (SATA system drive)
> >
> > Timing buffered disk reads: 118 MB in 3.08 seconds = 38.37 MB/sec
> > Timing buffered disk reads: 194 MB in 3.11 seconds = 62.47 MB/sec
> > Timing buffered disk reads: 322 MB in 3.01 seconds = 106.82 MB/sec
> > Timing buffered disk reads: 244 MB in 3.00 seconds = 81.21 MB/sec
> >
> > /dev/sdb: (PATA master)
> >
> > Timing buffered disk reads: 114 MB in 3.02 seconds = 37.70 MB/sec
> > Timing buffered disk reads: 114 MB in 3.00 seconds = 37.97 MB/sec
> > Timing buffered disk reads: 116 MB in 3.05 seconds = 38.06 MB/sec
> > Timing buffered disk reads: 116 MB in 3.05 seconds = 38.07 MB/sec
> >
> > /dev/sdc: (PATA slave)
> >
> > Timing buffered disk reads: 164 MB in 3.03 seconds = 54.21 MB/sec
> > Timing buffered disk reads: 166 MB in 3.02 seconds = 55.04 MB/sec
> > Timing buffered disk reads: 166 MB in 3.01 seconds = 55.10 MB/sec
> > Timing buffered disk reads: 158 MB in 3.01 seconds = 52.41 MB/sec
> >
> > /dev/sdd: (SATA backup drive)
> >
> > Timing buffered disk reads: 314 MB in 3.00 seconds = 104.55 MB/sec
> > Timing buffered disk reads: 312 MB in 3.01 seconds = 103.67 MB/sec
> > Timing buffered disk reads: 308 MB in 3.01 seconds = 102.34 MB/sec
> > Timing buffered disk reads: 314 MB in 3.00 seconds = 104.60 MB/sec
> >
> > The system drive throughput varies a lot, depending on other I/O.
>
> Those look ok to me, except that I would expect SATA-drives to be
> faster then PATA drives.
Well, they are, except for the sometimes busy system drive. And by now I
get similar results as for the 2nd SATA drive, throughput is between 90
and 110 MB/sec.
> > > You're running KDE4, guess you went for the default and use mysql
> > > for "app- office/akonadi-server".
> > >
> > > I switched to using "sqlite" for this due to issues getting it to
> > > work with mysql. I think this might help there?
> >
> > So I only have to set the sqlite use flag and remove the mysql use
> > flag for akonadi-server? I'm doing this now. And this also gives an
> > example for what is going on here:
> And unset mysql.
> There is one issue that needs to be resolved manually with getting it
> to work with sqlite.
> See:
> http://forums.gentoo.org/viewtopic-t-834883-view-
> next.html?sid=3ae77f5bfba5e006e8745eedb4b6cfc4
>
> Here is the bit that will solve the problem:
> --
> $ cd ~/.local/share/akonadi
> $ sqlite3 akonadi.db
> sqlite> INSERT INTO ResourceTable (name, isVirtual) VALUES
> ('akonadi_search_resource', 1);
> sqlite> .exit
I did nothing about this by now, but I probabyl will soon. Thanks for the
tip!
I also had trouble with akonadi in the past, and it still gives
warnings/errors at every startup, but at least it seems to work now.
> > > Disk I/O is, in my experience, a very common cause for
> > > "freeze-ups". Can you test with unencrypted disks to see if the
> > > issue occurs then as well?
> >
> > Yes, I can do this. It's some work, but I tried so much, why not
> > this. I have some free space, and already have written a backup
> > script that automatically creates LVM snapshots, decrypts them, and
> > backs it up, so I can do this from the running system.
>
> Ok, am interested to see if running unencrypted actually has benefits
> here.
Me too, but as things are quite faster now already, the priority for this
task is much lower than it was yesterday :)
> > > Can you post the result of: "ps axu"?
> > > This will give an indication which processes are running and using
> > > a lot of memory.
> >
> > First, here is f
Re: [gentoo-performance] Goodbye
J. Roeleveld writes: > I actually didn't find out about this list untill I was looking for a > different list on the Gentoo site. > > Most of the activity is on the gentoo-user list, but that tends to get > too busy at times. > > Also from me, goodbye to this list, it was nice while it lasted :) At least I will have the eternal high-score according to http://archives.gentoo.org/stats/gentoo-performance-per-year.xml . Not couting those empty cheating messages from imail...@capstonecomm.com. Wonko, winner of gentoo-performance@lists.gentoo.org
Re: [gentoo-performance] Goodbye
On Thursday 26 August 2010 15:05:26 Alex Schuster wrote: > Hi there! > > YFYI, this list is about to be closed. That's a pity, I'd like to have > such a list, but it is also true that there was nearly no traffic at all - > I count 24 posts this year, all in the same thread except for another > accidental posting. Maybe it was just that too few people knew about it? > Anyway, it will be gone soon. Thanks for the help, and see you on gentoo- > user :) > > http://bugs.gentoo.org/show_bug.cgi?id=334541 > > Wonko I actually didn't find out about this list untill I was looking for a different list on the Gentoo site. Most of the activity is on the gentoo-user list, but that tends to get too busy at times. Also from me, goodbye to this list, it was nice while it lasted :) -- Joost
Re: [gentoo-performance] Horrible performance
Joost Roeleveld writes: > On Thursday 26 August 2010 02:11:44 Alex Schuster wrote: > > I think I just found the problem! I decided to give the open source > > ATI drivers a try again, and replaced 'fglrx' by 'radeon' in my > > xorg.conf. And all is fast now. Wow. I had already forgotten how > > fast a desktop should be. > > > > I'm not yet sure all is fine. On the first start, kwin crashed and > > all windows were on the first desktop, this never happened before. > > But it's working on the second login. I have no direct rendering, > > Xorg.0.log shows (EE) RADEON(0): [dri] RADEONDRIGetVersion failed > > (libdri too old). I had this before with xorg-server-1.7.7, now I'm > > emerging 1.8.2. Stay tuned. > > > > Wonko > > Ok, looks like your video-driver was using up too much resources. > Either with all the desktop-effects or with something else. I often had desktop effects turned off. > I can't help further with ATI-issues as I don't use them myself. > > I do know that I need to change more then just the driver in my > xorg.conf when using nvidie-drivers. Am wondering if your crashes > occured because of a similar issue. > Eg. the "dri" module is only for the ATI-driver and not for the > open-source one? I did not configure modules like dri in my xorg.conf, this seems tzo work automatically. With ati-drivers, I got errors in Xorg.0.log about dri and dri2 missing, apparently those come with the fglrx driver. With radeon, I get messages that dr and dri2 are loaded. So I only had to replace 'fglrx' by 'radeon' inthe device section. And to reboot. Or something. At least it did not work until I did that. I had loaded the drm kernel module by hand, so this was not the problem. Whatever, I'm glad the radeon driver is runnign fine now, for the first time after several tries. Wonko
[gentoo-performance] Goodbye
Hi there! YFYI, this list is about to be closed. That's a pity, I'd like to have such a list, but it is also true that there was nearly no traffic at all - I count 24 posts this year, all in the same thread except for another accidental posting. Maybe it was just that too few people knew about it? Anyway, it will be gone soon. Thanks for the help, and see you on gentoo- user :) http://bugs.gentoo.org/show_bug.cgi?id=334541 Wonko
Re: [gentoo-performance] Horrible performance
On Thursday 26 August 2010 11:34:23 Alex Schuster wrote: > Update: > I first got no dri with xorg-server 1.8, too, and then weird effects > started happening, like the shift and ctrl keys not working. When tings > crashed, I got a block screen and still could move the mouse, but that was > all, even after Alt-SysRQ-R (which usually helps in these cases) I could > not switch to a text terminal. But after a reboot all seemed perfect, but > when I was writing a followe-up mail, X crashed. And I got several more > crashes after working 5 minutes in KDE. > > I downgraded back to xorg-server 1.7.7-r1, and now dri is working, too. > Looking good! KDE is running for over an hour now, I'm not using any swap > at all, amarok is emerging in the background, and mplayer is playing > smoothly. > > I experience little rendering problems in chromium (images appear over > text) and quake3 feels a little sluggish, but I can live with this. > > Thanks for anyone who helped! > > Wonko Glad you got it working properly now :) Enjoy the smoother desktop now -- Joost
Re: [gentoo-performance] Horrible performance
Update: > I think I just found the problem! I decided to give the open source ATI > drivers a try again, and replaced 'fglrx' by 'radeon' in my xorg.conf. > And all is fast now. Wow. I had already forgotten how fast a desktop > should be. > > I'm not yet sure all is fine. On the first start, kwin crashed and all > windows were on the first desktop, this never happened before. But it's > working on the second login. I have no direct rendering, Xorg.0.log > shows (EE) RADEON(0): [dri] RADEONDRIGetVersion failed (libdri too > old). I had this before with xorg-server-1.7.7, now I'm emerging > 1.8.2. Stay tuned. I first got no dri with xorg-server 1.8, too, and then weird effects started happening, like the shift and ctrl keys not working. When tings crashed, I got a block screen and still could move the mouse, but that was all, even after Alt-SysRQ-R (which usually helps in these cases) I could not switch to a text terminal. But after a reboot all seemed perfect, but when I was writing a followe-up mail, X crashed. And I got several more crashes after working 5 minutes in KDE. I downgraded back to xorg-server 1.7.7-r1, and now dri is working, too. Looking good! KDE is running for over an hour now, I'm not using any swap at all, amarok is emerging in the background, and mplayer is playing smoothly. I experience little rendering problems in chromium (images appear over text) and quake3 feels a little sluggish, but I can live with this. Thanks for anyone who helped! Wonko
Re: [gentoo-performance] Horrible performance
On Thursday 26 August 2010 02:11:44 Alex Schuster wrote: > I think I just found the problem! I decided to give the open source ATI > drivers a try again, and replaced 'fglrx' by 'radeon' in my xorg.conf. And > all is fast now. Wow. I had already forgotten how fast a desktop should > be. > > I'm not yet sure all is fine. On the first start, kwin crashed and all > windows were on the first desktop, this never happened before. But it's > working on the second login. I have no direct rendering, Xorg.0.log shows > (EE) RADEON(0): [dri] RADEONDRIGetVersion failed (libdri too old). I had > this before with xorg-server-1.7.7, now I'm emerging 1.8.2. Stay tuned. > > Wonko Ok, looks like your video-driver was using up too much resources. Either with all the desktop-effects or with something else. I can't help further with ATI-issues as I don't use them myself. I do know that I need to change more then just the driver in my xorg.conf when using nvidie-drivers. Am wondering if your crashes occured because of a similar issue. Eg. the "dri" module is only for the ATI-driver and not for the open-source one? -- Joost
Re: [gentoo-performance] Horrible performance
On Thursday 26 August 2010 01:06:59 Alex Schuster wrote:
> J. Roeleveld writes:
> > Can you post the result of: "ps axu"?
> > This will give an indication which processes are running and using a
> > lot of memory.
>
> The fifth colums gives the memory, right? Should this add up to the total
> of the 'used' column in free -m?
>
> Because it does not:
>
> wo...@weird ~ $ free -m
> total used free sharedbuffers cached
> Mem: 3452 3117334 0 39279
> -/+ buffers/cache: 2798653
> Swap: 4094 1472 2622
>
> wo...@weird ~ $ total=0; for rss in $( ps aux | grep -v USER | awk '{print
> $6}' ); do (( total += rss )); done; echo $(( total / 1024 ))
> 1984
>
> So, I get a sum of around 2 G with ps, while free -m shows 4.5 G. Whoops?
> Am I missing something here, or does it look like lots of RAM is not being
> freed?
>
> Wonko
Don't forget the buffers/cache.
The 2.5G you're missing is what is used in the buffers/cache line.
At least, that is how I see it.
Also, I generally look at the percentages used for the memory to find the
memory-hogs.
And your tv-viewer (java) is using quite a bit as well.
Maybe someone with more experience with tv-viewer apps can take a look and
maybe give a few pointers?
--
Joost
Re: [gentoo-performance] Horrible performance
On Wednesday 25 August 2010 21:38:10 Alex Schuster wrote:
> J. Roeleveld writes:
> > On Wednesday 25 August 2010 03:32:40 Alex Schuster wrote:
> > > I have an AMD Athlon(tm) Dual Core Processor 4850e CPU, on-board
> > > Radeon HD 3200 graphics, 4GB of memory, an 1.5 TB drive. Lots of LVM
> > > volumes, all encrypted, except for /usr/src and portage stuff. The
> > > system is ~amd64, and I have -march=k8-sse3 in my CFLAGS. Current
> > > kernel is 2.6.34-tuxonice, but I also tried others. I'm running KDE4
> > > with desktop effects enabled, X itself takes about 30-40% of CPU
> > > time according to top. After system startup and login into KDE, 3.5G
> > > of RAM are occupied. This increases after a while, and I need swap
> > > space. Nothing to worry about I think.
> >
> > Encrypted filesystems can cause additional with activity, but I would
> > expect that to remain the same over a long period.
>
> And I just moved my PORTAGE_TMPDIR to an unencrypted partition.
> Can LVM create noticeable overhead? I also resized my logical volumes a
> couple of times, could this lead to some LVM fragmentation?
Theoretically, LVM will create an additional overhead.
But I am extensibly using LVM on all my machines and haven't noticed any
significant performance drops.
LVM-fragmentation is a definite possibility.
To defragment it, have a look at the following:
http://bisqwit.iki.fi/source/lvm2defrag.html
http://www.linuxquestions.org/questions/linux-software-2/how-do-i-lvm2-defrag-
or-move-based-on-logical-volumes-689335/
I played with the first one on an older machine once and it does work quite
nicely.
> > However, how is the write and read performance on those disks?
>
> Here's the output of hdparm -t for all drives, 4 times.
>
> /dev/sda: (SATA system drive)
> Timing buffered disk reads: 118 MB in 3.08 seconds = 38.37 MB/sec
> Timing buffered disk reads: 194 MB in 3.11 seconds = 62.47 MB/sec
> Timing buffered disk reads: 322 MB in 3.01 seconds = 106.82 MB/sec
> Timing buffered disk reads: 244 MB in 3.00 seconds = 81.21 MB/sec
>
> /dev/sdb: (PATA master)
> Timing buffered disk reads: 114 MB in 3.02 seconds = 37.70 MB/sec
> Timing buffered disk reads: 114 MB in 3.00 seconds = 37.97 MB/sec
> Timing buffered disk reads: 116 MB in 3.05 seconds = 38.06 MB/sec
> Timing buffered disk reads: 116 MB in 3.05 seconds = 38.07 MB/sec
>
> /dev/sdc: (PATA slave)
> Timing buffered disk reads: 164 MB in 3.03 seconds = 54.21 MB/sec
> Timing buffered disk reads: 166 MB in 3.02 seconds = 55.04 MB/sec
> Timing buffered disk reads: 166 MB in 3.01 seconds = 55.10 MB/sec
> Timing buffered disk reads: 158 MB in 3.01 seconds = 52.41 MB/sec
>
> /dev/sdd: (SATA backup drive)
> Timing buffered disk reads: 314 MB in 3.00 seconds = 104.55 MB/sec
> Timing buffered disk reads: 312 MB in 3.01 seconds = 103.67 MB/sec
> Timing buffered disk reads: 308 MB in 3.01 seconds = 102.34 MB/sec
> Timing buffered disk reads: 314 MB in 3.00 seconds = 104.60 MB/sec
>
> The system drive throughput varies a lot, depending on other I/O.
Those look ok to me, except that I would expect SATA-drives to be faster then
PATA drives.
> > You're running KDE4, guess you went for the default and use mysql for
> > "app- office/akonadi-server".
> >
> > I switched to using "sqlite" for this due to issues getting it to work
> > with mysql. I think this might help there?
>
> So I only have to set the sqlite use flag and remove the mysql use flag
> for akonadi-server? I'm doing this now. And this also gives an example for
> what is going on here:
And unset mysql.
There is one issue that needs to be resolved manually with getting it to work
with sqlite.
See:
http://forums.gentoo.org/viewtopic-t-834883-view-
next.html?sid=3ae77f5bfba5e006e8745eedb4b6cfc4
Here is the bit that will solve the problem:
--
$ cd ~/.local/share/akonadi
$ sqlite3 akonadi.db
sqlite> INSERT INTO ResourceTable (name, isVirtual) VALUES
('akonadi_search_resource', 1);
sqlite> .exit
--
> I have just logged into KDE. I did not log out since yesterday, and when
> started a VM with vmplayer, the system swapped like crazy, I could not use
> it for minutes. After this, the panel did not react any more, and the
> desktop background did not redraw, so I logged out and in again. The VM
> started fine now. Well, could be faster, but maybe it was okay.
> Then I started answering your mail, and tried to reemerge akonadi-server,
> but I had a type, so portage took a long search for akomadi-server.
> meanwhile the dektop became quite unresponsive, load went high, and I made
> a screenshot [*]. If you look at the top right, gkrellm shows this above
> 'Proc'. The first increase at the left was after I started emerge, the 2nd
> at the right was after I pressed the PrtSc key.
VMWare allows virtual machines to use more memory then is actually available.
Also, there are settings in VMWare (possibly enabled by default) that cause
the memory to be duplicate
