Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-07 Thread Dimitrios Apostolou
Hi Jan, thanks for your help, with this commit I can't reproduce the problem. The problematic workload is no more inflating the ext4_inode_cache, in fact I've been able to run even heavier workloads with the ext4_inode_cache never surpassing 2-4MB. Thanks everyone for helping, Dimitris On

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-07 Thread Dimitrios Apostolou
Hi Jan, thanks for your help, with this commit I can't reproduce the problem. The problematic workload is no more inflating the ext4_inode_cache, in fact I've been able to run even heavier workloads with the ext4_inode_cache never surpassing 2-4MB. Thanks everyone for helping, Dimitris On

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-06 Thread Jan Kara
On Thu 06-12-12 17:15:37, Dimitrios Apostolou wrote: > On Thu, 6 Dec 2012, Jan Kara wrote: > >On Sun 25-11-12 21:30:00, Dimitrios Apostolou wrote: > >>On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: > >>>on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a > >>>backup

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-06 Thread Dimitrios Apostolou
On Thu, 6 Dec 2012, Jan Kara wrote: On Sun 25-11-12 21:30:00, Dimitrios Apostolou wrote: On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-06 Thread Jan Kara
On Sun 25-11-12 21:30:00, Dimitrios Apostolou wrote: > On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: > > on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a > > backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local > > ATA disk to ext3 on external

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-06 Thread Jan Kara
On Sun 25-11-12 21:30:00, Dimitrios Apostolou wrote: On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-06 Thread Dimitrios Apostolou
On Thu, 6 Dec 2012, Jan Kara wrote: On Sun 25-11-12 21:30:00, Dimitrios Apostolou wrote: On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-06 Thread Jan Kara
On Thu 06-12-12 17:15:37, Dimitrios Apostolou wrote: On Thu, 6 Dec 2012, Jan Kara wrote: On Sun 25-11-12 21:30:00, Dimitrios Apostolou wrote: On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Dimitrios Apostolou
On Mon, 3 Dec 2012, Dimitrios Apostolou wrote: I've managed to reproduce the scenario with concurrent "du" commands running on the filesystems. I'll try doing it once more, but it may take a while to get the dmesg/slabinfo etc output, since even a realtime root shell is non-responsive for

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Dimitrios Apostolou
On Mon, 3 Dec 2012, Eric Paris wrote: On Mon, Dec 3, 2012 at 12:43 PM, Theodore Ts'o wrote: If you are seeing a large number of inodes still in the ext4 inode cache after using drop_caches, then I'd look to see whether you have something like SELinux or auditing enabled which is pinning a

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Dimitrios Apostolou
On Mon, 3 Dec 2012, Roland Eggner wrote: On 2012-12-03 Monday at 01:56 +0200 Dimitrios Apostolou wrote: Dear Ronald, Excuse me, my name is Roland. Sorry, this was not intentional. Dimitris -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Eric Paris
On Mon, Dec 3, 2012 at 12:43 PM, Theodore Ts'o wrote: > If you are seeing a large number of inodes still in the ext4 inode > cache after using drop_caches, then I'd look to see whether you have > something like SELinux or auditing enabled which is pinning a bunch of > dentries or inodes You can

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Roland Eggner
On 2012-12-03 Monday at 01:56 +0200 Dimitrios Apostolou wrote: > Dear Ronald, Excuse me, my name is Roland. > sorry for not replying at your first message but I didn't consider changing > kernel as a resolution to my problem. > > … … > > … … > > One advantage of Linux compared to other OS is

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Theodore Ts'o
On Mon, Dec 03, 2012 at 01:56:27AM +0200, Dimitrios Apostolou wrote: > > Please take a look at [2], it's what is called in that case. Via > this call all slabs that have registered a shrinker, get actually > shrinked. Other fs do, but I can't find whether ext4 actually > registers a shrinker for

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Theodore Ts'o
On Mon, Dec 03, 2012 at 01:56:27AM +0200, Dimitrios Apostolou wrote: Please take a look at [2], it's what is called in that case. Via this call all slabs that have registered a shrinker, get actually shrinked. Other fs do, but I can't find whether ext4 actually registers a shrinker for its

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Roland Eggner
On 2012-12-03 Monday at 01:56 +0200 Dimitrios Apostolou wrote: Dear Ronald, Excuse me, my name is Roland. sorry for not replying at your first message but I didn't consider changing kernel as a resolution to my problem. … … … … One advantage of Linux compared to other OS is much more

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Eric Paris
On Mon, Dec 3, 2012 at 12:43 PM, Theodore Ts'o ty...@mit.edu wrote: If you are seeing a large number of inodes still in the ext4 inode cache after using drop_caches, then I'd look to see whether you have something like SELinux or auditing enabled which is pinning a bunch of dentries or inodes

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Dimitrios Apostolou
On Mon, 3 Dec 2012, Roland Eggner wrote: On 2012-12-03 Monday at 01:56 +0200 Dimitrios Apostolou wrote: Dear Ronald, Excuse me, my name is Roland. Sorry, this was not intentional. Dimitris -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Dimitrios Apostolou
On Mon, 3 Dec 2012, Eric Paris wrote: On Mon, Dec 3, 2012 at 12:43 PM, Theodore Ts'o ty...@mit.edu wrote: If you are seeing a large number of inodes still in the ext4 inode cache after using drop_caches, then I'd look to see whether you have something like SELinux or auditing enabled which is

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-03 Thread Dimitrios Apostolou
On Mon, 3 Dec 2012, Dimitrios Apostolou wrote: I've managed to reproduce the scenario with concurrent du commands running on the filesystems. I'll try doing it once more, but it may take a while to get the dmesg/slabinfo etc output, since even a realtime root shell is non-responsive for many

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-02 Thread Dimitrios Apostolou
Dear Ronald, sorry for not replying at your first message but I didn't consider changing kernel as a resolution to my problem. On Sun, 2 Dec 2012, Roland Eggner wrote: Hello Dimitrios! Which part of … “0,5 … 1 G RAM is usually occupied just by kernel slab [1]; this memory cannot be

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-02 Thread Roland Eggner
On 2012-11-25 Sun 23:59:55 +0100 Roland Eggner wrote: >On 2012-11-25 Sunday at 21:30 +0200 Dimitrios Apostolou wrote: > > On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: > > > on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a > > > backup process (tar|xz -4, nice'd

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-02 Thread Dimitrios Apostolou
Hi, the problem in the quoted message still happens, shouldn't all of ext4_inode_cache slab be emptied after "echo 3 > /proc/sys/vm/drop_caches"? In my case slab uses too much memory even after all processes finish and system is in bad shape due to lack of physical RAM. CC'ing tytso and Catalin

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-02 Thread Dimitrios Apostolou
Hi, the problem in the quoted message still happens, shouldn't all of ext4_inode_cache slab be emptied after echo 3 /proc/sys/vm/drop_caches? In my case slab uses too much memory even after all processes finish and system is in bad shape due to lack of physical RAM. CC'ing tytso and Catalin

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-02 Thread Roland Eggner
On 2012-11-25 Sun 23:59:55 +0100 Roland Eggner wrote: On 2012-11-25 Sunday at 21:30 +0200 Dimitrios Apostolou wrote: On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-12-02 Thread Dimitrios Apostolou
Dear Ronald, sorry for not replying at your first message but I didn't consider changing kernel as a resolution to my problem. On Sun, 2 Dec 2012, Roland Eggner wrote: Hello Dimitrios! Which part of … “0,5 … 1 G RAM is usually occupied just by kernel slab [1]; this memory cannot be

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-11-25 Thread Roland Eggner
On 2012-11-25 Sunday at 23:56 + Alan Cox wrote: > > Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel? > > Probably not a good idea, there are known exploitable holes in 2.6.27 era > kernels and nobody maintains anything that old. “old” is relative … cd

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-11-25 Thread Alan Cox
> Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel? Probably not a good idea, there are known exploitable holes in 2.6.27 era kernels and nobody maintains anything that old. I guess RHEL/Centos might work for you. I've not had any problems with leaks in 3.6 but there have

Re: kmemleak failure (was Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused)

2012-11-25 Thread Dimitrios Apostolou
On Mon, 2012-11-26 at 00:41 +0200, Dimitrios Apostolou wrote: > Hello, > > I tried running a kmemleak and DEBUG_SLUB enabled kernel, but almost > within 5 minutes of starting the backup process, I got: > > [ 486.863124] kmemleak: Cannot allocate a kmemleak_object structure > [ 486.887195]

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-11-25 Thread Roland Eggner
On 2012-11-25 Sunday at 21:30 +0200 Dimitrios Apostolou wrote: > On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: > > on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a > > backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local > > ATA disk to ext3 on

kmemleak failure (was Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused)

2012-11-25 Thread Dimitrios Apostolou
Hello, I tried running a kmemleak and DEBUG_SLUB enabled kernel, but almost within 5 minutes of starting the backup process, I got: [ 486.863124] kmemleak: Cannot allocate a kmemleak_object structure [ 486.887195] kmemleak: Automatic memory scanning thread ended [ 486.906604] kmemleak: Kernel

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-11-25 Thread Dimitrios Apostolou
On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: > on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a > backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local > ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). > Even though earlier

backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-11-25 Thread Dimitrios Apostolou
Hello list, on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier system load was minimal, free memory was plenty, the

backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-11-25 Thread Dimitrios Apostolou
Hello list, on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier system load was minimal, free memory was plenty, the

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-11-25 Thread Dimitrios Apostolou
On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card). Even though earlier

kmemleak failure (was Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused)

2012-11-25 Thread Dimitrios Apostolou
Hello, I tried running a kmemleak and DEBUG_SLUB enabled kernel, but almost within 5 minutes of starting the backup process, I got: [ 486.863124] kmemleak: Cannot allocate a kmemleak_object structure [ 486.887195] kmemleak: Automatic memory scanning thread ended [ 486.906604] kmemleak: Kernel

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-11-25 Thread Roland Eggner
On 2012-11-25 Sunday at 21:30 +0200 Dimitrios Apostolou wrote: On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote: on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local ATA disk to ext3 on external

Re: kmemleak failure (was Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused)

2012-11-25 Thread Dimitrios Apostolou
On Mon, 2012-11-26 at 00:41 +0200, Dimitrios Apostolou wrote: Hello, I tried running a kmemleak and DEBUG_SLUB enabled kernel, but almost within 5 minutes of starting the backup process, I got: [ 486.863124] kmemleak: Cannot allocate a kmemleak_object structure [ 486.887195] kmemleak:

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-11-25 Thread Alan Cox
Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel? Probably not a good idea, there are known exploitable holes in 2.6.27 era kernels and nobody maintains anything that old. I guess RHEL/Centos might work for you. I've not had any problems with leaks in 3.6 but there have

Re: backing up ext4 fs, system unresponsive, thrashing like crazy even though swap is unused

2012-11-25 Thread Roland Eggner
On 2012-11-25 Sunday at 23:56 + Alan Cox wrote: Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel? Probably not a good idea, there are known exploitable holes in 2.6.27 era kernels and nobody maintains anything that old. “old” is relative … cd git/linux-stable