Instead of IP addresses the kernel started to show zero's

[Alexey Vlasov]

Hi.

Here it writes LOG module (netfilter) in syslog:

Sep 21 22:24:04 l24 kernel: ipsec:SYN-OUTPUT-HTTP IN= OUT=eth0
SRC= DST= LEN=60 TOS=0x00 PREC=0x00
TTL=64 ID=9042 DF PROTO=TCP SPT=51169 DPT=80 WINDOW=14600 RES=0x00 SYN
URGP=0 UID=545369 GID=155


This is recent, here go zero's again.

cat /proc/net/xt_recent/j-brute
src= ttl: 117 last_seen: 4349942400 oldest_pkt: 1
4349942400
src= ttl: 119 last_seen: 4349968063 oldest_pkt: 1
4349968063


Can it be fixed without restarting the box?
Thanks.

# uname -a
Linux l24 3.4.6 ...

--
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Instead of IP addresses the kernel started to show zero's

[Alexey Vlasov]

Hi.

Here it writes LOG target from syslog:

Sep 25 03:23:49 l24 kernel: ip:SYN-OUTPUT-HTTP IN= OUT=eth0
SRC= DST= LEN=60 TOS=0x00 PREC=0x00
TTL=64 ID=22467 DF PROTO=TCP SPT=52829 DPT=80 WINDOW=14600 RES=0x00 SYN
URGP=0 UID=564373 GID=155

This is recent, here go zero's again:
# cat /proc/net/xt_recent/ssh-brute
...
src= ttl: 122 last_seen: 4371027622 oldest_pkt: 1
4371027622

Can it be fixed without restarting the box?
Thanks!

Kernel 3.4.6.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Attaching a process to cgroups

[Alexey Vlasov]

On Wed, Aug 08, 2012 at 09:51:29AM -0700, Paul E. McKenney wrote:
> On Wed, Aug 08, 2012 at 08:40:33PM +0400, Alexey Vlasov wrote:
> > 
> > In general I've changed it to synchronize_rcu_expedited () and all the
> > delays have gone both on writing and reading files from cgroups.
> 
> Is the writing and reading from cgroups something that your workload
> does all the time, or is it something that happens only on occasional
> updates to your cgroup configuration?

There always were some delay on writing. It reproduces easily, you have
to create some 1000 groups (may be it can be enough to create 1 group, I
didn't check it actually) and write pid to a task file of the group. I
described it in my first message.

Delays on reading appeared when there began an active rotation of
proccesses in task files and may be by renewing of counters
(cpuacct.stat, memory.stat) due to the cgroups hierarchy. LA has grown
from 10 to 500 and all the programms that read cgroups files in /proc
(htop for example) practically stopped working.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Attaching a process to cgroups

[Alexey Vlasov]

On Wed, Jul 25, 2012 at 03:57:47PM +0200, Mike Galbraith wrote:
> > Hanging on read():
> > 
> > # strace -ttT cat /proc/cgroups 
> > 
> > 17:30:43.825005 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 13), 
> > ...}) = 0 <0.05>
> > 17:30:43.825048 open("/proc/cgroups", O_RDONLY) = 3 <0.14>
> > 17:30:43.825085 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> > <0.04>
> > 17:30:43.825125 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.05>
> > 17:30:43.825161 read(3, "#subsys_name\thierarchy\tnum_cgrou"..., 32768) = 
> > 112 <7.447084>
 
In general I've changed it to synchronize_rcu_expedited () and all the
delays have gone both on writing and reading files from cgroups.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Instead of IP addresses the kernel started to show zero's

[Alexey Vlasov]

On Tue, Oct 09, 2012 at 02:50:10PM +0200, Eric Dumazet wrote:
> 
> Reporter was (kindly) requested to try 3.6-rc7 +, and we got no answer.

Now I don't have any free servers where I could test a new kernel.
But in 1-2 months we'll be placing a new server fo a shared-hosting and
then I will be able to test new kernels.
Unfortunatelly I don't think it is reasonable to test kernels on production 
servers.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Attaching a process to cgroups

[Alexey Vlasov]

Hi.

Now I've got almost 5k used groups and it got even worse. Now I've got
almost 5k used groups and it got even worse.

If only write was working slower, now everything connected with cgroups
is hardly working.

Could it be connected with synchronize_rcu()?

Hanging on read():

# strace -ttT cat /proc/cgroups 

17:30:43.825005 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 13), ...}) 
= 0 <0.05>
17:30:43.825048 open("/proc/cgroups", O_RDONLY) = 3 <0.14>
17:30:43.825085 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 <0.04>
17:30:43.825125 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.05>
17:30:43.825161 read(3, "#subsys_name\thierarchy\tnum_cgrou"..., 32768) = 112 
<7.447084>
17:30:51.272302 write(1, "#subsys_name\thierarchy\tnum_cgrou"..., 
112#subsys_name   hierarchy   num_cgroups enabled
cpuacct 1   16844   1
memory  2   16844   1
devices 3   16844   1
blkio   4   16844   1
) = 112 <0.18>
17:30:51.272363 read(3, "", 32768)  = 0 <0.07>
17:30:51.272405 brk(0x62d000)   = 0x62d000 <0.11>
17:30:51.272444 close(3)= 0 <0.11>
17:30:51.272488 close(1)= 0 <0.07>
17:30:51.272518 close(2)= 0 <0.000005>

On Wed, Jun 20, 2012 at 02:28:18PM +0200, Mike Galbraith wrote:
> On Tue, 2012-06-19 at 22:58 +0400, Alexey Vlasov wrote: 

> > Is it possible to somehow fasten a process of pid attaching to cgroup?
> > The problem is the pid attaches to a task-file with some strange delay:
> > 
> > 22:28:00.788224 
> > open("/sys/fs/cgroup/memory/virtwww/w_test-l24-apache1_4bdf3d13/apache/tasks",
> >  O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.35>
> > 22:28:00.788289 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 
> > <0.04>
> > 22:28:00.788326 mmap(NULL, 4096, PROT_READ|PROT_WRITE, 
> > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5e78074000 <0.05>
> > 22:28:00.788355 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 
> > <0.04>
> > 22:28:00.788389 lseek(3, 0, SEEK_SET)   = 0 <0.04>
> > 22:28:00.788426 write(3, "16317\n", 6)  = 6 <0.128094>
> > 22:28:00.916578 close(3)= 0 <0.06>
> 
> 
> kernel/cgroup.c::cgroup_attach_task()
> {
> ...
>   synchronize_rcu();
> ...
> }
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Attaching a process to cgroups

[Alexey Vlasov]

On Wed, Jul 25, 2012 at 03:57:47PM +0200, Mike Galbraith wrote:
> 
> I'd profile it with perf, and expect to find a large pile of cycles.

I did it the as following:
# perf stat cat /proc/self/cgroup 

4:blkio:/
3:devices:/
2:memory:/
1:cpuacct:/

 Performance counter stats for 'cat /proc/self/cgroup':

  0.472513 task-clock#0.000 CPUs utilized  
 1 context-switches  #0.002 M/sec  
 1 CPU-migrations#0.002 M/sec  
   169 page-faults   #0.358 M/sec  
   521 cycles#2.352 GHz
784737 stalled-cycles-frontend   #   70.60% frontend cycles idle   
445520 stalled-cycles-backend#   40.08% backend  cycles idle   
576622 instructions  #0.52  insns per cycle
 #1.36  stalled cycles per insn
120032 branches  #  254.029 M/sec  
  6577 branch-misses #5.48% of all branches

   9.114631804 seconds time elapsed

# perf report --sort comm,dso
Kernel address maps (/proc/{kallsyms,modules}) were restricted.

Check /proc/sys/kernel/kptr_restrict before running 'perf record'.

If some relocation was applied (e.g. kexec) symbols may be misresolved.

Samples in kernel modules can't be resolved as well.

# 
# captured on: Thu Jul 26 16:23:06 2012
# hostname : l24
# os release : 3.3.3-1gb-c-s-m
# perf version : 3.2
# arch : x86_64
# nrcpus online : 24
# nrcpus avail : 24
# cpudesc : Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
# total memory : 74181032 kB
# cmdline : /usr/sbin/perf record cat /proc/self/cgroup 
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, 
excl_usr = 0, excl_kern = 0, id = { 1758, 1759, 1760, 1761, 1762, 1763, 1764, 
1765, 1766, 1767, 1768, 1769, 1770, 1771, 1772, 1773, 1774, 1775, 1776, 1777, 
1778, 1779, 1780, 1781 }
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# 
#
# Events: 21  cycles
#
# Overhead  Command  Shared Object
#   ...  .
#
   100.00%  cat  [kernel.kallsyms]

but I don't know what next unfortunately.

I also checked the same thing on the other server with the 2.6.37 kernel,
there' some thousands cgroups too, and it somehow works there immediately.

-- 
Alexey Vlasov
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Kernel migration eat CPUs

[Alexey Vlasov]

Hi Peter,

On Sun, Aug 25, 2013 at 04:28:37PM +0200, Peter Zijlstra wrote:
>
> Gargh.. I've never seen anything like that. Nor ever had a report like
> this. Is there anything in particular one can do to try and reproduce
> this?

I don't know how to reproduce it. This happens by itself and only on
high-loaded servers. For example this happens almost every hour on one
server with kernel 3.8.11 with 10k web-sites and 5k MySQL databases. On
another server with kernel 3.9.4 with same load this can take place 3-5
times per day. Sometimes this happens almost synchronously on both
servers.
I returned to kernel 2.6.35 on servers where this often took place. Or
they are not high-loaded enough that this effect doesn't appear.

For example here is server which earlier worked on kernel 3.9.4. It is
high-loaded, but migration stopped to eat CPUs after downgrade to
2.6.35.

# uname -r
2.6.35.7

# uptime
13:56:34 up 32 days, 10:31, 10 users, load average: 24.44, 23.44, 24.13

# ps -u root -o user,bsdtime,comm | grep -E 'COMMAND|migration'
USER   TIME COMMAND
root   4:20 migration/0
root   6:07 migration/1
root  17:00 migration/2
root   5:23 migration/3
root  16:43 migration/4
root   3:48 migration/5
root  12:28 migration/6
root   3:44 migration/7
root  12:25 migration/8
root   3:49 migration/9
root   1:52 migration/10
root   2:51 migration/11
root   1:28 migration/12
root   2:43 migration/13
root   2:16 migration/14
root   4:53 migration/15
root   2:15 migration/16
root   4:13 migration/17
root   2:13 migration/18
root   4:21 migration/19
root   2:07 migration/20
root   4:13 migration/21
root   2:13 migration/22
root   3:26 migration/23

For comparison 3.9.4:
# uptime
13:55:49 up 11 days, 15:36, 11 users, load average: 24.62, 24.36, 23.63

USER   TIME COMMAND
root 233:51 migration/0
root 233:38 migration/1
root 231:57 migration/2
root 233:26 migration/3
root 231:46 migration/4
root 233:26 migration/5
root 231:37 migration/6
root 232:56 migration/7
root 231:09 migration/8
root 232:34 migration/9
root 231:04 migration/10
root 232:22 migration/11
root 230:50 migration/12
root 232:16 migration/13
root 230:38 migration/14
root 231:51 migration/15
root 230:04 migration/16
root 230:16 migration/17
root 230:06 migration/18
root 230:22 migration/19
root 229:45 migration/20
root 229:43 migration/21
root 229:27 migration/22
root 229:24 migration/23
root 229:11 migration/24
root 229:25 migration/25
root 229:16 migration/26
root 228:58 migration/27
root 228:48 migration/28
root 229:06 migration/29
root 228:25 migration/30
root 228:25 migration/31
 

> Could you perhaps send your .config and a function (or function-graph)
> trace for when this happens?

My .config
https://www.dropbox.com/s/vuwvalj58cfgahu/.config_3.9.4-1gb-csmb-tr

I can't make trace because it isn't turned on on my kernels. I will be
able to reboot servers on weekend as there are many clients there and
will send you trace.

> Also, do you use weird things like cgroup/cpusets or other such fancy
> stuff? If so, could you outline your setup?

Grsec patch is used on all kernels. Also there is following patch only on
kernel 3.8.11:

--- kernel/cgroup.c.orig
+++ kernel/cgroup.c 
@@ -1931,7 +1931,8 @@
   ss->attach(cgrp, &tset);
}
-   synchronize_rcu();
+   synchronize_rcu_expedited();

/*
 * wake up rmdir() waiter. the rmdir should fail since the

Aslo I use https://github.com/facebook/flashcache/

Actually I really use cgroup namely controllers cpuacct, memory, blkio.
I create cgroup for every user on server, where all users processes are
running. To make it work there are needed patches in Apache/prefork, SSH
and other users staff. There can be about 10k-15k users and accordingly
same amount of cgroups.

The other day I disabled all cgroups, but controllers are still mounted.

# cat /proc/cgroups
#subsys_namehierarchy   num_cgroups enabled
cpuset  2   1   1
cpuacct 3   1   1
memory  4   1   1
blkio   5   1   1

But migration still eats CPUs. However I also use cgroup on kernel
2.6.35.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: BUG: soft lockup on all kernels after 2.6.3x

[Alexey Vlasov]

On Sat, Feb 09, 2013 at 07:07:53AM -0800, Eric Dumazet wrote:
> Did you compile the kernel yourself, or is it a standard kernel (distro
> provided) ?
> 
> Your traces dont contain symbols, its quite hard to guess the issue.

I compile the kernel myself. Should I add CONFIG_DEBUG_INFO ? ok then
I'll try it. May be I should switch on anything else to get more info
for debug?
Thanks.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Unstable kernels work after 2.6.3x

[Alexey Vlasov]

Hello,

I've already written that after release of version 2.6.32 something
strange happened either in the kernel or with its base features, namely
since then there was not released even a single kernel, which would at
least on my hosting work consistently and without bugs.

That's the history of kernel crashes on one of my servers:
1.1 2013-02-14 22:47:01 - 3.7.3
1.2 2013-03-07 00:48:25 - 3.7.3
1.3 2013-03-07 13:59:45 - 3.7.3
1.4 2013-03-10 01:20 - 3.7.3
1.5 2013-03-14 00:47 - 3.8.2
1.6 2013-03-15 00:57 - 3.8.2
1.10 2013-03-17 04:00 - 3.8.2
1.14 2013-03-24 22:42 - 3.8.2
1.15 2013-03-25 01:30 - 3.8.2
1.16 2013-03-25 02:57 - 3.7.3
1.17 2013-04-02 06:09 - 3.8.4
1.18 2013-04-12 21:50 - 3.8.4
1.22 2013-06-02 20:12 - 3.8.4
1.23 2013-06-09 16:33 - 3.8.4
1.24 2013-07-28 02:00 - 3.9.4
1.25 2013-08-17 16:45 - 3.9.4
1.26 2013-09-03 21:07 - 3.10.10
1.27 2013-09-07 21:07 - 3.11.0
1.28 2013-09-15 06:18 - 3.11.0
1.29 2013-10-01 21:51 - 3.11.0
1.31 2013-10-11 22:02 - 3.11.4
1.34 2013-10-26 06:32 - 3.11.4
1.36 2013-11-05 06:26 - 3.11.4
1.37 2013-11-17 15:06 - 3.11.4
1.38 2013-12-07 08:29 - 3.9.11
(date - kernel version)

And this is a server with an old one:
# uname -a
Linux l7 2.6.25-r9-1gb-s #3 SMP Fri Nov 14 18:15:42 MSK 2008 x86_64
Intel(R) Xeon(R) CPU E5430 @ 2.66GHz GenuineIntel GNU/Linux
# uptime
 13:29:40 up 731 days, 14:26,  5 users,  load average: 27.56, 18.75, 18.86

Another:
# uname -a
Linux l5 2.6.25-r7-1gb #4 SMP Fri Aug 29 18:43:15 MSD 2008 x86_64
Intel(R) Xeon(R) CPU E5345 @ 2.33GHz GenuineIntel GNU/Linux
# uptime
 13:31:48 up 733 days, 16:18,  5 users,  load average: 13.12, 10.79, 10.67

And it's not just some trash standing in the closet, it's a highloaded
server with several thousand websites and that also perform function of
a gateway for VPS customers.

I certainly can, in some way, understand all these hangings and
failures, but it is difficult and sometimes impossible to explain them
to our clients when they see their non-performing web-sites.

Here's one of the latest of bugs:
(3.12.3_kpanic.txt in attach)

I'm completely satisfied with this kernel specifically, if I could
somehow manage to fix this particular bug, it would be great. I don't
want to try again to move to another new kernel, I've already had such an
experience. If it is possible, can you just release a patch for
particularly this kernel?

Thank you in advance.

-- 
BRGDS. Alexey Vlasov.
Jan 18 02:37:50 l24 [3461838.276970] BUG: unable to handle kernel paging 
request at ee55d9c8
Jan 18 02:37:50 l24 [3461838.319959] IP: [] 
kmem_cache_alloc+0x50/0xc0
Jan 18 02:37:50 l24 [3461838.356616] PGD 6ce07b067 PUD 0
Jan 18 02:37:50 l24 [3461838.377203] Oops:  [#1] SMP 
Jan 18 02:37:50 l24 [3461838.397794] Modules linked in: flashcache(O) netconsole
Jan 18 02:37:50 l24 [3461838.430454] CPU: 15 PID: 7797 Comm: v3w_http_loadti 
Tainted: G  IO 3.12.3-1gb-cm #1
Jan 18 02:37:50 l24 [3461838.481706] Hardware name: Intel Corporation 
S5520UR/S5520UR, BIOS S5500.86B.01.00.0061.030920121535 03/09/2012
Jan 18 02:37:50 l24 [3461838.543378] task: 881a0f11e0f0 ti: 
88018924 task.ti: 88018924
Jan 18 02:37:50 l24 [3461838.813383] RIP: 0010:[] 
[] kmem_cache_alloc+0x50/0xc0
Jan 18 02:37:50 l24 [3461838.864721] RSP: 0018:880189241dc0  EFLAGS: 
00010282
Jan 18 02:37:50 l24 [3461838.897674] RAX:  RBX: 
8810a842ac10 RCX: 01b6
Jan 18 02:37:50 l24 [3461838.941635] RDX: bb320459 RSI: 
00d0 RDI: 00014340
Jan 18 02:37:50 l24 [3461838.985599] RBP: 880189241de0 R08: 
881c6fcf4340 R09: 81062f68
Jan 18 02:37:51 l24 [3461839.029557] R10:  R11: 
 R12: 88102f803800
Jan 18 02:37:51 l24 [3461839.073525] R13: ee55d9c8 R14: 
00d0 R15: 7fbce95779d0
Jan 18 02:37:51 l24 [3461839.117497] FS:  7fbce9577700() 
GS:881c6fce() knlGS:
Jan 18 02:37:51 l24 [3461839.168441] CS:  0010 DS:  ES:  CR0: 
8005003b
Jan 18 02:37:51 l24 [3461839.204006] CR2: ee55d9c8 CR3: 
000a3900f000 CR4: 07e0
Jan 18 02:37:51 l24 [3461839.360725] Stack:
Jan 18 02:37:51 l24 [3461839.373892]  8810a842ac10  
8810a842ac10 
Jan 18 02:37:51 l24 [3461839.419631]  880189241e00 81062f68 
8810a842ac10 
Jan 18 02:37:51 l24 [3461839.465448]  880189241e30 81063531 
880189241e30 01200011
Jan 18 02:37:51 l24 [3461839.511230] Call Trace:
Jan 18 02:37:51 l24 [3461839.527013]  [] 
prepare_creds+0x18/0xc0
Jan 18 02:37:51 l24 [3461839.560494]  [] copy_creds+0x61/0x130
Jan 18 02:37:51 l24 [3461839.592935]  [] 
copy_process+0x384/0x1430
Jan 18 02:37:51 l24 [3461839.627455]  [] ? 
check_preempt_curr+0x75/0xa0
Jan 18 02:37:51 l24 [3461839.664581]  [] ? 
wake_up_new_task+0xff/0x140
Jan 18 02:37:51 l24 [3461839.701183]  [] do_fork+0x68/0

Re: Kernel migration eat CPUs

[Alexey Vlasov]

Hi Ingo,

On Wed, Sep 11, 2013 at 07:18:14PM +0400, Alexey Vlasov wrote:
> On Thu, Sep 05, 2013 at 01:12:52PM +0200, Ingo Molnar wrote:
> > 
> > Could you try the latest -tip tree at:
> > 
> >git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
> > 
> > In particular this fix:
> > 
> >5a8e01f8fa51 sched/cputime: Do not scale when utime == 0
> > 
> > Could perhaps fix the phantom CPU overhead you are seeing?
> 
> This kernel solved problems.
> Thank you.

One problem is solved, but there appeared another one. Now this kernel
began to crash.

Crashdump is in attachment.

-- 
BRGDS. Alexey Vlasov.
Oct  4 20:42:55 l25 [255033.586773] BUG: unable to handle kernel
Oct  4 20:42:55 l25 paging request
Oct  4 20:42:55 l25 at 0001
Oct  4 20:42:55 l25 [255033.594760] IP:
Oct  4 20:42:55 l25 [] kmem_cache_alloc+0x51/0x120
Oct  4 20:42:55 l25 [255033.601622] PGD 199b953067
Oct  4 20:42:55 l25 PUD 0
Oct  4 20:42:55 l25
Oct  4 20:42:55 l25 [255033.605481] Oops:  [#1]
Oct  4 20:42:55 l25 SMP
Oct  4 20:42:55 l25
Oct  4 20:42:55 l25 [255033.609249] Modules linked in:
Oct  4 20:42:55 l25 netconsole
Oct  4 20:42:55 l25 flashcache(O)
Oct  4 20:42:55 l25 x86_pkg_temp_thermal
Oct  4 20:42:55 l25
Oct  4 20:42:55 l25 [255033.617333] CPU: 13 PID: 21874 Comm: httpd Tainted: G   
   IO 3.11.0-1gb-cm-tr+ #1
Oct  4 20:42:55 l25 [255033.626297] Hardware name: Intel Corporation 
S2600WP/S2600WP, BIOS SE5C600.86B.01.06.0002.110120121539 11/01/2012
Oct  4 20:42:55 l25 [255033.637900] task: 880018108000 ti: 88003aafa000 
task.ti: 88003aafa000
Oct  4 20:42:55 l25 [255033.646386] RIP: 0010:[]
Oct  4 20:42:55 l25 [] kmem_cache_alloc+0x51/0x120
Oct  4 20:42:55 l25 [255033.656009] RSP: 0018:88003aafbe18  EFLAGS: 00010206
Oct  4 20:42:55 l25 [255033.662077] RAX:  RBX: 880e6fac6d80 
RCX: 
Oct  4 20:42:55 l25 [255033.670165] RDX: 0a4f5d48 RSI: 80d0 
RDI: 00015380
Oct  4 20:42:55 l25 [255033.678256] RBP: 88003aafbe58 R08: 881c3fcb5380 
R09: 81179fdf
Oct  4 20:42:55 l25 [255033.686442] R10: 01d26b20 R11: 0246 
R12: 8813ef803800
Oct  4 20:42:55 l25 [255033.694705] R13: 0001 R14: 880018108000 
R15: 88161f1eac00
Oct  4 20:42:55 l25 [255033.702968] FS:  7f3d46edc740() 
GS:881c3fca() knlGS:
Oct  4 20:42:55 l25 [255033.733352] CS:  0010 DS:  ES:  CR0: 
80050033
Oct  4 20:42:55 l25 [255033.739996] CR2: 0001 CR3: 001688961000 
CR4: 000407e0
Oct  4 20:42:55 l25 [255033.748263] Stack:
Oct  4 20:42:55 l25 [255033.750716]  
Oct  4 20:42:55 l25 80d0
Oct  4 20:42:55 l25 
Oct  4 20:42:55 l25 880e6fac6d80
Oct  4 20:42:55 l25
Oct  4 20:42:55 l25 [255033.759374]  88003aafbec8
Oct  4 20:42:55 l25 880e6fac6dc8
Oct  4 20:42:55 l25 880018108000
Oct  4 20:42:55 l25 88161f1eac00
Oct  4 20:42:55 l25
Oct  4 20:42:55 l25 [255033.768028]  88003aafbe78
Oct  4 20:42:55 l25 81179fdf
Oct  4 20:42:55 l25 0001
Oct  4 20:42:55 l25 880e6fac6d80
Oct  4 20:42:55 l25
Oct  4 20:42:55 l25 [255033.776672] Call Trace:
Oct  4 20:42:55 l25 [255033.779612]  [] 
locks_alloc_lock+0x1f/0x70
Oct  4 20:42:55 l25 [255033.786139]  [] 
flock_lock_file_wait+0x254/0x380
Oct  4 20:42:55 l25 [255033.793262]  [] ? 
kmem_cache_alloc+0xf8/0x120
Oct  4 20:42:55 l25 [255033.800085]  [] SyS_flock+0x1bb/0x1d0
Oct  4 20:42:55 l25 [255033.806145]  [] 
system_call_fastpath+0x16/0x1b
Oct  4 20:42:55 l25 [255033.813062] Code:
Oct  4 20:42:55 l25 03
Oct  4 20:42:55 l25 04
Oct  4 20:42:55 l25 25
Oct  4 20:42:55 l25 08
Oct  4 20:42:55 l25 cc
Oct  4 20:42:55 l25 00
Oct  4 20:42:55 l25 00
Oct  4 20:42:55 l25 49
Oct  4 20:42:55 l25 8b
Oct  4 20:42:55 l25 50
Oct  4 20:42:55 l25 08
Oct  4 20:42:55 l25 4d
Oct  4 20:42:55 l25 8b
Oct  4 20:42:55 l25 28
Oct  4 20:42:55 l25 49
Oct  4 20:42:55 l25 83
Oct  4 20:42:55 l25 78
Oct  4 20:42:55 l25 10
Oct  4 20:42:55 l25 00
Oct  4 20:42:55 l25 0f
Oct  4 20:42:55 l25 84
Oct  4 20:42:55 l25 be
Oct  4 20:42:55 l25 00
Oct  4 20:42:55 l25 00
Oct  4 20:42:55 l25 00
Oct  4 20:42:55 l25 4d
Oct  4 20:42:55 l25 85
Oct  4 20:42:55 l25 ed
Oct  4 20:42:55 l25 0f
Oct  4 20:42:55 l25 84
Oct  4 20:42:55 l25 b5
Oct  4 20:42:55 l25 00
Oct  4 20:42:55 l25 00
Oct  4 20:42:55 l25 00
Oct  4 20:42:55 l25 49
Oct  4 20:42:55 l25 63
Oct  4 20:42:55 l25 44
Oct  4 20:42:55 l25 24
Oct  4 20:42:55 l25 20
Oct  4 20:42:55 l25 49
Oct  4 20:42:55 l25 8b
Oct  4 20:42:55 l25 3c
Oct  4 20:42:55 l25 24
Oct  4 20:42:55 l25
Oct  4 20:42:55 l25 8b
Oct  4 20:42:55 l25 5c
Oct  4 20:42:55 l25 05
Oct  4 20:42:55 l25 00
Oct  4 20:42:55 l25 48
Oct  4 20:42:55 l25 8d
Oct  4 20:42:55 l25 4a
Oct  4 20:42:55 l25 01
Oct  4 20:42:55 l25 4c
Oct  4 20:42:55 l25 89
Oct  4 20:42:55 l25 e8
Oct  4 20:42:55 l25 65
Oct  4 20:42:55 l25 48
Oct

Re: Kernel migration eat CPUs

[Alexey Vlasov]

On Thu, Sep 05, 2013 at 01:12:52PM +0200, Ingo Molnar wrote:
> 
> Could you try the latest -tip tree at:
> 
>git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
> 
> In particular this fix:
> 
>5a8e01f8fa51 sched/cputime: Do not scale when utime == 0
> 
> Could perhaps fix the phantom CPU overhead you are seeing?

This kernel solved problems.
Thank you.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

There is strange packets dropping on new kernels

[Alexey Vlasov]

Hi.

I noticed that tcpdump shows many dropped packets on new kernels.
It looks like that:

# tcpdump -n -i eth0 -c 10
...
10 packets captured
10 packets received by filter
0 packets dropped by kernel
4294967268 packets dropped by interface
# uname -r
3.10.7

On old kernels the same command doesn't drop anything.
# tcpdump -n -i eth0 -c 10
10 packets captured
140 packets received by filter
0 packets dropped by kernel
# uname -r
3.8.11

What can I do to fix it? Or is it right?
Thanks.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: There is strange packets dropping on new kernels

[Alexey Vlasov]

On Thu, Aug 22, 2013 at 12:51:24PM +0200, Paul Rolland wrote:
> 
> Looks like you are not using the same tcpdump... Could it be a bug related
> to that application ?

You are right. Then problem is in new version of tcpdump.
Thanks.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Kernel migration eat CPUs

[Alexey Vlasov]

Hi,

Beginning from 3.7 including 3.10 kernel, I noticed that from time to
time all CPUs turn out to be busy 100% and at the moment websites of my
clients work slowly or don’t work at all.

top screen:
https://www.dropbox.com/s/psuk2cwn6exinyd/3.9.cpu_migration.jpg

All CPUs are busy with kernel threads migration for about 3-5 minutes.
Exactly how often it periodically happens I don’t know, but it happens
once in 1-3 hours.

Approximately 30 my servers work on kernel 2.6.35 and there’s no such
problem there. We use kernels 3.7 – 3.9 on 5 servers and I see such
effect everywhere. If you downgrade on 2.6.35, then migration doesn’t
eat CPUs.

These servers are used for usual LAMP shared hosting.

Can I get migration as if on the kernel 2.6.35 or can this feature be
disabled?

-- 
BRGDS. Alexey Vlasov.
jabber: ren...@1gb.ru
icq: 5128664
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Kernel migration eat CPUs

[Alexey Vlasov]

On Sun, Aug 25, 2013 at 04:28:37PM +0200, Peter Zijlstra wrote:

> and a function (or function-graph)
> trace for when this happens?

Unfortunately I could not make trace.
First when migration threads start to eat CPUs and when I turn on trace:
# echo function > current_tracer
# echo 1 > tracing_on
then server is down at once. LA comes up to several thousands, it stops
to receive commands. Sometimes I even have to reboot it.

And second when I manage to get trace then it shows little useful data.

https://www.dropbox.com/s/kxgwr9fuckddtgf/trace.txt.gz

Probably this junk appears from grsec patch? And I don't see migration
threads in it.

> Perhaps perf top -p  while the problem is happening will
> show something.

Also "perf top" doesn't show anything. It gives same values when
migration threads eat CPUs and when everything is OK on server.

=
PerfTop: 68932 irqs/sec kernel:60.3%  exact: 0.0% [4000Hz cycles], (all, 32 
CPUs)
-

64.11%  [netconsole]   [k] 0x81000de3
34.19%  [unknown]  [.] 0x00585c45
 0.40%  libc-2.15.so   [.] 0x00124b10
 0.23%  perf   [.] 0x0004df4f
 0.14%  ld-2.15.so [.] 0x000183e7
 0.08%  libc-2.15.so   [.] _nss_files_parse_grent
 0.06%  libc-2.15.so   [.] _nss_files_parse_pwent
 0.04%  libc-2.15.so   [.] memchr
 0.04%  php-cgi[.] match
 0.03%  libcrypt-2.15.so   [.] 0x4eab
 0.03%  bash   [.] 0x0001fe5a
 0.02%  libc-2.15.so   [.] _IO_getline_info
 0.02%  php-cgi[.] php_pcre_exec
 0.02%  libc-2.15.so   [.] _dl_addr
 0.02%  eaccelerator.so[.] eaccelerator_crc32
 0.02%  php-cgi[.] zend_hash_quick_find
 0.02%  libnss_compat-2.15.so  [.] 0x55f1
 0.01%  php-cgi[.] zendparse
 0.01%  libc-2.15.so   [.] fgets_unlocked
 0.01%  libnss_compat-2.15.so  [.] _nss_compat_initgroups_dyn
=

Now server is more loaded and eating CPUs by migration threads happens more 
often
and it lasts longer.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: BUG: soft lockup on all kernels after 2.6.3x

[Alexey Vlasov]

On Thu, Mar 07, 2013 at 08:57:28AM -0800, Eric Dumazet wrote:
> 
> Well, remove all alien patches and try to reproduce the bug with a
> pristine linux kernel.

I wrote to Spender (developer grsec) and he confirmed that it's possible
that a problem is with grsec patch.

Thank you greatly for your answers!

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: BUG: soft lockup on all kernels after 2.6.3x

[Alexey Vlasov]

Hi,

On Sat, Feb 09, 2013 at 07:07:53AM -0800, Eric Dumazet wrote:
> >
> > I used 2.6.2x kernel for a long time on my shared hosting and I didn't
> > have any problems. Kernels worked well and server uptime was about 2-3
> > years.
> > 
> > ...
> > 
> > it doesn't happen on an empty server, only on loaded ones. Unfortunately
> > I don't know how to provoke such hanging artificially.
> > 
 
> Your traces dont contain symbols, its quite hard to guess the issue.

Well the server got high loaded and began to crash almost once a day.

=
BUG: soft lockup - CPU#1 stuck for 23s! [httpd:21686]
Call Trace:
[] ? mntput_no_expire+0x25/0x170
[] ? path_lookupat+0x189/0x890
[] ? filename_lookup.clone.39+0xd7/0xe0
[] ? user_path_at_empty+0x5c/0xb0
[] ? __do_page_fault+0x1b9/0x480
[] ? vfs_fstatat+0x3e/0x90
[] ? remove_vma+0x5f/0x70
[] ? sys_newstat+0x1f/0x50
[] ? page_fault+0x22/0x30
[] ? system_call_fastpath+0x18/0x1d
=

There's a full trace in attachment. 

-- 
BRGDS. Alexey Vlasov.


bug_softlockup.txt.gz
Description: Binary data

Re: BUG: soft lockup on all kernels after 2.6.3x

[Alexey Vlasov]

Hi,

On Sat, Feb 09, 2013 at 07:07:53AM -0800, Eric Dumazet wrote:
> >
> > I used 2.6.2x kernel for a long time on my shared hosting and I
> > didn't
> > have any problems. Kernels worked well and server uptime was about
> > 2-3
> > years.
> >
> > ...
> >
> > it doesn't happen on an empty server, only on loaded ones.
> > Unfortunately
> > I don't know how to provoke such hanging artificially.
> >

> Your traces dont contain symbols, its quite hard to guess the issue.

Well the server got high loaded and began to crash almost once a day.

=
BUG: soft lockup - CPU#1 stuck for 23s! [httpd:21686]
Call Trace:
[] ? mntput_no_expire+0x25/0x170
[] ? path_lookupat+0x189/0x890
[] ? filename_lookup.clone.39+0xd7/0xe0
[] ? user_path_at_empty+0x5c/0xb0
[] ? __do_page_fault+0x1b9/0x480
[] ? vfs_fstatat+0x3e/0x90
[] ? remove_vma+0x5f/0x70
[] ? sys_newstat+0x1f/0x50
[] ? page_fault+0x22/0x30
[] ? system_call_fastpath+0x18/0x1d
=

There's a full trace in attachment.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: BUG: soft lockup on all kernels after 2.6.3x (include full log)

[Alexey Vlasov]

On Thu, Mar 07, 2013 at 05:34:14PM +0400, Alexey Vlasov wrote:
> 
> There's a full trace in attachment.
 
-- 
BRGDS. Alexey Vlasov.


bug_softlockup.txt.gz
Description: Binary data

Re: BUG: soft lockup on all kernels after 2.6.3x

[Alexey Vlasov]

On Thu, Mar 07, 2013 at 08:20:23AM -0800, Eric Dumazet wrote:
>
> What are gr_ symbols ?

This is grsecurity patches ;)
 
> Mar  7 00:50:00 l25 [1735187.889877]  [] ? 
> is_path_reachable+0x48/0x60
> Mar  7 00:50:00 l25 [1735187.889880]  [] ? 
> path_is_under+0x33/0x60
> Mar  7 00:50:00 l25 [1735187.889887]  [] ? 
> gr_is_outside_chroot+0x54/0x70
> Mar  7 00:50:00 l25 [1735187.889890]  [] ? 
> gr_chroot_fchdir+0x55/0x80
> Mar  7 00:50:00 l25 [1735187.889894]  [] ? 
> filename_lookup.clone.39+0x9e/0xe0
> Mar  7 00:50:00 l25 [1735187.889897]  [] ? 
> user_path_at_empty+0x5c/0xb0
> Mar  7 00:50:00 l25 [1735187.889903]  [] ? 
> __do_page_fault+0x1b9/0x480
> Mar  7 00:50:00 l25 [1735187.889907]  [] ? 
> page_fault+0x22/0x30
> Mar  7 00:50:00 l25 [1735187.889910]  [] ? 
> vfs_fstatat+0x3e/0x90
> Mar  7 00:50:00 l25 [1735187.889914]  [] ? 
> gr_learn_resource+0x3b/0x1e0
> Mar  7 00:50:00 l25 [1735187.889918]  [] ? 
> sys_newstat+0x1f/0x50
> Mar  7 00:50:00 l25 [1735187.889922]  [] ? 
> filp_close+0x54/0x80
> Mar  7 00:50:00 l25 [1735187.889925]  [] ? 
> page_fault+0x22/0x30
> Mar  7 00:50:00 l25 [1735187.889928]  [] ? 
> system_call_fastpath+0x18/0x1d
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Alexey Vlasov]

I have found a workaround preventing these hangs.
Primarily, disable THP:

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

and next, we should increase vm.min_free_kbytes, in my case 16Gb is
enough

vm.min_free_kbytes = 16777216

On Wed, Jul 15, 2020 at 01:04:38PM +0300, Alexey Vlasov wrote:
> Hi,
> 
> After upgrading from 3.14 to 4.14.173, I ran into exactly the same problem
> that the starter topic described. Namely, sometimes kswapd starts to consume 
> 100% 
> of the CPU, and the system freezes for several minutes.
> 
> Below is an example of such an event (orange - system cpu, red - total cpu):
> https://www.dropbox.com/s/5wr5su3p0fubq0a/kswapd_100.png?dl=0
> 
> Here is the top:
> 
> top - 23:44:16 up 9 days,  2:06, 14 users,  load average: 14.03, 12.32, 13.07
> Tasks: 7108 total,  16 running, 6921 sleeping,   0 stopped,   9 zombie
> %Cpu(s): 28.1 us, 18.1 sy,  0.0 ni, 51.7 id,  1.2 wa,  0.0 hi,  0.9 si,  0.0 
> st
> KiB Mem : 19803248+total,   596160 free, 11094233+used, 86493992 buff/cache
> KiB Swap: 62914556 total, 62302912 free,   611644 used. 71269504 avail Mem
> 
>   PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
>   134 root  20   0   0  0  0 R  86.2  0.0 383:21.35 kswapd0
>   135 root  20   0   0  0  0 R  84.9  0.0 344:00.17 kswapd1
> 
> this is a begin of the collapse, some minutes later the system has thousands 
> of D
> processes and does not answer:
> 
> top - 23:57:33 up 9 days,  2:19, 14 users,  load average: 1223.43, 1083.85, 
> 662.
> Tasks: 8356 total, 344 running, 7821 sleeping,   0 stopped,  44 zombie
> %Cpu(s): 28.1 us, 18.2 sy,  0.0 ni, 51.6 id,  1.2 wa,  0.0 hi,  0.9 si,  0.0 
> st
> KiB Mem : 19803248+total,   800516 free, 11587540+used, 81356560 buff/cache
> KiB Swap: 62914556 total, 62130072 free,   784484 used. 62231208 avail Mem
> 
>   PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
> 10704 w_defau+  20   0  393476 117160  15160 D 100.0  0.1   0:00.16 httpd
> 16056 w_sti46+  20   0  599048  21528   9504 S 100.0  0.0   0:00.00 httpd
> 12649 w_divan+  20   0   41764   8064   3904 D 100.0  0.0   0:06.62 menu1.pl
> 13739 w_defau+  20   0  248696  24168  14132 S 100.0  0.0   0:00.01 httpd
>  5172 mysql 20   0 6993508 2.310g   9660 D  38.9  1.2   3866:26 
> mysqld_aux3
>  4683 mysql 20   0 9974.1m 4.366g   8268 D  38.7  2.3   2553:14 mysqld
>  4791 mysql 20   0 10.359g 4.180g   9784 D  28.5  2.2   1659:40 
> mysqld_aux1
>  5078 mysql 20   0  9.871g 3.774g   9888 D  25.4  2.0   2445:08 
> mysqld_aux2
> 9 root  20   0   0  0  0 I   3.4  0.0  13:56.16 rcu_sched
>   135 root  20   0   0  0  0 D   2.8  0.0 344:29.12 kswapd1
>   134 root  20   0   0  0  0 D   2.6  0.0 383:49.86 kswapd0
> 
> Nevertheless there is not any I/O activity before after and during this 
> collapse.
> 
> I tried to use your patch about 
> "late_initcall(set_recommended_min_free_kbytes)",
> unfortunately it did not help.
> 
> In my experience this could be solved by adding RAM but unfortunately this 
> server
> no longer has free slots. 188 GB RAM is the maximum for it.
> 
> Also I cannot go back to 3.14 kernel, since one of the partitions contains 
> xfs with
> the superblock of the new version v5, which is not supported by 3.14 kernel.
> 
> If you need more information, for example, vmstat, /proc/meminfo, I can send.
> 
> Is there any solution to this problem?
> 
> > On Fri, Jan 22, 2016 at 12:28:10AM +1000, Nalorokk wrote:
> >> It appears that kernels newer than 4.1 have kswapd-related bug resulting in
> >> high CPU usage. CPU 100% usage could last for several minutes or several
> >> days, with CPU being busy entirely with serving kswapd. It happens usually
> >> after server being mostly idle, sometimes after days, sometimes after weeks
> >> of uptime. But the issue appears much sooner if the machine is loaded with
> >> something like building a kernel.
> >>
> >> Here are the graphs of CPU load: first
> >> <http://i.piccy.info/i9/9ee6c0620c9481a974908484b2a52a0f/1453384595/44012/994698/cpu_month.png>,
> >> second
> >> <http://i.piccy.info/i9/7c97c2f39620bb9d7ea93096312dbbb6/1453384649/41222/994698/cpu_year.png>.
> >> Perf top output is here <http://pastebin.com/aRzTjb2x>as well.
> >>
> >> To find the cause of this problem I've started with the fact that the issue
> >> appeared after 4.1 kernel update. Then I performed longterm test of 3.18,
> >> and discovered that 3.18 is unaffected by this bug. The

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Alexey Vlasov]

Hi,

After upgrading from 3.14 to 4.14.173, I ran into exactly the same problem
that the starter topic described. Namely, sometimes kswapd starts to consume 
100% 
of the CPU, and the system freezes for several minutes.

Below is an example of such an event (orange - system cpu, red - total cpu):
https://www.dropbox.com/s/5wr5su3p0fubq0a/kswapd_100.png?dl=0

Here is the top:

top - 23:44:16 up 9 days,  2:06, 14 users,  load average: 14.03, 12.32, 13.07
Tasks: 7108 total,  16 running, 6921 sleeping,   0 stopped,   9 zombie
%Cpu(s): 28.1 us, 18.1 sy,  0.0 ni, 51.7 id,  1.2 wa,  0.0 hi,  0.9 si,  0.0 st
KiB Mem : 19803248+total,   596160 free, 11094233+used, 86493992 buff/cache
KiB Swap: 62914556 total, 62302912 free,   611644 used. 71269504 avail Mem

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
  134 root  20   0   0  0  0 R  86.2  0.0 383:21.35 kswapd0
  135 root  20   0   0  0  0 R  84.9  0.0 344:00.17 kswapd1

this is a begin of the collapse, some minutes later the system has thousands of 
D
processes and does not answer:

top - 23:57:33 up 9 days,  2:19, 14 users,  load average: 1223.43, 1083.85, 662.
Tasks: 8356 total, 344 running, 7821 sleeping,   0 stopped,  44 zombie
%Cpu(s): 28.1 us, 18.2 sy,  0.0 ni, 51.6 id,  1.2 wa,  0.0 hi,  0.9 si,  0.0 st
KiB Mem : 19803248+total,   800516 free, 11587540+used, 81356560 buff/cache
KiB Swap: 62914556 total, 62130072 free,   784484 used. 62231208 avail Mem

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
10704 w_defau+  20   0  393476 117160  15160 D 100.0  0.1   0:00.16 httpd
16056 w_sti46+  20   0  599048  21528   9504 S 100.0  0.0   0:00.00 httpd
12649 w_divan+  20   0   41764   8064   3904 D 100.0  0.0   0:06.62 menu1.pl
13739 w_defau+  20   0  248696  24168  14132 S 100.0  0.0   0:00.01 httpd
 5172 mysql 20   0 6993508 2.310g   9660 D  38.9  1.2   3866:26 mysqld_aux3
 4683 mysql 20   0 9974.1m 4.366g   8268 D  38.7  2.3   2553:14 mysqld
 4791 mysql 20   0 10.359g 4.180g   9784 D  28.5  2.2   1659:40 mysqld_aux1
 5078 mysql 20   0  9.871g 3.774g   9888 D  25.4  2.0   2445:08 mysqld_aux2
9 root  20   0   0  0  0 I   3.4  0.0  13:56.16 rcu_sched
  135 root  20   0   0  0  0 D   2.8  0.0 344:29.12 kswapd1
  134 root  20   0   0  0  0 D   2.6  0.0 383:49.86 kswapd0

Nevertheless there is not any I/O activity before after and during this 
collapse.

I tried to use your patch about 
"late_initcall(set_recommended_min_free_kbytes)",
unfortunately it did not help.

In my experience this could be solved by adding RAM but unfortunately this 
server
no longer has free slots. 188 GB RAM is the maximum for it.

Also I cannot go back to 3.14 kernel, since one of the partitions contains xfs 
with
the superblock of the new version v5, which is not supported by 3.14 kernel.

If you need more information, for example, vmstat, /proc/meminfo, I can send.

Is there any solution to this problem?

> On Fri, Jan 22, 2016 at 12:28:10AM +1000, Nalorokk wrote:
>> It appears that kernels newer than 4.1 have kswapd-related bug resulting in
>> high CPU usage. CPU 100% usage could last for several minutes or several
>> days, with CPU being busy entirely with serving kswapd. It happens usually
>> after server being mostly idle, sometimes after days, sometimes after weeks
>> of uptime. But the issue appears much sooner if the machine is loaded with
>> something like building a kernel.
>>
>> Here are the graphs of CPU load: first
>> ,
>> second
>> .
>> Perf top output is here as well.
>>
>> To find the cause of this problem I've started with the fact that the issue
>> appeared after 4.1 kernel update. Then I performed longterm test of 3.18,
>> and discovered that 3.18 is unaffected by this bug. Then I did some tests
>> of 4.0 to confirm that this version behaves well too.
>>
>> Then I performed git bisect from tag v4.0 to v4.1-rc1 and found exact
>> commits that seem to be reason of high CPU usage.
>>
>> The first really "bad" commit is 79553da293d38d63097278de13e28a3b371f43c1.
>> 2 previous commits cause weird behavior as well resulting in kswapd
>> consuming more CPU than unaffected kernels, but not that much as the commit
>> pointed above. I believe those commits are related to the same mm tree
>> merge.
>>
>> I tried to add transparent_hugepage=never to kernel boot parameters, but it
>> did not change anything. Changing allocator to SLAB from SLUB alters
>> behavior and makes CPU load lower, but don't solve a problem at all.
>>
>> Here is kernel bugzilla
>> bugreport as well.
>>
>> Ideas? â
>
> Could you try to insert "late_initcall(set_recommended_min_free_kbytes);