Re: [PATCH v2 7/4] ipc: fine grained locking for semtimedop
On 03/06/2013 08:36 PM, Davidlohr Bueso wrote:
But my 8 socket 160 CPU box sure isn't happy. I'm getting all sorts of
issues (sometimes it will boot, sometimes it wont). When it does, linux
will hang as soon as I start my benchmarking:
I think I may have found the cause of this bug.
It looks like it would be possible for a caller with a single
semop to come into semtimedop, see that sma->complex_count is
elevated, and take the write lock.
Further down in semtimedop, that caller can call do_smart_update,
which calls update_queue, which in turn calls unlink_queue.
The function unlink_queue can then decrement sma->complex_count.
In other words, we correctly grab the write lock when changing
sma->complex_count. However, we do not remember that we took
the write lock, and call read_unlock when we leave semtimedop.
The next caller will get stuck.
To fix this bug, we need to remember whether or not we took
the lock in exclusive mode, and unlock it the right way.
Now that I have figured this bug out, it should be easy to
re-implement things the way Linus suggested, subject to the
same bug fixes :)
BUG: soft lockup - CPU#77 stuck for 23s! [oracle:129877]
Modules linked in: fuse autofs4 sunrpc pcc_cpufreq ipv6 dm_mirror
dm_region_hash dm_log dm_mod uinput iTCO_wdt iTCO_vendor_support sg freq_table
mperf coretemp kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr
lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core netxen_nic ext4 mbcache jbd2
sd_mod crc_t10dif aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul
hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
CPU 77
Pid: 129877, comm: oracle Tainted: G D W3.9.0-rc1+ #20 HP ProLiant
DL980 G7
RIP: 0010:[] [] __read_lock_failed+0xa/0x20
RSP: 0018:8b87b8cf9ca8 EFLAGS: 0297
RAX: c900293c1020 RBX: 00010007a021 RCX: d3a5
RDX: 0001 RSI: 8b87b8cf9d58 RDI: c900293c1020
RBP: 8b87b8cf9ca8 R08: R09:
R10: R11: R12: 8b87b8cf9c68
R13: 8b87b8cf9c68 R14: 0286 R15: 8b87caf10100
FS: 7f7a689b2700() GS:8987ff9c() knlGS:
CS: 0010 DS: ES: CR0: 80050033
CR2: 7fc49426d000 CR3: 0187cf08f000 CR4: 07e0
DR0: DR1: DR2:
DR3: DR6: 0ff0 DR7: 0400
Process oracle (pid: 129877, threadinfo 8b87b8cf8000, task 8b87caf10100)
Stack:
8b87b8cf9cb8 8155f374 8b87b8cf9ce8 81205245
0001 00090002 7fff82d3aa08
8b87b8cf9f78 812069e1 00cbc000 8b87b8cf9f38
Call Trace:
[] _raw_read_lock+0x14/0x20
[] sem_lock+0x85/0xa0
[] sys_semtimedop+0x521/0x7c0
[] ? task_sched_runtime+0x4c/0x90
[] ? native_sched_clock+0x13/0x80
[] ? sched_clock+0x9/0x10
[] ? sched_clock_cpu+0xcd/0x110
[] ? update_rq_clock+0x2b/0x50
[] ? task_sched_runtime+0x4c/0x90
[] ? thread_group_cputime+0x88/0xc0
[] ? cputime_adjust+0x3d/0x90
[] ? thread_group_cputime_adjusted+0x4e/0x60
[] system_call_fastpath+0x16/0x1b
Code: 90 55 48 89 e5 f0 ff 07 f3 90 83 3f 01 75 f9 f0 ff 0f 75 f1 5d c3 66 66 2e 0f
1f 84 00 00 00 00 00 55 48 89 e5 f0 48 ff 07 f3 90 <48> 83 3f 01 78 f8 f0 48 ff
0f 78 ee 5d c3 90 90 90 90 90 90 90
+static inline void sem_unlock(struct sem_array *sma, struct sembuf *sops,
+ int nsops)
+{
+ if (nsops == 1 && sma->sem_nsems > 1 && !sma->complex_count) {
+ struct sem *sem = sma->sem_base + sops->sem_num;
+ spin_unlock(>lock);
+ read_unlock(>sem_perm.lock);
+ } else
+ write_unlock(>sem_perm.lock);
+}
I believe (haven't tested it yet) the issue could be in sem_unlock() as
we aren't unlocking the RCU read section - before it was just a trivial
ipc_unlock call.
That is an additional bug. You are right that sem_unlock does need
to call rcu_read_unlock() and that my patches fail to do so.
Good spotting.
--
All rights reversed
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 7/4] ipc: fine grained locking for semtimedop
On 03/06/2013 08:36 PM, Davidlohr Bueso wrote:
But my 8 socket 160 CPU box sure isn't happy. I'm getting all sorts of
issues (sometimes it will boot, sometimes it wont). When it does, linux
will hang as soon as I start my benchmarking:
I think I may have found the cause of this bug.
It looks like it would be possible for a caller with a single
semop to come into semtimedop, see that sma-complex_count is
elevated, and take the write lock.
Further down in semtimedop, that caller can call do_smart_update,
which calls update_queue, which in turn calls unlink_queue.
The function unlink_queue can then decrement sma-complex_count.
In other words, we correctly grab the write lock when changing
sma-complex_count. However, we do not remember that we took
the write lock, and call read_unlock when we leave semtimedop.
The next caller will get stuck.
To fix this bug, we need to remember whether or not we took
the lock in exclusive mode, and unlock it the right way.
Now that I have figured this bug out, it should be easy to
re-implement things the way Linus suggested, subject to the
same bug fixes :)
BUG: soft lockup - CPU#77 stuck for 23s! [oracle:129877]
Modules linked in: fuse autofs4 sunrpc pcc_cpufreq ipv6 dm_mirror
dm_region_hash dm_log dm_mod uinput iTCO_wdt iTCO_vendor_support sg freq_table
mperf coretemp kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr
lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core netxen_nic ext4 mbcache jbd2
sd_mod crc_t10dif aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul
hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
CPU 77
Pid: 129877, comm: oracle Tainted: G D W3.9.0-rc1+ #20 HP ProLiant
DL980 G7
RIP: 0010:[812777fa] [812777fa] __read_lock_failed+0xa/0x20
RSP: 0018:8b87b8cf9ca8 EFLAGS: 0297
RAX: c900293c1020 RBX: 00010007a021 RCX: d3a5
RDX: 0001 RSI: 8b87b8cf9d58 RDI: c900293c1020
RBP: 8b87b8cf9ca8 R08: R09:
R10: R11: R12: 8b87b8cf9c68
R13: 8b87b8cf9c68 R14: 0286 R15: 8b87caf10100
FS: 7f7a689b2700() GS:8987ff9c() knlGS:
CS: 0010 DS: ES: CR0: 80050033
CR2: 7fc49426d000 CR3: 0187cf08f000 CR4: 07e0
DR0: DR1: DR2:
DR3: DR6: 0ff0 DR7: 0400
Process oracle (pid: 129877, threadinfo 8b87b8cf8000, task 8b87caf10100)
Stack:
8b87b8cf9cb8 8155f374 8b87b8cf9ce8 81205245
0001 00090002 7fff82d3aa08
8b87b8cf9f78 812069e1 00cbc000 8b87b8cf9f38
Call Trace:
[8155f374] _raw_read_lock+0x14/0x20
[81205245] sem_lock+0x85/0xa0
[812069e1] sys_semtimedop+0x521/0x7c0
[81089e2c] ? task_sched_runtime+0x4c/0x90
[8101c1b3] ? native_sched_clock+0x13/0x80
[8101b7b9] ? sched_clock+0x9/0x10
[8108f9ed] ? sched_clock_cpu+0xcd/0x110
[8108914b] ? update_rq_clock+0x2b/0x50
[81089e2c] ? task_sched_runtime+0x4c/0x90
[8108fe48] ? thread_group_cputime+0x88/0xc0
[8108fd1d] ? cputime_adjust+0x3d/0x90
[8108fece] ? thread_group_cputime_adjusted+0x4e/0x60
[81568119] system_call_fastpath+0x16/0x1b
Code: 90 55 48 89 e5 f0 ff 07 f3 90 83 3f 01 75 f9 f0 ff 0f 75 f1 5d c3 66 66 2e 0f
1f 84 00 00 00 00 00 55 48 89 e5 f0 48 ff 07 f3 90 48 83 3f 01 78 f8 f0 48 ff
0f 78 ee 5d c3 90 90 90 90 90 90 90
+static inline void sem_unlock(struct sem_array *sma, struct sembuf *sops,
+ int nsops)
+{
+ if (nsops == 1 sma-sem_nsems 1 !sma-complex_count) {
+ struct sem *sem = sma-sem_base + sops-sem_num;
+ spin_unlock(sem-lock);
+ read_unlock(sma-sem_perm.lock);
+ } else
+ write_unlock(sma-sem_perm.lock);
+}
I believe (haven't tested it yet) the issue could be in sem_unlock() as
we aren't unlocking the RCU read section - before it was just a trivial
ipc_unlock call.
That is an additional bug. You are right that sem_unlock does need
to call rcu_read_unlock() and that my patches fail to do so.
Good spotting.
--
All rights reversed
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 7/4] ipc: fine grained locking for semtimedop
On Wed, 2013-03-06 at 17:15 -0500, Rik van Riel wrote:
> Introduce finer grained locking for semtimedop, to handle the
> common case of a program wanting to manipulate one semaphore
> from an array with multiple semaphores.
>
> Each semaphore array has a read/write lock. If something
> complex is going on (manipulation of the array, of multiple
> semaphores in one syscall, etc), the lock is taken in exclusive
> mode.
>
> If the call is a semop manipulating just one semaphore in
> an array with multiple semaphores, the read/write lock for
> the semaphore array is taken in shared (read) mode, and the
> individual semaphore's lock is taken.
>
> On a 24 CPU system, performance numbers with the semop-multi
> test with N threads and N semaphores, look like this:
>
> vanilla Davidlohr's Davidlohr's +
> threads patches rwlock patches
> 10610652 726325 1783589
> 20341570 365699 1520453
> 30288102 307037 1498167
> 40290714 305955 1612665
> 50288620 312890 1733453
> 60289987 306043 1649360
> 70291298 306347 1723167
> 80290948 305662 1729545
> 90290996 306680 1736021
> 100 292243 306700 1773700
Lovely numbers :)
On my laptop:
cpus 4, threads: 256, semaphores: 128, test duration: 30 secs
total operations: 281430894, ops/sec 9381029
+ 20.87%a.out [kernel.kallsyms] [k] sys_semtimedop
+ 8.31%a.out [kernel.kallsyms] [k] ipc_has_perm.isra.21
+ 6.88%a.out [kernel.kallsyms] [k] _raw_read_lock
+ 6.78%a.out [kernel.kallsyms] [k] avc_has_perm_flags
+ 5.26%a.out [kernel.kallsyms] [k] ipcperms
+ 4.91%a.out [kernel.kallsyms] [k] ipc_obtain_object_check
+ 4.69%a.out [kernel.kallsyms] [k] __audit_syscall_exit
+ 4.21%a.out [kernel.kallsyms] [k]
copy_user_enhanced_fast_string
+ 3.61%a.out [kernel.kallsyms] [k] _raw_spin_lock
+ 3.55%a.out [kernel.kallsyms] [k] system_call
+ 3.35%a.out [kernel.kallsyms] [k] do_smart_update
+ 2.77%a.out [kernel.kallsyms] [k] __audit_syscall_entry
But my 8 socket 160 CPU box sure isn't happy. I'm getting all sorts of
issues (sometimes it will boot, sometimes it wont). When it does, linux
will hang as soon as I start my benchmarking:
BUG: soft lockup - CPU#77 stuck for 23s! [oracle:129877]
Modules linked in: fuse autofs4 sunrpc pcc_cpufreq ipv6 dm_mirror
dm_region_hash dm_log dm_mod uinput iTCO_wdt iTCO_vendor_support sg freq_table
mperf coretemp kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr
lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core netxen_nic ext4 mbcache jbd2
sd_mod crc_t10dif aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul
hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
CPU 77
Pid: 129877, comm: oracle Tainted: G D W3.9.0-rc1+ #20 HP ProLiant
DL980 G7
RIP: 0010:[] [] __read_lock_failed+0xa/0x20
RSP: 0018:8b87b8cf9ca8 EFLAGS: 0297
RAX: c900293c1020 RBX: 00010007a021 RCX: d3a5
RDX: 0001 RSI: 8b87b8cf9d58 RDI: c900293c1020
RBP: 8b87b8cf9ca8 R08: R09:
R10: R11: R12: 8b87b8cf9c68
R13: 8b87b8cf9c68 R14: 0286 R15: 8b87caf10100
FS: 7f7a689b2700() GS:8987ff9c() knlGS:
CS: 0010 DS: ES: CR0: 80050033
CR2: 7fc49426d000 CR3: 0187cf08f000 CR4: 07e0
DR0: DR1: DR2:
DR3: DR6: 0ff0 DR7: 0400
Process oracle (pid: 129877, threadinfo 8b87b8cf8000, task 8b87caf10100)
Stack:
8b87b8cf9cb8 8155f374 8b87b8cf9ce8 81205245
0001 00090002 7fff82d3aa08
8b87b8cf9f78 812069e1 00cbc000 8b87b8cf9f38
Call Trace:
[] _raw_read_lock+0x14/0x20
[] sem_lock+0x85/0xa0
[] sys_semtimedop+0x521/0x7c0
[] ? task_sched_runtime+0x4c/0x90
[] ? native_sched_clock+0x13/0x80
[] ? sched_clock+0x9/0x10
[] ? sched_clock_cpu+0xcd/0x110
[] ? update_rq_clock+0x2b/0x50
[] ? task_sched_runtime+0x4c/0x90
[] ? thread_group_cputime+0x88/0xc0
[] ? cputime_adjust+0x3d/0x90
[] ? thread_group_cputime_adjusted+0x4e/0x60
[] system_call_fastpath+0x16/0x1b
Code: 90 55 48 89 e5 f0 ff 07 f3 90 83 3f 01 75 f9 f0 ff 0f 75 f1 5d c3 66 66
2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 f0 48 ff 07 f3 90 <48> 83 3f 01 78 f8 f0
48 ff 0f 78 ee 5d c3 90 90 90 90 90 90 90
> +static inline void sem_unlock(struct sem_array *sma, struct sembuf *sops,
> + int nsops)
> +{
> + if (nsops == 1 && sma->sem_nsems > 1
Re: [PATCH v2 7/4] ipc: fine grained locking for semtimedop
On 03/06/2013 05:57 PM, Linus Torvalds wrote: On Wed, Mar 6, 2013 at 5:15 PM, Rik van Riel wrote: If the call is a semop manipulating just one semaphore in an array with multiple semaphores, the read/write lock for the semaphore array is taken in shared (read) mode, and the individual semaphore's lock is taken. You know, we do something like this already elsewhere, and I think we do it slightly better. See the mm_take_all_locks() logic in mm/mmap.c. That would work. If we are about to do one of the uncommon operations,or sma->complex_count is set, we need to take the outer lock and all of the inner locks. The only complication would be interactions with the non-semaphore code in ipc/util.c, which manipulates the kern_ipc_perm structures, which are part of the sem_array structure. That said, judging by your numbers, your read-write lock seems to work fine too, even though I'd worry about cacheline ping-pong (but not contention) on the readers. So it doesn't seem optimal, but it sure as hell seems better than what we do now ;) I can take a stab at implementing the take_all_locks approach tomorrow. If things turn out to be easier than I fear, I will send an updated patch. If the resulting changes to the rest of ipc/ turn out to be too ugly to live, the rwsem performance is likely to be good enough for a while, and I'll just send an email without a patch :) -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 7/4] ipc: fine grained locking for semtimedop
On Wed, Mar 6, 2013 at 5:15 PM, Rik van Riel wrote: > > If the call is a semop manipulating just one semaphore in > an array with multiple semaphores, the read/write lock for > the semaphore array is taken in shared (read) mode, and the > individual semaphore's lock is taken. You know, we do something like this already elsewhere, and I think we do it slightly better. See the mm_take_all_locks() logic in mm/mmap.c. The optimal strategy if you have many items, and the *common* case is that you need just one lock, is actually to only take that single lock for the common case. No top-level lock at all. Then, for the complex (but unlikely) case, you take a *separate* top-level lock, and then you take the lower-level locks one by one, while testing first if you already hold them (using a separate data structure that is protected by the top-level lock). This way, the common case only needs that single lock that protects just that particular data structure. That said, judging by your numbers, your read-write lock seems to work fine too, even though I'd worry about cacheline ping-pong (but not contention) on the readers. So it doesn't seem optimal, but it sure as hell seems better than what we do now ;) Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 7/4] ipc: fine grained locking for semtimedop
On Wed, Mar 6, 2013 at 5:15 PM, Rik van Riel r...@redhat.com wrote: If the call is a semop manipulating just one semaphore in an array with multiple semaphores, the read/write lock for the semaphore array is taken in shared (read) mode, and the individual semaphore's lock is taken. You know, we do something like this already elsewhere, and I think we do it slightly better. See the mm_take_all_locks() logic in mm/mmap.c. The optimal strategy if you have many items, and the *common* case is that you need just one lock, is actually to only take that single lock for the common case. No top-level lock at all. Then, for the complex (but unlikely) case, you take a *separate* top-level lock, and then you take the lower-level locks one by one, while testing first if you already hold them (using a separate data structure that is protected by the top-level lock). This way, the common case only needs that single lock that protects just that particular data structure. That said, judging by your numbers, your read-write lock seems to work fine too, even though I'd worry about cacheline ping-pong (but not contention) on the readers. So it doesn't seem optimal, but it sure as hell seems better than what we do now ;) Linus -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 7/4] ipc: fine grained locking for semtimedop
On 03/06/2013 05:57 PM, Linus Torvalds wrote: On Wed, Mar 6, 2013 at 5:15 PM, Rik van Riel r...@redhat.com wrote: If the call is a semop manipulating just one semaphore in an array with multiple semaphores, the read/write lock for the semaphore array is taken in shared (read) mode, and the individual semaphore's lock is taken. You know, we do something like this already elsewhere, and I think we do it slightly better. See the mm_take_all_locks() logic in mm/mmap.c. That would work. If we are about to do one of the uncommon operations,or sma-complex_count is set, we need to take the outer lock and all of the inner locks. The only complication would be interactions with the non-semaphore code in ipc/util.c, which manipulates the kern_ipc_perm structures, which are part of the sem_array structure. That said, judging by your numbers, your read-write lock seems to work fine too, even though I'd worry about cacheline ping-pong (but not contention) on the readers. So it doesn't seem optimal, but it sure as hell seems better than what we do now ;) I can take a stab at implementing the take_all_locks approach tomorrow. If things turn out to be easier than I fear, I will send an updated patch. If the resulting changes to the rest of ipc/ turn out to be too ugly to live, the rwsem performance is likely to be good enough for a while, and I'll just send an email without a patch :) -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 7/4] ipc: fine grained locking for semtimedop
On Wed, 2013-03-06 at 17:15 -0500, Rik van Riel wrote: Introduce finer grained locking for semtimedop, to handle the common case of a program wanting to manipulate one semaphore from an array with multiple semaphores. Each semaphore array has a read/write lock. If something complex is going on (manipulation of the array, of multiple semaphores in one syscall, etc), the lock is taken in exclusive mode. If the call is a semop manipulating just one semaphore in an array with multiple semaphores, the read/write lock for the semaphore array is taken in shared (read) mode, and the individual semaphore's lock is taken. On a 24 CPU system, performance numbers with the semop-multi test with N threads and N semaphores, look like this: vanilla Davidlohr's Davidlohr's + threads patches rwlock patches 10610652 726325 1783589 20341570 365699 1520453 30288102 307037 1498167 40290714 305955 1612665 50288620 312890 1733453 60289987 306043 1649360 70291298 306347 1723167 80290948 305662 1729545 90290996 306680 1736021 100 292243 306700 1773700 Lovely numbers :) On my laptop: cpus 4, threads: 256, semaphores: 128, test duration: 30 secs total operations: 281430894, ops/sec 9381029 + 20.87%a.out [kernel.kallsyms] [k] sys_semtimedop + 8.31%a.out [kernel.kallsyms] [k] ipc_has_perm.isra.21 + 6.88%a.out [kernel.kallsyms] [k] _raw_read_lock + 6.78%a.out [kernel.kallsyms] [k] avc_has_perm_flags + 5.26%a.out [kernel.kallsyms] [k] ipcperms + 4.91%a.out [kernel.kallsyms] [k] ipc_obtain_object_check + 4.69%a.out [kernel.kallsyms] [k] __audit_syscall_exit + 4.21%a.out [kernel.kallsyms] [k] copy_user_enhanced_fast_string + 3.61%a.out [kernel.kallsyms] [k] _raw_spin_lock + 3.55%a.out [kernel.kallsyms] [k] system_call + 3.35%a.out [kernel.kallsyms] [k] do_smart_update + 2.77%a.out [kernel.kallsyms] [k] __audit_syscall_entry But my 8 socket 160 CPU box sure isn't happy. I'm getting all sorts of issues (sometimes it will boot, sometimes it wont). When it does, linux will hang as soon as I start my benchmarking: BUG: soft lockup - CPU#77 stuck for 23s! [oracle:129877] Modules linked in: fuse autofs4 sunrpc pcc_cpufreq ipv6 dm_mirror dm_region_hash dm_log dm_mod uinput iTCO_wdt iTCO_vendor_support sg freq_table mperf coretemp kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core netxen_nic ext4 mbcache jbd2 sd_mod crc_t10dif aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core CPU 77 Pid: 129877, comm: oracle Tainted: G D W3.9.0-rc1+ #20 HP ProLiant DL980 G7 RIP: 0010:[812777fa] [812777fa] __read_lock_failed+0xa/0x20 RSP: 0018:8b87b8cf9ca8 EFLAGS: 0297 RAX: c900293c1020 RBX: 00010007a021 RCX: d3a5 RDX: 0001 RSI: 8b87b8cf9d58 RDI: c900293c1020 RBP: 8b87b8cf9ca8 R08: R09: R10: R11: R12: 8b87b8cf9c68 R13: 8b87b8cf9c68 R14: 0286 R15: 8b87caf10100 FS: 7f7a689b2700() GS:8987ff9c() knlGS: CS: 0010 DS: ES: CR0: 80050033 CR2: 7fc49426d000 CR3: 0187cf08f000 CR4: 07e0 DR0: DR1: DR2: DR3: DR6: 0ff0 DR7: 0400 Process oracle (pid: 129877, threadinfo 8b87b8cf8000, task 8b87caf10100) Stack: 8b87b8cf9cb8 8155f374 8b87b8cf9ce8 81205245 0001 00090002 7fff82d3aa08 8b87b8cf9f78 812069e1 00cbc000 8b87b8cf9f38 Call Trace: [8155f374] _raw_read_lock+0x14/0x20 [81205245] sem_lock+0x85/0xa0 [812069e1] sys_semtimedop+0x521/0x7c0 [81089e2c] ? task_sched_runtime+0x4c/0x90 [8101c1b3] ? native_sched_clock+0x13/0x80 [8101b7b9] ? sched_clock+0x9/0x10 [8108f9ed] ? sched_clock_cpu+0xcd/0x110 [8108914b] ? update_rq_clock+0x2b/0x50 [81089e2c] ? task_sched_runtime+0x4c/0x90 [8108fe48] ? thread_group_cputime+0x88/0xc0 [8108fd1d] ? cputime_adjust+0x3d/0x90 [8108fece] ? thread_group_cputime_adjusted+0x4e/0x60 [81568119] system_call_fastpath+0x16/0x1b Code: 90 55 48 89 e5 f0 ff 07 f3 90 83 3f 01 75 f9 f0 ff 0f 75 f1 5d c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 f0 48 ff 07 f3 90 48 83 3f 01 78 f8 f0 48 ff
