Hi Benjamin,
Great to hear that the early results look promising.
Yes, let's give it a week, so we can ensure that these patches do their
job under the high loads the git server faces each day over the week.
If things still look good early next week, I will go ahead and prepare
the patches for
The testing of the new kernel looks very promising. We don't observe any
of the latency/IOwait we had before even with the btree_shrinker
enabled.
We'll give it a week probably, but having a backport of those patches
would be fantastic for sure.
--
You received this bug notification because you
Thanks Matthew, we'll try this kernel tomorrow.
I tested it on an openstack instance, the only downside is that it
uninstalls the official 4.15.0-118-generic one.
Regards,
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
Hi Benjamin,
I think Dan is onto something.
The following commit was merged in upstream 5.5-rc1, but was backported
to 4.15.0-87-generic through upstream stable:
commit 9fcc34b1a6dd4b8e5337e2b6ef45e428897eca6b
Author: Coly Li
Date: Wed Nov 13 16:03:24 2019 +0800
Subject: bcache: at least try
Hi Mauricio,
The tests in [5][6] and [7] have been done with a 44GB memory VM.
This VM usually has 64GB of allocated memory.
The goal was to verify that the load of the whole btree in RAM was not
prevented (like in mca-reap -> down_write_trylock).
All this kernel stuff is rather new to me, so I
** Description changed:
Hello,
In short, we faced an issue with a huge IO wait on a bionic Ubuntu
4.15.0-118.119-generic kernel.
This is the full list of process and the kernel function they were stuck in
[0].
The main issue can probably be summarized by this perf reports
*
** Description changed:
Hello,
In short, we faced an issue with a huge IO wait on a bionic Ubuntu
4.15.0-118.119-generic kernel.
This is the full list of process and the kernel function they were stuck in
[0].
The main issue can probably be summarized by this perf reports
*
Hi Benjamin,
It's interesting that the memory shrinkers are being invoked without memory
pressure.
I'll try to understand it better, maybe I'm missing something.
I guess the before/after memory sizes for the test are swapped?
i.e., to alleviate potential memory pressure the VM's memory had
been
Hello Mauricio,
That was also one of the conclusion we reached yesterday.
I followed a wrong lead yesterday, after getting another more detailed
output with klockstat.
I'll update the description of the bug at the end, adding the bits I
found.
The system is not really having memory pressure
** Description changed:
Hello,
In short, we faced an issue with a huge IO wait on a bionic Ubuntu
4.15.0-118.119-generic kernel.
This is the full list of process and the kernel function they were stuck in
[0].
The main issue can probably be summarized by this perf reports
*
Hi Benjamin,
Thanks for the detailed report.
Does this system show signs of memory pressure?
bch_mca_scan() is part of bcache's memory shrinker, and thus should be
called when the system is trying to release memory from its several
caches.
Also, the bucket lock usage is widespread in bcache
Here is a trace of the bcache event related lock in the kernel obtained
with some bpfcc-tools.
klockstat-bpfcc -c bch_ -i 5 -s 3
The trace has been run in parallel with the following command line
echo "Shrinker disabled: $(date)"; sleep 60; echo "Enabling shrinker:
$(date)"; echo 0 | sudo tee
Actually we cannot be sure no. The server didn't have any metrics prior
to few days ago and the issue was already there.
It's worth nothing that few servers have this bcache configuration,
because the cache mode is configured as writethrough and the load is
pretty significant.
So no last "good"
** Also affects: linux (Ubuntu Bionic)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1898786
Title:
Issue with bcache bch_mca_scan causing
correction: from *a prior kernel version
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1898786
Title:
Issue with bcache bch_mca_scan causing huge IO wait
Status in linux package in
@Benjamin Allot is this a performance regression from the prior kernel
version? If so, can you confirm the last "good" version? Perhaps we can
do a bisect
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
** Description changed:
Hello,
In short, we faced an issue with a huge IO wait on a bionic Ubuntu
4.15.0-118.119-generic kernel.
- This is the full list of process and the kernel function they were stuck in
[2].
+ This is the full list of process and the kernel function they were stuck
17 matches
Mail list logo
