Attaching new debdiff for Jammy (ceph 17.2.9) based on feedback from
Seyeong.
** Changed in: ceph (Ubuntu Noble)
Status: Fix Released => In Progress
** Changed in: ceph (Ubuntu Noble)
Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel)
** Description changed:
[Impact]
The OSD crash in ScrubQueue::collect_ripe_jobs(), with the root cause and how
the patch fixes it.
[Test plan]
The race is between the OSD tick thread (OSD::sched_scrub ->
ScrubQueue::collect_ripe_jobs, which sorts the scrub jobs by scheduled_at under
jobs_lock) and PG worker threads (ScrubQueue::update_job, which
# rewrites scheduled_at without jobs_lock). Two sources of update_job() calls
are run in parallel to keep PG worker threads rewriting schedules all the time:
(a) toggling osd_scrub_min_interval -> OSD::resched_all_scrubs() ->
PG::reschedule_scrub() for every primary PG on every OSD
(b) operator-requested scrubs ("ceph osd scrub/deep-scrub N") ->
PgScrubber::scrub_requested() -> update_job() for every primary PG
This is hard to reproduce reliably in a typical cluster. But the
following is a reasonable test which loads the cluster with scrub
requests and can reproduce it.
1. Set up a ceph cluster
2. Use the following script to hammer it with scrubs:
```
#!/bin/bash
DURATION=${1:-7200}
END=$(( $(date +%s) + DURATION ))
echo "$(date -Is) stress start, duration ${DURATION}s, ceph-osd $(ceph-osd
--version)"
( while :; do
ceph config set osd osd_scrub_min_interval 86400
ceph config set osd osd_scrub_min_interval 86401
done ) &
A=$!
( while true; do
for i in 0 1 2; do ceph osd scrub "$i" >/dev/null 2>&1; ceph osd
deep-scrub "$i" >/dev/null 2>&1; done
done ) &
B=$!
trap 'kill $A $B 2>/dev/null; ceph config rm osd osd_scrub_min_interval' EXIT
while [ "$(date +%s)" -lt "$END" ]; do
sleep 10
n=$(ceph crash ls -f json 2>/dev/null | jq length)
if [ "${n:-0}" -gt 0 ]; then
echo "$(date -Is) CRASH detected after $(( DURATION - END + $(date
+%s) ))s:"
ceph crash ls
exit 1
fi
done
echo "$(date -Is) no crash during ${DURATION}s"
```
Then confirm one of the OSDs crash (the script stops at first crash).
3. Install the package with the fix and confirm the script doesn't exit
(i.e. no crash). The script typically crashed within a few mins, so if
there's no crash for about 15mins, that's a reasonable conclude the
test.
[Where problems could occur]
Deadlock could occur. `jobs_lock` is a non-recursive mutex, so a path that
already holds it and calls the public update_job() would hang. I checked the
call sites and none of them does.
PG worker threads now briefly contend on `jobs_lock`, so there could be a
performance penalty.
A regression could show up as scrubs not being scheduled.
[Other Info]
- This SRU is based on https://github.com/ceph/ceph/pull/58173 as well as the
latest code in main branch.
+ This SRU is based on https://github.com/ceph/ceph/pull/5817,
https://github.com/ceph/ceph/commit/b8003f6d746dd13060c65ee27e8d610097404ccc,
and the latest code in main branch.
Changes in https://github.com/ceph/ceph/pull/58173 refactor the
ownership model so that the scheduler queue no longer shares the same
ScrubJob object with the PG scrubber. Consequently, the queue’s schedule
fields and the PG-side schedule fields are separate, removing this
particular data race by the re-architecture.
+ Noble specific backport by upstream:
+ https://github.com/ceph/ceph/commit/b8003f6d746dd13060c65ee27e8d610097404ccc
+
I've also built a PPA to test:
https://launchpad.net/~pponnuvel/+archive/ubuntu/data-race-17.2.9
- This has been fixed in Reef onwards, so fixing only for Quincy.
+ This has been fixed in Tentacle onwards, so fixing only for Quincy and
+ Squid (Reef is EOL both upstream and in UCA).
=====
Original description:
ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs().
The crash is caused by a data race between the timer thread reading
ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG thread writing to it
via ScrubJob::update_schedule() (without jobs_lock). ceph-osd processes crash
intermittently in ScrubQueue::collect_ripe_jobs().
The crash appeears to be caused by a data race between the timer thread
reading ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG
thread writing to it via ScrubJob::update_schedule() (without
jobs_lock).
# ceph crash info
2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016
{
"backtrace": [
"/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x76e77e242520]",
"/usr/bin/ceph-osd(+0x8d198f) [0x5577b75b998f]",
"/usr/bin/ceph-osd(+0x8d1a23) [0x5577b75b9a23]",
"(ScrubQueue::collect_ripe_jobs(std::vector<boost::intrusive_ptr<ScrubQueue::ScrubJob>,
std::allocator<boost::intrusive_ptr<ScrubQueue::ScrubJob> > >&,
utime_t)+0x1db) [0x5577b75bce9b]",
"(ScrubQueue::select_pg_and_scrub(Scrub::ScrubPreconds&)+0x104)
[0x5577b75c2914]",
"(OSD::sched_scrub()+0x1f5) [0x5577b7382ca5]",
"(OSD::tick_without_osd_lock()+0x498) [0x5577b73bc2f8]",
"(Context::complete(int)+0xd) [0x5577b73cba2d]",
"(CommonSafeTimer<std::mutex>::timer_thread()+0x10c) [0x5577b7a6ddcc]",
"(CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x5577b7a6ede1]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x76e77e294ac3]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x76e77e3268c0]"
],
"ceph_version": "17.2.9",
"crash_id":
"2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016",
"entity_name": "osd.197",
"os_id": "22.04",
"os_name": "Ubuntu 22.04.5 LTS",
"os_version": "22.04.5 LTS (Jammy Jellyfish)",
"os_version_id": "22.04",
"process_name": "ceph-osd",
"stack_sig":
"8e59dd0b473e52714e12c295fb445f22d04e2f2239c1dabae9f65f7ebd0d8ee1",
"timestamp": "2026-01-24T19:55:40.134588Z",
"utsname_hostname": "dc1cpbaudosd010",
"utsname_machine": "x86_64",
"utsname_release": "6.8.0-85-generic",
"utsname_sysname": "Linux",
"utsname_version": "#85~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19
16:18:59 UTC 2"
}
The crash is intermittent and very likely to happen when several PGs are
rescheduled simultaneously.
** Also affects: cloud-archive/caracal
Importance: Undecided
Status: New
** Changed in: cloud-archive
Status: In Progress => Triaged
** Changed in: cloud-archive/caracal
Status: New => Triaged
** Changed in: cloud-archive/yoga
Status: In Progress => Triaged
** Changed in: ceph (Ubuntu Jammy)
Status: In Progress => Triaged
** Changed in: ceph (Ubuntu Noble)
Status: In Progress => Triaged
** Changed in: cloud-archive/caracal
Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel)
** Patch added: "ceph_17.2.9-0ubuntu0.22.04.5.debdiff"
https://bugs.launchpad.net/cloud-archive/+bug/2161211/+attachment/6010678/+files/ceph_17.2.9-0ubuntu0.22.04.5.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2161211
Title:
ceph-osd crashes due to a data race
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2161211/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
