Attaching the debdiff for jammy.
** Description changed:
+ [Impact]
+ The OSD crash in ScrubQueue::collect_ripe_jobs(), with the root cause and how
the patch fixes it.
+
+ [Test plan]
+
+ The race is between the OSD tick thread (OSD::sched_scrub ->
ScrubQueue::collect_ripe_jobs, which sorts the scrub jobs by scheduled_at under
jobs_lock) and PG worker threads (ScrubQueue::update_job, which
+ # rewrites scheduled_at without jobs_lock). Two sources of update_job() calls
are run in parallel to keep PG worker threads rewriting schedules all the time:
+ (a) toggling osd_scrub_min_interval -> OSD::resched_all_scrubs() ->
PG::reschedule_scrub() for every primary PG on every OSD
+ (b) operator-requested scrubs ("ceph osd scrub/deep-scrub N") ->
PgScrubber::scrub_requested() -> update_job() for every primary PG
+
+ This is hard to reproduce reliably in a typical cluster. But the
+ following is a reasonable test which loads the cluster with scrub
+ requests and can reproduce it.
+
+ Setup a ceph cluster and use the following script to hammer it with scrubs:
+ ```
+ #!/bin/bash
+
+ DURATION=${1:-7200}
+ END=$(( $(date +%s) + DURATION ))
+ echo "$(date -Is) stress start, duration ${DURATION}s, ceph-osd $(ceph-osd
--version)"
+
+ ( while :; do
+ ceph config set osd osd_scrub_min_interval 86400
+ ceph config set osd osd_scrub_min_interval 86401
+ done ) &
+ A=$!
+ ( while true; do
+ for i in 0 1 2; do ceph osd scrub "$i" >/dev/null 2>&1; ceph osd
deep-scrub "$i" >/dev/null 2>&1; done
+ done ) &
+ B=$!
+ trap 'kill $A $B 2>/dev/null; ceph config rm osd osd_scrub_min_interval' EXIT
+
+ while [ "$(date +%s)" -lt "$END" ]; do
+ sleep 10
+ n=$(ceph crash ls -f json 2>/dev/null | jq length)
+ if [ "${n:-0}" -gt 0 ]; then
+ echo "$(date -Is) CRASH detected after $(( DURATION - END + $(date
+%s) ))s:"
+ ceph crash ls
+ exit 1
+ fi
+ done
+ echo "$(date -Is) no crash during ${DURATION}s"
+ ```
+
+ [Where problems could occur]
+ Deadlock could occur. `jobs_lock` is a non-recursive mutex, so a path that
already holds it and calls the public update_job() would hang. I checked the
call sites and none of them does.
+ PG worker threads now briefly contend on `jobs_lock`, so there could be a
performance penalty.
+
+ A regression could show up as scrubs not being scheduled.
+
+ [Other Info]
+ This patch is based on https://github.com/ceph/ceph/pull/58173 as well as the
latest code in main branch.
+
+ Changes in https://github.com/ceph/ceph/pull/58173 refactor the
+ ownership model so that the scheduler queue no longer shares the same
+ ScrubJob object with the PG scrubber. Consequently, the queue’s schedule
+ fields and the PG-side schedule fields are separate, removing this
+ particular data race by the re-architecture.
+
+ I've also built a PPA to test:
+ https://launchpad.net/~pponnuvel/+archive/ubuntu/data-race-17.2.9
+
+ =====
+
+ Original description:
+
ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs().
The crash is caused by a data race between the timer thread reading
ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG thread writing to it
via ScrubJob::update_schedule() (without jobs_lock). ceph-osd processes crash
intermittently in ScrubQueue::collect_ripe_jobs().
The crash appeears to be caused by a data race between the timer thread
reading ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG
thread writing to it via ScrubJob::update_schedule() (without
jobs_lock).
# ceph crash info
2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016
{
"backtrace": [
"/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x76e77e242520]",
"/usr/bin/ceph-osd(+0x8d198f) [0x5577b75b998f]",
"/usr/bin/ceph-osd(+0x8d1a23) [0x5577b75b9a23]",
"(ScrubQueue::collect_ripe_jobs(std::vector<boost::intrusive_ptr<ScrubQueue::ScrubJob>,
std::allocator<boost::intrusive_ptr<ScrubQueue::ScrubJob> > >&,
utime_t)+0x1db) [0x5577b75bce9b]",
"(ScrubQueue::select_pg_and_scrub(Scrub::ScrubPreconds&)+0x104)
[0x5577b75c2914]",
"(OSD::sched_scrub()+0x1f5) [0x5577b7382ca5]",
"(OSD::tick_without_osd_lock()+0x498) [0x5577b73bc2f8]",
"(Context::complete(int)+0xd) [0x5577b73cba2d]",
"(CommonSafeTimer<std::mutex>::timer_thread()+0x10c) [0x5577b7a6ddcc]",
"(CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x5577b7a6ede1]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x76e77e294ac3]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x76e77e3268c0]"
],
"ceph_version": "17.2.9",
"crash_id":
"2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016",
"entity_name": "osd.197",
"os_id": "22.04",
"os_name": "Ubuntu 22.04.5 LTS",
"os_version": "22.04.5 LTS (Jammy Jellyfish)",
"os_version_id": "22.04",
"process_name": "ceph-osd",
"stack_sig":
"8e59dd0b473e52714e12c295fb445f22d04e2f2239c1dabae9f65f7ebd0d8ee1",
"timestamp": "2026-01-24T19:55:40.134588Z",
"utsname_hostname": "dc1cpbaudosd010",
"utsname_machine": "x86_64",
"utsname_release": "6.8.0-85-generic",
"utsname_sysname": "Linux",
"utsname_version": "#85~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19
16:18:59 UTC 2"
}
The crash is intermittent and very likely to happen when several PGs are
rescheduled simultaneously.
** Patch added: "lp2161211-scrub-job-schedule-race.patch"
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2161211/+attachment/6010342/+files/lp2161211-scrub-job-schedule-race.patch
** Also affects: ceph (Ubuntu Jammy)
Importance: Undecided
Status: New
** Description changed:
[Impact]
The OSD crash in ScrubQueue::collect_ripe_jobs(), with the root cause and how
the patch fixes it.
[Test plan]
The race is between the OSD tick thread (OSD::sched_scrub ->
ScrubQueue::collect_ripe_jobs, which sorts the scrub jobs by scheduled_at under
jobs_lock) and PG worker threads (ScrubQueue::update_job, which
# rewrites scheduled_at without jobs_lock). Two sources of update_job() calls
are run in parallel to keep PG worker threads rewriting schedules all the time:
- (a) toggling osd_scrub_min_interval -> OSD::resched_all_scrubs() ->
PG::reschedule_scrub() for every primary PG on every OSD
- (b) operator-requested scrubs ("ceph osd scrub/deep-scrub N") ->
PgScrubber::scrub_requested() -> update_job() for every primary PG
+ (a) toggling osd_scrub_min_interval -> OSD::resched_all_scrubs() ->
PG::reschedule_scrub() for every primary PG on every OSD
+ (b) operator-requested scrubs ("ceph osd scrub/deep-scrub N") ->
PgScrubber::scrub_requested() -> update_job() for every primary PG
This is hard to reproduce reliably in a typical cluster. But the
following is a reasonable test which loads the cluster with scrub
requests and can reproduce it.
Setup a ceph cluster and use the following script to hammer it with scrubs:
```
#!/bin/bash
DURATION=${1:-7200}
END=$(( $(date +%s) + DURATION ))
echo "$(date -Is) stress start, duration ${DURATION}s, ceph-osd $(ceph-osd
--version)"
( while :; do
- ceph config set osd osd_scrub_min_interval 86400
- ceph config set osd osd_scrub_min_interval 86401
- done ) &
+ ceph config set osd osd_scrub_min_interval 86400
+ ceph config set osd osd_scrub_min_interval 86401
+ done ) &
A=$!
( while true; do
- for i in 0 1 2; do ceph osd scrub "$i" >/dev/null 2>&1; ceph osd
deep-scrub "$i" >/dev/null 2>&1; done
- done ) &
+ for i in 0 1 2; do ceph osd scrub "$i" >/dev/null 2>&1; ceph osd
deep-scrub "$i" >/dev/null 2>&1; done
+ done ) &
B=$!
trap 'kill $A $B 2>/dev/null; ceph config rm osd osd_scrub_min_interval' EXIT
while [ "$(date +%s)" -lt "$END" ]; do
- sleep 10
- n=$(ceph crash ls -f json 2>/dev/null | jq length)
- if [ "${n:-0}" -gt 0 ]; then
- echo "$(date -Is) CRASH detected after $(( DURATION - END + $(date
+%s) ))s:"
- ceph crash ls
- exit 1
- fi
+ sleep 10
+ n=$(ceph crash ls -f json 2>/dev/null | jq length)
+ if [ "${n:-0}" -gt 0 ]; then
+ echo "$(date -Is) CRASH detected after $(( DURATION - END + $(date
+%s) ))s:"
+ ceph crash ls
+ exit 1
+ fi
done
echo "$(date -Is) no crash during ${DURATION}s"
```
[Where problems could occur]
Deadlock could occur. `jobs_lock` is a non-recursive mutex, so a path that
already holds it and calls the public update_job() would hang. I checked the
call sites and none of them does.
PG worker threads now briefly contend on `jobs_lock`, so there could be a
performance penalty.
A regression could show up as scrubs not being scheduled.
[Other Info]
This patch is based on https://github.com/ceph/ceph/pull/58173 as well as the
latest code in main branch.
Changes in https://github.com/ceph/ceph/pull/58173 refactor the
ownership model so that the scheduler queue no longer shares the same
ScrubJob object with the PG scrubber. Consequently, the queue’s schedule
fields and the PG-side schedule fields are separate, removing this
particular data race by the re-architecture.
I've also built a PPA to test:
https://launchpad.net/~pponnuvel/+archive/ubuntu/data-race-17.2.9
+
+ This has been fixed in Reef onwards, so fixing only for Quincy.
=====
Original description:
ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs().
The crash is caused by a data race between the timer thread reading
ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG thread writing to it
via ScrubJob::update_schedule() (without jobs_lock). ceph-osd processes crash
intermittently in ScrubQueue::collect_ripe_jobs().
The crash appeears to be caused by a data race between the timer thread
reading ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG
thread writing to it via ScrubJob::update_schedule() (without
jobs_lock).
# ceph crash info
2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016
{
"backtrace": [
"/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x76e77e242520]",
"/usr/bin/ceph-osd(+0x8d198f) [0x5577b75b998f]",
"/usr/bin/ceph-osd(+0x8d1a23) [0x5577b75b9a23]",
"(ScrubQueue::collect_ripe_jobs(std::vector<boost::intrusive_ptr<ScrubQueue::ScrubJob>,
std::allocator<boost::intrusive_ptr<ScrubQueue::ScrubJob> > >&,
utime_t)+0x1db) [0x5577b75bce9b]",
"(ScrubQueue::select_pg_and_scrub(Scrub::ScrubPreconds&)+0x104)
[0x5577b75c2914]",
"(OSD::sched_scrub()+0x1f5) [0x5577b7382ca5]",
"(OSD::tick_without_osd_lock()+0x498) [0x5577b73bc2f8]",
"(Context::complete(int)+0xd) [0x5577b73cba2d]",
"(CommonSafeTimer<std::mutex>::timer_thread()+0x10c) [0x5577b7a6ddcc]",
"(CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x5577b7a6ede1]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x76e77e294ac3]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x76e77e3268c0]"
],
"ceph_version": "17.2.9",
"crash_id":
"2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016",
"entity_name": "osd.197",
"os_id": "22.04",
"os_name": "Ubuntu 22.04.5 LTS",
"os_version": "22.04.5 LTS (Jammy Jellyfish)",
"os_version_id": "22.04",
"process_name": "ceph-osd",
"stack_sig":
"8e59dd0b473e52714e12c295fb445f22d04e2f2239c1dabae9f65f7ebd0d8ee1",
"timestamp": "2026-01-24T19:55:40.134588Z",
"utsname_hostname": "dc1cpbaudosd010",
"utsname_machine": "x86_64",
"utsname_release": "6.8.0-85-generic",
"utsname_sysname": "Linux",
"utsname_version": "#85~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19
16:18:59 UTC 2"
}
The crash is intermittent and very likely to happen when several PGs are
rescheduled simultaneously.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2161211
Title:
ceph-osd crashes due to a data race
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2161211/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
