Attached are debdiffs for the SRU: - boost1.90, resolute: boost1.90_1.90.0-6ubuntu1.1.debdiff - boost1.90, stonking: boost1.90_1.90.0-6.1ubuntu1.1.debdiff - boost1.88, resolute: boost1.88_1.88.0-1.4ubuntu4.1.debdiff
All three add the upstream fix as d/p/context-manage-exception-state- noinline.patch, at the end of the series, with a changelog entry, and run update-maintainer. All patches apply cleanly with quilt, and the unpacked sources contain the patched header. boost1.88 is affected as well: manage_exception_state was introduced in Boost 1.88 without BOOST_NOINLINE. 1.83 and 1.87 do not have it, and 1.92 has the fix. In resolute the only user of libboost-context1.88.0 outside Boost itself is icinga2 2.15.1-1, which is built against boost1.88. The attached reproducer, built purely against the boost1.88 sources, fails the same way: 2.7M of 11M catch blocks without a current exception, and "terminate called without an active exception" on rethrow. With the patch it reports 0. A small objdump-based check also finds 27 affected fiber switches in icinga2-bin 2.15.1-1 (amd64); I can attach it if that helps. I have added a boost1.88 task for this. The development release is already fixed: tenacious uses Boost 1.92 (boost-defaults 1.92.0.1ubuntu2), and nothing in tenacious depends on libboost-context1.90.0 any more apart from boost1.90's own packages. So I don't think an upload of boost1.90 to tenacious is needed. If you want one anyway, I can attach that debdiff too (1.90.0-6.1ubuntu2). Since the bug is in a header, the affected packages need no-change rebuilds after the fix: - resolute: ceph (radosgw, librgw2, ceph-common), pdns-recursor, nix, simgrid, graph-tool, wesnoth-1.18 and yuzu, all on boost1.90, and icinga2, on boost1.88 - stonking: the same set on boost1.90, including icinga2 Setting the status back to New. ** Patch added: "boost1.90_1.90.0-6.1ubuntu1.1.debdiff" https://bugs.launchpad.net/ubuntu/+source/boost1.90/+bug/2173538/+attachment/6017072/+files/boost1.90_1.90.0-6.1ubuntu1.1.debdiff ** Changed in: boost1.90 (Ubuntu) Status: Incomplete => New ** Description changed: [ Impact ] - Since Boost 1.90, boost::context fibers can overwrite the libstdc++ - exception state (__cxa_get_globals) of another thread when a fiber - resumes on a different thread. manage_exception_state in - boost/context/fiber_fcontext.hpp lost its BOOST_NOINLINE, so gcc caches - the TLS address across the fiber switch. + Boost.Context 1.88 to 1.91 save and restore the libstdc++ exception + state (__cxa_eh_globals) around each fiber switch, in + manage_exception_state in boost/context/fiber_fcontext.hpp. + __cxa_get_globals() is declared __attribute__((const)). Once the + constructor and destructor are inlined, gcc computes the TLS address + once before the switch and reuses it afterwards. - User-visible result: `throw;` inside a catch block in a - boost::asio::spawn coroutine aborts with "terminate called without an - active exception". With Icinga 2 on 26.04, about 1 in 240 cancelled - connections crashed an I/O thread; with the patched header, 0 in 12,675. - radosgw (ceph) is affected in the archive: it runs asio::spawn - coroutines on strands over a pool of rgw_thread_pool_size (default 512) - threads, and its amd64 binaries contain the broken code (see Other Info). + If the fiber is resumed on a different thread, it writes the saved state + into the thread it came from, while that thread is running other code. A + thread inside a catch block then loses its current exception: throw; + calls std::terminate() ("terminate called without an active exception") + and std::current_exception() returns null. This is easy to hit with + boost::asio::spawn on an io_context run by several threads. - This is a regression from noble: Boost 1.83 has no - manage_exception_state, so 24.04 is not affected. + Upstream issue: https://github.com/boostorg/context/issues/323. It is + fixed in Boost 1.92 by commit 0921b9fd5c + (https://github.com/boostorg/context/pull/331), which re-adds + BOOST_NOINLINE to both functions. - The fix is the upstream commit 0921b9fd5c - (https://github.com/boostorg/context/pull/331), in Boost 1.92. It - restores BOOST_NOINLINE. The attached - context-manage-exception-state-noinline.patch is that commit as a quilt - patch for the package layout and applies cleanly on 1.90.0-6ubuntu1. + Affected source packages: + + - resolute: boost1.90 (1.90.0-6ubuntu1) and boost1.88 (1.88.0-1.4ubuntu4) + - stonking: boost1.90 (1.90.0-6.1ubuntu1) + + The development release is not affected: tenacious defaults to Boost + 1.92, and nothing there uses libboost-context1.90.0 apart from + boost1.90's own packages. + + Real-world impact: Icinga 2 on 26.04 (the packages.icinga.com build, + against boost1.90). On a production monitoring master, 17 of 4,026 + cancelled outgoing connections crashed an I/O thread within 21.6 hours. + After rebuilding the same sources against the patched header there were + 0 of 12,675. Ubuntu's own icinga2 is affected too: 2.15.1-1 in resolute + against boost1.88, and 2.16.5-1 in stonking against boost1.90. + + The bug is in a header, so it is compiled into the binaries that use it. + Those need no-change rebuilds after the update to pick up the fix: + + - resolute: ceph (radosgw, librgw2, ceph-common), pdns-recursor, nix, simgrid, graph-tool, wesnoth-1.18 and yuzu, all on boost1.90, and icinga2, on boost1.88 + - stonking: the same set on boost1.90, including icinga2 [ Test Plan ] - 1. On resolute (amd64 and arm64), install libboost-context1.90-dev and - build the attached reproducer: - g++ -O2 -std=c++17 -pthread -o repro repro.cpp -lboost_context - 2. Run: ./repro rethrow 10 - - Without the fix: aborts within milliseconds with - "terminate called without an active exception". - - With the fix: runs for 10 seconds and exits 0. - 3. Optionally run: ./repro check 10 - - Without the fix: reports a large number of catch blocks without - a current exception and exits 1. - - With the fix: reports 0 and exits 0. - 4. Run the boost1.90 autopkgtests and check for regressions. + Build the attached repro.cpp against the installed libboost-context-dev + and run it: + + g++ -O2 -std=c++17 -pthread -o repro repro.cpp -lboost_context + ./repro check 10 + ./repro rethrow 10 + + With the current packages, check reports a large number of catch blocks + without a current exception (millions within 10 s) and exits 1. rethrow + aborts within milliseconds with "terminate called without an active + exception" (exit 134). + + With the fixed packages, rebuild repro after upgrading. check then + reports 0 and exits 0, and rethrow runs the full 10 s and exits 0. + + Do this for boost1.90 in resolute and stonking. For boost1.88 in + resolute, build against libboost1.88-dev / libboost-context1.88-dev + instead. + + Results so far, using the patched header via an include overlay: + + boost1.90, amd64 2 vCPU, check 10 2 8 unpatched 1845846 of 15032671 bad patched 0 + boost1.90, arm64, check 10 unpatched 3739253 of 12523609 bad patched 0 + boost1.88 sources, arm64, check 10 unpatched 2733288 of 11038347 bad patched 0 + rethrow (all of the above) unpatched terminate patched 10 s, exit 0 + + For rebuilt reverse dependencies, we can provide a small objdump-based + check on request. It counts fiber switches that write through a cached + eh_globals address: currently 27 for icinga2-bin 2.15.1-1 in resolute, + and 0 is expected after the rebuild. [ Where problems could occur ] - The patch only adds a noinline attribute to one header-only function, - so it does not change ABI. The worst case is a small slowdown on each - fiber switch, because the function is no longer inlined. + The patch adds BOOST_NOINLINE to the constructor and destructor of + boost::context::detail::manage_exception_state. Each of them copies 16 + bytes. Nothing else changes, and there is no change to ABI, API or the + shared libraries. - The fix sits in a header, so already-built packages stay affected until - they are rebuilt. In resolute, only ceph needs a no-change rebuild - (radosgw, librgw2, ceph-common). That rebuild could show unrelated - FTBFS or pick up other changes in ceph's build dependencies. + A fiber switch costs two extra calls, which is negligible next to the + switch itself. A regression would show up in programs using + Boost.Context fibers (asio::spawn, Coroutine2, Fiber), as unexpected + exception behaviour around a switch. The identical change is part of + Boost 1.92.0. - The other resolute packages that depend on libboost-context1.90.0 do - not need a rebuild: - - pdns-recursor, simgrid, yuzu: call jump_fcontext/make_fcontext - directly and never use fiber_fcontext.hpp. - - wesnoth-1.18 (server): asio::spawn, but io_context only runs on the - main thread. - - nix: coroutine2 sources/sinks are deliberately kept on the calling - thread (see the comment in src/libstore/filetransfer.cc). - - graph-tool: coroutine2 generators exposed to Python; only affected - if one is resumed from another thread while the first thread is - inside a C++ catch block, which the GIL makes impractical. - - libboost-fiber1.90.0 is built from boost1.90 itself and gets fixed - by this upload. - Third-party binaries built against the 1.90 headers (e.g. Icinga 2 from - packages.icinga.com) need to be rebuilt by their vendors. + Binaries built against the old header stay affected until they are + rebuilt. The update does not make them worse. [ Other Info ] - Upstream: https://github.com/boostorg/context/issues/323 - - Affected versions (resolute, amd64 and arm64): - libboost-context1.90-dev / libboost1.90-dev 1.90.0-6ubuntu1 - g++ 15.2.0-16ubuntu1 - Icinga 2 was installed from packages.icinga.com. - - Affected binaries in ceph 20.2.0-0ubuntu2.1 (amd64): radosgw, - radosgw-admin, radosgw-object-expirer, librgw.so.2, denc-mod-rgw.so and - libceph_crypto_qat.so. In radosgw, - boost::asio::detail::spawned_fiber_thread::resume() keeps the address - returned by __cxa_get_globals() in a register across jump_fcontext and - writes the saved state back through it: - - 7e91b1: call __cxa_get_globals@plt - 7e91c3: mov (%rax),%r14 - 7e91c6: mov 0x8(%rax),%r13d - 7e91ca: mov %rax,%r12 - 7e91cd: call boost::context::detail::jump_fcontext@plt - 7e91d2: mov %r14,(%r12) <- stale TLS address after the switch - 7e91da: mov %r13d,0x8(%r12) - - spawned_fiber_thread::suspend_with() at 0x7f114d has the same sequence. + - Upstream: https://github.com/boostorg/context/issues/323, fix https://github.com/boostorg/context/pull/331 (0921b9fd5c). The earlier attempt, #324, was reverted. + - Boost 1.83 and 1.87 do not have manage_exception_state and are not affected. 25.10 (Boost 1.88) is affected but end of life. + - Downstream: https://github.com/Icinga/icinga2/issues/11160 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2173538 Title: [SRU] boost1.90: fibers clobber the libstdc++ exception state of another thread (upstream boostorg/context#323) To manage notifications about this bug go to: https://bugs.launchpad.net/boost/+bug/2173538/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
