Hello again Samuel,
First of all, I want to apologize again for the patch churn over the
past week. I wanted to put this to rest properly, and I am now sending
my final, stable version.
This is it. I have applied numerous fixes, performance tweaks, and
cleanups. I am happy to report that this now performs on par with
unjournaled ext2 on normal workloads, such as configuring/compiling
the Hurd, installing and reinstalling packages via APT, and untarring
large archives (like the Linux kernel). I have also heavily tested it
against artificial stress conditions (which I am happy to share if
there is interest), and it handles highly concurrent loads beautifully
without deadlocks or memory leaks.
Progressive checkpointing ensures the filesystem runs smoothly, and
the feature remains strictly opt-in (until a partition is tuned with
tune2fs -j, the journal is completely inactive).
The new API in libdiskfs is minimal but expressive enough to wrap all
filesystem operations in transactions and handle strict POSIX sync
barriers.
Since v4, I have made several major architectural improvements:
Smart Auto-Commit: diskfs_journal_stop_transaction now automatically
commits to disk if needs_sync has been flagged anywhere in the nested
RPC chain and the reference count drops to zero.
Cleaned ext2 internal Journal API: I have exposed journal_store_write
and journal_store_read as block-device filter layers. Internal state
checks (journal_has_active_transaction, etc.) are now strictly hidden.
How the journal preserves the WAL property is now very obvious, as it
directly intercepts physical store operations.
The "Lifeboat" Cache: Those store wrappers now utilize a small,
temporary internal cache to handle situations where the Mach VM pager
rushes blocks due to memory pressure. The Lifeboat seamlessly
intercepts and absorbs these hazard blocks without blocking the pager
or emitting warnings, even at peak write throughput.
As before, I have added detailed comments across the patch to explain
the state machine and locking hierarchy. I know this is a complex
subsystem, so I am more than happy to write additional documentation
in whatever form is needed.
Once again, apologies for the rapid iterations. I won't be touching
this code further until I hear your feedback.
Kind regards,
Milos
On Sun, Mar 15, 2026 at 9:01 PM Milos Nikic <[email protected]>
wrote:
Hi Samuel,
I am writing to sincerely apologize for the insane amount of patch
churn over the last week. I know the rapid version bumps from v2 up to
v4 have been incredibly noisy, and I want to hit the brakes before you
spend any more time reviewing the current code.
While running some extreme stress tests on a very small ext2 partition
with the tiniest (allowed by the tooling) journal, I caught a few
critical edge cases. While fixing those, I also realized that my
libdiskfs VFS API boundary is clunkier than it needs to be. I am
currently rewriting it to more closely match Linux's JBD2 semantics,
where the VFS simply flags a transaction for sync and calls stop,
allowing the journal to auto-commit when the reference count drops to
zero.
I'm also adding handling for cases where the Mach VM pager rushes
blocks to the disk while they are in the process of committing. This
safely intercepts them and will remove those warnings and WAL
violations in almost all cases.
Please completely disregard v4.
I promise the churn is coming to an end. I am going to take a little
time to finish this API contraction, stress-test it, polish it, and
make sure it is 100% rock-solid. I will be back soon with a finalized
v5.
Thanks for your patience with my crazy iteration process!
Best, Milos
On Thu, Mar 12, 2026 at 8:53 AM Milos Nikic <[email protected]>
wrote:
Hi Samuel,
As promised, here is the thoroughly tested and benchmarked V4 revision
of the JBD2 Journal for Hurd.
This revision addresses a major performance bottleneck present in V3
under heavy concurrent workloads. The new design restores performance
to match vanilla Hurd's unjournaled ext2fs while preserving full crash
consistency.
Changes since V3:
- Removed eager memcpy() from the journal_dirty_block() hot-path.
- Introduced deferred block copying that triggers only when the
transaction becomes quiescent.
- Added a `needs_copy` flag to prevent redundant memory copies.
- Eliminated the severe lock contention and memory bandwidth pressure
observed in V3.
Why the changes in v4 vs v3?
I have previously identified that the last remaining performance
bottleneck is memcpy of 4k byte every time journal_dirty_block is
called. And i was thinking about how to improve it.
Deferred copy comes to mind, But...
The Hurd VFS locks at the node level rather than the physical block
level (as Linux does). Because multiple nodes may share the same 4KB
disk block, naively deferring the journal copy until commit time can
capture torn writes if another thread is actively modifying a
neighboring node in the same block.
Precisely because of this V3 performed a 4KB memcpy immediately inside
journal_dirty_block() (copy on write) while the node lock was held.
While safe, this placed expensive memory operations and global journal
lock contention directly in the VFS hot-path, causing severe slowdowns
under heavy parallel workloads.
V4 removes this eager copy entirely by leveraging an existing
transaction invariant:
All VFS threads increment and decrement the active transaction's
`t_updates` counter via the start/stop transaction functions. A
transaction cannot commit until this counter reaches zero.
When `t_updates == 0`, we are mathematically guaranteed that no VFS
threads are mutating blocks belonging to the transaction. At that exact
moment, the memory backing those blocks has fully settled and can be
safely copied without risk of torn writes. A perfect place for a
deferred write!
journal_dirty_block() now simply records the dirty block id in a hash
table, making the hot-path strictly O(1). (and this is why we have an
amazing performance boost between v3 and v4)
But we also need to avoiding redundant copies:
Because transactions remain open for several seconds, `t_updates` may
bounce to zero and back up many times during a heavy workload (as
multiple VFS threads start/stop the transaction). To avoid repeatedly
copying the same unchanged blocks every time the counter hits zero,
each shadow buffer now contains a `needs_copy` flag.
When a block is dirtied, the flag is set. When `t_updates` reaches
zero, only buffers with `needs_copy == 1` are copied to the shadow
buffers, after which the flag is cleared.
So two things need to be true in order for a block to be copied: 1)
t_updates must just hit 0 and 2) needs_copy needs to be 1
This architecture completely removes the hot-path bottleneck. Journaled
ext2fs now achieves performance virtually identical to vanilla ext2fs,
even under brutal concurrency (e.g. scripts doing heavy writes from
multiple shells at the same time).
I know this is a dense patch with a lot to unpack. I've documented the
locking and Mach VM interactions as thoroughly as possible in the code
itself (roughly 1/3 of the lines are comments in ext2fs/journal.c), but
I understand there is only so much nuance that can fit into C comments.
If it would be helpful, I would be happy to draft a dedicated document
detailing the journal's lifecycle, its hooks into libdiskfs/ext2, and
the rationale behind the macro-transaction design, so future developers
have a clear reference.
Looking forward to your thoughts.
Best,
Milos
On Tue, Mar 10, 2026 at 9:25 PM Milos Nikic <[email protected]>
wrote:
Hi Samuel,
Just a quick heads-up: please hold off on reviewing this V3 series.
While V3 version works fast for simple scenarios in single threaded
situations (like configure or make ext2fs etc) I fund that while
running some heavy multi-threaded stress tests on V3, a significant
performance degradation happens due to lock contention bottleneck
caused by the eager VFS memcpy hot-path. (memcpy inside
journal_dirty_block which is called 1000s of time a second really
becomes a performance problem.)
I have been working on a much cleaner approach that safely defers the
block copying to the quiescent transaction stop state. It completely
eliminates the VFS lock contention and brings the journaled performance
back to vanilla ext2fs levels even with many threads competing at
writing/reading/renaming in the same place.
I am going to test this new architecture thoroughly over the next few
days and will send it as V4 once I am certain it is rock solid.
Thanks!
On Mon, Mar 9, 2026 at 12:15 PM Milos Nikic <[email protected]>
wrote:
Hello Samuel and the Hurd team,
I am sending over v3 of the journaling patch. I know v2 is still
pending review, but while testing and profiling based on previous
feedback, I realized the standard mapping wasn't scaling well for
metadata-heavy workloads. I wanted to send this updated architecture
your way to save you from spending time reviewing the obsolete v2 code.
This version keeps the core JBD2 logic from v2 but introduces several
structural optimizations, bug fixes, and code cleanups:
- Robin Hood Hash Map: Replaced ihash with a custom map for
significantly tighter cache locality and faster lookups.
- O(1) Slab Allocator: Added a pre-allocated pool to make transaction
buffers zero-allocation in the hot path.
- Unified Buffer Tracking: Eliminated the dual linked-list/map
structure in favor of just the map, fixing a synchronization bug from
v2 and simplifying the code.
- Few other small bug fixes
- Refactored Dirty Block Hooks: Moved the journal_dirty_block calls
from inode.c directly into the ext2fs.h low-level block computation
functions (record_global_poke, sync_global_ptr, record_indir_poke, and
alloc_sync). This feels like a more natural fit and makes it much
easier to ensure we aren't missing any call sites.
Performance Benchmarks:
I ran repeated tests on my machine to measure the overhead, comparing
this v3 journal implementation against Vanilla Hurd.
make ext2fs (CPU/Data bound - 5 runs):
Vanilla Hurd Average: ~2m 40.6s
Journal v3 Average: ~2m 41.3s
Result: Statistical tie. Journal overhead is practically zero.
make clean && ../configure (Metadata bound - 5 runs):
Vanilla Hurd Average: ~3.90s (with latency spikes up to 4.29s)
Journal v3 Average: ~3.72s (rock-solid consistency, never breaking
3.9s)
Result: Journaled ext2 is actually faster and more predictable here due
to the WAL absorbing random I/O.
Crash Consistency Proof:
Beyond performance, I wanted to demonstrate the actual crash recovery
in action.
Boot Hurd, log in, create a directory (/home/loshmi/test-dir3).
Wait for the 5-second kjournald commit tick.
Hard crash the machine (kill -9 the QEMU process on the host).
Inspecting from the Linux host before recovery shows the inode is
completely busted (as expected): sudo debugfs -R "stat
/home/loshmi/test-dir3" /dev/nbd0 debugfs 1.47.3 (8-Jul-2025)
Inode: 373911 Type: bad type Mode: 0000 Flags: 0x0
Generation: 0 Version: 0x00000000
User: 0 Group: 0 Size: 0
File ACL: 0 Translator: 0
Links: 0 Blockcount: 0
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x00000000 -- Wed Dec 31 16:00:00 1969
atime: 0x00000000 -- Wed Dec 31 16:00:00 1969
mtime: 0x00000000 -- Wed Dec 31 16:00:00 1969
BLOCKS:
Note: On Vanilla Hurd, running fsck here would permanently lose the
directory or potentially cause further damage depending on luck.
Triggering the journal replay:
sudo e2fsck -fy /dev/nbd0
Inspecting immediately after recovery: sudo debugfs -R "stat
/home/loshmi/test-dir3" /dev/nbd0 debugfs 1.47.3 (8-Jul-2025)
Inode: 373911 Type: directory Mode: 0775 Flags: 0x0
Generation: 1773077012 Version: 0x00000000
User: 1001 Group: 1001 Size: 4096
File ACL: 0 Translator: 0
Links: 2 Blockcount: 8
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x69af0213 -- Mon Mar 9 10:23:31 2026
atime: 0x69af0213 -- Mon Mar 9 10:23:31 2026
mtime: 0x69af0213 -- Mon Mar 9 10:23:31 2026
BLOCKS:
(0):1507738
TOTAL: 1
The journal successfully reconstructed the directory, and logdump
confirms the transactions were consumed perfectly.
I have run similar hard-crash tests for rename, chmod, and chown etc
with the same successful recovery results.
I've attached the v3 diff. Let me know what you think of the new hash
map and slab allocator approach!
Best,
Milos
On Fri, Mar 6, 2026 at 10:06 PM Milos Nikic <[email protected]>
wrote:
And here is the last one...
I hacked up an improvement for journal_dirty_block to try and see if i
could speed it up a bit.
1) Used specialized robin hood based hash table for speed (no
tombstones etc) (I took it from one of my personal projects....just
specialized it here a bit)
2) used a small slab allocator to avoid malloc-ing in the hot path
3) liberally sprinkled __rdtsc() to get a sense of cycle time inside
journal_dirty_block
Got to say, just this simple local change managed to shave off 3-5% of
slowness.
So my test is:
- Boot Hurd
- Inside Hurd go to the Hurd build directory
- run:
$ make clean && ../configure
$ time make ext2fs
I do it multiple times for 3 different versions of ext2 libraries
1) Vanilla Hurd (No Journal): ~avg, 151 seconds
2) Enhanced JBD2 (Slab + Custom Hash): ~159 seconds (5% slower!)
3) Baseline JBD2 (malloc + libihash what was sent in V2): ~168 seconds
Of course there is a lot of variability, and my laptop is not a perfect
environment for these kinds of benchmarks, but this is what i have.
My printouts on the screen show this:
ext2fs: part:5:device:wd0: warning: === JBD2 STATS ===
ext2fs: part:5:device:wd0: warning: Total Dirty Calls: 339105
ext2fs: part:5:device:wd0: warning: Total Function: 217101909
cycles
ext2fs: part:5:device:wd0: warning: Total Lock Wait: 16741691
cycles
ext2fs: part:5:device:wd0: warning: Total Alloc: 673363
cycles
ext2fs: part:5:device:wd0: warning: Total Memcpy: 137938008
cycles
ext2fs: part:5:device:wd0: warning: Total Hash Add: 258533
cycles
ext2fs: part:5:device:wd0: warning: Total Hash Find: 29501960
cycles
ext2fs: part:5:device:wd0: warning: --- AVERAGES (Amortized per call)
---
ext2fs: part:5:device:wd0: warning: Avg Function Time: 640 cycles
ext2fs: part:5:device:wd0: warning: Avg Lock Wait: 49 cycles
ext2fs: part:5:device:wd0: warning: Avg Memcpy: 406 cycles
ext2fs: part:5:device:wd0: warning: Avg Malloc 1: 1 cycles
ext2fs: part:5:device:wd0: warning: Avg Hash Add: 0 cycles
ext2fs: part:5:device:wd0: warning: Avg Hash Find: 86 cycles
ext2fs: part:5:device:wd0: warning: ==================
Averages here say a lot...with these improvements we are now down to
basically Memcpy time...and for copying 4096 bytes of ram Im not sure
we can make it take less than 400 cycles...so we are hitting hardware
limitations.
It would be great if we could avoid memcpy here altogether or delay it
until commit or similar, and i have some ideas, but they all require
drastic changes across libdiskfs and ext2fs, not sure if a few
remaining percentage points of slowdown warrant that.
Also, wow during ext2 compilation...this function (journal_dirty_block)
is being called a bit more than 1000 times per second (for each and
every block that is ever being touched by the compiler)
I am attaching here the altered journal.c with these changes if one is
interested in seeing the localized changes.
Regards,
Milos
On Fri, Mar 6, 2026 at 11:09 AM Milos Nikic <[email protected]>
wrote:
Hi Samuel,
One quick detail I forgot to mention regarding the performance
analysis:
The entire ~0.4s performance impact I measured is isolated exclusively
to journal_dirty_block.
To verify this, I ran an experiment where I stubbed out
journal_dirty_block so it just returned immediately (which obviously
makes for a very fast, but not very useful, journal!). With that single
function bypassed, the filesystem performs identically to vanilla Hurd.
This confirms that the background kjournald flusher, the transaction
reference counting, and the checkpointing logic add absolutely no
noticeable latency to the VFS. The overhead is strictly tied to the
physics of the memory copying and hashmap lookups in that one block
which we can improve in subsequent patches.
Thanks, Milos
On Fri, Mar 6, 2026 at 10:55 AM Milos Nikic <[email protected]>
wrote:
Hi Samuel,
Thanks for reviewing my mental model on V1; I appreciate the detailed
feedback.
Attached is the v2 patch. Here is a breakdown of the architectural
changes and refactors based on your review:
1. diskfs_node_update and the Pager
Regarding the question, "Do we really want to update the node?": Yes,
we must update it with every change. JBD2 works strictly at the
physical block level, not the abstract node cache level. To capture a
node change in the journal, the block content must be physically
serialized to the transaction buffer. Currently, this path is
diskfs_node_update -> diskfs_write_disknode -> journal_dirty_block.
When wait is 0, this just copies the node details from the node-cache
to the pager. It is strictly an in-memory serialization and is
extremely fast. I have updated the documentation for diskfs_node_update
to explicitly describe this behavior so future maintainers understand
it isn't triggering synchronous disk I/O and doesn't measurably
increase the latency of the file system.
journal_dirty_block is not one of the most hammered functions in
libdiskfs/ext2 and more on that below.
2. Synchronous Wait & Factorization
I completely agree with your factorization advice:
write_disknode_journaled has been folded directly into
diskfs_write_disknode, making it much cleaner.
Regarding the wait flag: we are no longer ignoring it! Instead of
blocking the VFS deeply in the stack, we now set an "IOU" flag on the
transaction. This bubbles the sync requirement up to the outer RPC
layer, which is the only place safe enough to actually sleep on the
commit and thus maintain the POSIX sync requirement without deadlocking
etc.
3. Multiple Writes to the Same Metadata Block
"Can it happen that we write several times to the same metadata block?"
Yes, multiple nodes can live in the same block. However, because the
Mach pager always flushes the "latest snapshot" of the block, we don't
have an issue with mixed or stale data hitting the disk.
If RPCs hit while pager is actively writing that is all captured in the
"RUNNING TRANSACTION". If it happens that that RUNNING TRANSACTION has
the same blocks pager is committing RUNNING TRANSACTION will be
forcebliy committed.
4. The New libdiskfs API
I added two new opaque accessors to diskfs.h:
diskfs_journal_set_sync
diskfs_journal_needs_sync
This allows inner nested functions to declare a strict need for a POSIX
sync without causing lock inversions. We only commit at the top RPC
layer once the operation is fully complete and locks are dropped.
5. Cleanups & Ordering
Removed the redundant record_global_poke calls.
Reordered the pager write notification in journal.c to sit after the
committing function, as the pager write happens after the journal
commit.
Merged the ext2_journal checks inside diskfs_journal_start_transaction
to return early.
Reverted the bold unlock moves.
Fixed the information leaks.
Elevated the deadlock/WAL bypass logs to ext2_warning.
Performance:
I investigated the ~0.4s (increase from 4.9s to 5.3s) regression on my
SSD during a heavy Hurd ../configure test. By stubbing out
journal_dirty_block, performance returned to vanilla Hurd speeds,
isolating the overhead to that specific function.
A nanosecond profile reveals the cost is evenly split across the
mandatory physics of a block journal:
25%: Lock Contention (Global transaction serialization)
22%: Memcpy (Shadowing the 4KB blocks)
21%: Hash Find (hurd_ihash lookups for block deduplication)
I was surprised to see hurd_ihash taking up nearly a quarter of the
overhead. I added some collision mitigation, but left a further
improvements of this patch to keep the scope tight. In the future, we
could drop the malloc entirely using a slab allocator and optimize the
hashmap to get this overhead closer to zero (along with introducing a
"frozen data" concept like Linux does but that would be a bigger non
localized change).
Final Note on Lock Hierarchy
The intended, deadlock-free use of the journal in libdiskfs is best
illustrated by the CHANGE_NODE_FIELD macro in libdiskfs/priv.h
txn = diskfs_journal_start_transaction ();
pthread_mutex_lock (&np->lock);
(OPERATION);
diskfs_node_update (np, diskfs_synchronous);
pthread_mutex_unlock (&np->lock);
if (diskfs_synchronous || diskfs_journal_needs_sync (txn))
diskfs_journal_commit_transaction (txn);
else
diskfs_journal_stop_transaction (txn);
By keeping journal operations strictly outside of the node
locking/unlocking phases, we treat it as the outermost "lock" on the
file system, mathematically preventing deadlocks.
Kind regards,
Milos
On Thu, Mar 5, 2026 at 12:41 PM Samuel Thibault
<[email protected]> wrote:
Hello,
Milos Nikic, le jeu. 05 mars 2026 09:31:26 -0800, a ecrit: Hurd VFS
works in 3 layers:
1. Node cache layer: The abstract node lives here and it is the ground
truth
of a running file system. When one does a stat myfile.txt, we get the
information straight from the cache. When we create a new file, it gets
placed in the cache, etc.
2. Pager layer: This is where nodes are serialized into the actual
physical
representation (4KB blocks) that will later be written to disk.
3. Hard drive: The physical storage that receives the bytes from the
pager.
During normal operations (not a sync mount, fsync, etc.), the VFS
operates
almost entirely on Layer 1: The Node cache layer. This is why it's
super fast.
User changed atime? No problem. It just fetches a node from the node
cache
(hash table lookup, amortized to O(1)) and updates the struct in
memory. And
that is it.
Yes, so that we get as efficient as possible.
Only when the sync interval hits (every 30 seconds by default) does the
Node
cache get iterated and serialized to the pager layer
(diskfs_sync_everything ->
write_all_disknodes -> write_node -> pager_sync). So basically, at that
moment, we create a snapshot of the state of the node cache and place
it onto
the pager(s).
It's not exactly a snapshot because the coherency between inodes and
data is not completely enforced (we write all disknodes before asking
the kernel to write back dirty pages, and then poke the writes).
Even then, pager_sync is called with wait = 0. It is handed to the
pager, which
sends it to Mach. At some later time (seconds or so later), Mach sends
it back
to the ext2 pager, which finally issues store_write to write it to
Layer 3 (The
Hard drive). And even that depends on how the driver reorders or delays
it.
The effect of this architecture is that when store_write is finally
called, the
absolute latest version of the node cache snapshot is what gets written
to the
storage. Is this basically correct?
It seems to be so indeed.
Are there any edge cases or mechanics that are wrong in this model
that would make us receive a "stale" node cache snapshot?
Well, it can be "stale" if another RPC hasn't called
diskfs_node_update() yet, but that's what "safe" FS are all about: not
actually provide more than coherency of the content on the disk so fsck
is not suppposed to be needed. Then, if a program really wants
coherency
between some files etc. it has to issue sync calls, dpkg does it for
instance.
Samuel
