It ended up taking a little work to get another environment to trigger
this reliably, but I think I have something now.
I modified the iterate_handles() function a bit so that it keeps
scanning over and over again indefinitely rather than letting the server
start up. This forces the code path in question without having to
restart the servers. Using this setup I'm able to trigger it on an
empty 8 node file system, but I have to leave all of the servers running
on it anywhere from a few minutes to half an hour before one of them
crashes. Oddly enough, with this environment it crashes faster on an
emtpy file system than one with 500,000 files.
I repeated this test with the latest HEAD version from trunk, and that
didn't seem to make any difference.
I'll try the mutex suggestion next.
-Phil
Sam Lang wrote:
On Feb 20, 2007, at 11:32 AM, Pete Wyckoff wrote:
[EMAIL PROTECTED] wrote on Tue, 20 Feb 2007 07:29 -0500:
dbpf-dspace.c:1371
assert(!dbpf_op_queue_empty(dbpf_completion_queue_array [context_id]));
According to the stack trace, this test() call followed a
trove_dspace_iterate_handles() call within the
trove_check_handle_ranges() function. This is part of the logic on
startup that scans all of the handles in the storage space to update
the
list of available/used handles in trove-handle-mgmt.
Another thought for Sam, who knows this code better.
(1) DBPF_COMPLETION_START modifies cur_op->op.state without holding the
dbpf_completion_queue_array_mutex[cid] mutex. Then it grabs the
mutex and puts the op on the completion array.
(2) dbpf_dspace_test grabs that mutex, looks at op.state, then asserts
that the queue must not be empty.
Perhaps (1) modifies the state but doesn't get around to putting it
on the completion array. (Possibly because the lock is held in
(2).)
Good point Pete. Given that this seems to be race Phil is seeing, your
theory seems more likely.
Maybe (1) should put the op on the array before modifying its state,
and hold the array mutex the whole time. I'm not sure what kind of
locking rules are involved between the mutex in the op and the mutex
on the completion array, though. Or what else might break with such
a change.
I don't think there should be any problems with doing this. It
probably doesn't matter when the op is added to the completion queue
(before or after its state gets changed), just that the completion
queue's mutex gets locked before either (at the top of
DBPF_COMPLETION_START). I wonder if Phil could make this change and
run his tests again.
-- Pete
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
