Aloha -
I've been trying to debug a problem in rpctrace which causes rpctrace to
crash when I use it to wrap /hurd/ext2fs.
The bug is triggered by a memory_object_lock_request /
memory_object_lock_completed sequence. Specifically, ext2fs sends a lock
request to the kernel with a send-once reply_to port. Once the lock is
complete, the kernel sends a memory_object_lock_completed message to the
send-once right, including a send right to the memory control port (for
identification purposes) and that's where the trouble starts in rpctrace.
rpctrace is designed to trace multiple tasks simultaneously, and to
identify which task is doing an RPC, it allocates separate ports for each
task (even if they wrap the same port). So, if you pass out a send right
to three tasks, three different receive rights will be allocated in
rpctrace. Which receive right a message comes in on indicates which task
is sending the message.
For this to work right, we need to identify which task is on the receiving
end of a send right. Not which task the send right came from, mind you,
but the ultimate destination. In the previous example, we transferring a
copy of the original send right, we need to pick which of the three new
send rights should be transferred, based on the ultimate destination of the
message, to ensure that the right task gets the right version of the port.
There's also a fourth case - we're transferring to a task that we're not
tracing.
All of this complexity is already built into rpctrace. It plays games like
looking at a task's port space, extracting a send right from each remote
receive right, and checking to see if it matches a local send right, in
order to determine that the local send right's final destination is the
remote receive right on the task in question. See discover_receive_right().
In my case, problems arise because a send-once right is used to return the
lock completed message, and there's no way to know which task the send-once
right ultimately goes to. There's a bad pointer deference involved, but
even once that's fixed, how do you know which send right to transfer? It's
important to get it right, since the send right is used by the memory
manager to identify different clients.
I've patched it up by assuming that the task sending the send-once right is
the ultimate destination, which works in my case, but obviously it isn't
right in general.
The more I think about it, the more I'm thinking that it's a design flaw in
rpctrace. We need to identify ultimate destinations, but can't do that
reliably.
I read on the website's hurd/debugging/rpctrace page that somebody (zhenga)
had come with a new version of rpctrace. Do we have a copy of it around
somewhere?
I could submit the patches that I've got, but they're not right, and I
don't see any way to make them right. I'm thinking now that the way to fix
it is to redesign rpctrace so that each wrapped task gets a separate
rpctrace task wrapping it. That way, we should be able to determine which
task makes which RPC without the problems I've described above.
I'm also thinking that I don't want to undertake rewriting rpctrace right
now. I was just trying to fix it so that I could understand what ext2fs
was doing.
Comments?
agape
brent
