Re: SMP, barriers, etc.
Hi, Samuel Thibault wrote: After I read this introduction, I checked some atomic operations implementation such as atomic_add_return in Linux for Aphla processors. Before and after these operations change the variable, they put memory barriers. So it's something like this: smp_mb(); operations; smp_mb(); But the problem is: on SMP, a variable's value can always been changed by another processor after the first memory barrier is called. Thus, the CPU does an operation on the stale value. There seems to be no way that we can guarantee that the variable has the latest value at the moment the CPU is doing the operation on it. Yes, that's why the operations have a loop that keeps retrying until the result is as expected. I didn't notice there was a loop in the implementation of atomic operations. It seems most architectures provide a special load instruction that sets a flag/register, and a conditional store instruction. I found some information about these instructions but I still have something I'm not sure of. It seems to me that it should work as follows: If processor A loads a variable in memory to the register with the special load instruction, it should monitor *all* cache lines specified by the address (including the ones on other processors). If any other processors change the variable in their own cache, processor A has to clear its flag. After it changes the variable and tries to store the value to memory, the conditional store instruction might fail. If the store fails, processor A has to *invalidate* its own cache line so that it can load the latest value in the next iteration. Is the process above correct? Zheng Da
Re: Reauthentication implementation flaw due to EINTR
On Sat, Dec 26, 2009 at 09:12:08PM +0100, Samuel Thibault wrote: I've checked again the result Carl Fredrik Hammar, le Sat 26 Dec 2009 19:58:12 +0100, a écrit : There is this issue as well, which I have fixed already in commit 041baa80 (and indeed seen cases where it helped), but that's not enough, because not only auth gets EINTR here and can fix things, but ext2fs also gets an EINTR but can't able to restart the call in iohelp_reauth since the rendez-vous port is dead and thus gets EINVAL. Why does this happen? I don't know, but I know for sure that auth_server_authenticate returns EINTR in ext2fs (and next call returns EINVAL), even if auth never returned EINTR. OK, I think I have a vague picture of what is going on: ports_interrupt_self_on_port_death ports_interrupt_self_on_notification ports_interrupt_rpc_on_notification, which requests notification (to the same port as auth_server_authenticate). When rendezvous port dies we get the notification: ports_notify_server ports_do_mach_notify_dead_name ports_dead_name ports_interrupt_notified_rpcs hurd_thread_cancel (in glibc) _hurdsig_abort_rpcs, which does some funky stuff that seems to hijack any pending RPCs to make them return EINTR. If the notification request is canceled before auth_user_authenticate returns there should be no problem. I don't know how to do that, as libports doesn't seem to have a function for it. The primitive function would be mach_port_request_notification with a null port, but I assume there is other stuff to clean up. Alternatively, the notification can be handled separately by the auth server, perhaps by overriding libports's notify server. Regards, Fredrik
Re: SMP, barriers, etc.
Da Zheng, le Sun 27 Dec 2009 16:39:04 +0800, a écrit : Is the process above correct? I have never actually programmed the architectures where things work like this (powerpc such), but that's what I have understood from the code and explanations here and there, yes. It's a sort of transactional memory actually. Samuel
Re: Reauthentication implementation flaw due to EINTR
On Sun, Dec 27, 2009 at 08:56:21PM +0100, Carl Fredrik Hammar wrote: On Sat, Dec 26, 2009 at 09:12:08PM +0100, Samuel Thibault wrote: I've checked again the result Carl Fredrik Hammar, le Sat 26 Dec 2009 19:58:12 +0100, a écrit : There is this issue as well, which I have fixed already in commit 041baa80 (and indeed seen cases where it helped), but that's not enough, because not only auth gets EINTR here and can fix things, but ext2fs also gets an EINTR but can't able to restart the call in iohelp_reauth since the rendez-vous port is dead and thus gets EINVAL. Why does this happen? I don't know, but I know for sure that auth_server_authenticate returns EINTR in ext2fs (and next call returns EINVAL), even if auth never returned EINTR. OK, I think I have a vague picture of what is going on: ports_interrupt_self_on_port_death ports_interrupt_self_on_notification ports_interrupt_rpc_on_notification, which requests notification (to the same port as auth_server_authenticate). When rendezvous port dies we get the notification: ports_notify_server ports_do_mach_notify_dead_name ports_dead_name ports_interrupt_notified_rpcs hurd_thread_cancel (in glibc) _hurdsig_abort_rpcs, which does some funky stuff that seems to hijack any pending RPCs to make them return EINTR. If the notification request is canceled before auth_user_authenticate returns there should be no problem. I don't know how to do that, as libports doesn't seem to have a function for it. The primitive function would be mach_port_request_notification with a null port, but I assume there is other stuff to clean up. Alternatively, the notification can be handled separately by the auth server, perhaps by overriding libports's notify server. Oh sorry, I did not think through how this affects your original solution. AFAICT, after auth_server_authenticate_reply returns _hurdsig_abort_rpcs can not hijack it, so your solution should work. But all this is a bit over my head so I can't tell for sure. Canceling the notification would be safer in this respect, if you can figure out how to do it. Regards, Fredrik
Re: Reauthentication implementation flaw due to EINTR
Carl Fredrik Hammar, le Sun 27 Dec 2009 22:24:24 +0100, a écrit : AFAICT, after auth_server_authenticate_reply returns _hurdsig_abort_rpcs can not hijack it, so your solution should work. I've checked more in the kernel internals, maybe there is a race window, even if very unlikely. ext2fs is in auth_server_authenticate()'s receive part, let's assume it's already queued on the receive mqueue-imq_threads from ipc_mqueue_receive(). In that case, auth_server_authenticate_reply() finds it and sets ith_state to MACH_MSG_SUCCESS. When ipc_mqueue_receive wakes up, it finds that first, before checking whether ith_wait_result is THREAD_INTERRUPTED, so we win. However, can it be that ext2fs may have sent its message (triggering everything in auth etc.) but not yet blocked in ipc_mqueue_receive() when the auth_server_authenticate_reply comes? In that case, it may not have the time to receive the message before auth_user_authenticate() returns and the reply port gets dead, triggering an interrupt. I can see locks on spaces and ports, but in mach_msg_trap, it seems that at label slow_get_rcv_port we don't have any lock left on send part. Samuel
Re: Reauthentication implementation flaw due to EINTR
Carl Fredrik Hammar, le Sun 27 Dec 2009 20:56:21 +0100, a écrit : If the notification request is canceled before auth_user_authenticate returns there should be no problem. Ok, I see. Samuel
Re: Reauthentication implementation flaw due to EINTR
Carl Fredrik Hammar, le Sun 27 Dec 2009 20:56:21 +0100, a écrit : If the notification request is canceled before auth_user_authenticate returns there should be no problem. I tried a crude error_t err2 = mach_port_request_notification(mach_task_self(), rendezvous, MACH_NOTIFY_DEAD_NAME, 0, MACH_PORT_NULL, MACH_MSG_TYPE_MAKE_SEND_ONCE, old); after the complete if (hurd_condition_wait (s.wakeup, pending_lock) statement, with no luck: ext2fs got an EINTR anyway. Samuel
Re: Reauthentication implementation flaw due to EINTR
Carl Fredrik Hammar, le Sun 27 Dec 2009 22:24:24 +0100, a écrit : OK, I think I have a vague picture of what is going on: ports_interrupt_self_on_port_death ports_interrupt_self_on_notification ports_interrupt_rpc_on_notification, which requests notification (to the same port as auth_server_authenticate). But in another task (auth vs ext2fs). That's where I've still not found what makes ext2fs return EINTR. When rendezvous port dies we get the notification: Here we seemingly see ext2fs get the notification, but why? ports_notify_server ports_do_mach_notify_dead_name ports_dead_name ports_interrupt_notified_rpcs hurd_thread_cancel (in glibc) _hurdsig_abort_rpcs, which does some funky stuff that seems to hijack any pending RPCs to make them return EINTR. Samuel
