[openib-general] Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages
Quoting r. Libor Michalek [EMAIL PROTECTED]: Subject: Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages On Fri, May 13, 2005 at 04:51:45PM +0300, Michael S. Tsirkin wrote: Quoting r. Roland Dreier [EMAIL PROTECTED]: Libor Always call aio_complete() immediately when Libor iocb_complete() is called, and only spawn the work thread Libor to unlock the memory after the aio_complete() call. The Libor patch is below. Libor, I dont think its a good idea - this will break other assumptions, like the assumption that the task mm isnt destroyed before we unlock the memory. That's a good point. Another alternative would be to always complete aios asynchronously, which should preserve the order. I guess this would hurt latency for small ios... To avoid hurting latency, lets count the number of outstanding asynchronous AIOs, and if there are asynchronous AIOs complete all of them asynchronously. Does this make sense? Yes, except that the current iocb code does not reference individual sockets anywhere, and do_iocb_complete would have to be the function which decremented the per connection counter of outstanding AIOs. Also, since we don't have to do get_user_pages a second time on the send path, this would only need to be done on the recv path. -Libor I thought about this some more: what if we set users to 1 before releasing the irq spinlock, and call sdp_conn_unlock in thread context after completing the aio iocb? Any synchronous transfer would then wait till socket is unloacked. -- MST - Michael S. Tsirkin ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages
On Wed, May 25, 2005 at 11:21:28PM +0300, Michael S. Tsirkin wrote: Quoting r. Libor Michalek [EMAIL PROTECTED]: Subject: Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages On Fri, May 13, 2005 at 04:51:45PM +0300, Michael S. Tsirkin wrote: Quoting r. Roland Dreier [EMAIL PROTECTED]: Libor Always call aio_complete() immediately when Libor iocb_complete() is called, and only spawn the work thread Libor to unlock the memory after the aio_complete() call. The Libor patch is below. Libor, I dont think its a good idea - this will break other assumptions, like the assumption that the task mm isnt destroyed before we unlock the memory. That's a good point. Another alternative would be to always complete aios asynchronously, which should preserve the order. I guess this would hurt latency for small ios... To avoid hurting latency, lets count the number of outstanding asynchronous AIOs, and if there are asynchronous AIOs complete all of them asynchronously. Does this make sense? Yes, except that the current iocb code does not reference individual sockets anywhere, and do_iocb_complete would have to be the function which decremented the per connection counter of outstanding AIOs. Also, since we don't have to do get_user_pages a second time on the send path, this would only need to be done on the recv path. I thought about this some more: what if we set users to 1 before releasing the irq spinlock, and call sdp_conn_unlock in thread context after completing the aio iocb? Any synchronous transfer would then wait till socket is unloacked. So use 'users' as a reference count, and basically increment the lock before spawning do_iocb_complete. do_iocb_complete would then unlock one reference and the function calling iocb_complete would unlock the other reference? This could work... Remeber that it's possible for many do_iocb_complete functions to be in flight for a given connection. FYI, The problem I saw occured when the iocb_complete was called during the sdp_conn_unlock CQ poll, which is done with IRQ diabled. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages
On Sat, May 14, 2005 at 10:19:30PM +0300, Michael S. Tsirkin wrote: Quoting r. Libor Michalek [EMAIL PROTECTED]: Another alternative would be to always complete aios asynchronously, which should preserve the order. I guess this would hurt latency for small ios... To avoid hurting latency, lets count the number of outstanding asynchronous AIOs, and if there are asynchronous AIOs complete all of them asynchronously. Does this make sense? Yes, except that the current iocb code does not reference individual sockets anywhere, and do_iocb_complete would have to be the function which decremented the per connection counter of outstanding AIOs. But do_iocb_complete is always called with socket locked, is it not? If so its not hard to add that counting. iocb_complete() is always called with the socket locked, but by the time do_iocb_complete() executes it's unknown what the socket is doing since it's not referenced anywhere in sdp_iocb. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages
Quoting r. Libor Michalek [EMAIL PROTECTED]: Subject: Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages On Sat, May 14, 2005 at 10:19:30PM +0300, Michael S. Tsirkin wrote: Quoting r. Libor Michalek [EMAIL PROTECTED]: Another alternative would be to always complete aios asynchronously, which should preserve the order. I guess this would hurt latency for small ios... To avoid hurting latency, lets count the number of outstanding asynchronous AIOs, and if there are asynchronous AIOs complete all of them asynchronously. Does this make sense? Yes, except that the current iocb code does not reference individual sockets anywhere, and do_iocb_complete would have to be the function which decremented the per connection counter of outstanding AIOs. But do_iocb_complete is always called with socket locked, is it not? If so its not hard to add that counting. iocb_complete() is always called with the socket locked, but by the time do_iocb_complete() executes it's unknown what the socket is doing since it's not referenced anywhere in sdp_iocb. -Libor Wait, thats exactly the case we were trying to handle - spawning a thread to complete the iocb. So if we add a conn pointer to the iocb, we can increment a counter before spawning a thread to complete the iocb and decrement after the iocb is complete. If counter 0, spawn a thread to complete synchronous io. -- MST - Michael S. Tsirkin ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
