Re: Is it really safe to use workqueues to drive expedited grace periods?
On Fri, Mar 03, 2017 at 11:30:49AM -0800, Tejun Heo wrote: > Hello, Paul. > > Sorry about the long delay. Travel + sickness. Just starting to > catch up with things now. No problem, I have been distracted by other projects in any case. ;-) > On Mon, Feb 13, 2017 at 04:16:00PM -0800, Paul E. McKenney wrote: > > Thank you for the information! So if I am to continue using workqueues > > for expedited RCU grace periods, I believe that need to do the following: > > > > 1. Use alloc_workqueue() to create my own WQ_MEM_RECLAIM > > workqueue. > > This is only necessary if RCU can be in a dependency chain in the > memory reclaim path - e.g. somebody doing synchronize_expedited_rcu() > in some obscure writeback path; however, given how wildly RCU is used, > I don't think this is a bad idea. The only extra overhead which comes > from it is memory used for an extra workqueue and a rescuer thread > after all. I suspect that SLAB_DESTROY_BY_RCU qualifies -- such a slab allocator could have a great many slabs waiting for an RCU grace period. > > 2. Rework my workqueue handler to avoid blocking waiting for > > the expedited grace period to complete. I should be able > > to do a small number of timed wait, but if I actually > > wait for the grace period to complete, I might end up > > hogging the reserved items. (Or does my workqueue supply > > them for me? If so, so much the better!) > > So, what the dedicated workqueue w/ WQ_MEMRECLAIM guarantees is that > there will always be at least one worker thread which is executing > work items from the workqueue. > > As long as your workqueue usage guarantees forward progress - that is, > as long as one work item in the workqueue won't block indefinitely on > another work item on the same workqueue, you shouldn't need to reword > the workqueue handler. > > If there can be dependency chain among work items of the same > WQ_MEMRECLAIM workqueue, often the best approach is breaking up the > chain and putting them into their own WQ_MEMRECLAIM workqueues. Thank you, good to know! > > 3. Concurrency would not be a problem -- there can be no more > > four work elements in flight across both possible flavors > > of expedited grace periods. > > You usually don't have to worry about concurrency all that much with > workqueues. They'll provide the maximum the system can as long as > that's possible. > > If the four work items can depend on each other, it'd be best to put > them in separate workqueues. If not, there's nothing to worry about. For a given pair, one can be acquiring a mutext that the other holds. Ah, so if only one task was running, that would fail to make forward progress. That said, splitting would be a bit weird in this case, because alternating grace periods for a given RCU flavor would need to schedule work on a different workqueue. Might be easier to do mutex_trylock() and reschedule... And thank you for the info, very helpful! Thanx, Paul
Re: Is it really safe to use workqueues to drive expedited grace periods?
On Fri, Mar 03, 2017 at 11:30:49AM -0800, Tejun Heo wrote: > Hello, Paul. > > Sorry about the long delay. Travel + sickness. Just starting to > catch up with things now. No problem, I have been distracted by other projects in any case. ;-) > On Mon, Feb 13, 2017 at 04:16:00PM -0800, Paul E. McKenney wrote: > > Thank you for the information! So if I am to continue using workqueues > > for expedited RCU grace periods, I believe that need to do the following: > > > > 1. Use alloc_workqueue() to create my own WQ_MEM_RECLAIM > > workqueue. > > This is only necessary if RCU can be in a dependency chain in the > memory reclaim path - e.g. somebody doing synchronize_expedited_rcu() > in some obscure writeback path; however, given how wildly RCU is used, > I don't think this is a bad idea. The only extra overhead which comes > from it is memory used for an extra workqueue and a rescuer thread > after all. I suspect that SLAB_DESTROY_BY_RCU qualifies -- such a slab allocator could have a great many slabs waiting for an RCU grace period. > > 2. Rework my workqueue handler to avoid blocking waiting for > > the expedited grace period to complete. I should be able > > to do a small number of timed wait, but if I actually > > wait for the grace period to complete, I might end up > > hogging the reserved items. (Or does my workqueue supply > > them for me? If so, so much the better!) > > So, what the dedicated workqueue w/ WQ_MEMRECLAIM guarantees is that > there will always be at least one worker thread which is executing > work items from the workqueue. > > As long as your workqueue usage guarantees forward progress - that is, > as long as one work item in the workqueue won't block indefinitely on > another work item on the same workqueue, you shouldn't need to reword > the workqueue handler. > > If there can be dependency chain among work items of the same > WQ_MEMRECLAIM workqueue, often the best approach is breaking up the > chain and putting them into their own WQ_MEMRECLAIM workqueues. Thank you, good to know! > > 3. Concurrency would not be a problem -- there can be no more > > four work elements in flight across both possible flavors > > of expedited grace periods. > > You usually don't have to worry about concurrency all that much with > workqueues. They'll provide the maximum the system can as long as > that's possible. > > If the four work items can depend on each other, it'd be best to put > them in separate workqueues. If not, there's nothing to worry about. For a given pair, one can be acquiring a mutext that the other holds. Ah, so if only one task was running, that would fail to make forward progress. That said, splitting would be a bit weird in this case, because alternating grace periods for a given RCU flavor would need to schedule work on a different workqueue. Might be easier to do mutex_trylock() and reschedule... And thank you for the info, very helpful! Thanx, Paul
Re: Is it really safe to use workqueues to drive expedited grace periods?
Hello, Paul. Sorry about the long delay. Travel + sickness. Just starting to catch up with things now. On Mon, Feb 13, 2017 at 04:16:00PM -0800, Paul E. McKenney wrote: > Thank you for the information! So if I am to continue using workqueues > for expedited RCU grace periods, I believe that need to do the following: > > 1.Use alloc_workqueue() to create my own WQ_MEM_RECLAIM > workqueue. This is only necessary if RCU can be in a dependency chain in the memory reclaim path - e.g. somebody doing synchronize_expedited_rcu() in some obscure writeback path; however, given how wildly RCU is used, I don't think this is a bad idea. The only extra overhead which comes from it is memory used for an extra workqueue and a rescuer thread after all. > 2.Rework my workqueue handler to avoid blocking waiting for > the expedited grace period to complete. I should be able > to do a small number of timed wait, but if I actually > wait for the grace period to complete, I might end up > hogging the reserved items. (Or does my workqueue supply > them for me? If so, so much the better!) So, what the dedicated workqueue w/ WQ_MEMRECLAIM guarantees is that there will always be at least one worker thread which is executing work items from the workqueue. As long as your workqueue usage guarantees forward progress - that is, as long as one work item in the workqueue won't block indefinitely on another work item on the same workqueue, you shouldn't need to reword the workqueue handler. If there can be dependency chain among work items of the same WQ_MEMRECLAIM workqueue, often the best approach is breaking up the chain and putting them into their own WQ_MEMRECLAIM workqueues. > 3.Concurrency would not be a problem -- there can be no more > four work elements in flight across both possible flavors > of expedited grace periods. You usually don't have to worry about concurrency all that much with workqueues. They'll provide the maximum the system can as long as that's possible. If the four work items can depend on each other, it'd be best to put them in separate workqueues. If not, there's nothing to worry about. Thanks. -- tejun
Re: Is it really safe to use workqueues to drive expedited grace periods?
Hello, Paul. Sorry about the long delay. Travel + sickness. Just starting to catch up with things now. On Mon, Feb 13, 2017 at 04:16:00PM -0800, Paul E. McKenney wrote: > Thank you for the information! So if I am to continue using workqueues > for expedited RCU grace periods, I believe that need to do the following: > > 1.Use alloc_workqueue() to create my own WQ_MEM_RECLAIM > workqueue. This is only necessary if RCU can be in a dependency chain in the memory reclaim path - e.g. somebody doing synchronize_expedited_rcu() in some obscure writeback path; however, given how wildly RCU is used, I don't think this is a bad idea. The only extra overhead which comes from it is memory used for an extra workqueue and a rescuer thread after all. > 2.Rework my workqueue handler to avoid blocking waiting for > the expedited grace period to complete. I should be able > to do a small number of timed wait, but if I actually > wait for the grace period to complete, I might end up > hogging the reserved items. (Or does my workqueue supply > them for me? If so, so much the better!) So, what the dedicated workqueue w/ WQ_MEMRECLAIM guarantees is that there will always be at least one worker thread which is executing work items from the workqueue. As long as your workqueue usage guarantees forward progress - that is, as long as one work item in the workqueue won't block indefinitely on another work item on the same workqueue, you shouldn't need to reword the workqueue handler. If there can be dependency chain among work items of the same WQ_MEMRECLAIM workqueue, often the best approach is breaking up the chain and putting them into their own WQ_MEMRECLAIM workqueues. > 3.Concurrency would not be a problem -- there can be no more > four work elements in flight across both possible flavors > of expedited grace periods. You usually don't have to worry about concurrency all that much with workqueues. They'll provide the maximum the system can as long as that's possible. If the four work items can depend on each other, it'd be best to put them in separate workqueues. If not, there's nothing to worry about. Thanks. -- tejun
Re: Is it really safe to use workqueues to drive expedited grace periods?
On Sat, Feb 11, 2017 at 11:35:41AM +0900, Tejun Heo wrote: > Hello, Paul. > > On Fri, Feb 10, 2017 at 01:21:58PM -0800, Paul E. McKenney wrote: > > So RCU's expedited grace periods have been using workqueues for a > > little while, and things seem to be working. But as usual, I worry... > > Is this use subject to some sort of deadlock where RCU's workqueue cannot > > start running until after a grace period completes, but that grace > > period is the one needing the workqueue? Note that there are ways to > > set up your kernel so that all RCU grace periods are expedited. > > > > Should I be worried? If not, what prevents this from being a problem, > > especially given that workqueue handlers are allowed to wait for RCU > > grace periods to complete? > > A per-cpu (normal) workqueue's concurrency is regulated automatically > so that there are at least one worker running for the worker pool on a > given CPU. > > Let's say there are two work items queued on a workqueue. The first > one is something which will do synchronize_rcu() and the second is the > expedited grace period work item. When the first one runs > synchronize_rcu(), it'd block. If there are no other work items > running at the time, workqueue will dispatch another worker so that > there's at least one actively running, which in this case will be the > expedited rcu grace period work item. > > The dispatching of a new worker can be delayed by two things - memory > pressure preventing creation of a new worker and the workqueue hitting > maximum concurrency limit. > > If expedited RCU grace period is something that memory reclaim path > may depend on, the workqueue that it executes on should have > WQ_MEM_RECLAIM set, which will guarantee that there's at least one > worker (across all CPUs) which is ready to serve the work items on > that workqueue regardless of memory pressure. > > The latter, concurrency limit, would only matter if the RCU work items > use system_wq. system_wq's concurrency limit is very high (512 per > CPU), but it is theoretically possible to fill all up with work items > doing synchronize_rcu() with the expedited RCU work item scheduled > behind it. The system would already be in a very messed up state > outside the RCU situation tho. Thank you for the information! So if I am to continue using workqueues for expedited RCU grace periods, I believe that need to do the following: 1. Use alloc_workqueue() to create my own WQ_MEM_RECLAIM workqueue. 2. Rework my workqueue handler to avoid blocking waiting for the expedited grace period to complete. I should be able to do a small number of timed wait, but if I actually wait for the grace period to complete, I might end up hogging the reserved items. (Or does my workqueue supply them for me? If so, so much the better!) 3. Concurrency would not be a problem -- there can be no more four work elements in flight across both possible flavors of expedited grace periods. Anything I am missing here? Thanx, Paul
Re: Is it really safe to use workqueues to drive expedited grace periods?
On Sat, Feb 11, 2017 at 11:35:41AM +0900, Tejun Heo wrote: > Hello, Paul. > > On Fri, Feb 10, 2017 at 01:21:58PM -0800, Paul E. McKenney wrote: > > So RCU's expedited grace periods have been using workqueues for a > > little while, and things seem to be working. But as usual, I worry... > > Is this use subject to some sort of deadlock where RCU's workqueue cannot > > start running until after a grace period completes, but that grace > > period is the one needing the workqueue? Note that there are ways to > > set up your kernel so that all RCU grace periods are expedited. > > > > Should I be worried? If not, what prevents this from being a problem, > > especially given that workqueue handlers are allowed to wait for RCU > > grace periods to complete? > > A per-cpu (normal) workqueue's concurrency is regulated automatically > so that there are at least one worker running for the worker pool on a > given CPU. > > Let's say there are two work items queued on a workqueue. The first > one is something which will do synchronize_rcu() and the second is the > expedited grace period work item. When the first one runs > synchronize_rcu(), it'd block. If there are no other work items > running at the time, workqueue will dispatch another worker so that > there's at least one actively running, which in this case will be the > expedited rcu grace period work item. > > The dispatching of a new worker can be delayed by two things - memory > pressure preventing creation of a new worker and the workqueue hitting > maximum concurrency limit. > > If expedited RCU grace period is something that memory reclaim path > may depend on, the workqueue that it executes on should have > WQ_MEM_RECLAIM set, which will guarantee that there's at least one > worker (across all CPUs) which is ready to serve the work items on > that workqueue regardless of memory pressure. > > The latter, concurrency limit, would only matter if the RCU work items > use system_wq. system_wq's concurrency limit is very high (512 per > CPU), but it is theoretically possible to fill all up with work items > doing synchronize_rcu() with the expedited RCU work item scheduled > behind it. The system would already be in a very messed up state > outside the RCU situation tho. Thank you for the information! So if I am to continue using workqueues for expedited RCU grace periods, I believe that need to do the following: 1. Use alloc_workqueue() to create my own WQ_MEM_RECLAIM workqueue. 2. Rework my workqueue handler to avoid blocking waiting for the expedited grace period to complete. I should be able to do a small number of timed wait, but if I actually wait for the grace period to complete, I might end up hogging the reserved items. (Or does my workqueue supply them for me? If so, so much the better!) 3. Concurrency would not be a problem -- there can be no more four work elements in flight across both possible flavors of expedited grace periods. Anything I am missing here? Thanx, Paul
Re: Is it really safe to use workqueues to drive expedited grace periods?
Hello, Paul. On Fri, Feb 10, 2017 at 01:21:58PM -0800, Paul E. McKenney wrote: > So RCU's expedited grace periods have been using workqueues for a > little while, and things seem to be working. But as usual, I worry... > Is this use subject to some sort of deadlock where RCU's workqueue cannot > start running until after a grace period completes, but that grace > period is the one needing the workqueue? Note that there are ways to > set up your kernel so that all RCU grace periods are expedited. > > Should I be worried? If not, what prevents this from being a problem, > especially given that workqueue handlers are allowed to wait for RCU > grace periods to complete? A per-cpu (normal) workqueue's concurrency is regulated automatically so that there are at least one worker running for the worker pool on a given CPU. Let's say there are two work items queued on a workqueue. The first one is something which will do synchronize_rcu() and the second is the expedited grace period work item. When the first one runs synchronize_rcu(), it'd block. If there are no other work items running at the time, workqueue will dispatch another worker so that there's at least one actively running, which in this case will be the expedited rcu grace period work item. The dispatching of a new worker can be delayed by two things - memory pressure preventing creation of a new worker and the workqueue hitting maximum concurrency limit. If expedited RCU grace period is something that memory reclaim path may depend on, the workqueue that it executes on should have WQ_MEM_RECLAIM set, which will guarantee that there's at least one worker (across all CPUs) which is ready to serve the work items on that workqueue regardless of memory pressure. The latter, concurrency limit, would only matter if the RCU work items use system_wq. system_wq's concurrency limit is very high (512 per CPU), but it is theoretically possible to fill all up with work items doing synchronize_rcu() with the expedited RCU work item scheduled behind it. The system would already be in a very messed up state outside the RCU situation tho. Thanks. -- tejun
Re: Is it really safe to use workqueues to drive expedited grace periods?
Hello, Paul. On Fri, Feb 10, 2017 at 01:21:58PM -0800, Paul E. McKenney wrote: > So RCU's expedited grace periods have been using workqueues for a > little while, and things seem to be working. But as usual, I worry... > Is this use subject to some sort of deadlock where RCU's workqueue cannot > start running until after a grace period completes, but that grace > period is the one needing the workqueue? Note that there are ways to > set up your kernel so that all RCU grace periods are expedited. > > Should I be worried? If not, what prevents this from being a problem, > especially given that workqueue handlers are allowed to wait for RCU > grace periods to complete? A per-cpu (normal) workqueue's concurrency is regulated automatically so that there are at least one worker running for the worker pool on a given CPU. Let's say there are two work items queued on a workqueue. The first one is something which will do synchronize_rcu() and the second is the expedited grace period work item. When the first one runs synchronize_rcu(), it'd block. If there are no other work items running at the time, workqueue will dispatch another worker so that there's at least one actively running, which in this case will be the expedited rcu grace period work item. The dispatching of a new worker can be delayed by two things - memory pressure preventing creation of a new worker and the workqueue hitting maximum concurrency limit. If expedited RCU grace period is something that memory reclaim path may depend on, the workqueue that it executes on should have WQ_MEM_RECLAIM set, which will guarantee that there's at least one worker (across all CPUs) which is ready to serve the work items on that workqueue regardless of memory pressure. The latter, concurrency limit, would only matter if the RCU work items use system_wq. system_wq's concurrency limit is very high (512 per CPU), but it is theoretically possible to fill all up with work items doing synchronize_rcu() with the expedited RCU work item scheduled behind it. The system would already be in a very messed up state outside the RCU situation tho. Thanks. -- tejun
Is it really safe to use workqueues to drive expedited grace periods?
Hello! So RCU's expedited grace periods have been using workqueues for a little while, and things seem to be working. But as usual, I worry... Is this use subject to some sort of deadlock where RCU's workqueue cannot start running until after a grace period completes, but that grace period is the one needing the workqueue? Note that there are ways to set up your kernel so that all RCU grace periods are expedited. Should I be worried? If not, what prevents this from being a problem, especially given that workqueue handlers are allowed to wait for RCU grace periods to complete? Thanx, Paul
Is it really safe to use workqueues to drive expedited grace periods?
Hello! So RCU's expedited grace periods have been using workqueues for a little while, and things seem to be working. But as usual, I worry... Is this use subject to some sort of deadlock where RCU's workqueue cannot start running until after a grace period completes, but that grace period is the one needing the workqueue? Note that there are ways to set up your kernel so that all RCU grace periods are expedited. Should I be worried? If not, what prevents this from being a problem, especially given that workqueue handlers are allowed to wait for RCU grace periods to complete? Thanx, Paul
