Re: [m5-dev] does drain need to be so complicated?
So I think I figured out the right way to handle drain, but I don't know how soon I'll have time to get to it, so I wanted to write it down and send it out for feedback, in case someone else has time to do it, and so I don't forget what I came up with. So here's the idea: Basically the C++ mechanisms all hang off of SimObject, so any SimObject can serve as the root of a subtree that gets drained. The callback pointer that gets passed to each SimObject's drain() method is just the pointer to the root of the hierarchy that's being drained; when the object wants to invoke the callback, it calls a specific new method on that root SimObject (like notifyDrainComplete() or something) that replaces the current DrainEvent::process() method. As I mentioned before, this logically requires an extra int per SimObject so that the SimObject can maintain the drain count while it's acting as root. Given that this is not performance critical, I suggest we have a global data structure that's logically a map from SimObject* to int. Then when we initiate a drain we just allocate a SimObject*,int pair for the root and use that as the drain counter, and we can deallocate it when the drain is over. In the near term since we don't support concurrent drains we could simplify this as just two global variables, the drain count and a SimObject* just to track the current root and print a warning if someone does manage to fire off two concurrent drains somehow. I'll even provide the diff for the Python side of the change: --- a/src/python/m5/simulate.py +++ b/src/python/m5/simulate.py @@ -147,15 +147,13 @@ # be drained. def drain(root): all_drained = False -drain_event = internal.event.createCountedDrain() -unready_objs = sum(obj.drain(drain_event) for obj in root.descendants()) +unready_objs = sum(obj.drain(root) for obj in root.descendants()) # If we've got some objects that can't drain immediately, then simulate if unready_objs 0: -drain_event.setCount(unready_objs) +root.setDrainCount(unready_objs) simulate() else: all_drained = True -internal.event.cleanupCountedDrain(drain_event) return all_drained (Not counting all the CountedDrainEvent swig stuff that can get whacked.) One nice side-effect is that we don't have to expose any particular callback or event object to python for it to allocate dynamically... whatever dynamic allocation happens is hidden behind the SimObject interface. (I know Nate will say this isn't hard, but I still feel like we should be keeping the python/c++ interface as narrow and clean as possible.) Another nice thing about this setup is that once a node knows that it is the root of a drain, it can easily detect if a drain is started somewhere further up the hierarchy, and in the long run maybe we'd want to do something to combine the two drain operations dynamically. We don't need to figure out the details yet, but I feel like this puts us in a good position to handle it intelligently if we need to. Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] does drain need to be so complicated?
On Mon, Dec 27, 2010 at 5:11 AM, nathan binkert n...@binkert.org wrote: I don't think we always drain on a per-System basis though... e.g., when we checkpoint, don't we do a drain on the entire config? And in that case there are things like Ethernet links that may need to be drained but aren't necessarily below any System object in the hierarchy. Yes, but when we switch over in a two system setting, I don't believe that we drain the non-switching. I think we're talking past each other here... I agree (and always have) that we sometimes drain system A but not system B, and that we want to keep that ability. Right here I'm just saying, in response to Ali's proposal, that there are also times when we drain the entire configuration, and in those cases, only being able to drain on a per-system basis is inadequate because there could be SimObjects that need draining outside of any system. I'd also honestly worry that you could have two independent drains going on if there were sampling on systems. It seems that making all systems wait for the slowest drainer would be a bad idea. Thinking forward to multithreaded simulation, drain would be annoying as a global. Finally, if we're going to eventually have to drain on functional accesses (or writes) for ruby, that seems bad in a multisystem setting. One system doing a syscall should not affect another system as this would really be confusing when trying to understand performance. Good points... I had forgotten about the drain-for-functional-accesses proposal. That's probably enough to kill the global right there. One option would be to have a set of SimObject classes that can serve as the root of a draining subtree, and include both System and Root in that set. Or we could have every SimObject have that potential at the cost of an extra int per SimObject. I don't have a strong opinion on this. It would certainly be nice to cut down on the amount of data, but not critical. I guess I don't have a good feeling about how difficult any of this is going to be. It seems to me that the global is a bit rough for either multithreaded or multisystem simulations or for functional drain requirements. Yea, I think it's either some per-SimObject thing or wrapping the callback, I think. One danger this brings up is how do we handle potential drains that overlap in both the set of objects they want to drain and in time? E.g., what if while I'm draining system A for a mode switch, I hit some other event that wants to initiate a checkpoint so it kicks off a config-wide drain? I don't know that we need to handle this, but we should at least detect it and block one of them, if we're serious about allowing concurrent drains. Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] does drain need to be so complicated?
On Fri, Dec 24, 2010 at 7:01 PM, nathan binkert n...@binkert.org wrote: I'm wondering why we need to dynamically allocate this object at all... the only benefit I see is that in theory, we could have multiple disjoint parts of the system draining concurrently but with independent completion detection. In reality, I don't see how we would really do that, unless we had multiple python threads calling the python drain() function concurrently, which doesn't seem likely to happen in our lifetimes. Is anyone opposed to just using a global variable for the drain counter, and a global function to decrement it and exit the sim loop when it hits zero? I don't think a global variable will work because if you have two systems, they may not drain at the same time. I think that a drain count on the system object could work fine though. That's what I was referring to... you can indeed have different parts of the config (e.g., different systems) draining at different times, but that's compatible with a global variable as long as those independent drain operations don't overlap in time. That is, the global variable can be used by just a subset of the config, the only constraint is that you can only have one drain operation in process at a time. Given that the python drain() function doesn't return until the drain operation is complete, it's not even possible to express multiple concurrent drain operations in a script right now. Even if it were expressible, I cannot envision a scenario where it would be useful. Thus I'm thinking that the current setup is overkill. Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] does drain need to be so complicated?
On Dec 26, 2010, at 10:40 PM, Steve Reinhardt wrote: On Fri, Dec 24, 2010 at 7:01 PM, nathan binkert n...@binkert.org wrote: I'm wondering why we need to dynamically allocate this object at all... the only benefit I see is that in theory, we could have multiple disjoint parts of the system draining concurrently but with independent completion detection. In reality, I don't see how we would really do that, unless we had multiple python threads calling the python drain() function concurrently, which doesn't seem likely to happen in our lifetimes. Is anyone opposed to just using a global variable for the drain counter, and a global function to decrement it and exit the sim loop when it hits zero? I don't think a global variable will work because if you have two systems, they may not drain at the same time. I think that a drain count on the system object could work fine though. That's what I was referring to... you can indeed have different parts of the config (e.g., different systems) draining at different times, but that's compatible with a global variable as long as those independent drain operations don't overlap in time. That is, the global variable can be used by just a subset of the config, the only constraint is that you can only have one drain operation in process at a time. Given that the python drain() function doesn't return until the drain operation is complete, it's not even possible to express multiple concurrent drain operations in a script right now. Even if it were expressible, I cannot envision a scenario where it would be useful. Thus I'm thinking that the current setup is overkill. Why not just have a drain object be part of the System class? It seems like that should solve the problem and allow to concurrent drain events to occur if we decided that functionality was required at some point. Ali ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] does drain need to be so complicated?
Why not just have a drain object be part of the System class? It seems like that should solve the problem and allow to concurrent drain events to occur if we decided that functionality was required at some point. I don't think we always drain on a per-System basis though... e.g., when we checkpoint, don't we do a drain on the entire config? And in that case there are things like Ethernet links that may need to be drained but aren't necessarily below any System object in the hierarchy. One option would be to have a set of SimObject classes that can serve as the root of a draining subtree, and include both System and Root in that set. Or we could have every SimObject have that potential at the cost of an extra int per SimObject. Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] does drain need to be so complicated?
Long story, but CountedDrainEvent is bugging me (again)... it's not an event (it never gets scheduled; it's just a callback) and it derives unnecessarily from SimLoopExitEvent (it just uses the inherited cause and code fields to set the cause and code fields of a new SimLoopExitEvent it creates when it's really ready to exit). The resulting code is very confusing since it's not what it appears to be. One solution would be to make it a Callback object instead of an Event; this would at least make the code match the usage and purpose. However, that's not totally trivial, since the current event is exposed through swig to python so that the count value can be set from the python code. The result is that that simple change requires more swig and python tweaking than you might expect. It's not that much :) I'm wondering why we need to dynamically allocate this object at all... the only benefit I see is that in theory, we could have multiple disjoint parts of the system draining concurrently but with independent completion detection. In reality, I don't see how we would really do that, unless we had multiple python threads calling the python drain() function concurrently, which doesn't seem likely to happen in our lifetimes. Is anyone opposed to just using a global variable for the drain counter, and a global function to decrement it and exit the sim loop when it hits zero? I don't think a global variable will work because if you have two systems, they may not drain at the same time. I think that a drain count on the system object could work fine though. Nate ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
[m5-dev] does drain need to be so complicated?
Long story, but CountedDrainEvent is bugging me (again)... it's not an event (it never gets scheduled; it's just a callback) and it derives unnecessarily from SimLoopExitEvent (it just uses the inherited cause and code fields to set the cause and code fields of a new SimLoopExitEvent it creates when it's really ready to exit). The resulting code is very confusing since it's not what it appears to be. One solution would be to make it a Callback object instead of an Event; this would at least make the code match the usage and purpose. However, that's not totally trivial, since the current event is exposed through swig to python so that the count value can be set from the python code. The result is that that simple change requires more swig and python tweaking than you might expect. I'm wondering why we need to dynamically allocate this object at all... the only benefit I see is that in theory, we could have multiple disjoint parts of the system draining concurrently but with independent completion detection. In reality, I don't see how we would really do that, unless we had multiple python threads calling the python drain() function concurrently, which doesn't seem likely to happen in our lifetimes. Is anyone opposed to just using a global variable for the drain counter, and a global function to decrement it and exit the sim loop when it hits zero? Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
