On Thu, Mar 5, 2009 at 1:48 PM, Steve Lawrence <stephen.lawre...@sun.com> wrote: > On Thu, Mar 05, 2009 at 01:22:25PM -0500, Jeff Victor wrote: >> On Thu, Mar 5, 2009 at 11:00 AM, Gael <gael.marti...@gmail.com> wrote: >> > On Wed, Mar 4, 2009 at 9:06 AM, Jeff Victor <jeff.j.vic...@gmail.com> >> > wrote: >> >> >> >> Some questions: >> >> 1. Do you use "set pool=" anymore, now that the dedicated-cpu feature >> >> exists? >> > >> It is now clear to me that this feature would need to support >> disabling interrupts when a zone uses "set pool=". Currently, all pool >> attributes are configured using the pool tools (poolcfg, pooladm) and >> I don't see any reason to not continue. When I write this up, it will >> fulfill that need. > > Ae you proposing that we add support for pset-interrupt disposition config > to the pools framework? Such as a property on a pool-pset > "boolean pset.interrupts = false"??
The short answer is "yes." BobN and I came to the same conclusion just a few hours ago... :-) CPUs already have cpu.status which can be on-line, no-intr (LWPs but no interrupt handlers), or off-line (no LWPs but still able to handle interrupts). A pset.interrupts field would allow Solaris to set cpu.status on CPUs as they enter the pset. Zones could then use that so we can increase their isolation. When a CPU re-enters the default pset, it becomes able to handle interrupts again. When needed, intrd will give it one (or more). > I think the right solution for "pool=" is this or similar. It could also > be a string value, such as: > > "none" no interrupts handled on cpus in the pool-pset. > "zone" Device interrupts for bound zones are serviced. > "any" Any device interrupts can be dispatched to the pset. I don't see how we could do "zone" in all situations - there isn't a 1:1 mapping between zone and device (except for exclusive-IP). Imagine zoneA and zoneB on a pset (psetAB) with pset.interrupts=zone. Further, zoneA and zoneC share e1000g0, but zoneB doesn't. Finally, zoneC has its own pset. Where does the interrupt handler for e1000g0 go - psetAB or psetC? Or are you suggesting that interrupts from one device can be intercepted and diverted to a CPU associated with a specific pset, based on which process the interrupt is/should be associated with? Or am I misunderstanding the description of "zone"? > Zonecfg could make use of these pool-pset properties to implement the > desired behavior for "dedicated-cpu". Exactly. > The default value should be "any". zonecfg should set "zone" for all > dedicated-cpu zones. zoneadm could warn if "pool=" is set, the zone has > dedicated devices, zone the pset for that pool has not been configured to > be "zone". The only devices we can be sure are dedicated for the boot-session of a zone are NICs. So this whole "segregate the interrupts per zone/pset combo" will be limited at best. It would be nice if we could generalize it like you say, but I don't think it's workable yet. > legacy psets (psrset) could be extended to support this property via some new > flags. > > Ther other part of this is how to reconsile zonecfg and/or pools settings > for interrupts, with device-cpu mappings that are specified via dladm. > Currently, dladm allows the specification of a list of cpu ids. Another > way to approach this would be to point dladm directly at the desired pool. Which "currently" are you on? :-) I'm on NV94 and I don't see anything like that in dladm(1M) I'm beginning to think this is really a two-phase project: * Phase 1: make it easier to disable interrupts on a zone's pset (one configured with the pool property or dedicated-cpu resource) * Phase 2: optimize this by enabling a zone's pset to handle interrupts from a device which is exclusively bound to this zone. I think that most people that need any of this only need Phase 1. Philosophically, shifting interrupt handlers into the default pset is consistent with the original zones principles: hardware is part of the platform, not part of a zone. So I'm not even convinced that we should be allowing zones' psets to selectively "attract" interrupt handlers. Great conversation! --JeffV >> >> 2. Is it sufficient to simply disable interrupts on a zone's pset? >> > >> > In our case, we do pset only when licensing requires it (aka >> > oracle,datastage,sybase,borland apps) or when the applications behave >> > poorly >> > and we keep hearing that by lack of budget/resources, the issue cannot be >> > addressed and without direct impact on the business itself, nothing will >> > change. >> >> Gael, I realized that my question was vague. When you use a pool, >> you're using a pset. Do you mean that you only use pools and psets >> when licensing requires it? >> >> Also, I couldn't tell how the comment responded to the question. >> >> > What about creating an IO pset, and then disabling the interrupt on >> > everything else while using it as a FSS pool or psets pools ? Very similar >> > to ldom I would think... >> >> Yes, that occurred to me, too. You can do that now, either with a pset >> that's being used by a zone or with the default pset. But I'm not >> convinced there's enough reason to separate an I/O pset from the >> default pset. There's great potential for wasted CPU cycles. >> >> >> --JeffV > - Show quoted text - >> _______________________________________________ >> zones-discuss mailing list >> zones-discuss@opensolaris.org > -- --JeffV _______________________________________________ zones-discuss mailing list zones-discuss@opensolaris.org
