Re: Clustering
Eric Shubert wrote: > Has anyone here implemented any clusters? > Is any particular distro better or worse at clustering? > Any pointers regarding clustering you'd like to share? Thanks for everyone's input. -- -Eric 'shubes' --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
Re: Clustering
SuperMicro has a number of servers built with infiniband on-baord in 1u, twin 1u and the like... not sureif they have onbaord infiniband with 4 pci-e 16x slots... On Tue, Jun 30, 2009 at 1:14 AM, Michael Butash wrote: > Exactly as Nadeem said, start local, and if you need to expand, look at > infiniband. Buying IB gear directly from the vendor, supported, > warranted, etc, will get expensive quick. You'll probably find > programming info fairly obscure, as most of this is very proprietary, as > supercomputing is a cash cow to certain vars. > > Obscurity seems to make it cheap on the secondary market, as most app > dev's won't/don't take it to that level, and not many really need it. > Tesla's change things as they provide an extensible way to increase > generic crunching local, and can scale significantly with pcie bandwith > and hardware with enough slots. > > For me IB was interesting as I was investigating upping my house to > gige, fiber-channel, and other things to geek out on. IB actually > looked fairly promising as it's device stack allows for IP/FC > emululation layer, as well as it's own native socket stack where > parallel processing plays into. Then I found out it was cheaper than > buying decent 4g fiber channel gear off ebay, and got even more > interesting. Might provide some ideal scalability if you need to expand > the processing outside of one box eventually. > > -mb > > > On Tue, 2009-06-30 at 00:35 -0700, Stephen wrote: >> Are you looking at the 1u Tesla or the x58 personal supercomputer they >> are talking about... >> >> I would look into RH mostly for compatability fo ahrdware as it is >> tested by them, and for most packages... but the distro is really >> dependant on your app. >> >> Man they have really changed tesla, it used to be a bunch of gpu cards >> in a box chained to a header and then 16 of those can be tied together >> to a single machine. >> >> but now you can buy a preconfigured clsuter >> >> http://www.penguincomputing.com/products/linux/server/web_promotion/tesla_workgroup_cluster >> >> man i stop following for just 1 year... >> >> >> On Mon, Jun 29, 2009 at 6:29 PM, Eric Shubert wrote: >> > I'm looking at clustering together a handful of hosts, each running dual >> > nvidia tesla cards. Modeling applications of some sort. I honestly don't >> > know much more than that. >> > >> > Michael Butash wrote: >> >> You're probably talking infiniband switching, infiniband hba's, >> >> pci-e/htx interfaces, fiber channel disk arrays, etc. Linux seems to >> >> support infiniband hba's reasonably well, and 10g 4x infiniband hba's >> >> tend to be cheap these days on ebay. We're talking $100 used hba's for >> >> the nodes, and ~$1200 for a 12 port Cisco/Topspin switch. I thought >> >> about buying some to play with, as it ends up cheaper than IP or Fiber >> >> channel technologies, yet can replace them all to some extents. >> >> >> >> IB is quite versatile, emulating fiber-channel, IP network, or raw >> >> interrupt switching to a cpu and memory via different driver socket >> >> interface api's. Cray always used a similar means to make theirs with >> >> proprietary north bridge and software, but IB is more of a standard now, >> >> enabling (relatively) cheap supercomputing on the fly with commodity >> >> hardware. Well, hardware-wise at least... >> >> >> >> So yeah, what apps are you talking about utilizing it? >> >> >> >> -mb >> >> >> >> >> >> On Mon, 2009-06-29 at 13:47 -0700, Matt Graham wrote: >> >>> From: Eric Shubert >> Has anyone here implemented any clusters? >> >>> I've only set one up, but I maintain the ones that my predecessors >> >>> set up. It's not rocket science. >> >>> >> Is any particular distro better or worse at clustering? >> >>> Not really. Every distro has heartbeat/DRBD/LVS available. >> >>> >> Any pointers regarding clustering you'd like to share? >> >>> Define the problem you're trying to solve more rigorously than just >> >>> "clustering" first. Do you want flailover between 2 boxes? Do you >> >>> want a frontend box with N service-providing boxes behind it? The >> >>> answers to that greatly affect what you will end up doing, as does >> >>> the question "What services is this cluster going to provide?" >> >>> >> >>> Basically, all I can do is handwave without answers to "what services?" >> >>> and "how many machines?". >> >>> >> >> >> > >> > >> > -- >> > -Eric 'shubes' >> > >> > --- >> > PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us >> > To subscribe, unsubscribe, or to change your mail settings: >> > http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss >> > >> >> >> > > --- > PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us > To subscribe, unsubscribe, or to change your mail settings: > http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss > -- A mouse trap, placed on top
Re: Clustering
Exactly as Nadeem said, start local, and if you need to expand, look at infiniband. Buying IB gear directly from the vendor, supported, warranted, etc, will get expensive quick. You'll probably find programming info fairly obscure, as most of this is very proprietary, as supercomputing is a cash cow to certain vars. Obscurity seems to make it cheap on the secondary market, as most app dev's won't/don't take it to that level, and not many really need it. Tesla's change things as they provide an extensible way to increase generic crunching local, and can scale significantly with pcie bandwith and hardware with enough slots. For me IB was interesting as I was investigating upping my house to gige, fiber-channel, and other things to geek out on. IB actually looked fairly promising as it's device stack allows for IP/FC emululation layer, as well as it's own native socket stack where parallel processing plays into. Then I found out it was cheaper than buying decent 4g fiber channel gear off ebay, and got even more interesting. Might provide some ideal scalability if you need to expand the processing outside of one box eventually. -mb On Tue, 2009-06-30 at 00:35 -0700, Stephen wrote: > Are you looking at the 1u Tesla or the x58 personal supercomputer they > are talking about... > > I would look into RH mostly for compatability fo ahrdware as it is > tested by them, and for most packages... but the distro is really > dependant on your app. > > Man they have really changed tesla, it used to be a bunch of gpu cards > in a box chained to a header and then 16 of those can be tied together > to a single machine. > > but now you can buy a preconfigured clsuter > > http://www.penguincomputing.com/products/linux/server/web_promotion/tesla_workgroup_cluster > > man i stop following for just 1 year... > > > On Mon, Jun 29, 2009 at 6:29 PM, Eric Shubert wrote: > > I'm looking at clustering together a handful of hosts, each running dual > > nvidia tesla cards. Modeling applications of some sort. I honestly don't > > know much more than that. > > > > Michael Butash wrote: > >> You're probably talking infiniband switching, infiniband hba's, > >> pci-e/htx interfaces, fiber channel disk arrays, etc. Linux seems to > >> support infiniband hba's reasonably well, and 10g 4x infiniband hba's > >> tend to be cheap these days on ebay. We're talking $100 used hba's for > >> the nodes, and ~$1200 for a 12 port Cisco/Topspin switch. I thought > >> about buying some to play with, as it ends up cheaper than IP or Fiber > >> channel technologies, yet can replace them all to some extents. > >> > >> IB is quite versatile, emulating fiber-channel, IP network, or raw > >> interrupt switching to a cpu and memory via different driver socket > >> interface api's. Cray always used a similar means to make theirs with > >> proprietary north bridge and software, but IB is more of a standard now, > >> enabling (relatively) cheap supercomputing on the fly with commodity > >> hardware. Well, hardware-wise at least... > >> > >> So yeah, what apps are you talking about utilizing it? > >> > >> -mb > >> > >> > >> On Mon, 2009-06-29 at 13:47 -0700, Matt Graham wrote: > >>> From: Eric Shubert > Has anyone here implemented any clusters? > >>> I've only set one up, but I maintain the ones that my predecessors > >>> set up. It's not rocket science. > >>> > Is any particular distro better or worse at clustering? > >>> Not really. Every distro has heartbeat/DRBD/LVS available. > >>> > Any pointers regarding clustering you'd like to share? > >>> Define the problem you're trying to solve more rigorously than just > >>> "clustering" first. Do you want flailover between 2 boxes? Do you > >>> want a frontend box with N service-providing boxes behind it? The > >>> answers to that greatly affect what you will end up doing, as does > >>> the question "What services is this cluster going to provide?" > >>> > >>> Basically, all I can do is handwave without answers to "what services?" > >>> and "how many machines?". > >>> > >> > > > > > > -- > > -Eric 'shubes' > > > > --- > > PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us > > To subscribe, unsubscribe, or to change your mail settings: > > http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss > > > > > --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
Re: Clustering
Are you looking at the 1u Tesla or the x58 personal supercomputer they are talking about... I would look into RH mostly for compatability fo ahrdware as it is tested by them, and for most packages... but the distro is really dependant on your app. Man they have really changed tesla, it used to be a bunch of gpu cards in a box chained to a header and then 16 of those can be tied together to a single machine. but now you can buy a preconfigured clsuter http://www.penguincomputing.com/products/linux/server/web_promotion/tesla_workgroup_cluster man i stop following for just 1 year... On Mon, Jun 29, 2009 at 6:29 PM, Eric Shubert wrote: > I'm looking at clustering together a handful of hosts, each running dual > nvidia tesla cards. Modeling applications of some sort. I honestly don't > know much more than that. > > Michael Butash wrote: >> You're probably talking infiniband switching, infiniband hba's, >> pci-e/htx interfaces, fiber channel disk arrays, etc. Linux seems to >> support infiniband hba's reasonably well, and 10g 4x infiniband hba's >> tend to be cheap these days on ebay. We're talking $100 used hba's for >> the nodes, and ~$1200 for a 12 port Cisco/Topspin switch. I thought >> about buying some to play with, as it ends up cheaper than IP or Fiber >> channel technologies, yet can replace them all to some extents. >> >> IB is quite versatile, emulating fiber-channel, IP network, or raw >> interrupt switching to a cpu and memory via different driver socket >> interface api's. Cray always used a similar means to make theirs with >> proprietary north bridge and software, but IB is more of a standard now, >> enabling (relatively) cheap supercomputing on the fly with commodity >> hardware. Well, hardware-wise at least... >> >> So yeah, what apps are you talking about utilizing it? >> >> -mb >> >> >> On Mon, 2009-06-29 at 13:47 -0700, Matt Graham wrote: >>> From: Eric Shubert Has anyone here implemented any clusters? >>> I've only set one up, but I maintain the ones that my predecessors >>> set up. It's not rocket science. >>> Is any particular distro better or worse at clustering? >>> Not really. Every distro has heartbeat/DRBD/LVS available. >>> Any pointers regarding clustering you'd like to share? >>> Define the problem you're trying to solve more rigorously than just >>> "clustering" first. Do you want flailover between 2 boxes? Do you >>> want a frontend box with N service-providing boxes behind it? The >>> answers to that greatly affect what you will end up doing, as does >>> the question "What services is this cluster going to provide?" >>> >>> Basically, all I can do is handwave without answers to "what services?" >>> and "how many machines?". >>> >> > > > -- > -Eric 'shubes' > > --- > PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us > To subscribe, unsubscribe, or to change your mail settings: > http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss > -- A mouse trap, placed on top of your alarm clock, will prevent you from rolling over and going back to sleep after you hit the snooze button. Stephen --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
Re: Clustering
Since u are using tesla I would suggest first that u put 4 gpus in each cluster and make sure you have at least a quad core each. If you are going with modeling applications than go with infiniband, but if you are running hpc/parallel computing stuff than u probably need only gigabit or 10 gb ethernet the reason being is that the gpus will need hours to compute. I hope this might help. Nadim Hoque Cell: 480-518-6235 Address: 6302 West Kent Drive Chandler, Arizona 85226 Sent via BlackBerry from T-Mobile -Original Message- From: Eric Shubert Date: Mon, 29 Jun 2009 18:29:11 To: Subject: Re: Clustering I'm looking at clustering together a handful of hosts, each running dual nvidia tesla cards. Modeling applications of some sort. I honestly don't know much more than that. Michael Butash wrote: > You're probably talking infiniband switching, infiniband hba's, > pci-e/htx interfaces, fiber channel disk arrays, etc. Linux seems to > support infiniband hba's reasonably well, and 10g 4x infiniband hba's > tend to be cheap these days on ebay. We're talking $100 used hba's for > the nodes, and ~$1200 for a 12 port Cisco/Topspin switch. I thought > about buying some to play with, as it ends up cheaper than IP or Fiber > channel technologies, yet can replace them all to some extents. > > IB is quite versatile, emulating fiber-channel, IP network, or raw > interrupt switching to a cpu and memory via different driver socket > interface api's. Cray always used a similar means to make theirs with > proprietary north bridge and software, but IB is more of a standard now, > enabling (relatively) cheap supercomputing on the fly with commodity > hardware. Well, hardware-wise at least... > > So yeah, what apps are you talking about utilizing it? > > -mb > > > On Mon, 2009-06-29 at 13:47 -0700, Matt Graham wrote: >> From: Eric Shubert >>> Has anyone here implemented any clusters? >> I've only set one up, but I maintain the ones that my predecessors >> set up. It's not rocket science. >> >>> Is any particular distro better or worse at clustering? >> Not really. Every distro has heartbeat/DRBD/LVS available. >> >>> Any pointers regarding clustering you'd like to share? >> Define the problem you're trying to solve more rigorously than just >> "clustering" first. Do you want flailover between 2 boxes? Do you >> want a frontend box with N service-providing boxes behind it? The >> answers to that greatly affect what you will end up doing, as does >> the question "What services is this cluster going to provide?" >> >> Basically, all I can do is handwave without answers to "what services?" >> and "how many machines?". >> > -- -Eric 'shubes' --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
Re: Clustering
I'm looking at clustering together a handful of hosts, each running dual nvidia tesla cards. Modeling applications of some sort. I honestly don't know much more than that. Michael Butash wrote: > You're probably talking infiniband switching, infiniband hba's, > pci-e/htx interfaces, fiber channel disk arrays, etc. Linux seems to > support infiniband hba's reasonably well, and 10g 4x infiniband hba's > tend to be cheap these days on ebay. We're talking $100 used hba's for > the nodes, and ~$1200 for a 12 port Cisco/Topspin switch. I thought > about buying some to play with, as it ends up cheaper than IP or Fiber > channel technologies, yet can replace them all to some extents. > > IB is quite versatile, emulating fiber-channel, IP network, or raw > interrupt switching to a cpu and memory via different driver socket > interface api's. Cray always used a similar means to make theirs with > proprietary north bridge and software, but IB is more of a standard now, > enabling (relatively) cheap supercomputing on the fly with commodity > hardware. Well, hardware-wise at least... > > So yeah, what apps are you talking about utilizing it? > > -mb > > > On Mon, 2009-06-29 at 13:47 -0700, Matt Graham wrote: >> From: Eric Shubert >>> Has anyone here implemented any clusters? >> I've only set one up, but I maintain the ones that my predecessors >> set up. It's not rocket science. >> >>> Is any particular distro better or worse at clustering? >> Not really. Every distro has heartbeat/DRBD/LVS available. >> >>> Any pointers regarding clustering you'd like to share? >> Define the problem you're trying to solve more rigorously than just >> "clustering" first. Do you want flailover between 2 boxes? Do you >> want a frontend box with N service-providing boxes behind it? The >> answers to that greatly affect what you will end up doing, as does >> the question "What services is this cluster going to provide?" >> >> Basically, all I can do is handwave without answers to "what services?" >> and "how many machines?". >> > -- -Eric 'shubes' --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
Re: Clustering
You're probably talking infiniband switching, infiniband hba's, pci-e/htx interfaces, fiber channel disk arrays, etc. Linux seems to support infiniband hba's reasonably well, and 10g 4x infiniband hba's tend to be cheap these days on ebay. We're talking $100 used hba's for the nodes, and ~$1200 for a 12 port Cisco/Topspin switch. I thought about buying some to play with, as it ends up cheaper than IP or Fiber channel technologies, yet can replace them all to some extents. IB is quite versatile, emulating fiber-channel, IP network, or raw interrupt switching to a cpu and memory via different driver socket interface api's. Cray always used a similar means to make theirs with proprietary north bridge and software, but IB is more of a standard now, enabling (relatively) cheap supercomputing on the fly with commodity hardware. Well, hardware-wise at least... So yeah, what apps are you talking about utilizing it? -mb On Mon, 2009-06-29 at 13:47 -0700, Matt Graham wrote: > From: Eric Shubert > > Has anyone here implemented any clusters? > > I've only set one up, but I maintain the ones that my predecessors > set up. It's not rocket science. > > > Is any particular distro better or worse at clustering? > > Not really. Every distro has heartbeat/DRBD/LVS available. > > > Any pointers regarding clustering you'd like to share? > > Define the problem you're trying to solve more rigorously than just > "clustering" first. Do you want flailover between 2 boxes? Do you > want a frontend box with N service-providing boxes behind it? The > answers to that greatly affect what you will end up doing, as does > the question "What services is this cluster going to provide?" > > Basically, all I can do is handwave without answers to "what services?" > and "how many machines?". > --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
Re: Clustering
Eric Shubert wrote: > Has anyone here implemented any clusters? > Is any particular distro better or worse at clustering? > Any pointers regarding clustering you'd like to share? > In my experience, the base distro doesn't matter much unless the apps you plan on running have specific requirements. I have run a couple hundred node batch cluster (not much if any IPC on the jobs). During the time I managed that system we used PBS Pro (Commercial IIRC, there is OpenPBS as an alternative), torque (free, open source), maui, and Moab(commercial). The torque was pretty straight forward. The general concept is you have a cluster manager that tracks resources (resource manager, like torque) and schedules jobs (like pbs_sched that comes with torque or maui/moab) and then each node has an agent that gets managed by that cluster manager. Typically there is shared storage between nodes but that is not required. http://www.clusterresources.com/products/torque-resource-manager.php http://www.clusterresources.com/torquedocs21/ http://www.clusterresources.com/torquedocs21/a.ltorquequickstart.shtml There were some other options we looked at and briefly tested. But I was most fond of torque/maui combination IIRC. Moab seemed to be overkill for our needs at the time. Austin --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
Re: Clustering
The new Ubuntu server is supposed to make it child's play to set up, but I have not had an opportunity to play with it yet. On Mon, Jun 29, 2009 at 1:32 PM, Eric Shubert wrote: > Has anyone here implemented any clusters? > Is any particular distro better or worse at clustering? > Any pointers regarding clustering you'd like to share? > -- > -Eric 'shubes' > > --- > PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us > To subscribe, unsubscribe, or to change your mail settings: > http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss > --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
Re: Clustering
Matt Graham wrote: > From: Eric Shubert >> Has anyone here implemented any clusters? > > I've only set one up, but I maintain the ones that my predecessors > set up. It's not rocket science. > >> Is any particular distro better or worse at clustering? > > Not really. Every distro has heartbeat/DRBD/LVS available. > >> Any pointers regarding clustering you'd like to share? > > Define the problem you're trying to solve more rigorously than just > "clustering" first. Do you want flailover between 2 boxes? Do you > want a frontend box with N service-providing boxes behind it? The > answers to that greatly affect what you will end up doing, as does > the question "What services is this cluster going to provide?" > > Basically, all I can do is handwave without answers to "what services?" > and "how many machines?". > I'm talking about a supercomputer sort of thing. Not failover. Not redundancy. A group of machines (let's say 6 for example) acting as a single number crunching behemoth. Something that can deliver tens of teraflops. -- -Eric 'shubes' --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
Re: Clustering
the only cluster i messed with was a Beowulf cluster. it worked (i think) but i was unsure how to really make it do more than talk to itself. in any case id look at various features of the distro first (GFS support, cluster management tools and application to be used) i think ubuntu has a cloud computing app comparable with or related to the amazon S3 stuff. and GFS is a RH technology but available all over. On Mon, Jun 29, 2009 at 1:47 PM, Matt Graham wrote: > From: Eric Shubert >> Has anyone here implemented any clusters? > > I've only set one up, but I maintain the ones that my predecessors > set up. It's not rocket science. > >> Is any particular distro better or worse at clustering? > > Not really. Every distro has heartbeat/DRBD/LVS available. > >> Any pointers regarding clustering you'd like to share? > > Define the problem you're trying to solve more rigorously than just > "clustering" first. Do you want flailover between 2 boxes? Do you > want a frontend box with N service-providing boxes behind it? The > answers to that greatly affect what you will end up doing, as does > the question "What services is this cluster going to provide?" > > Basically, all I can do is handwave without answers to "what services?" > and "how many machines?". > > -- > Matt G / Dances With Crows > The Crow202 Blog: http://crow202.org/wordpress/ > There is no Darkness in Eternity/But only Light too dim for us to see > > > --- > PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us > To subscribe, unsubscribe, or to change your mail settings: > http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss > -- A mouse trap, placed on top of your alarm clock, will prevent you from rolling over and going back to sleep after you hit the snooze button. Stephen --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
Re: Clustering
From: Eric Shubert > Has anyone here implemented any clusters? I've only set one up, but I maintain the ones that my predecessors set up. It's not rocket science. > Is any particular distro better or worse at clustering? Not really. Every distro has heartbeat/DRBD/LVS available. > Any pointers regarding clustering you'd like to share? Define the problem you're trying to solve more rigorously than just "clustering" first. Do you want flailover between 2 boxes? Do you want a frontend box with N service-providing boxes behind it? The answers to that greatly affect what you will end up doing, as does the question "What services is this cluster going to provide?" Basically, all I can do is handwave without answers to "what services?" and "how many machines?". -- Matt G / Dances With Crows The Crow202 Blog: http://crow202.org/wordpress/ There is no Darkness in Eternity/But only Light too dim for us to see --- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
