On Wed, Nov 24, 2021 at 9:05 PM Ishan Chattopadhyaya < [email protected]> wrote:
> > > On Wed, Nov 24, 2021 at 8:19 PM Gus Heck <[email protected]> wrote: > >> Left some additional comments in the google doc, extracting the key point >> for others here so they can know if they want to read the google doc >> commentary: >> > > I'm addressing them here, inline. > > >> There is presently disagreement with my proposal that the sip specify >> both how we track configuration (which I have been referring to as >> "capability") and runtime state ("providing"). I see the primary value of >> having a role concept as one of organizing the code and configuration in a >> way that is easy to understand, which is why I'm in favor of this SiP >> overall. >> > > I purposely don't want to complicate this design by allowing for > "capability" vs "currently providing" to be represented as first class > citizens. Besides the OVERSEER role (preferred overseer), I can't think of > concrete scenarios where such a concept could be applicable, and hence I > don't want to get ahead of myself in trying to address how to solve that. > Since the current proposal for dealing with roles is flexible and generic > enough, I don't think adding such extensions would be a problem. However, I > don't want to go down that rabbithole at this point in time. > > > >> >> If you don't want to tackle implementing the runtime state bit I'm ok >> with that as long as we have a clear plan in the SIP of how this type of >> information should be organized in the future. >> The structure proposed in the SIP for information in zk seems to map more >> clearly to runtime state than configuration, and seems like it would >> inhibit a natural representation of such. >> > > The structure proposed allows for role specific configurations (if > needed). It doesn't provide for node specific configuration in ZK, since > all node properties should come from either solr.xml or sysprops. At > present, I don't think we ever have node specific info *configuration* in > ZK in any part of Solr, and I think we shouldn't go that route here. > Correction: "At present, I don't think we ever have node specific *configuration* in ZK in any part of Solr, and I think we shouldn't go that route here." > > >> >> Finally the structure in the SIP for zk information doesn't specify if >> some of the nodes are ephemeral, and reliable for a "current" list or if >> they are persistent over time. I feel like we may not yet have a plan for >> how much information about nodes (n general) should be retained while a >> node is down, which maybe needs to underlie this design. >> > > All the node names under the roles znodes are ephemeral. Added this to the > document (confluence and gdoc both). > > >> >> -Gus >> >> >> On Tue, Nov 23, 2021 at 10:34 PM Ishan Chattopadhyaya < >> [email protected]> wrote: >> >>> I've synced back all changes to confluence. Both represent the same >>> document as of now. >>> Please review and suggest if there are any outstanding concerns. Thanks >>> for all the feedback. >>> >>> I wish to proceed with the implementation based on lazy consensus again. >>> >>> On Tue, Nov 23, 2021 at 9:38 PM Mike Drob <[email protected]> wrote: >>> >>>> I should clarify, that my -1 was specifically about reaching lazy >>>> consensus and not about that proposal itself. >>>> >>>> On Tue, Nov 23, 2021 at 9:42 AM Ishan Chattopadhyaya < >>>> [email protected]> wrote: >>>> >>>>> > -1, I would like to see a proposal on list where it will be >>>>> permanently available >>>>> >>>>> It will be available in the confluence document (as it currently is). >>>>> The google docs is just for collecting temporary feedback, thanks to easy >>>>> inline commenting capability. We promised to consolidate feedback in the >>>>> gdoc and sync back to the SIP document in confluence. This is reasonable >>>>> enough expectations, and shouldn't be a worry enough for a -1. >>>>> >>>>> As of right now, while trying to address some concerns, some edits >>>>> have been made in gdocs by Noble that we'll sync back into confluence; so >>>>> that's the only divergence. >>>>> >>>>> On Tue, Nov 23, 2021 at 9:09 PM Ishan Chattopadhyaya < >>>>> [email protected]> wrote: >>>>> >>>>>> > -1, I would like to see a proposal on list where it will be >>>>>> permanently available >>>>>> > in the archives instead of being directed to a Google doc which can >>>>>> be edited at any point in time. >>>>>> > This has been an incredibly difficult proposal to keep track of. >>>>>> >>>>>> This entire SIP process has been incredibly difficult to coordinate. >>>>>> The idea of doing this on a mail thread was a terrible one. There's no >>>>>> archive available publicly for this list that doesn't break the threading >>>>>> functionality (I guess someone among us is using some funky mail client >>>>>> that's messing with Pony Mail). JIRA for discussion would've been much >>>>>> better. I suppose JIRA used to have threaded discussions once upon a >>>>>> time, >>>>>> that would've been very handy here. >>>>>> >>>>>> On Tue, Nov 23, 2021 at 9:07 PM Ishan Chattopadhyaya < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Okay, sure, I'll rescind my assertion about having reached a lazy >>>>>>> consensus. I'll work through all the recent comments and feedback. >>>>>>> I think at the last moment, there was an edit by Noble to the Google >>>>>>> Doc that hasn't made it into the SIP document, and I'll review that >>>>>>> change, >>>>>>> the recent comments and get back here tomorrow. Other than that last >>>>>>> moment >>>>>>> change, the SIP document is up to date. >>>>>>> >>>>>>> On Tue, Nov 23, 2021 at 8:24 PM Gus Heck <[email protected]> wrote: >>>>>>> >>>>>>>> Also, I don't know that it's defined anywhere, but I feel that >>>>>>>> declaring lazy consensus on a SIP in less than a week is rushing >>>>>>>> things. >>>>>>>> Some folks won't have the flexibility to address things daily and I've >>>>>>>> been >>>>>>>> uncomfortably pulled away from paid work trying to keep up with this >>>>>>>> as is. >>>>>>>> Lazy consensus should include at least one weekend (IMHO) >>>>>>>> >>>>>>>> On Tue, Nov 23, 2021 at 9:49 AM Gus Heck <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> IMHO two things must happen before lazy consensus wait period can >>>>>>>>> begin: >>>>>>>>> 1) The wiki must be updated to reflect the results of discussion >>>>>>>>> in the google doc. >>>>>>>>> 2) The fact that the wiki has been updated, and that you think >>>>>>>>> the discussion is at an end must be posted here, preferably with a >>>>>>>>> summary >>>>>>>>> on the list. >>>>>>>>> >>>>>>>>> If it didn't happen on the list it didn't happen. >>>>>>>>> >>>>>>>>> Also the google doc has not been notifying me of >>>>>>>>> changes/responses. So a note here when you've responded and some time >>>>>>>>> to >>>>>>>>> respond before the above steps are taken would be appreciated. >>>>>>>>> >>>>>>>>> -Gus >>>>>>>>> >>>>>>>>> On Tue, Nov 23, 2021 at 9:40 AM Mike Drob <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> -1, I would like to see a proposal on list where it will be >>>>>>>>>> permanently available in the archives instead of being directed to a >>>>>>>>>> Google >>>>>>>>>> doc which can be edited at any point in time. >>>>>>>>>> >>>>>>>>>> This has been an incredibly difficult proposal to keep track of. >>>>>>>>>> >>>>>>>>>> Mike >>>>>>>>>> >>>>>>>>>> On Tue, Nov 23, 2021 at 6:09 AM Ishan Chattopadhyaya < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> This proposal has passed with lazy consensus. We can proceed to >>>>>>>>>>> the implementation phase. >>>>>>>>>>> Thanks to everyone for feedback, esp. Gus for the patience. >>>>>>>>>>> >>>>>>>>>>> On Sun, Nov 21, 2021 at 2:24 PM Gus Heck <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks for adding that :) and Thanks to Nobel for translating >>>>>>>>>>>> my ravings :). Sorry about the acronym. Amusingly I had to look up >>>>>>>>>>>> gish >>>>>>>>>>>> gallop to understand what you meant :) I can assure you that I >>>>>>>>>>>> will never >>>>>>>>>>>> gish gallop you or anyone on the list, and I find such techniques >>>>>>>>>>>> repugnant. If I'm not making sense, I'm likely covering ground too >>>>>>>>>>>> fast, >>>>>>>>>>>> skipping things that are in my head but not yours, expressing too >>>>>>>>>>>> many >>>>>>>>>>>> tangents (when I start nesting parentheses this is (usually) a >>>>>>>>>>>> sign I'm >>>>>>>>>>>> wandering too much), or I am simply mistaken about something. >>>>>>>>>>>> Always feel >>>>>>>>>>>> free to ask me to explain! >>>>>>>>>>>> >>>>>>>>>>>> -Gus >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Nov 21, 2021 at 12:17 AM Ishan Chattopadhyaya < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Added both your suggestions to Google Docs for inline >>>>>>>>>>>>> commenting and the SIP document at confluence. Thanks for the >>>>>>>>>>>>> feedback. >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Nov 21, 2021 at 10:28 AM Ishan Chattopadhyaya < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I'll add the lifecycle details to the document as well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, 21 Nov, 2021, 9:08 am Ishan Chattopadhyaya, < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Gus, >>>>>>>>>>>>>>> Thanks for bringing it up again. Initially, I wasn't clear >>>>>>>>>>>>>>> what you meant and mistook them for gish gallop. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These were the things I am/was confused about (Noble >>>>>>>>>>>>>>> explained it to me what you mean): >>>>>>>>>>>>>>> - adherence to DRY via zk >>>>>>>>>>>>>>> - runtime config info should come from zk >>>>>>>>>>>>>>> - add dynamic features >>>>>>>>>>>>>>> - same interaction patterns for consulting config values. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Now, after Noble's help, I think I understand your >>>>>>>>>>>>>>> motivations. >>>>>>>>>>>>>>> - DRY = Don't Repeat Yourself ;-) >>>>>>>>>>>>>>> - You're suggesting that we have a standard way to have role >>>>>>>>>>>>>>> specific configurations (in ZK), so that a new role added later >>>>>>>>>>>>>>> can access >>>>>>>>>>>>>>> the configurations in a standard way >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a very good suggestion, and makes complete sense >>>>>>>>>>>>>>> now. This was definitely a missing piece! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Here's a proposal to address that (adding it to Google Docs: >>>>>>>>>>>>>>> https://docs.google.com/document/d/1hijvM1WX9u2TOUdLEkFYVCofLeJlv2MRZqe-ncobJVw/edit >>>>>>>>>>>>>>> ): >>>>>>>>>>>>>>> - Per Role ZNode >>>>>>>>>>>>>>> - Every role's ZNode to hold configuration data (this is >>>>>>>>>>>>>>> the key addition that you're asking for, IIUC) >>>>>>>>>>>>>>> - Children of role ZNodes to be list of ephemeral solr nodes >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We can iterate over this on the Google Docs (internal >>>>>>>>>>>>>>> representation ZK section) with inline commenting if we need to. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please let us know how that sounds. >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Ishan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sun, Nov 21, 2021 at 4:49 AM Gus Heck <[email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It is difficult to track everything in this thread for >>>>>>>>>>>>>>>> sure. Specifically, I don't feel my email of Wed, Nov 17, 7:43 >>>>>>>>>>>>>>>> PM (EST) has >>>>>>>>>>>>>>>> been addressed or responded to except for one difference of >>>>>>>>>>>>>>>> opinion with >>>>>>>>>>>>>>>> Jan on whether or not we should prohibit the notion of any >>>>>>>>>>>>>>>> role ever >>>>>>>>>>>>>>>> changing at runtime. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Note that I made a specific proposal for an addition to the >>>>>>>>>>>>>>>> sip, regarding start up lifecycle and adherence to DRY via zk, >>>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>>> (probably not clearly expressed) hope that we can move to a >>>>>>>>>>>>>>>> general overall >>>>>>>>>>>>>>>> application principle that at runtime config info should come >>>>>>>>>>>>>>>> from zk. If >>>>>>>>>>>>>>>> we follow such a principle, we will always be in a position to >>>>>>>>>>>>>>>> add dynamic >>>>>>>>>>>>>>>> features without significant rework, and all code could >>>>>>>>>>>>>>>> hopefully reuse the >>>>>>>>>>>>>>>> same interaction patterns for consulting config values. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Neither item is addressed in the SIP currently. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Also I had put that out there as a single item so it could >>>>>>>>>>>>>>>> be focused on (simplifying the discussion I hope), and >>>>>>>>>>>>>>>> depending on how >>>>>>>>>>>>>>>> discussion proceeds, I may have other follow on items. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Gus >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sat, Nov 20, 2021 at 8:45 AM Ishan Chattopadhyaya < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, 19 Nov, 2021, 7:03 pm Ilan Ginzburg, < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Is the request here for everybody to express again the >>>>>>>>>>>>>>>>>> concerns >>>>>>>>>>>>>>>>>> already expressed in this email thread and not addressed? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I think the expectation now (after we've expressed our >>>>>>>>>>>>>>>>> intention to reach a lazy consensus) is that if someone wants >>>>>>>>>>>>>>>>> to block this >>>>>>>>>>>>>>>>> SIP from adoption, then to put forward the objections. Sort >>>>>>>>>>>>>>>>> of like a veto. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I suggest instead the authors review the thread, match >>>>>>>>>>>>>>>>>> expressed >>>>>>>>>>>>>>>>>> concerns with how the concern was addressed (or not >>>>>>>>>>>>>>>>>> addressed) and >>>>>>>>>>>>>>>>>> provide an exhaustive list. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I've summarized most of the discussion in the SIP >>>>>>>>>>>>>>>>> document. If you feel I could do a better job with it, please >>>>>>>>>>>>>>>>> help me with >>>>>>>>>>>>>>>>> areas of improvement. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This proposal in its current form (data and overseer >>>>>>>>>>>>>>>>>> roles) doesn't >>>>>>>>>>>>>>>>>> offer much that can't be reasonably achieved by other >>>>>>>>>>>>>>>>>> means. I'd find >>>>>>>>>>>>>>>>>> much more value in making sure what is done now is a >>>>>>>>>>>>>>>>>> solid foundation >>>>>>>>>>>>>>>>>> for the future. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Fair enough, I understand your perspective. Thanks for >>>>>>>>>>>>>>>>> your feedback. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Ilan >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, Nov 18, 2021 at 11:24 PM Noble Paul < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > After so many back and forth mails, I just can't say >>>>>>>>>>>>>>>>>> who has an outstanding concern and if they are already >>>>>>>>>>>>>>>>>> addressed or not. I >>>>>>>>>>>>>>>>>> think the Google doc would help us get clarity on that. >>>>>>>>>>>>>>>>>> Please take a >>>>>>>>>>>>>>>>>> moment to give your inputs >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > On Fri, Nov 19, 2021, 9:18 AM Ishan Chattopadhyaya < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> Apologies, the vote hasn't passed formally and I was >>>>>>>>>>>>>>>>>> under some confusion on the process. >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> I'd like to proceed with a lazy consensus and proceed >>>>>>>>>>>>>>>>>> to the implementation phase now. >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> However, I would appreciate it if someone wants to >>>>>>>>>>>>>>>>>> bring out any outstanding concerns about the SIP document. >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> To facilitate in-line comments, here's a temporary >>>>>>>>>>>>>>>>>> Google Docs version of this document. >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1hijvM1WX9u2TOUdLEkFYVCofLeJlv2MRZqe-ncobJVw/edit?usp=sharing >>>>>>>>>>>>>>>>>> >> (I shall copy changes back to confluence eventually) >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> Thanks and apologies again regarding the confusion >>>>>>>>>>>>>>>>>> with the voting, >>>>>>>>>>>>>>>>>> >> Regards, >>>>>>>>>>>>>>>>>> >> Ishan >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> On Thu, Nov 18, 2021 at 9:50 PM Ishan Chattopadhyaya < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> The SIP passed the voting phase. Thanks for all for >>>>>>>>>>>>>>>>>> the feedback and insights. >>>>>>>>>>>>>>>>>> >>> Looking forward to your collaboration and reviews as >>>>>>>>>>>>>>>>>> we implement this. >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> On Thu, Nov 18, 2021 at 9:42 PM Ishan Chattopadhyaya < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> > It's fine if we don't provide any ability for >>>>>>>>>>>>>>>>>> runtime modification of roles at this time but I'm leery of >>>>>>>>>>>>>>>>>> precluding it >>>>>>>>>>>>>>>>>> in the future. >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> In future, the necessity for such a facility can >>>>>>>>>>>>>>>>>> dictate our course of action. We cannot lay down rules cast >>>>>>>>>>>>>>>>>> in stone for >>>>>>>>>>>>>>>>>> functionality that we can't foresee yet. >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> On Thu, Nov 18, 2021 at 9:40 PM Ishan Chattopadhyaya >>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> Thanks Jan, I added both those points to the SIP >>>>>>>>>>>>>>>>>> document in the Notes section. >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> On Thu, Nov 18, 2021 at 7:18 PM Jan Høydahl < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> 18. nov. 2021 kl. 01:43 skrev Gus Heck < >>>>>>>>>>>>>>>>>> [email protected]>: >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> 2) Roles will not be checked by loading config >>>>>>>>>>>>>>>>>> from disk or caching disk config in memory. (zk ONLY source >>>>>>>>>>>>>>>>>> of truth) >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> It sounds a bit backward for a local node to first >>>>>>>>>>>>>>>>>> parse solr.node.roles, determine its local set of roles, >>>>>>>>>>>>>>>>>> then publish them >>>>>>>>>>>>>>>>>> to Zookeeper, and then read back its own roles from ZK. >>>>>>>>>>>>>>>>>> >>>>>> Code that only needs to determine "Do I have the >>>>>>>>>>>>>>>>>> XXX role?" or find out "What roles do I have" should be able >>>>>>>>>>>>>>>>>> to fetch the >>>>>>>>>>>>>>>>>> (static) roles from some roles utility class without >>>>>>>>>>>>>>>>>> consulting ZK. >>>>>>>>>>>>>>>>>> >>>>>> Code that needs to check what nodes have a certain >>>>>>>>>>>>>>>>>> role (such as placement) would obviously need ot consult ZK. >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> Perhaps the SIP should also state some Non-goals >>>>>>>>>>>>>>>>>> or assertions such as >>>>>>>>>>>>>>>>>> >>>>>> * Roles are static and immutable (also in zk) for >>>>>>>>>>>>>>>>>> the entire life cycle of a node >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> I also think we should state that the bar for >>>>>>>>>>>>>>>>>> adding new roles should be high so it is not abused as any >>>>>>>>>>>>>>>>>> other tag or >>>>>>>>>>>>>>>>>> label for any tiny feature. It should be reserved for >>>>>>>>>>>>>>>>>> functionality that >>>>>>>>>>>>>>>>>> may benefit from a dedicated set of nodes. That may be clear >>>>>>>>>>>>>>>>>> already, but >>>>>>>>>>>>>>>>>> you never know... >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> Jan >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>>>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>>>>>>>>>>> http://www.the111shift.com (play) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>>>>>>> http://www.the111shift.com (play) >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>>>> http://www.the111shift.com (play) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> http://www.needhamsoftware.com (work) >>>>>>>> http://www.the111shift.com (play) >>>>>>>> >>>>>>> >> >> -- >> http://www.needhamsoftware.com (work) >> http://www.the111shift.com (play) >> >
