+1 for ApplicationCluster. I think it is generic and not tied to a specific concept (machine,VM, service etc)
On Wed, Mar 1, 2017 at 1:54 AM, Daan Hoogland <daan.hoogl...@shapeblue.com> wrote: > Syed, Kilham, et. al. > > After some thought I came up with ApplicationCluster. It is not purely > machines or instances but includes network and storage resources and maybe > more. Next to that these are meant for running application like k8, mesos > or DBaaS. I don't like service as prefix because it implies the cluster is > for servicing the cloud or VMs that may be broken or need a kind of extra > feature while from user perspective they are an addition, hence application > to cloudstack. > > Any push back? > > Sent from Nine<http://www.9folders.com/> > ________________________________ > > daan.hoogl...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > From: Daan Hoogland > Sent: 28 Feb 2017 6:49 pm > To: dev@cloudstack.apache.org > Subject: Re: [PROPOSAL] add native vm-cluster orchestration service (was: > [PROPOSAL] add native container orchestration service) > > Syed, I chose machine as they might be bare metal in some cases. > > Sent from Nine<http://www.9folders.com/> > ________________________________ > From: Syed Ahmed <sah...@cloudops.com> > Sent: 28 Feb 2017 4:22 pm > To: dev@cloudstack.apache.org > Subject: Re: [PROPOSAL] add native vm-cluster orchestration service (was: > [PROPOSAL] add native container orchestration service) > > We already call the VMs as Instances. So, InstanceCluster would be a better > name imo. > > On Tue, Feb 28, 2017 at 10:05 AM, Daan Hoogland < > daan.hoogl...@shapeblue.com > > wrote: > > > Kishan, I see some sensible additions but also some unnecessary > omissions. > > Most of it seems to be Murali’s text so I’ll c&p your improvements back > and > > rename the page to the more sensible title of “MachineCluster service” > and > > delete the other. > > > > About naming, I was thinking MachineCluster instead of ServiceCluster, > > makes sense? Or even GuestMachineCluster. ServiceCluster could mean a > > supporting cluster that delivers e.g. backup as a service to guests, or > > maybe some build, artifact or networking service. For this ambiguity im > am > > :-1: on the name ServiceCluster. > > > > > > On 28/02/17 11:16, "Kishan Kavala" <kishan.kav...@accelerite.com> wrote: > > > > Daan, > > I've updated the earlier spec to support any Vm cluster. Please let > > me know your thoughts on this. > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/ > > Service+Cluster+Functional+Specification > > > > regards, > > Kishan > > > > -----Original Message----- > > From: Daan Hoogland [mailto:daan.hoogl...@shapeblue.com] > > Sent: 27 February 2017 04:02 PM > > To: dev@cloudstack.apache.org > > Subject: Re: [PROPOSAL] add native vm-cluster orchestration service > > (was: [PROPOSAL] add native container orchestration service) > > > > Any follow up Koushik? I want to refactor our proof of concept and > > integrate it in master. > > > > On 21/02/17 10:42, "Kishan Kavala" <kishan.kav...@accelerite.com> > > wrote: > > > > Sure Daan. I'll publish the design on cwiki and share the link. > > > > -----Original Message----- > > From: Daan Hoogland [mailto:daan.hoogl...@shapeblue.com] > > Sent: Monday, February 20, 2017 7:27 PM > > To: dev@cloudstack.apache.org > > Subject: [PROPOSAL] add native vm-cluster orchestration service > > (was: [PROPOSAL] add native container orchestration service) > > > > So, being very late in the discussion but havingread the whole > > thread before editting the title of this thread, > > > > Can we agree that we want a generic vm-cluster service and leave > > the container bits to containers? Kishan can you share your design? > > Shapeblue wants to rebase their k8 service on top of this and I would > like > > yours and Murali's work to not conflict. > > > > daan.hoogl...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com> > > 53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 > > VENetherlands @shapeblue > > > > > > > > > > -----Original Message----- > > From: Paul Angus [mailto:paul.an...@shapeblue.com] > > Sent: dinsdag 7 februari 2017 08:14 > > To: dev@cloudstack.apache.org > > Subject: Re: [PROPOSAL] add native container orchestration > service > > > > Will is 100% correct. As I mentioned the Title is misleading. > > However, Murali did clarify in his explanation; this is really about vm > > cluster orchestration. > > > > > > > > ________________________________ > > From: Will Stevens <wstev...@cloudops.com> > > Sent: 6 Feb 2017 22:54 > > To: dev@cloudstack.apache.org > > Subject: Re: [PROPOSAL] add native container orchestration > service > > > > My understanding is that what Paul is talking about is what is > > already built and IS what the thread is talking about. > > > > *Will STEVENS* > > Lead Developer > > > > <https://goo.gl/NYZ8KK> > > > > On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani < > > rajesh.ramchand...@accelerite.com> wrote: > > > > > Hi Paul - I think this is different from what the thread was > > about. > > > The conversation was specifically about adding support for > > container > > > orchestrators. You are talking about provisioning a group of > VMs. > > > Although, this is something I think several Cloudstack users > have > > > requested before and we should propose a solution to this. > > > > > > Raj > > > > > > ________________________________ > > > From: Paul Angus <paul.an...@shapeblue.com> > > > Sent: Monday, February 6, 2017 11:16:41 AM > > > To: dev@cloudstack.apache.org > > > Subject: RE: [PROPOSAL] add native container orchestration > > service > > > > > > #WhatHeSaid > > > > > > The title is misleading. The proposal is purely to add the > > notion of > > > Clusters of VMs to CloudStack. These may be for databases, > > containers > > > or anything else that needs 'clusters' of compute. Self-healing > > and > > > autoscaling are logical next steps to be added. > > > > > > Those guys at ShapeBlue have open-sourced their whole k8s > > container > > > service piece. If/when the 'cluster' part of that work is > added > > into > > > CloudStack, the k8s specific pieces can be used by anyone who > > wants > > > to, alternatively they could be used for reference in order to > > create > > > another types of cluster. (or ignored completely). > > > > > > > > > > > > > > > paul.an...@shapeblue.com > > > www.shapeblue.com<http://www.shapeblue.com> > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > > > > > > > > > > > -----Original Message----- > > > From: Will Stevens [mailto:williamstev...@gmail.com] > > > Sent: 31 January 2017 13:26 > > > To: dev@cloudstack.apache.org > > > Subject: Re: [PROPOSAL] add native container orchestration > > service > > > > > > s/cloud-init/cloud-config/ > > > > > > On Jan 31, 2017 7:24 AM, "Will Stevens" < > > williamstev...@gmail.com> wrote: > > > > > > > I think that is covered in this proposal. There is nothing > k8s > > > > specific in this integration (from what I understand), all > the > > k8s > > > > details are passed in via the cloud-init configuration after > > the > > > > cluster > > > has been provisioned. > > > > > > > > On Jan 31, 2017 3:06 AM, "Lianghwa Jou" > > > > <lianghwa....@accelerite.com> > > > > wrote: > > > > > > > > > > > > There are many container orchestrators. Those container > > > > orchestrators are happy to run on any VMs or bare metal > > machines. > > > > K8s is just one of them and there will be more in the future. > > It may > > > > not be a good idea to make CloudStack to be k8s aware. IMO, > the > > > > relationship between k8s and cloudstack should be similar to > > > > application and os. It probably not a good idea to make your > > OS to > > > > be aware of any specific applications so IMHO I don’t think > > k8s should be native to CloudStack. > > > > It makes more sense to provide some generic services that > many > > > > applications can take advantages of. Some generic resource > > grouping > > > > service makes sense so a group of VMs, baremetal machines or > > network > > > > can be treated as a single entity. Some life cycle management > > will > > > > be necessary for these entities too. We can deploy k8s, > swarm, > > dcos > > > > or even non-container specific services on top of CloudStack. > > The > > > > k8s is changing fast. One single tenant installation may need > > more > > > > than one VM group and may actually requires more (k8s > > federation). > > > > It will be a struggle to be in sync if we try to bring k8s > > specific > > > > knowledge into > > > cloudstack. > > > > > > > > Regards, > > > > > > > > > > > > -- > > > > Lianghwa Jou > > > > VP Engineering, Accelerite > > > > email: lianghwa....@accelerite.com > > > > > > > > > > > > > > > > > > > > > > > > On 1/29/17, 11:54 PM, "Murali Reddy" < > > murali.re...@shapeblue.com> wrote: > > > > > > > > > > > > I agree with some good views Will has shared and I also > > agree to > > > > the concerns raised by Wido and Eric. IMO we need balance of > > staying > > > > relevant/add new features vs stability of CloudStack and take > > > > corrective action if needed. We have two good examples for > > both. > > > > When SDN was hot technology CloudStack ended up with bunch of > > SDN > > > > controller > > > integrations. > > > > Few years later, now CloudStack is carrying baggage with no > > > > maintainers for those plugins, with probably not many of > > CloudStack > > > users needing overlays. > > > > On the other hand, other than attempt to liaison with ETSI > for > > NFV > > > > no effort was done. OpenStack has become de-facto for NFV. > Now > > for > > > > OpenStack, arguably NFV has become larger use case than cloud > > it self. > > > > I concur with Will’s point that with minimal viable solution > > that > > > > does not change the core of CloudStack, and can keep > CloudStack > > > > relevant is worth to be taken in. > > > > > > > > Will, > > > > > > > > To your question of how different is from ShapeBlue’s > > container > > > > service, its design, implementation and API semantics etc > > remain same. > > > > ShapeBlue’s container service was true drop in plug-in to > > > > CloudStack, with this proposal I am trying to re-work to make > > it a > > > > core CloudStack pluggable service which is part of > CloudStack. > > > > > > > > Key concepts that this proposal is trying to add > > > > > > > > - add notion of ‘container cluster’ as a first class > > entity > > > > in CloudStack. Which is bacially collection of other > CloudStack > > > > resources (like VM’s, networks, public ip, network rules etc) > > > > - life cycle operation to manage ‘container cluster’ > > like > > > > create, delete, start, stop, scale-up, scale-down, heal etc > > > > - orchestrate container orchestrator control plane on > > top of > > > > provisioned resources > > > > > > > > At-least for k8s (which is what this proposal is > > targeting), > > > > integration with k8s is bare minimum. There are cloud-config > > scripts > > > > that automatically setup k8s cluster master and node VM’s. > All > > > > CloudStack is doing in passing the cloud-config to the core > OS > > VM’s > > > > as > > > user data. > > > > > > > > Regards > > > > Murali Reddy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 29/01/17, 8:54 AM, "Will Stevens" < > > williamstev...@gmail.com > > > > on behalf of wstev...@cloudops.com> wrote: > > > > > > > > >I agree that we need to be careful what we take on and > > own inside > > > > >CloudStack. I feel like some of the plugins or > > integrations > > > > which we have > > > > >been "maintaining" may serve us better to abandon, but I > > feel > > > > like that is > > > > >a whole discussion on its own. > > > > > > > > > >In this case, I feel like there is a minimum viable > > solution > > > > which puts > > > > >CloudStack in a pretty good place to enable container > > orchestration. > > > > For > > > > >example, one of the biggest challenges with K8S is the > > fact > > > > that it > > > is > > > > >single tenant. CloudStack has good multi tenancy > support > > and > > > > has > > > the > > > > >ability to orchestrate the underlying infra quite well. > > We > > > > will have to be > > > > >very careful not to try to own too deep into the K8S > world > > > > though, in my > > > > >opinion. We only want to be responsible for providing > the > > > > infra (and a way > > > > >to bootstrap K8S ideally) and be able to scale the > infra, > > > > everything else > > > > >should be owned by the K8S on top. That is the way I > see > > it > > > > anyway, but > > > > >please add your input. > > > > > > > > > >I think it is a liability to try to go too deep, for the > > same > > > > reasons Wido > > > > >and Erik have mentioned. But I also think we need to > > take it > > > > seriously > > > > >because that train is moving and this may be a good > > opportunity > > > > to stay > > > > >relevant in a rapidly changing market. > > > > > > > > > >*Will STEVENS* > > > > >Lead Developer > > > > > > > > > ><https://goo.gl/NYZ8KK> > > > > > > > > > >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander > > > > <w...@widodh.nl> > > > > wrote: > > > > > > > > > >> > > > > >> > Op 27 januari 2017 om 16:08 schreef Will Stevens < > > > > wstev...@cloudops.com > > > > >> >: > > > > >> > > > > > >> > > > > > >> > Hey Murali, > > > > >> > How different is this proposal than what ShapeBlue > > already > > > > built. It > > > > >> looks > > > > >> > pretty consistent with the functionality that you > > guys open > > > > sourced in > > > > >> > Seville. > > > > >> > > > > > >> > I have not yet used this functionality, but I have > > reports > > > > that it works > > > > >> > quite well. > > > > >> > > > > > >> > I believe the premise here is to only orchestrate > the > > VM > > > > layer > > > and > > > > >> > basically expose a "group" of running VMs to the > > user. The > > > > user is > > > > >> > responsible for configuring K8S or whatever other > > container > > > > orchestrator > > > > >> on > > > > >> > top. I saw mention of the "cloud-config" scripts in > > the > > > > FS, how are > > > > >> those > > > > >> > exposed to the cluster? Maybe the FS can expand on > > that a bit? > > > > >> > > > > > >> > I believe the core feature that is being requested > to > > be > > > > added is the > > > > >> > ability to create a group of VMs which will be kept > > active > > > > as a group if > > > > >> at > > > > >> > all possible. ACS would be responsible for making > > sure > > > > that the number > > > > >> of > > > > >> > VMs specified for the group are in running state and > > it > > > > would spin up new > > > > >> > VMs as needed in order to satisfy the group > > settings. In > > > > general, it is > > > > >> > understood that any application running on this > group > > would > > > > have to be > > > > >> > fault tolerant enough to be able to rediscover a new > > VM if > > > > one fails and > > > > >> is > > > > >> > replaced by a fresh copy. Is that fair to say? How > > is it > > > > expected that > > > > >> > this service discovery is done, just by VMs being > > present > > > > on the network? > > > > >> > > > > > >> > As for some of the other people's concerns in this > > thread. > > > > >> > > > > > >> > - Regarding Wido's remarks. I understand that there > > is > > > > some > > > added > > > > >> > complexity, but I don't feel like the scope of the > > addition is > > > > >> > unrealistic. I think the LXC integration was a lot > > farther > > > > out of the > > > > >> > scope of what ACS does then this is. This does not > > change > > > > the "things" > > > > >> > which ACS orchestrates, it just adds the concept of > a > > > > grouping of things > > > > >> > which ACS already manages. I think this is the > right > > > > approach since it > > > > >> is > > > > >> > not trying to be a container orchestrator. We will > > never > > > > compete with > > > > >> K8S, > > > > >> > for example, and we should not try, but K8S is here > > and the > > > > market wants > > > > >> > it. I do think we should be keeping our head up > > about that > > > > fact because > > > > >> > being able to provide a the underlay for K8S is very > > > > valuable in the > > > > >> > current marketplace. I see this functionality as a > > way to > > > > enable K8S > > > > >> > adoption on top of ACS without changing our core > > values. > > > > >> > > > > > >> > - Regarding Erik's remarks. The container space is > > moving > > > > fast, but so > > > > >> is > > > > >> > the industry. If we want to remain relevant, we > need > > to be > > > > able to > > > > >> adapt a > > > > >> > bit. I don't think this is a big shift in what we > > do, but > > > > it is one that > > > > >> > enables people to be able to start running with > > something > > > > like K8S on top > > > > >> > of their existing ACS. This is something we are > > interested > > > > in doing and > > > > >> so > > > > >> > are our customers. If we can have a thin layer in > ACS > > > > which helps enable > > > > >> > the use of K8S (or other container orchestrators) by > > > orchestrating > > > > >> > infrastructure, as we already do, and making it > > easier to > > > > adopt > > > a > > > > >> container > > > > >> > orchestrator running on top of ACS, I think that > > gives us a > > > > nice foothold > > > > >> > in the market. I don't really feel it is fair to > > compare > > > > containers to > > > > >> > IPv6. IPv6 has been out forever and it has taken > > almost a > > > > decade to get > > > > >> > anyone to adopt it. Containers have really only > been > > here > > > > for like 2 > > > > >> years > > > > >> > and they are changing the market landscape in a very > > real way. > > > > >> > > > > > >> > Kind of on topic and kind of off topic. I think > > > > understanding > > > our > > > > >> approach > > > > >> > to containers is going to be important for the ACS > > > > community as a whole. > > > > >> > If we don't offer that market anything, then we will > > not be > > > > considered > > > > >> and > > > > >> > we will lose market share we can't afford to lose. > > If we > > > > try to hitch > > > > >> our > > > > >> > horse to that cart too much, we will not be able to > be > > > > agile enough and > > > > >> > will fail. I feel like the right approach is for us > > to > > > > know that it is a > > > > >> > thriving market and continue to do what we do, but > to > > > > extend an olive > > > > >> > branch to that market. I think this sort of > > implementation > > > > is the right > > > > >> > approach because we are not trying to do too much. > > We are > > > > simply giving > > > > >> a > > > > >> > foundation on which the next big thing in the > > container > > > > orchestration > > > > >> world > > > > >> > can adopt without us having to compete directly in > > that > > > > space. I think > > > > >> we > > > > >> > have to focus on what we do best, but at the same > > time, > > > > think about what > > > > >> we > > > > >> > can do to enable that huge market of users to adopt > > ACS as their > > > > >> > foundation. The ability to offer VMs and containers > > in the > > > > same data > > > > >> plane > > > > >> > is something we have the ability to do, especially > > with > > > > this approach, > > > > >> and > > > > >> > is something that most other softwares can not do. > > The > > > > adoption of > > > > >> > containers by bigger organizations will be only part > > of > > > > their workload, > > > > >> > they will still be running VMs for the foreseeable > > future. > > > > Being able to > > > > >> > appeal to that market is going to be important for > us. > > > > >> > > > > > >> > Hopefully I don't have too many strong opinions > here, > > but I > > > > do think we > > > > >> > need to be thinking about how we move forward in a > > world > > > > which > > > is > > > > >> adopting > > > > >> > containers in a very real way. > > > > >> > > > > > >> > > > > >> Understood. I just want to prevent that we add more > > features > > > > to CloudStack > > > > >> which are poorly maintained. Not judging Murali here > at > > all, > > > > but I'd rather > > > > >> see CloudStack loose code then have it added. > > > > >> > > > > >> Thinking about LXC, would like to see that removed > > together > > > > with various > > > > >> other network plugins which I think are rarely used. > > > > >> > > > > >> Now, the idea of Murali was misunderstood by me. I > > think it > > > > would be worth > > > > >> more if we would improve our cloud-init support and > > > > integration in various > > > > >> tools which makes it much easier to deploy VMs running > > > > containers > > > ON > > > > >> CloudStack. > > > > >> > > > > >> Not so much changing CloudStack code, but rather > tooling > > > > around > > > it. > > > > >> > > > > >> If we have CloudStack talking to Kubernetes we > suddenly > > have > > > > to maintain > > > > >> that API integration. Who's going to do that. > > Realistically. > > > > >> > > > > >> That's my main worry in this regard. > > > > >> > > > > >> We have so much more work to do in ACS in total that I > > don't > > > > know if this > > > > >> is the realistic route. I talk to many people who are > > not > > > > looking > > > at > > > > >> containers and are still working with VMs. > > > > >> > > > > >> There is not a single truth which is true, it really > > depends > > > > on who you > > > > >> ask. > > > > >> > > > > >> Wido > > > > >> > > > > >> > Cheers, > > > > >> > > > > > >> > Will > > > > >> > > > > > >> > *Will STEVENS* > > > > >> > Lead Developer > > > > >> > > > > > >> > <https://goo.gl/NYZ8KK> > > > > >> > > > > > >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber > > > > <terbol...@gmail.com> > > > > wrote: > > > > >> > > > > > >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy < > > > > muralimmre...@gmail.com > > > > >> > > > > > >> > > wrote: > > > > >> > > > All, > > > > >> > > > > > > > >> > > > I would like propose native functionality into > > > > CloudStack to provide > > > > >> a > > > > >> > > container service through which users out-of-the > > box can > > > > use to launch > > > > >> > > container based application. Idea is to support > > ability > > > > to orchestrate > > > > >> the > > > > >> > > resources and automate aspects of setting up > > container > > > > orchestrator > > > > >> through > > > > >> > > CloudStack. Public IAAS service providers AWS with > > its > > > > ECS [1] and > > > > >> google > > > > >> > > with GKE [2] already provides ability container > > applications. > > > > >> Competitive > > > > >> > > cloud orchestration platforms already have native > > support > > > > for container > > > > >> > > service. Users of CloudStack both as public cloud > > > > providers and users > > > > >> with > > > > >> > > private clouds will benefit with such > functionality. > > > > >> > > > > > > > >> > > > While container orchestrator of user choice can > be > > > > provisioned on > > > > >> top of > > > > >> > > CloudStack (with out CloudStack being involved) > with > > > > tools > > > like > > > > >> > > TerraForm[3], Ansible[4] etc, advantage of having > > native > > > > orchestration > > > > >> is > > > > >> > > giving user a nice cohesive integration. This > > proposal > > > > would like add a > > > > >> > > notion of first class CloudStack entity called > > container > > > > cluster which > > > > >> can > > > > >> > > be used to provision resources, scale up, scale > > down, > > > > start and stop > > > > >> the > > > > >> > > cluster of VM’s on which containerised > applications > > can > > > > be > > > run. > > > > For > > > > >> actual > > > > >> > > container orchestration we will still need > container > > > > orchestrator like > > > > >> > > docker swarm, marathon, kubernetes, but CloudStack > > > > container service > > > > >> can > > > > >> > > automate setting up of control place > automatically. > > > > >> > > > > > > > >> > > > > > > >> > > To be honest I'm torn on this one. > > > > >> > > > > > > >> > > Containers are a rapid changing thing, and while > > docker swam, > > > > >> > > kubernetes, rancher or whatnot is popular today, > > they > > > > might not be > > > > >> > > tomorrow. > > > > >> > > They might use CoreOS today, but might not > tomorrow. > > > > >> > > > > > > >> > > We have a rather poor track record of staying up > to > > date > > > > with new > > > > >> > > features/versions, and adding a feature that is so > > > > rapidly changing > > > > >> > > is, I fear, going to be hard to maintain. > > > > >> > > Want an example, look at xenserver. It is one of > > the most used > > > > >> > > hypervisors we support, yet it took 6 months or so > > for us > > > > to support > > > > >> > > the latest release. > > > > >> > > Or IPv6... > > > > >> > > > > > > >> > > I don't mean to bash at maintainers/implementers > of > > those > > > > features, I > > > > >> > > appreciate all the work being done in every > aspect, > > but I > > > > believe we > > > > >> > > should be realistic and realize that we have > issues > > with > > > > keeping stuff > > > > >> > > up to date. > > > > >> > > > > > > >> > > I'd say focus on making sure other tools can do > > their job > > > > well against > > > > >> > > CloudStack (kops, rancher, ++), but that does not > > mean I > > > > will > > > > -1 the > > > > >> > > idea if anyone really wants to go through with it. > > > > >> > > > > > > >> > > -- > > > > >> > > Erik > > > > >> > > > > > > >> > > > > > > > > murali.re...@shapeblue.com > > > > www.shapeblue.com<http://www.shapeblue.com> > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > > @shapeblue > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > DISCLAIMER > > > > ========== > > > > This e-mail may contain privileged and confidential > information > > > > which is the property of Accelerite, a Persistent Systems > > business. > > > > It is intended only for the use of the individual or entity > to > > which > > > > it is addressed. If you are not the intended recipient, you > > are not > > > > authorized to read, retain, copy, print, distribute or use > this > > > > message. If you have received this communication in error, > > please > > > > notify the sender and delete all copies of this message. > > Accelerite, > > > > a Persistent Systems business does not accept any liability > for > > > > virus > > > infected mails. > > > > > > > > > > > > > > > > > > > > > > > > DISCLAIMER > > > ========== > > > This e-mail may contain privileged and confidential information > > which > > > is the property of Accelerite, a Persistent Systems business. > It > > is > > > intended only for the use of the individual or entity to which > > it is > > > addressed. If you are not the intended recipient, you are not > > > authorized to read, retain, copy, print, distribute or use this > > > message. If you have received this communication in error, > please > > > notify the sender and delete all copies of this message. > > Accelerite, a > > > Persistent Systems business does not accept any liability for > > virus infected mails. > > > > > > > paul.an...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > > > > > > > > > > DISCLAIMER > > ========== > > This e-mail may contain privileged and confidential information > > which is the property of Accelerite, a Persistent Systems business. It is > > intended only for the use of the individual or entity to which it is > > addressed. If you are not the intended recipient, you are not authorized > to > > read, retain, copy, print, distribute or use this message. If you have > > received this communication in error, please notify the sender and delete > > all copies of this message. Accelerite, a Persistent Systems business > does > > not accept any liability for virus infected mails. > > > > > > > > daan.hoogl...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com> > > 53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands > > @shapeblue > > > > > > > > > > > > > > DISCLAIMER > > ========== > > This e-mail may contain privileged and confidential information which > > is the property of Accelerite, a Persistent Systems business. It is > > intended only for the use of the individual or entity to which it is > > addressed. If you are not the intended recipient, you are not authorized > to > > read, retain, copy, print, distribute or use this message. If you have > > received this communication in error, please notify the sender and delete > > all copies of this message. Accelerite, a Persistent Systems business > does > > not accept any liability for virus infected mails. > > > > > > > > daan.hoogl...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > > > > > daan.hoogl...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > >
