[PROPOSAL] Support pure Xen as a hypervisor

[Tim Mackey]

Historically CloudStack has used Xen and XenServer interchangeably to refer
to any XenAPI based implementation.  With the recent release of Xen Project
4.4 (http://blog.xen.org/index.php/2014/03/10/xen-4-4-released/), and
interest in alternate architectures like ARM, the loose definition of our
Xen support could be confusing.  In this two part effort I propose that
CloudStack 4.4 be cleansed to ensure that all Xen references become
XenServer references, and second that an alternate hypervisor type of
"XenProject" be introduced for pure Xen which could either support libvirt
or  libxl (preference for libvirt given the 4.4 work to improve the
interface and broader support for libvirt in general).

Cross posted to users to for broader comment.

-tim

Re: Cloudstack support for Xen (on Ubuntu)

[Tim Mackey]

Tracy,

XCP itself was made EOL last year in favor of XenServer. You can of course
still continue to use XCP 1.6, but there will be no future releases and
development is now XenServer based.  If the dom0 distro matters, you should
take a look at the xenserver-core work going on.

-tim


On Fri, Apr 11, 2014 at 12:09 PM, Tracy Phillips <
tracy.phill...@weberize.com> wrote:

> Rafael,
>
> Once you put XCP on, do you gain features that belong to Xenserver or is it
> limited in someway?
>
> The only reason I am not using Xenserver is that I have to manage it
> differently than my other hosts.
>
> Tracy
>
> Tracy Phillips
> Weberize, Inc.
>
>
> On Fri, Apr 11, 2014 at 8:26 AM, Rafael Weingartner <
> rafaelweingart...@gmail.com> wrote:
>
> > True that, just xen hypervisor is not yet supported. However, if you also
> > install the XCP over xen hypervisor it will work.
> >
> >
> > On Fri, Apr 11, 2014 at 7:05 AM, Nux!  wrote:
> >
> > > On 11.04.2014 10:10, Pradeep Cloudstack wrote:
> > >
> > >> I have a server running Ubuntu 13.04. I am planning to install Xen
> > >> Hypervisor on that
> > >> (https://help.ubuntu.com/community/XenProposed#Xen_and_XAPI)
> > >>
> > >> Can I then have it managed (as a host) by Cloudstack 4.2 ?
> > >>
> > >
> > > Not yet, there are plans to make it happen, but Xen is not yet
> supported.
> > >
> > > Lucian
> > >
> > > --
> > > Sent from the Delta quadrant using Borg technology!
> > >
> > > Nux!
> > > www.nux.ro
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>

Re: [ACS4.5] move from xen 2 xenserver

[Tim Mackey]

This was a little more than a straight renaming, and I'll post all the
changes to the wiki for review in the morning.  At a high level this patch
did the following:

- Move the old plugins/hypervisors/xen to /plugins/hypervisors/xenserver
and change all namespaces to reflect the move (bulk of the work)
- Change any DDL/DML containing "xen"/"Xen" to become "XenServer" (took
care of both create and upgrade paths)
- Change any display areas containing "Xen" to become "XenServer"

iirc there was one API change in the network label.  It was
xennetworklabel, and I updated it to become xenservernetworklabel.  Since I
don't want to break backwards compatibility either, I'm open to how best to
handle the situation.  Also would like to understand a test case I can use
to validate.

-tim


On Tue, Apr 15, 2014 at 6:27 AM, Sebastien Goasguen wrote:

>
> On Apr 15, 2014, at 4:46 AM, Stephen Turner 
> wrote:
>
> > As I said in the previous threaed, I'm +1 on the principle. It will
> avoid confusion between straight Xen and XenServer. It will also allow us
> to offer a non-XenServer Xen implementation.
> >
> > Sebastien, as all the filenames have changed, it's not clear from the
> diff whether this is just a straight renaming with no other changes. Could
> you confirm that?
> >
>
> That's what it looks like, basically it creates a 'xenserver' plugin
> I know Tim has a prototype of Xen support as well, but it was not in this
> commit it seems.
>
>
> > Also, are there any backwards compatibility implications for the API? Or
> did we already use "XenServer" instead of "Xen" there?
> >
>
> In the CloudStack API ?
>
> We are probably using Xen as value in several api calls…so we will need to
> check this carefully before any merge, we definitely don't want to break
> backward compatibility, or if we do we will need to move to 5.0
>
> > --
> > Stephen Turner
> >
> >
> > -----Original Message-
> > From: Sebastien Goasguen [mailto:run...@gmail.com]
> > Sent: 15 April 2014 08:36
> > To: dev@cloudstack.apache.org
> > Subject: [ACS4.5] move from xen 2 xenserver
> >
> > Folks,
> >
> > I just applied a patch from Tim Mackey:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commit;h=748b575af8a66c58b0d52e7457e4d4e1e875231f
> >
> > commit: 748b575af8a66c58b0d52e7457e4d4e1e875231f
> >
> > This followed a [PROPOSAL] thread [1]
> >
> > This was pushed into a separate branch: xen2server
> >
> > This is a significant change that we should get consensus on and merge
> to master relatively quickly to avoid any conflicts later on.
> >
> > Basically, we have been treating xen and xenserver the same so far,
> since the integration with Xen (i.e Xen Project) was/is done via xapi.
> >
> > There has been discussion to integrate with Xen using straight up
> libvirt, by creating a new Xen agent probably based on the KVM agent and
> there was some discussion to that effect in the Denver Hackathon.
> >
> > Making a clear split between Xen and Xenserver type of hypervisors will
> help support different integration methods.
> >
> > I have asked Tim (in the review message) to create a wiki page,
> discussing all of this, but I wanted to give a heads up that this just hit
> the repo and that we should see a [MERGE] thread quickly.
> >
> > thoughts, flames ?
> >
> >
> > [1] http://markmail.org/thread/yrl3ii7gqlaaexij
> >
> > -Sebastien
> >
>
>

Re: [ACS4.5] move from xen 2 xenserver

[Tim Mackey]

Stephen,

I'm going to look into how to construct a test case for that, and here is
the wiki page covering the feature:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Convert+Xen+usage+to+XenServer

Please feel free to suggest additional test cases, and I'll add them to the
list.

-tim


On Wed, Apr 16, 2014 at 5:20 AM, Stephen Turner
wrote:

> Thanks, Tim, that's helpful. My question about API backwards compatibility
> was also whether any API arguments or return values are of the form "Xen"
> currently and would become "XenServer" after this change, for example
> because of changes in some parsing code.
>
> --
> Stephen Turner
>
>
> -Original Message-
> From: Tim Mackey [mailto:tmac...@gmail.com]
> Sent: 16 April 2014 01:31
> To: dev@cloudstack.apache.org
> Subject: Re: [ACS4.5] move from xen 2 xenserver
>
> This was a little more than a straight renaming, and I'll post all the
> changes to the wiki for review in the morning.  At a high level this patch
> did the following:
>
> - Move the old plugins/hypervisors/xen to /plugins/hypervisors/xenserver
> and change all namespaces to reflect the move (bulk of the work)
> - Change any DDL/DML containing "xen"/"Xen" to become "XenServer" (took
> care of both create and upgrade paths)
> - Change any display areas containing "Xen" to become "XenServer"
>
> iirc there was one API change in the network label.  It was
> xennetworklabel, and I updated it to become xenservernetworklabel.  Since I
> don't want to break backwards compatibility either, I'm open to how best to
> handle the situation.  Also would like to understand a test case I can use
> to validate.
>
> -tim
>
>
> On Tue, Apr 15, 2014 at 6:27 AM, Sebastien Goasguen  >wrote:
>
> >
> > On Apr 15, 2014, at 4:46 AM, Stephen Turner
> > 
> > wrote:
> >
> > > As I said in the previous threaed, I'm +1 on the principle. It will
> > avoid confusion between straight Xen and XenServer. It will also allow
> > us to offer a non-XenServer Xen implementation.
> > >
> > > Sebastien, as all the filenames have changed, it's not clear from
> > > the
> > diff whether this is just a straight renaming with no other changes.
> > Could you confirm that?
> > >
> >
> > That's what it looks like, basically it creates a 'xenserver' plugin I
> > know Tim has a prototype of Xen support as well, but it was not in
> > this commit it seems.
> >
> >
> > > Also, are there any backwards compatibility implications for the
> > > API? Or
> > did we already use "XenServer" instead of "Xen" there?
> > >
> >
> > In the CloudStack API ?
> >
> > We are probably using Xen as value in several api calls…so we will
> > need to check this carefully before any merge, we definitely don't
> > want to break backward compatibility, or if we do we will need to move
> > to 5.0
> >
> > > --
> > > Stephen Turner
> > >
> > >
> > > -Original Message-
> > > From: Sebastien Goasguen [mailto:run...@gmail.com]
> > > Sent: 15 April 2014 08:36
> > > To: dev@cloudstack.apache.org
> > > Subject: [ACS4.5] move from xen 2 xenserver
> > >
> > > Folks,
> > >
> > > I just applied a patch from Tim Mackey:
> > >
> > >
> > https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commit;h=74
> > 8b575af8a66c58b0d52e7457e4d4e1e875231f
> > >
> > > commit: 748b575af8a66c58b0d52e7457e4d4e1e875231f
> > >
> > > This followed a [PROPOSAL] thread [1]
> > >
> > > This was pushed into a separate branch: xen2server
> > >
> > > This is a significant change that we should get consensus on and
> > > merge
> > to master relatively quickly to avoid any conflicts later on.
> > >
> > > Basically, we have been treating xen and xenserver the same so far,
> > since the integration with Xen (i.e Xen Project) was/is done via xapi.
> > >
> > > There has been discussion to integrate with Xen using straight up
> > libvirt, by creating a new Xen agent probably based on the KVM agent
> > and there was some discussion to that effect in the Denver Hackathon.
> > >
> > > Making a clear split between Xen and Xenserver type of hypervisors
> > > will
> > help support different integration methods.
> > >
> > > I have asked Tim (in the review message) to create a wiki page,
> > discussing all of this, but I wanted to give a heads up that this just
> > hit the repo and that we should see a [MERGE] thread quickly.
> > >
> > > thoughts, flames ?
> > >
> > >
> > > [1] http://markmail.org/thread/yrl3ii7gqlaaexij
> > >
> > > -Sebastien
> > >
> >
> >
>

Hypervisor version and XenServer

[Tim Mackey]

I'm running through some bugs with my Xen->XenServer work and just ran
across the HYPERVISOR_VERSION being the Xen version and not the XenServer
version.  Does anyone know why that is?  Given that feature/function in
XenServer is tied to the XenServer version, I see using the Xen version as
a bug waiting to bite us.  Since I don't want to break API compatability,
I'm not inclined to change it for this work, but do wonder if we shouldn't
change it when the API revs next.

Thoughts?

-tim

Re: Hypervisor version and XenServer

[Tim Mackey]

It's an ApiConstant (ApiConstants.HYPERVISOR_VERSION), and is exposed in a
number of places such as the HostForMigration, HypervisorCapabilities and
GuestOSMapping responses.  It's set for XenServer in the
XcpServerDiscoverer, and for XenServer 6.2 it really says the hypervisor
version is 4.1.5, not 6.2.0.  Once I've gotten everything working in the
Xen->XenServer work, I'll look a bit deeper into how we use it internally
(if at all).

-tim


On Wed, Apr 23, 2014 at 11:11 PM, Yitao Jiang  wrote:

> cloud.host table contains hypervisor_version column.
> Hopes can help.
>
>
> ---
> Thanks,
> Yitao
> jiangyt.github.io
>
>
> On Thu, Apr 24, 2014 at 7:22 AM, Chiradeep Vittal <
> chiradeep.vit...@citrix.com> wrote:
>
> > Where is this HYPERVISOR_VERSION? In the code? Docs?
> >
> > From: Tim Mackey mailto:tmac...@gmail.com>>
> > Reply-To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
> <
> > dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> > Date: Wednesday, April 23, 2014 at 2:50 PM
> > To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <
> > dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> > Subject: Hypervisor version and XenServer
> >
> > I'm running through some bugs with my Xen->XenServer work and just ran
> > across the HYPERVISOR_VERSION being the Xen version and not the XenServer
> > version.  Does anyone know why that is?  Given that feature/function in
> > XenServer is tied to the XenServer version, I see using the Xen version
> as
> > a bug waiting to bite us.  Since I don't want to break API compatability,
> > I'm not inclined to change it for this work, but do wonder if we
> shouldn't
> > change it when the API revs next.
> >
> > Thoughts?
> >
> > -tim
> >
> >
>

Merge Review Request 22270: Xen 2 XenServer refactoring

[Tim Mackey]

I've just submitted a review request which is essentially a merge of
the xen2server feature branch back into master.  Since this is a
refactoring of the Xen plugin to make it more explicitly a XenServer
plugin per the feature:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Convert+Xen+usage+to+XenServer,
I wanted to ensure everyone was aware of what is changing. Diff
details can be found here: https://reviews.apache.org/r/22270/

The key item in this change is that what was the xen plugin has now
moved to become the xenserver plugin to make way for a pure xen
plugin.  If you are working on something which is XenServer specific,
you'll probably want to take a look at what I've done, sanity check it
against your plans and ask questions.  Additionally, if you've done
some work in XenServer code since the start of April, you might want
to make certain I didn't accidentally revert any of your changes
during conflict resolution.  I tried to be careful, but accidents do
happen.

The merge point was commit 603eab7 (HEAD yesterday), and from there I
did a bunch of sanity testing using XenServer 6.2.  I also tested and
validated with the current HEAD (8b5ec64).  If you were looking at or
testing anything on the xen2server branch, no new functionality was
introduced from that branch; this is effectively a merge with
conflicts resolved.

Thanks

-tim

Re: [DISCUSS] Increasing VM IOPS by separating golden image in high IOPS partition in Xen Server ?

[Tim Mackey]

Hieu,

If I understand the objective correctly, you are trying to reduce the
IO associated with a desktop "start of day" boot storm.  In your
proposal, you're effectively wanting to move the CloudStack secondary
storage concept to include a locally attached storage device which is
SSD based.  While that seems viable in concept, in practice with
XenServer your proposed flow could cause a bunch of issues.  Some of
the challenges I see include:

- XenServer hosts with multiple independent local storage are very
rare.  See this KB article covering how to create such storage:
http://support.citrix.com/article/CTX121313
- By default local storage is LVM based, but to enable thin
provisioning you'll want EXT3.  See this blog for how to convert to
EXT3: 
http://thinworldblog.blogspot.com/2011/08/enabling-thin-provisioning-on-existing.html
- It seems like you're planning on using Storage XenMotion to move the
VHD from the golden primary storage to normal primary storage, but
that's going to move the entire VHD chain and it will do so over the
network.  Here's a blog article describing a bit about how it works:
http://blogs.citrix.com/2012/08/24/storage_xenmotion/.  I'm reasonably
certain the design parameters didn't include local->local without
network.
- If someone wants to take a snapshot of the VM, will that snapshot
then got to normal secondary storage or back to the golden master?
- To Punith's point, I *think* VM start will occur post clone, so the
clone will consume network to occur and then will start on local
storage.

The big test I'd like to see first would be creating the golden master
and from it creating a few VMs.  Then once you have those VMs run some
normal XenServer operations like moving a VM within a pool, moving
that VM across pools and assigning a home server.  If those pass, then
things might work out, but if those fail then you'll need to sort
things out within the XenServer code first. If these basic tests do
work, then I'd look at the network usage to see if things did indeed
get better.

-tim

On Thu, Jun 5, 2014 at 8:11 AM, Punith S  wrote:
> hi Hieu,
>
> after going through your  "Golden Primary Storage" proposal , from my
> understanding you are creating a SSD golden PS for holding parent
> VDH(nothing but the template which go copied from secondary storage) and a
> normal primary storage for ROOT volumes(child VHD) for the corresponding
> vm's.
>
> from the following flowchart , i have the following questions,
>
> 1. since you are having problem with slow boot time of the vm's, will the
> booting of the vm's happen in golden PS, ie while cloning ?
>  if so, the spawning of the vm's will be always fast .
>
> but i see you are starting the vm after moving the cloned vhd to the
> ROOT PS and pointing the child vhd to its parent vhd on the GOLDEN PS,
> hence , there will be a network traffic between these two
> primary storages, which will obviously slow down the vm's performance
> forever.
>
> 2. what if someone removes the golden primary storage containing the the
> parent VHD(template) where all the child VDH's in the root primary storage
> are been pointed to ?
>if so, all vm's running will be crashed immediately. since its child
> vhd's parent is removed.
>
> thanks
>
>
> On Thu, Jun 5, 2014 at 8:59 AM, Hieu LE  wrote:
>
>> Mike, Punith,
>>
>> Please review "Golden Primary Storage" proposal. [1]
>>
>> Thank you.
>>
>> [1]:
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage
>>
>>
>> On Wed, Jun 4, 2014 at 10:32 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> Daan helped out with this. You should be good to go now.
>>>
>>>
>>> On Tue, Jun 3, 2014 at 8:50 PM, Hieu LE  wrote:
>>>
>>> > Hi Mike,
>>> >
>>> > Could you please give edit/create permission on ASF Jira/Wiki
>>> confluence ?
>>> > I can not add a new Wiki page.
>>> >
>>> > My Jira ID: hieulq
>>> > Wiki: hieulq89
>>> > Review Board: hieulq
>>> >
>>> > Thanks !
>>> >
>>> >
>>> > On Wed, Jun 4, 2014 at 9:17 AM, Mike Tutkowski <
>>> > mike.tutkow...@solidfire.com
>>> > > wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > Yes, please feel free to add a new Wiki page for your design.
>>> > >
>>> > > Here is a link to applicable design info:
>>> > >
>>> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Design
>>> > >
>>> > > Also, feel free to ask more questions and have me review your design.
>>> > >
>>> > > Thanks!
>>> > > Mike
>>> > >
>>> > >
>>> > > On Tue, Jun 3, 2014 at 7:29 PM, Hieu LE  wrote:
>>> > >
>>> > > > Hi Mike,
>>> > > >
>>> > > > You are right, performance will be decreased over time because
>>> writes
>>> > > IOPS
>>> > > > will always end up on slower storage pool.
>>> > > >
>>> > > > In our case, we are using CloudStack integrated in VDI solution to
>>> > > provived
>>> > > > pooled VM type[1]. So may be my approach can bring better UX for
>>> user
>>> > > with
>>> > > > lower bootime ...
>>> > > >
>>> > > > A short change in design are followings
>>> 

Re: Support pure Xen as a hypervisor follow-up

[Tim Mackey]

Dave,

I've submitted a merge review request
(https://reviews.apache.org/r/22270/) yesterday.  If you want to avoid
having to potentially deal with a bunch of conflicts, you might want
to see if your patches apply cleanly there and let me know.  Happy to
help with any conflict resolution.

btw, I didn't see a design document up on the
wiki(https://cwiki.apache.org/confluence/display/CLOUDSTACK/4.5+Design+Documents).
 Can you get one there and start a DISCUSS thread?  It'll probably
tease out some gotchas you might not be aware of.

-tim

On Fri, Jun 6, 2014 at 11:04 AM, Dave Scott  wrote:
> Hi,
>
> Here’s a quick status update:
>
> On 16 May 2014, at 15:22, Dave Scott  wrote:
>
>> Hi,
>>
>> On 14 May 2014, at 09:53, sebgoa  wrote:
>>
>>>
>>> On Apr 9, 2014, at 2:37 PM, Dave Scott  wrote:
>>>
 Hi,

 Following up from Tim's "Support pure Xen as a hypervisor" proposal last 
 month[1] I'd like to start working on this and maybe even make a little 
 bit of progress while I'm at CCC in Denver.

 Helpfully James Bulpin managed to get CS + libvirt + xen to start an 
 instance in a simple configuration. Although the patches[2] are not 
 intended to be production-ready :) they help highlight some of the areas 
 we need to change.
>>>
>>> Dave, just to let you know that Tim has done some important "refactoring" 
>>> to split up XenServer hypervisor in CS between Xen and XenServer. That way 
>>> we could keep using xapi for XS but start moving to libvirt for Xen.
>>>
>>> Tim worked in the xen2server branch (don't ask about the name, I messed it 
>>> up…:) ).
>>>
>>> https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=shortlog;h=refs/heads/xen2server
>>>
>>> Would be nice to see some of the libvirt stuff in that branch to handle a 
>>> new driver for Xen.
>>>
>>> Since the two hypervisors will be split up, we could still drop in some 
>>> early libvirt patches to handle Xen and put this in 4.5 as a wip.
>>
>> Thanks for the links.
>>
>> I’m slowly building up a set of patches here:
>>
>> https://github.com/djs55/cloudstack/tree/virsh-capabilities
>>
>> I think once I’ve gotten to a stable-ish point I’ll rebase on top of Tim’s 
>> branch.
>>
>> So far I’ve
>> * changed the hypervisor detection to use ‘virsh capabilities’ in one place 
>> and ‘cat /sys/hypervisor/type’ in another. Thinking about it again, I think 
>> it’s probably best to standardise on /sys/hypervisor/type since that will 
>> succeed irrespective of whether the libvirtd service is chkconfig’d on or 
>> not.
>>
>> * in the python cloudutils system setup stuff isKvmEnabled() has become 
>> isHypervisorEnabled()
>>
>> * added a XenLibvirtDiscoverer similar to the LXC one
>>
>> * fixed what I believe is a race in sshExecuteCmdOneShotWithExitCode (which 
>> seems to hit me every time, I don’t know why other people seem to be immune 
>> from it): see CLOUDSTACK-6621 and review board request 21261
>>
>> * added the new hypervisor to hypervisor.list and 
>> system.vm.default.hypervisor, so it appears in the UI properly
>>
>> * registered a system VM template in the database, using the same qcow2 
>> image as KVM
>>
>> For my test host I’m using a XenServer nightly snapshot which comes with a 
>> nice modern xen and kernel, and is easy to install bleeding-edge libvirt on 
>> top. I had to tweak the kernel configuration and the network configuration 
>> but I’m hoping to make it work out of the box in future.
>>
>> When I deploy my ‘datacenter’ the discovery phase works, the agent connects 
>> and looks healthy in the logs and the UI. The next step is to figure out why 
>> the system VM template isn’t being copied to primary storage — for some 
>> reason the copy isn’t being attempted but I can’t see any obvious reason why.
>
> I’m now at the stage of getting my system VMs to start via libvirt. The main 
> missing feature is support for : low-bandwidth private host<->guest 
> control channels. These channels are generally useful things and are needed 
> by other projects (like oVirt), so I’d like to add them to libxl and 
> libvirt’s libxl driver. There’s a thread on xen-devel and libvir-devel if 
> anyone’s interested:
>
> http://www.redhat.com/archives/libvir-list/2014-June/msg00180.html
>
> Once the  are sorted, basic VM operations should work. The next 
> step would be to rebase my patches on top of Tim’s renaming changes and tidy 
> them up for review.
>
> Cheers,
> Dave
>
>>
>> Cheers,
>> Dave
>>
>>>
>>> -Sebastien
>>>

 Some of the areas are:

 1. hypervisor detection

 Where we currently look for KVM specifically ("lsmod | grep kvm") we could 
 switch to either detecting any Linux hypervisor (by reading 
 /sys/hypervisor/type) and assuming if a hypervisor is present then we can 
 use libvirt on it (is this a fair assumption?) Or we could white-list 
 “kvm” or “xen”. Or we could query libvirt directly (perhaps via 'virsh 
 capabilities'?)

 2. fiddling with t

Re: Support pure Xen as a hypervisor follow-up

[Tim Mackey]

If you can give me the commit ID for your HEAD, I'll check and see
what's up.  It could be another conflict needing resolution.

-tim

On Fri, Jun 6, 2014 at 12:16 PM, sebgoa  wrote:
>
> On Jun 6, 2014, at 5:17 PM, Tim Mackey  wrote:
>
>> Dave,
>>
>> I've submitted a merge review request
>> (https://reviews.apache.org/r/22270/) yesterday.
>
> Unfortunately that patch did not seem to apply cleanly. I have not dug deep 
> though..
>
>> If you want to avoid
>> having to potentially deal with a bunch of conflicts, you might want
>> to see if your patches apply cleanly there and let me know.  Happy to
>> help with any conflict resolution.
>>
>> btw, I didn't see a design document up on the
>> wiki(https://cwiki.apache.org/confluence/display/CLOUDSTACK/4.5+Design+Documents).
>> Can you get one there and start a DISCUSS thread?  It'll probably
>> tease out some gotchas you might not be aware of.
>>
>> -tim
>>
>> On Fri, Jun 6, 2014 at 11:04 AM, Dave Scott  wrote:
>>> Hi,
>>>
>>> Here’s a quick status update:
>>>
>>> On 16 May 2014, at 15:22, Dave Scott  wrote:
>>>
>>>> Hi,
>>>>
>>>> On 14 May 2014, at 09:53, sebgoa  wrote:
>>>>
>>>>>
>>>>> On Apr 9, 2014, at 2:37 PM, Dave Scott  wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Following up from Tim's "Support pure Xen as a hypervisor" proposal last 
>>>>>> month[1] I'd like to start working on this and maybe even make a little 
>>>>>> bit of progress while I'm at CCC in Denver.
>>>>>>
>>>>>> Helpfully James Bulpin managed to get CS + libvirt + xen to start an 
>>>>>> instance in a simple configuration. Although the patches[2] are not 
>>>>>> intended to be production-ready :) they help highlight some of the areas 
>>>>>> we need to change.
>>>>>
>>>>> Dave, just to let you know that Tim has done some important "refactoring" 
>>>>> to split up XenServer hypervisor in CS between Xen and XenServer. That 
>>>>> way we could keep using xapi for XS but start moving to libvirt for Xen.
>>>>>
>>>>> Tim worked in the xen2server branch (don't ask about the name, I messed 
>>>>> it up…:) ).
>>>>>
>>>>> https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=shortlog;h=refs/heads/xen2server
>>>>>
>>>>> Would be nice to see some of the libvirt stuff in that branch to handle a 
>>>>> new driver for Xen.
>>>>>
>>>>> Since the two hypervisors will be split up, we could still drop in some 
>>>>> early libvirt patches to handle Xen and put this in 4.5 as a wip.
>>>>
>>>> Thanks for the links.
>>>>
>>>> I’m slowly building up a set of patches here:
>>>>
>>>> https://github.com/djs55/cloudstack/tree/virsh-capabilities
>>>>
>>>> I think once I’ve gotten to a stable-ish point I’ll rebase on top of Tim’s 
>>>> branch.
>>>>
>>>> So far I’ve
>>>> * changed the hypervisor detection to use ‘virsh capabilities’ in one 
>>>> place and ‘cat /sys/hypervisor/type’ in another. Thinking about it again, 
>>>> I think it’s probably best to standardise on /sys/hypervisor/type since 
>>>> that will succeed irrespective of whether the libvirtd service is 
>>>> chkconfig’d on or not.
>>>>
>>>> * in the python cloudutils system setup stuff isKvmEnabled() has become 
>>>> isHypervisorEnabled()
>>>>
>>>> * added a XenLibvirtDiscoverer similar to the LXC one
>>>>
>>>> * fixed what I believe is a race in sshExecuteCmdOneShotWithExitCode 
>>>> (which seems to hit me every time, I don’t know why other people seem to 
>>>> be immune from it): see CLOUDSTACK-6621 and review board request 21261
>>>>
>>>> * added the new hypervisor to hypervisor.list and 
>>>> system.vm.default.hypervisor, so it appears in the UI properly
>>>>
>>>> * registered a system VM template in the database, using the same qcow2 
>>>> image as KVM
>>>>
>>>> For my test host I’m using a XenServer nightly snapshot which comes with a 
>>>> nice modern xen and kernel, and is easy to install bleeding-edge libvirt 
>&

Re: [DISCUSS] extending the libvirt/KVM plugin to also support libvirt/Xen

[Tim Mackey]

Dave,

Thanks for putting this up on the wiki. A few things jumped out at me...

- Please change "Xen" to "XenProject" or "Xen Project" as appropriate.
There's already a ton of confusion out there, and I'd like to see us
get our terms correct from the outset where ever possible.

- It would be good to see a UI mock up for how users would configure
the Xen Project hypervisor option.  I think that would go a long way
to helping with the mixed hypervisor cluster concept and how it could
be blocked.

- It would also be good to see examples of how the APIs might need to
be changed to support this.  Minimally I'd expect to see things like
supported disk/network/os types and that sort of thing.

- I see you have a todo to document the supported Xen Project
hypervisor and libvirt versions, but also dependencies on libxl
changes.  Are these critical dependencies, or if someone doesn't have
latest upstream will things work in a reduced feature set?

- C6.1 talks about exposing a config setting.  Is that really
required?  Couldn't that be set correctly based on hypervisor type?

- Would QCOW2 be used for the Xen Project disk type for all templates
to keep with KVM consistency?  I'm actually thinking  about support
for VMDK, but perhaps that's a different proposal?

- Since we're talking about sharing a libvirt plugin, I'm not clear on
if the shared work is done in a new libvirt plugin which is then
exposed to a KVM and a XenProject plugin or if the existing KVM plugin
is refactored to encompass both.

-tim




On Mon, Jun 9, 2014 at 2:35 AM, Wido den Hollander  wrote:
>
>
> On 06/08/2014 11:14 PM, Dave Scott wrote:
>>
>> Hi Wido,
>>
>> Thanks for your mail!
>>
>> On 8 Jun 2014, at 19:02, Wido den Hollander  wrote:
>>
>>> On 06/08/2014 06:23 PM, Dave Scott wrote:

 Hi,

 Following on from the earlier "[PROPOSAL] Support pure Xen as a
 hypervisor”, I’ve added a design doc to the wiki:


 https://cwiki.apache.org/confluence/display/CLOUDSTACK/Allow+hosts+running+the+Xen+hypervisor+to+be+managed+via+libvirt

 This design would allow people who want to manage their hypervisors
 purely through the libvirt tools to choose the Xen hypervisor.

  From the code point of view, I want to maximise sharing between the KVM
 and Xen code paths, partly to make QA easier and partly to maximise the
 chance that adding a feature for “Xen” causes it to work for “KVM” and
 vice-versa. In particular this means that, if a genuinely-useful capability
 is currently missing from the libvirt libxl driver, I want to implement it
 rather than work around it.

>>>
>>> Seems like a great route to me! You also want to support Xen+Qemu with
>>> this way?
>>
>>
>> Yes, it should be possible to run fully virtualised VMs with Xen + Qemu. I
>> think we’ll be able to choose whether to run VMs as PV or HVM.
>
>
> Ok, but those will be different code paths at some level.
>
>
>>
>>> We have to be aware that there might be some storage differences between
>>> KVM and Xen like Ceph which is not fully supported yet by Xen.
>>
>>
>> Ceph is an interesting one. Xen itself doesn’t know anything about
>> storage— instead the dom0 takes care of it either via a kernel driver
>> (blkback) or userspace program (qemu or tapdisk). When I tried to make Ceph
>> work about a year ago[1] I hit a bug in libxl (the Xen control library). The
>> good news is the fix made it into Xen 4.4, so with luck we can get it to
>> work.
>>
>
> When Xen runs with Qemu as full HVM it's Qemu which takes care of the Ceph
> storage, so in that case it's fixed.
>
> I haven't got a lot of experience with PV Xen. I heard stories of Ceph being
> integrated in blktap(2), but never tested it.
>
>
>>
>>> If anything is missing in libvirt or the Java bindings we have to fix
>>> that indeed instead of hacking around it.
>>
>>
>> Great :)
>>
>> Cheers,
>> Dave
>>
>> [1]
>> http://xenserver.org/discuss-virtualization/virtualization-blog/entry/tech-preview-of-xenserver-libvirt-ceph.html
>>
>>>
>>> Wido
>>>
 Comments appreciated!

 Cheers,
 Dave

 [1]
 http://mail-archives.apache.org/mod_mbox/cloudstack-users/201403.mbox/%3ccajgxtbnbmqtq81ralgh2kma7v5wjyzkr3xnyasmkc_br+uk...@mail.gmail.com%3e

>>
>

Re: [DISCUSS] extending the libvirt/KVM plugin to also support libvirt/Xen

[Tim Mackey]

On Mon, Jun 9, 2014 at 1:02 PM, Dave Scott  wrote:

>
> > - It would be good to see a UI mock up for how users would configure
> > the Xen Project hypervisor option.  I think that would go a long way
> > to helping with the mixed hypervisor cluster concept and how it could
> > be blocked.
>
> My comments about clusters were to emphasise that you shouldn’t expect to
> live migrate a VM between KVM and Xen, even if both are using libvirt
> underneath. Actually perhaps this is a misunderstanding of mine: is it
> possible today to mix hypervisors within a single CloudStack cluster? I’m
> not trying to change anything, but just point out the obvious. Maybe I got
> that wrong :-)
>

This comes down to your implementation.  Today CS clusters need to have
uniform hypervisor types, but if you're exposing a libvirt plugin, then
those protections won't exist since CS will see things as just libvirt. If
you create a KVM plugin and a separate XenProject one both of which
implement a libvirt base class then you should get the protections for
free.  It should also allow you to define hypervisor specific details in a
cleaner way.


> > - It would also be good to see examples of how the APIs might need to
> > be changed to support this.  Minimally I'd expect to see things like
> > supported disk/network/os types and that sort of thing.
>
> I can talk about some of this more explicitly in the doc. Since Xen can
> use qemu for disks (for both PV and HVM guests) there should be no
> difference in supported disk formats between this and the existing KVM
> support. I’m not proposing to add anything to the Xen support which isn’t
> supported by KVM such as .vhd via tapdisk. Similarly, networking is handled
> by the regular Linux network stack so that should all work in the same way.
>

I wouldn't assume the existing KVM plugin is generic. For example, only
QCOW2 is supported for disk images today.


> > - I see you have a todo to document the supported Xen Project
> > hypervisor and libvirt versions, but also dependencies on libxl
> > changes.  Are these critical dependencies, or if someone doesn't have
> > latest upstream will things work in a reduced feature set?
>
> In this proposal they would be critical dependencies (I’ll go make that
> clear). It is possible to make transitional arrangements but I didn’t want
> to overburden this proposal with backwards compat.
>
> > - C6.1 talks about exposing a config setting.  Is that really
> > required?  Couldn't that be set correctly based on hypervisor type?
>
> You’re right that this would be a hypervisor-specific thing.
>
> I’m still pondering the choice between PV and HVM. Using PV mode is
> convenient because it would allow the VMs to boot under Xen under
> virtualbox, like current devcloud. Using HVM might be more future-proof.
>
> > - Would QCOW2 be used for the Xen Project disk type for all templates
> > to keep with KVM consistency?  I'm actually thinking  about support
> > for VMDK, but perhaps that's a different proposal?
>
> That sounds a separate proposal, but it shouldn’t conflict with this one
> (assuming the VMDK support is in qemu)
>

VMDK support is there, but the existing KVM plugin only supports QCOW2, so
some of this is up to you for what you want to have supported with
XenProject.


>
> > - Since we're talking about sharing a libvirt plugin, I'm not clear on
> > if the shared work is done in a new libvirt plugin which is then
> > exposed to a KVM and a XenProject plugin or if the existing KVM plugin
> > is refactored to encompass both.
>
> Not totally sure what the best thing to do is — I’ll have to play with the
> code a little more.
>

I'd suggest looking at libvirt as a base class for KVM and XenProject and
see if that cleans anything up.  It might not, but worth looking into.


>
> Thanks,
> Dave
>
>
> >
> > -tim
> >
> >
> >
> >
> > On Mon, Jun 9, 2014 at 2:35 AM, Wido den Hollander 
> wrote:
> >>
> >>
> >> On 06/08/2014 11:14 PM, Dave Scott wrote:
> >>>
> >>> Hi Wido,
> >>>
> >>> Thanks for your mail!
> >>>
> >>> On 8 Jun 2014, at 19:02, Wido den Hollander  wrote:
> >>>
>  On 06/08/2014 06:23 PM, Dave Scott wrote:
> >
> > Hi,
> >
> > Following on from the earlier "[PROPOSAL] Support pure Xen as a
> > hypervisor”, I’ve added a design doc to the wiki:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Allow+hosts+running+the+Xen+hypervisor+to+be+managed+via+libvirt
> >
> > This design would allow people who want to manage their hypervisors
> > purely through the libvirt tools to choose the Xen hypervisor.
> >
> > From the code point of view, I want to maximise sharing between the
> KVM
> > and Xen code paths, partly to make QA easier and partly to maximise
> the
> > chance that adding a feature for “Xen” causes it to work for “KVM”
> and
> > vice-versa. In particular this means that, if a genuinely-useful
> capability
> > is currently missing from the libvirt libxl drive

Re: [DISCUSS] Increasing VM IOPS by separating golden image in high IOPS partition in Xen Server ?

[Tim Mackey]

Hieu,

I made a couple of minor edits to your design to ensure everything is
"XenServer" based.  If you haven't done so already, please also fetch the
most recent master and base off of that.  I refactored the old Xen plugin
into a XenServer specific one since Xen Project isn't currently supported,
and files have moved.  Also please ensure you don't use the term "Xen" in
your code/docs to avoid any future confusion when the Xen Project work
starts to materialize.

Looking forward to seeing your work!!

-tim


On Mon, Jun 9, 2014 at 4:31 PM, Mike Tutkowski  wrote:

> Thanks, Hieu!
>
> I have reviewed your design (making only minor changes to your Wiki).
>
> Please feel free to have me review your code when you are ready.
>
> Also, do you have a plan for integration testing? It would be great if you
> could update your Wiki page to include what your plans on in this regard.
>
> Thanks!
> Mike
>
>
> On Mon, Jun 9, 2014 at 4:24 AM, Hieu LE  wrote:
>
> > Hi guys,
> >
> > I have updated this proposal wiki[1], included diagram for VM migrate,
> > volume migrate and snapshot.
> >
> > Please review and give feedback.
> >
> > [1]:
> >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage
> >
> >
> > On Fri, Jun 6, 2014 at 7:14 PM, Todd Pigram  wrote:
> >
> > > Sorry, thought you were based off the link you provided in this reply.
> > >
> > > "In our case, we are using CloudStack integrated in VDI solution to
> > > provived
> > > pooled VM type[1]. So may be my approach can bring better UX for user
> > with
> > > lower bootime ...
> > >
> > > A short change in design are followings
> > > - VM will be deployed with golden primary storage if primary storage is
> > > marked golden and this VM template is also marked as golden.
> > > - Choosing the best deploy destionation for both golden primary storage
> > and
> > > normal root volume primary storage. Chosen host can also access both
> > > storage pools.
> > > - New Xen Server plug-in for modifying VHD parent id.
> > >
> > > Is there some place for me to submit my design and code. Can I write a
> > new
> > > proposal in CS wiki ?
> > >
> > > [1]:
> > >
> > >
> >
> http://support.citrix.com/proddocs/topic/xendesktop-rho/cds-choose-scheme-type-rho.html
> > >  "
> > >
> > >
> > > On Thu, Jun 5, 2014 at 11:55 PM, Hieu LE  wrote:
> > >
> > > > Hi Todd,
> > > >
> > > >
> > > > On Fri, Jun 6, 2014 at 9:17 AM, Todd Pigram 
> > wrote:
> > > >
> > > > > Hieu,
> > > > >
> > > > > I assume you are using MCS for you golden image? What version of
> XD?
> > > > Given
> > > > > you are using pooled desktops, have you thought about using a PVS
> BDM
> > > iso
> > > > > and mount it with in your 1000 VMs? This way you can stagger
> reboots
> > > via
> > > > > PVS console or Studio. This would require a change to your delivery
> > > > group.
> > > > >
> > > > >
> > > > Sorry but I did not use MCS or XenDesktop in my company :-)
> > > >
> > > >
> > > > >
> > > > > On Thu, Jun 5, 2014 at 9:28 PM, Mike Tutkowski <
> > > > > mike.tutkow...@solidfire.com
> > > > > > wrote:
> > > > >
> > > > > > 6) The copy_vhd_from_secondarystorage XenServer plug-in is not
> used
> > > > when
> > > > > > you're using XenServer + XS62ESP1 + XS62ESP1004. In that case,
> > please
> > > > > refer
> > > > > > to copyTemplateToPrimaryStorage(CopyCommand) method in the
> > > > > > Xenserver625StorageProcessor class.
> > > > > >
> > > > >
> > > >
> > > > Thank Mike, I will take note of that.
> > > >
> > > >
> > > > >  >
> > > > > > On Thu, Jun 5, 2014 at 1:56 PM, Mike Tutkowski <
> > > > > > mike.tutkow...@solidfire.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Other than going through a "for" loop and deploying VM after
> VM,
> > I
> > > > > don't
> > > > > > > think CloudStack currently supports a bulk-VM-deploy operation.
> > > > > > >
> > > > > > > It would be nice if CS did so at some point in the future;
> > however,
> > > > > that
> > > > > > > is probably a separate proposal from Hieu's.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 5, 2014 at 12:13 AM, Amit Das <
> > amit@cloudbyte.com>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Hieu,
> > > > > > >>
> > > > > > >> Will it be good to include bulk operation of this feature? In
> > > > > addition,
> > > > > > >> does Xen support parallel execution of these operations ?
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Amit
> > > > > > >> *CloudByte Inc.* 
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Jun 5, 2014 at 8:59 AM, Hieu LE 
> > > wrote:
> > > > > > >>
> > > > > > >> > Mike, Punith,
> > > > > > >> >
> > > > > > >> > Please review "Golden Primary Storage" proposal. [1]
> > > > > > >> >
> > > > > > >> > Thank you.
> > > > > > >> >
> > > > > > >> > [1]:
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Jun 

Re: [ANNOUNCE] Pierre-Luc Dion as a committer

[Tim Mackey]

Félicitations mon ami!


On Tue, Jun 10, 2014 at 1:46 PM, Ian Duffy  wrote:

> Congrats Pierre-Luc!
>
>
> On 10 June 2014 17:07, Nguyen Anh Tu  wrote:
>
> > Congratulation, Pierre-Luc!
> >
> > --Tuna
> >
> >
> > On Tue, Jun 10, 2014 at 10:34 PM, Mike Tutkowski <
> > mike.tutkow...@solidfire.com> wrote:
> >
> > > Congratulations!
> > >
> > > On Tuesday, June 10, 2014, Hugo Trippaers  wrote:
> > >
> > > > Congrats Pierre-Luc!
> > > >
> > > > Cheers,
> > > >
> > > > Hugo
> > > >
> > > >
> > > > On 10 jun. 2014, at 16:40, Chip Childers  > > > > wrote:
> > > >
> > > > > On Tue, Jun 10, 2014 at 01:45:36PM +0200, sebgoa wrote:
> > > > >> The Project Management Committee (PMC) for Apache CloudStack has
> > > > >> asked Pierre-Luc Dion to become a committer and we are pleased to
> > > > announce
> > > > >> that he has accepted.
> > > > >>
> > > > >> Being a committer allows many contributors to contribute more
> > > > autonomously. For
> > > > >> developers, it makes it easier to submit changes and eliminates
> the
> > > > need to
> > > > >> have contributions reviewed via the patch submission process.
> > Whether
> > > > >> contributions are development-related or otherwise, it is a
> > > recognition
> > > > of a
> > > > >> contributor's participation in the project and commitment to the
> > > > project and
> > > > >> the Apache Way.
> > > > >>
> > > > >> Please join me in congratulating Pierre-Luc!
> > > > >>
> > > > >> PS: Nice work on the documentation
> > > > >>
> > > > >> -Sebastien, on behalf of the CloudStack PMC
> > > > >
> > > > > Yay Pierre-Luc!
> > > >
> > > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkow...@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > *™*
> > >
> >
>

Re: Xen Project support and state of APIs

[Tim Mackey]

Rohit,

We've only supported xapi so far, so I'm curious if you had Xen Project 4.4
working before with xapi and if that is now broken. Nothing I did should
have impacted that, but it wasn't in my test suite so anything is possible
On Jun 17, 2014 11:21 AM, "Rohit Yadav"  wrote:

> Hi,
>
> Xen Project (previously xen.org xenserver) is widely accessible in several
> Linux distributions such as Arch, Fedora etc. Therefore, people may want to
> use it instead of XenServer (6.2 or later). The ACS 4.5 design doc [1]
> explains about XenServer and how it should be tested etc. but what about
> using Xen Project?
>
> Since the recent Xen plugin refactoring as xenserver plugin [1], I'm unable
> to get ACS to work with Xen Project 4.4 (on a real host and on a new custom
> DevCloud). Using the XenServer ACS plugin, it identifies a Xen Project host
> as some version of Xen but then fails. Has anyone tried this with ACS
> 4.4/master? Are we going to test/support Xen Project (4.x) soon? If so, can
> people involved share their plan/vision/roadmap on it?
>
> Lastly, as the design doc [1] suggested XCP's development has stopped, are
> we going to favour using XAPI [1] now or libVirt [2] or something else?
>
> [1]
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Convert+Xen+usage+to+XenServer
>
> [2] http://blog.xen.org/index.php/2014/03/10/xen-4-4-released
>
> Regards.
>

Re: Xen Project support and state of APIs

[Tim Mackey]

On Tue, Jun 17, 2014 at 7:30 AM, Rohit Yadav  wrote:

> Tim,
>
> Thanks for replying, let me clean up everything and re-setup my test infra
> after my $dayjob hours and get back to you. The test cases in the design
> doc does not discuss Xen Project so I thought I should ask if ACS works
> with Xen Project 4.x.
>

The longer term objective is to provide support for XenProject without
XAPI, most notably with libvirt.  Effectively XenServer implies XAPI and
XenProject would imply libvirt as the tooling.


>
> The Xen/XenProject/XenServer/XAPI/XCP is confusing [1]. From what I
> understand, XCP was some sort of an opensource version of XenServer, but
> since XenServer distribution (which consists of Xen, XenCenter, XAPI, Linux
> etc.) is opensource, XCP has been discontinued since 1.6 release?
>

XCP was rolled into XenServer as of XenServer 6.2 which means 1.6 was the
last version of XCP.  I tried to sum it up in this blog:
http://open.citrix.com/blog/342-knowing-what-you-ve-got-avoiding-the-xen-vs-xenserver-confusion.html


>
> Now, on many Linux distro XAPI is still available as 'xcp-xapi' such as in
> Debian/Ubuntu, so what package(s) do you suggest one needs to install and
> configure so I can get DevCloud [2] work again? DevCloud is a VM that runs
> Xen (Xen Project) inside it and can be used as a virtual host to
> test/develop ACS against it.
>
>
I see no reason why any other XAPI implementation wouldn't work, but it's
entirely possible the version of xcp-xapi you have has issues with
XenProject 4.4 due to normal versioning type issues.  There also was a
different "productization" attempt called xenserver-core, but that looks to
have died on the vine.  That being said, I could easily have broken
something.  If I've broken something, I probably won't be able to test
until next week due to travels.  The question of xapi inclusion in distros
is one I'd need to check on.



> [1] http://www.xenproject.org/developers/teams/xapi.html
> [2] http://bhaisaab.org/logs/devcloud
>
> Regards.
>
>
>
>
> On Tue, Jun 17, 2014 at 4:27 PM, Tim Mackey  wrote:
>
> > Rohit,
> >
> > We've only supported xapi so far, so I'm curious if you had Xen Project
> 4.4
> > working before with xapi and if that is now broken. Nothing I did should
> > have impacted that, but it wasn't in my test suite so anything is
> possible
> > On Jun 17, 2014 11:21 AM, "Rohit Yadav"  wrote:
> >
> > > Hi,
> > >
> > > Xen Project (previously xen.org xenserver) is widely accessible in
> > several
> > > Linux distributions such as Arch, Fedora etc. Therefore, people may
> want
> > to
> > > use it instead of XenServer (6.2 or later). The ACS 4.5 design doc [1]
> > > explains about XenServer and how it should be tested etc. but what
> about
> > > using Xen Project?
> > >
> > > Since the recent Xen plugin refactoring as xenserver plugin [1], I'm
> > unable
> > > to get ACS to work with Xen Project 4.4 (on a real host and on a new
> > custom
> > > DevCloud). Using the XenServer ACS plugin, it identifies a Xen Project
> > host
> > > as some version of Xen but then fails. Has anyone tried this with ACS
> > > 4.4/master? Are we going to test/support Xen Project (4.x) soon? If so,
> > can
> > > people involved share their plan/vision/roadmap on it?
> > >
> > > Lastly, as the design doc [1] suggested XCP's development has stopped,
> > are
> > > we going to favour using XAPI [1] now or libVirt [2] or something else?
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Convert+Xen+usage+to+XenServer
> > >
> > > [2] http://blog.xen.org/index.php/2014/03/10/xen-4-4-released
> > >
> > > Regards.
> > >
> >
>

Re: Managing individual ESXi instances

[Tim Mackey]

Mike,

I wouldn't expect things with the VMware Hypervisor (what they refer to
standalone ESXi) to work out of the box.  Since you can't cluster things,
I'd expect only raw iSCSI to work, but it's been years since I've worked
with raw ESXi.

-tim


On Wed, Jun 18, 2014 at 11:40 AM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Or I'd like to know if it doesn't work (as is the case for Hyper-V until I
> get time to add that kind of support for it).
>
>
> On Wed, Jun 18, 2014 at 9:37 AM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
> > I know, for example, that I'd like to test out that managed storage works
> > with it.
> >
> > I've added support for managed storage to XenServer, ESX/vCenter, and KVM
> > for CloudStack.
> >
> > Another hypervisor type - to me personally - means I'd to verify managed
> > storage works with it.
> >
> > Depending on how radical the changes for an ESX-only solution are, it may
> > or may not work "out of the box" for managed storage.
> >
> >
> > On Wed, Jun 18, 2014 at 9:30 AM, Ivan Efremov  wrote:
> >
> >> Hi Alex,
> >>
> >> How do you think, what is the rough estimation of adding ESX API support
> >> to CloudStack?
> >> AFAIU the main point of integration of the new API is
> >> plugins/hypervisors. Are there any other major points that should be
> >> patched when adding a new hypervisor type?
> >>
> >>
> >> Thanks,
> >> Ivan
> >>
> >> 18.06.2014, 18:24, "Alex Huang" :
> >> > IIRC, the reason is because the vCenter API is more powerful than the
> >> ESX API.  At the time (before Apache), the features that requested
> needed
> >> vCenter. There's currently no proposal to use plain ESXi.  Would love to
> >> see one though.
> >> >
> >> > --Alex
> >> >>  -Original Message-
> >> >>  From: Ivan Efremov [mailto:e...@yandex.ru]
> >> >>  Sent: Tuesday, June 17, 2014 8:26 PM
> >> >>  To: dev@cloudstack.apache.org
> >> >>  Subject: Managing individual ESXi instances
> >> >>
> >> >>  Hi all,
> >> >>
> >> >>  I've sent this mail to the users list but this one looks as the
> >> better destination.
> >> >>
> >> >>  I'm new to the CloudStack platform and I'm wondering why the
> platform
> >> >>  does need the vCenter API and can not use ESXi directly,
> >> >>
> >> >>  Can anyone elaborate on this?
> >> >>  Are there any proposals for adding ESXi integration to CloudStack?
> >> >>
> >> >>  Thanks,
> >> >>  Ivan
> >>
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkow...@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > *™*
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> *™*
>

Re: Review Request 22799: Golden (Base) Primary Storage feature

[Tim Mackey]

> On June 23, 2014, 9:16 p.m., Mike Tutkowski wrote:
> > I was wondering if you could fill out the section about tests that have 
> > been performed?
> > 
> > I would be interested in both new and regression testing.
> > 
> > Thanks!

It looked like some functions added parameters, but didn't include descriptions 
of those parameters.  

Having just gone through some refactoring, I'd like to see some of the 
abbreviations made clearer.  I'm thinking of future people who might not be as 
familiar with this code.

You made changes to schema-421to430, but that represents shipping versions.  
Please ensure schema changes are done in 4.5 (aka master).

I didn't see anything which limited this to XenServer only.  I could easily 
have missed it, but if this is only XenServer, I'd prefer to see some check to 
keep people from accidentally configuring something which isn't expected to 
work for them.

Thanks

-tim 


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22799/#review46451
---


On June 20, 2014, 3:46 a.m., Hieu LE wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22799/
> ---
> 
> (Updated June 20, 2014, 3:46 a.m.)
> 
> 
> Review request for cloudstack, Mike Tutkowski and Tim Mackey.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> As discussed in mailing list, this patch is applied for golden primary 
> storage in [1].
> I have changed the term from "golden" to "base" because there are some 
> functions and variables in CloudStack also use "base" for base image.
> This patch only apply for Xen Server.
> 
> [1]: 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage
> 
> 
> Diffs
> -
> 
>   api/src/com/cloud/deploy/DeployDestination.java 
> 4ded5ebe7a18252da471ee25019856f2b2f772e0 
>   api/src/com/cloud/storage/StoragePool.java 
> 8e03c3348f3a6dd3156ab9e440126ea317957dc0 
>   api/src/com/cloud/template/VirtualMachineTemplate.java 
> 599212bb039fdbb78511019e8f0a6ea4b4a84440 
>   api/src/org/apache/cloudstack/api/ApiConstants.java 
> ae5d6f05b6b52f60b151369a641cb11fcbb558af 
>   api/src/org/apache/cloudstack/api/BaseUpdateTemplateOrIsoCmd.java 
> 2350f6b389203e2c6cc2182fe03fe9a95e936b81 
>   
> api/src/org/apache/cloudstack/api/command/admin/storage/CreateStoragePoolCmd.java
>  ae44bc9373232d242e4ebdcf76844969f0fe69fc 
>   
> api/src/org/apache/cloudstack/api/command/admin/storage/UpdateStoragePoolCmd.java
>  3d1a77353257c814efaf60875ffdf99603bc414e 
>   
> api/src/org/apache/cloudstack/api/command/user/template/RegisterTemplateCmd.java
>  f478c9bc8eebf867a03deb4add1bf695ac3ec0ad 
>   api/src/org/apache/cloudstack/api/response/StoragePoolResponse.java 
> 3571866fe74dca9aa5fe0d11373313eab97e94ac 
>   api/src/org/apache/cloudstack/api/response/TemplateResponse.java 
> 3e21043e339103c021d3c9e767acac8b3837f760 
>   core/src/com/cloud/agent/api/CheckPoolBelongToHostAnswer.java PRE-CREATION 
>   core/src/com/cloud/agent/api/CheckPoolBelongToHostCommand.java PRE-CREATION 
>   core/src/org/apache/cloudstack/storage/to/PrimaryDataStoreTO.java 
> 29e53b0d9581f764a17ea285606213d2c045b029 
>   core/src/org/apache/cloudstack/storage/to/TemplateObjectTO.java 
> b201c386f4975913f13c575d7685e50cedc7d92f 
>   
> core/test/org/apache/cloudstack/api/agent/test/BackupSnapshotCommandTest.java 
> 33361e87265df05e00bfa6dba810d2b68ae8d923 
>   core/test/org/apache/cloudstack/api/agent/test/CheckNetworkAnswerTest.java 
> 66feaecb5ef20053db50956e2801fec096a350c9 
>   core/test/org/apache/cloudstack/api/agent/test/SnapshotCommandTest.java 
> 114c8854d1504436523aa99c78bf2b4d84a12077 
>   
> engine/api/src/org/apache/cloudstack/engine/subsystem/api/storage/PrimaryDataStoreParameters.java
>  1dbff59a8911ad8f0933ef17a2c2b1d3e33523b9 
>   
> engine/api/src/org/apache/cloudstack/engine/subsystem/api/storage/StoragePoolAllocator.java
>  dfdbd8ab92c47799f6ad23637fa63e030f0be968 
>   
> engine/api/src/org/apache/cloudstack/engine/subsystem/api/storage/VolumeInfo.java
>  f93f4efac83c565cd33eb7eb67dcaca335f1c226 
>   engine/components-api/src/com/cloud/deploy/DeploymentPlanningManager.java 
> ee6721ab445a5222d0087dc9170e0b58f9eef91a 
>   engine/orchestration/src/com/cloud/vm/VirtualMachineManagerImpl.java 
> 4aa5fc80d9660d2f985db98124c33465bd99767f 
>   
> engine/orchestration/src/org/apache/cloudstack/engine/cloud/enti

Re: Review Request 22799: Golden (Base) Primary Storage feature

[Tim Mackey]

Hieu,

It would also be good in your design doc and testing to validate which
XenServer versions you expect this to work with, and don't forget about
xcp-xapi in pure Linux as part of that.  If you have time, I'd also be
curious if XenServer Creedence Read cache has any impact on your
results/implementation.  That was part of the alpha.2 drop.

-tim


On Mon, Jun 23, 2014 at 11:24 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Hi,
>
> Yes, Marvin is great for automated regression testing of CloudStack.
>
> However, I was hoping you could note even manual testing (new and
> regression tests) you may have run to verify correctness of the system
> within reason (even if these are not automated).
>
> If you have time, automated tests would be excellent, as well.
>
> Thanks!
> Mike
>
>
> On Mon, Jun 23, 2014 at 8:50 PM, Hieu LE  wrote:
>
>> Hi Tim and Mike,
>>
>>
>> On Tue, Jun 24, 2014 at 5:59 AM, Tim Mackey  wrote:
>>
>>>This is an automatically generated e-mail. To reply, visit:
>>> https://reviews.apache.org/r/22799/
>>>
>>> On June 23rd, 2014, 9:16 p.m. UTC, *Mike Tutkowski* wrote:
>>>
>>> I was wondering if you could fill out the section about tests that have 
>>> been performed?
>>>
>>> I would be interested in both new and regression testing.
>>>
>>> Thanks!
>>>
>>> Thank you Mike, will I use marvin for regression testing ?
>>
>>
>>>   It looked like some functions added parameters, but didn't include 
>>> descriptions of those parameters.
>>>
>>> Having just gone through some refactoring, I'd like to see some of the 
>>> abbreviations made clearer.  I'm thinking of future people who might not be 
>>> as familiar with this code.
>>>
>>> Thank you, I will refactor my code to make it clearer.
>>
>>
>>>
>>>
>>> You made changes to schema-421to430, but that represents shipping versions. 
>>>  Please ensure schema changes are done in 4.5 (aka master).
>>>
>>> Sure !
>>
>>> I didn't see anything which limited this to XenServer only.  I could easily 
>>> have missed it, but if this is only XenServer, I'd prefer to see some check 
>>> to keep people from accidentally configuring something which isn't expected 
>>> to work for them.
>>>
>>> I have implemented some checks in this patch which limited to XenServer,
>> e.g: in DeploymentPlanningManagerImpl class, function planBaseDeployment,
>> line 298, or basePsHostCheck function to check that host can communicate
>> with pool.
>>
>>
>>>
>>>
>>> Thanks
>>>
>>> -tim
>>>
>>>
>>> - Tim
>>>
>>> On June 20th, 2014, 3:46 a.m. UTC, Hieu LE wrote:
>>>Review request for cloudstack, Mike Tutkowski and Tim Mackey.
>>> By Hieu LE.
>>>
>>> *Updated June 20, 2014, 3:46 a.m.*
>>>  *Repository: * cloudstack-git
>>> Description
>>>
>>> As discussed in mailing list, this patch is applied for golden primary 
>>> storage in [1].
>>> I have changed the term from "golden" to "base" because there are some 
>>> functions and variables in CloudStack also use "base" for base image.
>>> This patch only apply for Xen Server.
>>>
>>> [1]: 
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage
>>>
>>>   Diffs
>>>
>>>- api/src/com/cloud/deploy/DeployDestination.java
>>>(4ded5ebe7a18252da471ee25019856f2b2f772e0)
>>>- api/src/com/cloud/storage/StoragePool.java
>>>(8e03c3348f3a6dd3156ab9e440126ea317957dc0)
>>>- api/src/com/cloud/template/VirtualMachineTemplate.java
>>>(599212bb039fdbb78511019e8f0a6ea4b4a84440)
>>>- api/src/org/apache/cloudstack/api/ApiConstants.java
>>>(ae5d6f05b6b52f60b151369a641cb11fcbb558af)
>>>- api/src/org/apache/cloudstack/api/BaseUpdateTemplateOrIsoCmd.java
>>>(2350f6b389203e2c6cc2182fe03fe9a95e936b81)
>>>- 
>>> api/src/org/apache/cloudstack/api/command/admin/storage/CreateStoragePoolCmd.java
>>>(ae44bc9373232d242e4ebdcf76844969f0fe69fc)
>>>- 
>>> api/src/org/apache/cloudstack/api/command/admin/storage/UpdateStoragePoolCmd.java
>>>(3d1a77353257c814efaf60875ffdf99603bc414e)
>>>- 
>>> api/src/org/apache/cloudstack/api/command/user/template/RegisterTemp

Review: Hypervisor selection PDF

[Tim Mackey]

Good evening everyone.  I'm presenting at Collab this week on hypervisor
selection in CloudStack.  While I've been running CloudStack since the
pre-Apache days, my experience is obviously limited to what I've personally
implemented.  Since I want to keep this session factual and avoid any bias,
I'm hoping that as the devs behind the features some of you on this list
can review my deck for errors and omissions.

Here's a link to a PDF of my deck, and thanks to anyone who takes the time
to see if I've made any errors.

https://citrix.sharefile.com/d/s8e4036e1da04c748

Thanks

-tim

Re: XenServer SR Question

[Tim Mackey]

Unfortunately what you're experiencing is how it works.  While XenServer
does support different CHAP credentials by SR, it only supports a single
CHAP credential for discovery.  It can be made to work, but you'd need to
either modify how the storage manager works to pull it off, or rewrite some
of the init scripts to cache the discovery data.


On Wed, Dec 18, 2013 at 3:55 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Hi,
>
> I just noticed a problem today while creating SRs in XenServer. Perhaps
> someone with related experience could point me in the right direction.
>
> Let's say my SAN's management IP address is X.
>
> I can have XenServer create a shared SR using IP address X with CHAP
> credentials Y.
>
> If I try to have XenServer create another shared SR using IP address X that
> uses different CHAP credentials (ex. CHAP credentials Z), XenServer returns
> a discovery failure.
>
> It's like XenServer is expecting all iSCSI targets at the same IP address
> to have the same CHAP credentials.
>
> Does anyone know if I am mistaken here? This seems like a critical defect
> if this is true.
>
> Thanks!
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud
> *™*
>

Re: XenServer SR Question

[Tim Mackey]

Mike,

I'm referring to the open-iscsi code used by XAPI.  XAPI has a storage
manager API which deals with all the SR management.  It's also where the
issue you're running into exists.

In terms of clearing the connections and credentials, the easiest way is
via a reboot.  Since your using multiple CHAP credentials, only one set
will be used and any SRs which use a different CHAP secret will fail to
have their targets discovered during the pdb-plug phase of storage
initialization.  You can then destroy the SRs which failed to come up and
move forward.

-tim


On Wed, Dec 18, 2013 at 4:35 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Hey Tim,
>
> When you refer to modifying the storage manager, are you referring to
> CloudStack?
>
> Perhaps you are referring to CitrixResourceBase, which is where we discover
> and log in to iSCSI targets.
>
> Do you know of a way to delete those cached CHAP credentials via XAPI so
> when new ones are used for discovery they can work?
>
> Thanks!
>
>
> On Wed, Dec 18, 2013 at 2:22 PM, Tim Mackey  wrote:
>
> > Unfortunately what you're experiencing is how it works.  While XenServer
> > does support different CHAP credentials by SR, it only supports a single
> > CHAP credential for discovery.  It can be made to work, but you'd need to
> > either modify how the storage manager works to pull it off, or rewrite
> some
> > of the init scripts to cache the discovery data.
> >
> >
> > On Wed, Dec 18, 2013 at 3:55 PM, Mike Tutkowski <
> > mike.tutkow...@solidfire.com> wrote:
> >
> > > Hi,
> > >
> > > I just noticed a problem today while creating SRs in XenServer. Perhaps
> > > someone with related experience could point me in the right direction.
> > >
> > > Let's say my SAN's management IP address is X.
> > >
> > > I can have XenServer create a shared SR using IP address X with CHAP
> > > credentials Y.
> > >
> > > If I try to have XenServer create another shared SR using IP address X
> > that
> > > uses different CHAP credentials (ex. CHAP credentials Z), XenServer
> > returns
> > > a discovery failure.
> > >
> > > It's like XenServer is expecting all iSCSI targets at the same IP
> address
> > > to have the same CHAP credentials.
> > >
> > > Does anyone know if I am mistaken here? This seems like a critical
> defect
> > > if this is true.
> > >
> > > Thanks!
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkow...@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the
> > > cloud<http://solidfire.com/solution/overview/?video=play>
> > > *™*
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *™*
>

Re: XenServer SR Question

[Tim Mackey]

The problem with doing that is during host reboot. Then only one of the
credential sets will be used to do the discovery, and on each reboot a
discovery will occur. It'll also have issues with multipath.

There will also be an issue during pool join, and there could be
replication issues in xenstore.

Net is that you can make things work by doing that, but error recovery
paths and reboots break.
On Dec 18, 2013 6:07 PM, "Mike Tutkowski" 
wrote:

> We have noticed if I ssh into XenServer and delete the file /etc/iscsi/
> 10.10.8.108,3260 (where 10.10.8.108 is our storage's IP address and 3260 is
> the port) that I can create SRs using different CHAP credentials.
>
> Can anyone think of a "got-cha" here?
>
> Thanks!
>
>
> On Wed, Dec 18, 2013 at 3:18 PM, Tim Mackey  wrote:
>
> > Mike,
> >
> > I'm referring to the open-iscsi code used by XAPI.  XAPI has a storage
> > manager API which deals with all the SR management.  It's also where the
> > issue you're running into exists.
> >
> > In terms of clearing the connections and credentials, the easiest way is
> > via a reboot.  Since your using multiple CHAP credentials, only one set
> > will be used and any SRs which use a different CHAP secret will fail to
> > have their targets discovered during the pdb-plug phase of storage
> > initialization.  You can then destroy the SRs which failed to come up and
> > move forward.
> >
> > -tim
> >
> >
> > On Wed, Dec 18, 2013 at 4:35 PM, Mike Tutkowski <
> > mike.tutkow...@solidfire.com> wrote:
> >
> > > Hey Tim,
> > >
> > > When you refer to modifying the storage manager, are you referring to
> > > CloudStack?
> > >
> > > Perhaps you are referring to CitrixResourceBase, which is where we
> > discover
> > > and log in to iSCSI targets.
> > >
> > > Do you know of a way to delete those cached CHAP credentials via XAPI
> so
> > > when new ones are used for discovery they can work?
> > >
> > > Thanks!
> > >
> > >
> > > On Wed, Dec 18, 2013 at 2:22 PM, Tim Mackey  wrote:
> > >
> > > > Unfortunately what you're experiencing is how it works.  While
> > XenServer
> > > > does support different CHAP credentials by SR, it only supports a
> > single
> > > > CHAP credential for discovery.  It can be made to work, but you'd
> need
> > to
> > > > either modify how the storage manager works to pull it off, or
> rewrite
> > > some
> > > > of the init scripts to cache the discovery data.
> > > >
> > > >
> > > > On Wed, Dec 18, 2013 at 3:55 PM, Mike Tutkowski <
> > > > mike.tutkow...@solidfire.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I just noticed a problem today while creating SRs in XenServer.
> > Perhaps
> > > > > someone with related experience could point me in the right
> > direction.
> > > > >
> > > > > Let's say my SAN's management IP address is X.
> > > > >
> > > > > I can have XenServer create a shared SR using IP address X with
> CHAP
> > > > > credentials Y.
> > > > >
> > > > > If I try to have XenServer create another shared SR using IP
> address
> > X
> > > > that
> > > > > uses different CHAP credentials (ex. CHAP credentials Z), XenServer
> > > > returns
> > > > > a discovery failure.
> > > > >
> > > > > It's like XenServer is expecting all iSCSI targets at the same IP
> > > address
> > > > > to have the same CHAP credentials.
> > > > >
> > > > > Does anyone know if I am mistaken here? This seems like a critical
> > > defect
> > > > > if this is true.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkow...@solidfire.com
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the
> > > > > cloud<http://solidfire.com/solution/overview/?video=play>
> > > > > *™*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkow...@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the
> > > cloud<http://solidfire.com/solution/overview/?video=play>
> > > *™*
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *™*
>

Re: Question about XenServer Snapshots

[Tim Mackey]

Mike, you might want to post this to xs-devel on XenServer.org.  Some of
the storage engineers there would be in s better position to diagnose.
On Dec 19, 2013 5:10 PM, "Mike Tutkowski" 
wrote:

> Hi,
>
> I have been experimenting with VM snapshots on XenServer and have noticed a
> problem that I hope someone might be able to shed some light on.
>
> In a normal flow of taking a VM snapshot, reverting to it, then deleting
> the VM snapshot, I have observed the following (which looks just fine):
>
> *SR:*
>
> uuid ( RO): 2a06-a8c8-11db-4a8b-f8d519f9ac3e
>   name-label ( RW): Test
> name-description ( RW): iSCSI SR [10.10.8.108
> (iqn.2010-01.com.solidfire:3y8w.test.15; LUN 0:
> 33793877000ff47acc01: 93.1 GB (SolidFir))]
> host ( RO): XenServer-6.1-Tut-2
> type ( RO): lvmoiscsi
> content-type ( RO):
>
>
>- *Before VM snap:*
>
>
> *Active:*
>
> uuid ( RO): b4587018-9679-4fe7-ba72-5523cb988cec
>   name-label ( RW): i-2-21-VM-DATA
> name-description ( RW):
>  sr-uuid ( RO): 2a06-a8c8-11db-4a8b-f8d519f9ac3e
> virtual-size ( RO): 16106127360
> sharable ( RO): false
>read-only ( RO): false
>
>
>- *After VM snap:*
>
>
> *Base copy (contains the data of the previously active VDI):*
>
> uuid ( RO): d167952d-deb4-4942-9ea8-c8b3777d885e
>   name-label ( RW): base copy
> name-description ( RW):
>  sr-uuid ( RO): 2a06-a8c8-11db-4a8b-f8d519f9ac3e
> virtual-size ( RO): 16106127360
> sharable ( RO): false
>read-only ( RO): true
>
> *Snapshot:*
>
> uuid ( RO): 613dc799-cf69-445a-a2fe-611653e0b0c9
>   name-label ( RW): i-2-21-VM-DATA
> name-description ( RW):
>  sr-uuid ( RO): 2a06-a8c8-11db-4a8b-f8d519f9ac3e
> virtual-size ( RO): 16106127360
> sharable ( RO): false
>read-only ( RO): false
>
> *Active (has the same UUID as the previously active VDI):*
>
> uuid ( RO): b4587018-9679-4fe7-ba72-5523cb988cec
>   name-label ( RW): i-2-21-VM-DATA
> name-description ( RW):
>  sr-uuid ( RO): 2a06-a8c8-11db-4a8b-f8d519f9ac3e
> virtual-size ( RO): 16106127360
> sharable ( RO): false
>read-only ( RO): false
>
>
>- *After revert to VM snap:*
>
>
> *Base copy:*
>
> uuid ( RO): d167952d-deb4-4942-9ea8-c8b3777d885e
>   name-label ( RW): base copy
> name-description ( RW):
>  sr-uuid ( RO): 2a06-a8c8-11db-4a8b-f8d519f9ac3e
> virtual-size ( RO): 16106127360
> sharable ( RO): false
>read-only ( RO): true
>
> *Snapshot (this VDI is un-touched):*
>
> uuid ( RO): 613dc799-cf69-445a-a2fe-611653e0b0c9
>   name-label ( RW): i-2-21-VM-DATA
> name-description ( RW):
>  sr-uuid ( RO): 2a06-a8c8-11db-4a8b-f8d519f9ac3e
> virtual-size ( RO): 16106127360
> sharable ( RO): false
>read-only ( RO): false
>
> *Active (this is a new VDI - the old active VDI was deleted):*
>
> uuid ( RO): b21284fa-347a-459a-a8bf-0fcd7717a134
>   name-label ( RW): i-2-21-VM-DATA
> name-description ( RW):
>  sr-uuid ( RO): 2a06-a8c8-11db-4a8b-f8d519f9ac3e
> virtual-size ( RO): 16106127360
> sharable ( RO): false
>read-only ( RO): false
>
>
>- *After deleting VM snap:*
>
>
> *Active (the snapshot is gone as is the base copy...the base copy was
> rolled up into this VDI):*
>
> uuid ( RO): b21284fa-347a-459a-a8bf-0fcd7717a134
>   name-label ( RW): i-2-21-VM-DATA
> name-description ( RW):
>  sr-uuid ( RO): 2a06-a8c8-11db-4a8b-f8d519f9ac3e
> virtual-size ( RO): 16106127360
> sharable ( RO): false
>read-only ( RO): false
>
> Now, in my case, where I create an SR on the fly (in response to attaching
> a CloudStack volume to a VM on XenServer for the first time) to house a
> single VDI (which has guaranteed IOPS), I see the following erroneous
> behavior when it comes to hypervisor snapshots:
>
> *SR:*
>
> uuid ( RO): 70e06f08-2c9d-f9cf-4e64-2f064c11325a
>   name-label ( RW): /iqn.2010-01.com.solidfire:3y8w.test.19/0
> name-description ( RW): /iqn.2010-01.com.solidfire:3y8w.test.19/0
> host ( RO): XenServer-6.1-Tut-2
> type ( RO): lvmoiscsi
> content-type ( RO): user
>
>
>- *Before VM snap:*
>
>
> *Active:*
>
> uuid ( RO): 067572a8-fa4d-45b5-9365-2d7790a4b202
>   name-label ( RW): i-2-21-VM-DATA
> name-description ( RW):
>  sr-uuid ( RO): 70e06f08-2c9d-f9cf-4e64-2f064c11325a
> virtual-size ( RO): 10737418240
> sharable ( RO): false
>read-only ( RO): false

Re: [DISCUSS] Move to Debian9 systemvmtemplate

[Tim Mackey]

Syed,

I did a bunch of work on XenServer with Packer [1] before leaving Citrix.
My stuff works rather well and was tested with XS 6.2, 6.5 and 7. It
shouldn't be hard to validate with newest XS and updated Packer - I just
lack the infra to do the testing.

[1] https://github.com/xenserverarmy/packer

-tim

On Thu, Jul 27, 2017 at 11:19 AM, Syed Ahmed  wrote:

> -1 on Arch as well. Moving to Debian 9 seems the wiser choice IMO. I've
> used Packer before and I really like it, the only downside that I see is
> that Packer lacks support for XenServer VHD images. There is some work on a
> XenServer plugin but I haven't tested that. If the community decides to use
> Packer, I can do some initial validation of it on XenServer.
>
> Thanks,
> -Syed
>
> On Tue, Jul 25, 2017 at 3:19 AM, Wido den Hollander 
> wrote:
>
> >
> > > Op 24 juli 2017 om 19:07 schreef Rene Moser :
> > >
> > >
> > > Hi Rohit
> > >
> > >
> > > On 07/23/2017 06:08 PM, Rohit Yadav wrote:
> > > > All,
> > > >
> > > >
> > > > Just want to kick an initial discussion around migration to Debian9
> > based systemvmtemplate, and get your feedback on the same.
> > > >
> > > > Here's a work-in-progress PR: https://github.com/apache/
> > cloudstack/pull/2198
> > >
> > > Have you considered to replace veewee by packer?
> > >
> >
> > Packer is really nice indeed. We use it to build our templates [0] which
> > we use on CloudStack.
> >
> > Building the SSVM using Packer should be rather easy I think.
> >
> > [0]: https://github.com/pcextreme/packer-templates
> >
> > > Our friends from schubergphilis have already done some work here
> > > https://github.com/MissionCriticalCloud/systemvm-packer.
> > >
> > > However there would be also an official way to convert the definitions
> > > https://www.packer.io/guides/veewee-to-packer.html
> > >
> > > Regards René
> >
>

Re: [XenServer] meltdown-spectre

[Tim Mackey]

PLD,

One thing to add to your testing is template management. When I was doing
all the Packer stuff with XS 6.5 and 7, ACS needed to know if the template
was PV or HVM to provision properly. No idea if the ACS template logic has
changed since then, but something to be aware of.

>From a performance perspective with XS 6.5 started having newer XS
templates follow an HVM model. e.g. CentOS 7 on XS 6.5 was HVM not PV.
iirc, the performance difference was negligible on reasonably new CPUs
(Sandy Bridge+ I think).

-tim

On Mon, Jan 8, 2018 at 9:39 PM, Ivan Kudryavtsev 
wrote:

> Hi. Every kind of virtualization is affected according to qemu developers.
>
> 9 янв. 2018 г. 9:32 пользователь "Pierre-Luc Dion" 
> написал:
>
> >  Hi,
> >
> > From recent blog post, I've read that system using full virtualization
> such
> > as KVM, VMware or Xen-HVM are not affected?  Anyhow, from the latest
> hotfix
> > of XenServer 7.1cu1 hf8, it look like they systematically convert VM from
> > PV to HVM, so in the case of a VM stop/start by CloudStack, a PV vm would
> > be restarted as HVM.
> >
> > Look like this could be problematic if your VM kernel does not support
> > both, we've just starting tested and so far look like our Debian systemvm
> > template work fine, it can be created as HVM.
> >
> > Another point is that Citrix released an hotfix for xs7.2, 7.3 but not
> for
> > 7.1, you need to cumulative update to remain on 7.1 which is LTS.
> >
> > And last, does anyone did some benchmark before and after the kernel fix
> > for Meltdown ?  Some report state 30-35% cpu usage increase (not
> hypervisor
> > specific) and  Lucian [1] might indicate it would depend on the cpu
> model.
> > Any metrics to share ?  We are doing some tests on our side we should be
> > able to share some stuff soon...
> >
> > Regards,
> >
> > [1] http://markmail.org/thread/wkzze3n24mns274x
> >
>

Re: [DISCUSS] running sVM and VR as HVM on XenServer

[Tim Mackey]

dom0 already has a DHCP server listening for requests on internal
management networks. I'd be wary trying to manage it from an external
service like cloudstack lest it get reset upon XenServer patch. This alone
makes me favor option #2. I also think option #2 simplifies network design
for users.

Agreed on making this as consistent across flows as possible.



On Fri, Jan 12, 2018 at 9:44 AM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> It looks reasonable to manage VRs via management IP network. We should
> focus on using the same work flow for different deployment scenarios.
>
>
> On Fri, Jan 12, 2018 at 12:13 PM, Pierre-Luc Dion 
> wrote:
>
> > Hi,
> >
> > We need to start a architecture discussion about running SystemVM and
> > Virtual-Router as HVM instances in XenServer. With recent
> Meltdown-Spectre,
> > one of the mitigation step is currently to run VMs as HVM on XenServer to
> > self contain a user space attack from a guest OS.
> >
> > Recent hotfix from Citrix XenServer (XS71ECU1009) enforce VMs to start
> has
> > HVM. This is currently problematic for Virtual Routers and SystemVM
> because
> > CloudStack use PV "OS boot Options" to preconfigure the VR eth0:
> > cloud_link_local. While using HVM the "OS boot Options" is not accessible
> > to the VM so the VR fail to be properly configured.
> >
> > I currently see 2 potential approaches for this:
> > 1. Run a dhcpserver in dom0 managed by cloudstack so VR eth0 would
> receive
> > is network configuration at boot.
> > 2. Change the current way of managing VR, SVMs on XenServer, potentiall
> do
> > same has with VMware: use pod management networks and assign a POD IP to
> > each VR.
> >
> > I don't know how it's implemented in KVM, maybe cloning KVM approach
> would
> > work too, could someone explain how it work on this thread?
> >
> > I'd a bit fan of a potential #2 aproach because it could facilitate VR
> > monitoring and logging, although a migration path for an existing cloud
> > could be complex.
> >
> > Cheers,
> >
> >
> > Pierre-Luc
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: [DISCUSS] running sVM and VR as HVM on XenServer

[Tim Mackey]

> We found that we can use xenstore-read / xenstore-write to send data from
dom0 to domU which are in our case  VRs or SVMs. Any reason not using this
approach ?

xenstore has had some issues in the past. The most notable of which were
limitations on the number of event channels in use, followed by overall
performance impact. iirc, the event channel stuff was fully resolved with
XenServer 6.5, but they do speak to a need to test if there are any changes
to the maximum number of VMs which can be reliably supported. It also
limits legacy support (in case that matters).

Architecturally I think this is a reasonable approach to the problem. One
other thing to note is that xapi replicates xenstore information to all
members of a pool. That might impact RVRs.

-tim

[1] "xenstore is not a high-performance facility and should beused only for
small amounts of control plane data."
https://xenbits.xen.org/docs/4.6-testing/misc/xenstore.txt

On Fri, Jan 12, 2018 at 4:56 PM, Pierre-Luc Dion  wrote:

> After some verification with Syed and Khosrow,
>
> We found that we can use xenstore-read / xenstore-write to send data from
> dom0 to domU which are in our case  VRs or SVMs. Any reason not using this
> approach ?  that way we would not need a architectural change for XenServer
> pods, and this would support HVM and PV virtual-router. more test required,
> for sure, VR would need to have xentools pre-installed.
>
>
> *Pierre-Luc DION*
> Architecte de Solution Cloud | Cloud Solutions Architect
> t 855.652.5683
>
> *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Fri, Jan 12, 2018 at 4:07 PM, Syed Ahmed  wrote:
>
> > KVM uses a VirtIO channel to send information about the IP address and
> > other params to the SystemVMs. We could use a similar strategy in
> XenServer
> > using XenStore. This would involve minimal changes to the code while
> > keeping backward compatibility.
> >
> >
> >
> > On Fri, Jan 12, 2018 at 3:07 PM, Simon Weller 
> > wrote:
> >
> > > They do not. They receive a link-local ip address that is used for host
> > > agent to VR communication. All VR commands are proxied through the host
> > > agent. Host agent to VR communication is over SSH.
> > >
> > >
> > > 
> > > From: Rafael Weingärtner 
> > > Sent: Friday, January 12, 2018 1:42 PM
> > > To: dev
> > > Subject: Re: [DISCUSS] running sVM and VR as HVM on XenServer
> > >
> > > but we are already using this design in vmware deployments (not sure
> > about
> > > KVM). The management network is already an isolated network only used
> by
> > > system vms and ACS. Unless we are attacked by some internal agent, we
> are
> > > safe from customer attack through management networks. Also, we can (if
> > we
> > > don't do yet) restrict access only via these management interfaces in
> > > system VMs(VRs, SSVM, console proxy and others to come).
> > >
> > >
> > >
> > > Can someone confirm if VRs receive management IPs in KVM deployments?
> > >
> > > On Fri, Jan 12, 2018 at 5:36 PM, Syed Ahmed 
> wrote:
> > >
> > > > The reason why we used link local in the first place was to isolate
> the
> > > VR
> > > > from directly accessing the management network. This provides another
> > > layer
> > > > of security in case of a VR exploit. This will also have a side
> effect
> > of
> > > > making all VRs visible to each other. Are we okay accepting this?
> > > >
> > > > Thanks,
> > > > -Syed
> > > >
> > > > On Fri, Jan 12, 2018 at 11:37 AM, Tim Mackey 
> > wrote:
> > > >
> > > > > dom0 already has a DHCP server listening for requests on internal
> > > > > management networks. I'd be wary trying to manage it from an
> external
> > > > > service like cloudstack lest it get reset upon XenServer patch.
> This
> > > > alone
> > > > > makes me favor option #2. I also think option #2 simplifies network
> > > > design
> > > > > for users.
> > > > >
> > > > > Agreed on making this as consistent across flows as possible.
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Jan 12, 2018 at 9:44 AM, Rafael Weingärtner <
> > > > > rafaelweingart...@gmail.com> wrote:
> > > > >
> > &g

Re: [DISCUSS] running sVM and VR as HVM on XenServer

[Tim Mackey]

There isn't anything I can think of wrt reliability. If the usage is
limited to VR boot, then I don't see anything on the surface to limit
performance.

In other words, xenstore as a solution seems a reasonable approach.

-tim

On Mon, Jan 15, 2018 at 8:26 PM, Pierre-Luc Dion  wrote:

> Hi Tim,
>
> As long as it work, since it's only used to for the initial instruction set
> at the VR boot so eth0 can be configure, I think xenstore would work just
> fine.
> unless you are saying we could just not rely on xenstore in terms of
> reliability?
>
>
> *Pierre-Luc DION*
> Architecte de Solution Cloud | Cloud Solutions Architect
> t 855.652.5683
>
> *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Fri, Jan 12, 2018 at 7:34 PM, Tim Mackey  wrote:
>
> > > We found that we can use xenstore-read / xenstore-write to send data
> from
> > dom0 to domU which are in our case  VRs or SVMs. Any reason not using
> this
> > approach ?
> >
> > xenstore has had some issues in the past. The most notable of which were
> > limitations on the number of event channels in use, followed by overall
> > performance impact. iirc, the event channel stuff was fully resolved with
> > XenServer 6.5, but they do speak to a need to test if there are any
> changes
> > to the maximum number of VMs which can be reliably supported. It also
> > limits legacy support (in case that matters).
> >
> > Architecturally I think this is a reasonable approach to the problem. One
> > other thing to note is that xapi replicates xenstore information to all
> > members of a pool. That might impact RVRs.
> >
> > -tim
> >
> > [1] "xenstore is not a high-performance facility and should beused only
> for
> > small amounts of control plane data."
> > https://xenbits.xen.org/docs/4.6-testing/misc/xenstore.txt
> >
> > On Fri, Jan 12, 2018 at 4:56 PM, Pierre-Luc Dion 
> > wrote:
> >
> > > After some verification with Syed and Khosrow,
> > >
> > > We found that we can use xenstore-read / xenstore-write to send data
> from
> > > dom0 to domU which are in our case  VRs or SVMs. Any reason not using
> > this
> > > approach ?  that way we would not need a architectural change for
> > XenServer
> > > pods, and this would support HVM and PV virtual-router. more test
> > required,
> > > for sure, VR would need to have xentools pre-installed.
> > >
> > >
> > > *Pierre-Luc DION*
> > > Architecte de Solution Cloud | Cloud Solutions Architect
> > > t 855.652.5683
> > >
> > > *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts
> > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > w cloudops.com *|* tw @CloudOps_
> > >
> > > On Fri, Jan 12, 2018 at 4:07 PM, Syed Ahmed 
> wrote:
> > >
> > > > KVM uses a VirtIO channel to send information about the IP address
> and
> > > > other params to the SystemVMs. We could use a similar strategy in
> > > XenServer
> > > > using XenStore. This would involve minimal changes to the code while
> > > > keeping backward compatibility.
> > > >
> > > >
> > > >
> > > > On Fri, Jan 12, 2018 at 3:07 PM, Simon Weller
>  > >
> > > > wrote:
> > > >
> > > > > They do not. They receive a link-local ip address that is used for
> > host
> > > > > agent to VR communication. All VR commands are proxied through the
> > host
> > > > > agent. Host agent to VR communication is over SSH.
> > > > >
> > > > >
> > > > > 
> > > > > From: Rafael Weingärtner 
> > > > > Sent: Friday, January 12, 2018 1:42 PM
> > > > > To: dev
> > > > > Subject: Re: [DISCUSS] running sVM and VR as HVM on XenServer
> > > > >
> > > > > but we are already using this design in vmware deployments (not
> sure
> > > > about
> > > > > KVM). The management network is already an isolated network only
> used
> > > by
> > > > > system vms and ACS. Unless we are attacked by some internal agent,
> we
> > > are
> > > > > safe from customer attack through management networks. Also, we can
> > (if
> > > > we
> > > > > don't do yet) restrict access only via these manag

Re: [PROPOSE] EOL for supported OSes & Hypervisors

[Tim Mackey]

I think there are three pieces in play. First there are guest OSes, second
the management server and third the hypervisors themselves. For the
hypervisors and management server I can see a more stringent set of
requirements. Going by the XenServer experience, legacy guests should
continue to work, but removing any "unsupported" guest OS versions from
template definition should be done.

When communicating end of support status, I've always been a fan of
something which can be quickly understood. e.g. "XenServer support ends six
months after Citrix drops primary support for the version". Then follow
that in a table with a date, but I feel tying it to the "upstream" support
statement more clearly communicates the real dependency.

>From a release notes perspective, I'd recommend communicating any impending
support statement change early and often. e.g. For any "upstream" EOL/EOS
statement occurring in the next 12 months we add a notice to the release
notes reminding people of the impending change.

For new versions of things, maybe adopting a statement that within X months
after launch of the dependency we'd have support for it?

-tim

On Tue, Jan 16, 2018 at 2:57 PM, Ron Wheeler  wrote:

> I am having a hard time imagining why a new Cloudstack user would want to
> use CentOS 6.
> I have not heard complaints from any forum about applications running on
> CentOS 6 that could not run on CentOS 7.
>
> It does have an impact of DevOps but even there it is pretty minor and
> well documented and once one gets one's head around systemctl, journalctl
> and firewalld, it really is a lot better
>
> As you pointed out, support for CentOS 7 is mandatory regardless of the
> decision on CentOS 6.
> It will be a big black mark against CloudStack if the current
> RedHat/CentOS release is not supported 2.5 years after its release.
>
> For existing Cloudstack users, is there a large risk that dropping
> official support for CentOS6 as a VM would result in a CentOS 6 VM not
> running after an upgrade to the latest Cloudstack?
> Is there anyone in the community that absolutely can not upgrade? Would
> they take over the testing of CentOS 6 if official support was dropped?
>
> Anyone using CentOS6 as a hypervisor will have to upgrade soon in any
> event and as you point out Cloudstack will soon be forced to move from
> CentOS 6 by dependencies dropping support for CentOS 6.
>
> It would seem to make sense to declare an EOL date for support for CentOS
> 6 as a hypervisor as soon as possible so that system administrators are put
> on notice.
>
> Ron
>
>
>
>
> On 16/01/2018 12:58 PM, Paul Angus wrote:
>
>> Hi Eric,
>>
>> This is the type of discussion that I wanted to open - the argument that
>> I see for earlier dropping of v6 is that - Between May 2018 and q2 2020
>> RHEL/CentOS 6.x will only receive security and mission critical updates,
>> meanwhile packages on which we depend or may want to utilise in the future
>> are been deprecated or not developed for v6.x
>> Also the testing and development burden on the CloudStack community
>> increases as we try to maintain backward compatibility while including new
>> versions.
>>
>> Needing installation documentation for centos 7 is a great point, and
>> something that we need to address regardless.
>>
>>
>> Does anyone else have a view, I'd really like to here from a wide range
>> of people.
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.an...@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>> -Original Message-
>> From: Eric Green [mailto:eric.lee.gr...@gmail.com]
>> Sent: 12 January 2018 17:24
>> To: us...@cloudstack.apache.org
>> Cc: dev@cloudstack.apache.org
>> Subject: Re: [PROPOSE] EOL for supported OSes & Hypervisors
>>
>> Official EOL for Centos 6 / RHEL 6 as declared by Red Hat Software is
>> 11/30/2020. Jumping the gun a bit there, padme.
>>
>> People on Centos 6 should certainly be working on a migration strategy
>> right now, but the end is not here *yet*. Furthermore, the install
>> documentation is still written for Centos 6 rather than Centos 7. That
>> needs to be fixed before discontinuing support for Centos 6, eh?
>>
>> On Jan 12, 2018, at 04:35, Rohit Yadav  wrote:
>>>
>>> +1 I've updated the page with upcoming Ubuntu 18.04 LTS.
>>>
>>>
>>> After 4.11, I think 4.12 (assuming releases by mid of 2018) should
>>> remove "declared" (they might still work with 4.12+ but in docs and by
>>> project we should officially support them) support for following:
>>>
>>>
>>> a. Hypervisor:
>>>
>>> XenServer - 6.2, 6.5,
>>>
>>> KVM - CentOS6, RHEL6, Ubuntu12.04 (I think this is already removed,
>>> packages don't work I think?)
>>>
>>> vSphere/Vmware - 4.x, 5.0, 5.1, 5.5
>>>
>>>
>>> b. Remove packaging for CentOS6.x, RHEL 6.x (the el6 packages), and
>>> Ubuntu 12.04 (any non-systemd debian distro).
>>>
>>>
>>> Thoughts, comments?
>>>
>>>
> --
> Ron Wheeler
> President
> Artifact Software Inc
> email: rwhee...@a

Re: CloudStack Collab in Brazil

[Tim Mackey]

Gabriel,

I'm happy to help review proposals if required.

-tim

On Wed, Dec 19, 2018 at 12:35 PM Gabriel Beims Bräscher <
gabrasc...@gmail.com> wrote:

> Hi Rafael,
>
> I am available to help, count on me!
> I have one question. Can anyone (one that is not a PMC/Committer) help to
> review presentations?
>
> The divisions for the CFP looks good, adding security aspects as Ricardo
> Makino proposed is also interesting.
>
> Regards,
> Gabriel.
>
> Em qua, 19 de dez de 2018 às 11:12, Cristian Latapiat 
> escreveu:
>
> > Hi Rafael ,
> >
> > I am, therefore, available to collaborate and to help you in everything
> > that may be necessary.
> >
> > Regards,
> >
> > Cristian
> >
> > Em seg, 17 de dez de 2018 às 18:49, Rafael Weingärtner <
> > rafaelweingart...@gmail.com> escreveu:
> >
> > > Hey guys,
> > >
> > > Have you guys had time to read through this e-mail? Are there
> volunteers
> > to
> > > help us make CCC happen in Brazil? We need to provide them the topics
> of
> > > tracks that we will be participating until 21/12/2018.
> > >
> > > On Thu, Dec 13, 2018 at 7:11 PM Rafael Weingärtner <
> > > rafaelweingart...@gmail.com> wrote:
> > >
> > > > Hello CloudStackers,
> > > >
> > > > I had a few meetings with the TDC folks, and we seem to be moving on.
> > > They
> > > > have a slightly different organization than ApacheCon though.
> > Therefore,
> > > we
> > > > were asked to provide them with some “track topics” that fit in the
> > area
> > > of
> > > > Cloud Computing. Then, we could direct presentations to one of these
> > > > tracks. The idea is that the international tracks (the ones that will
> > be
> > > in
> > > > English) will not be parallelized to enable the audience to attend
> all
> > of
> > > > them (this means, one for each day). Also, the tracks will receive
> > > > presentations from other people that are not in our bubble, and this
> is
> > > > great (at least I found this awesome), because different people with
> > > > different backgrounds would come together on the same track, which in
> > > turn
> > > > means, people that might not know ACS would have the opportunity not
> > just
> > > > to meet the solution, but also the people behind it.
> > > >
> > > > So, this is what I have in mind:
> > > >
> > > >- Cloud computing (area/topic)
> > > >   - cloud orchestration -- this would be the track where topics
> > > >   regarding features, and cloud orchestration systems (e.g.
> > > CloudStack)
> > > >   design and structure would be presented
> > > >   - DevOps -- track for presentations that address the day-to-day
> > of
> > > >   CloudStack (or OpenStack) developers and the daily life of
> > > operators with
> > > >   tasks such as debugging and troubleshooting
> > > >   - tests -- track for discussing the Q&A process and testing
> > methods
> > > >   for clouds
> > > >   - cloud open source ecosystem -- track focusing on the cloud
> > > >   ecosystem, where people can address things relating the job
> > market,
> > > >   business opportunities, and the management process of highly
> > > heterogeneous
> > > >   and distributed communities in OpenSource (such as CloudStack)
> > > >
> > > >
> > > > What do you guys think of these divisions for the CFP?
> > > > Also, we might need help to review and select presentation proposals.
> > > > Would some of you guys be willing to help on this process?
> > > >
> > > > And last, but not least, it would be awesome if companies linked to
> ACS
> > > > are interested to be the sponsors of tracks or the event. They have
> > sent
> > > me
> > > > the brochure and sponsorship prospects from 2018 so we can get to
> know
> > > > better the conference [1]. The attendance report and prospectus are
> in
> > > > English, and for instance, in 2018 the TDC event in Florianopolis
> > (where
> > > we
> > > > are proposing to have CCC in 2019) received about 4000 people. The
> > > > sponsorship prospectus for 2019 events is being prepared, and I guess
> > if
> > > > there are interested parties on this, you can reach them directly, or
> > if
> > > > you have some problems to do that, I can help you guys as well.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://www.dropbox.com/sh/53ujp2usf402dlj/AAA1a2jZPddGcAT8ZosRiGCAa?dl=0
> > > >
> > > > On Wed, Oct 24, 2018 at 8:16 PM Tutkowski, Mike <
> > > mike.tutkow...@netapp.com>
> > > > wrote:
> > > >
> > > >> Thanks, Rafael!
> > > >>
> > > >> The dates work for me.
> > > >>
> > > >> Get Outlook for iOS
> > > >> 
> > > >> From: Rafael Weingärtner 
> > > >> Sent: Wednesday, October 24, 2018 5:02:14 PM
> > > >> To: users
> > > >> Cc: dev
> > > >> Subject: Re: CloudStack Collab in Brazil
> > > >>
> > > >> NetApp Security WARNING: This is an external email. Do not click
> links
> > > or
> > > >> open attachments unless you recognize the sender and know the
> content
> > is
> > > >> safe.
> > > >>
> > > >>
> > > >>
> > > >>
> 

Re: xapi jar as separate release

[Tim Mackey]

Curious if XAPI being LGPL2 matters for this discussion.

http://www.xenproject.org/developers/teams/xapi.html

-tim


On Wed, Jul 16, 2014 at 9:28 AM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Doh! I mixed up master (4.5) with 4.4 in my e-mail. My mistake. I
> understand your question now, Daan.
>
> On Wednesday, July 16, 2014, Daan Hoogland 
> wrote:
>
> > Hey, I missed this mail. Should I have picked something into 4.4
> > before calling a vote?
> >
> > On Wed, Jul 16, 2014 at 6:42 AM, Mike Tutkowski
> > > wrote:
> > > That's exactly what I was wondering, Alex. :)
> > >
> > >
> > > On Tue, Jul 15, 2014 at 3:13 PM, Alex Huang  > > wrote:
> > >
> > >> I checked into master the changes that remove our copy and just use
> the
> > >> copy the XenServer team checked into maven.  Is there any reason we're
> > >> talking about this?
> > >>
> > >> --Alex
> > >>
> > >> > -Original Message-
> > >> > From: David Nalley [mailto:da...@gnsa.us ]
> > >> > Sent: Tuesday, July 15, 2014 1:40 AM
> > >> > To: dev@cloudstack.apache.org 
> > >> > Subject: Re: xapi jar as separate release
> > >> >
> > >> > XAPI is not an apache project. I do not believe that we can sanely
> > make a
> > >> > separate release. Compared to bundling it with our release as we We
> > have
> > >> > essentially 'forked' XAPI from the upstream at Xen Project and made
> > our
> > >> > own changes. Those changes are in the latest version AIUI, but not
> the
> > >> > version that we are currently using.
> > >> >
> > >> > --David
> > >> >
> > >> >
> > >> > On Mon, Jul 14, 2014 at 6:17 AM, Daan Hoogland <
> > daan.hoogl...@gmail.com >
> > >> > wrote:
> > >> > > LS,
> > >> > >
> > >> > > One of the issues with the last RC was that the cloudstack xapi
> was
> > >> > > not a released version (i.e. 6.2.0-1-SNAPSHOT). In the release
> build
> > >> > > it was numbered as 6.2.0-1 but not referred by that release number
> > but
> > >> > > by the snapshot id. There are several solutions to this. The most
> > >> > > correct would seem to be to release the xapi module separately and
> > >> > > create a release for it. I know that there has been some shifting
> > back
> > >> > > and forth with this module and I would like to get consensus to
> > create
> > >> > > a separate module and release for it.
> > >> > >
> > >> > > So my question: How do I get a version into the maven repo?
> > >> > >
> > >> > > thanks,
> > >> > > --
> > >> > > Daan
> > >>
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkow...@solidfire.com 
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > *™*
> >
> >
> >
> > --
> > Daan
> >
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> *™*
>

Re: CLOUDSTACK-7184

[Tim Mackey]

Remi,

Did you have the native XenServer HA enabled at the same time?

-tim
On Jul 25, 2014 8:31 AM, "Remi Bergsma"  wrote:

> Hi guys,
>
> We had some serious corruption today, described in CLOUDSTACK-7184
> (XenServer specific).
> https://issues.apache.org/jira/browse/CLOUDSTACK-7184
>
> Some discussion/help on how to prevent these issues is appreciated. Thx!
>
> Kind Regards,
> Remi Bergsma
>
>

Re: CLOUDSTACK-7184

[Tim Mackey]

Ok. I'm boarding a plane now, but would love to get a copy of the XenServer
logs from the relevant hosts, and the management server logs.  I'm wanting
to see why XenServer allowed a vm to start with the same disk on two hosts.
On Jul 25, 2014 8:53 AM, "Remi Bergsma"  wrote:

> Hi Tim,
>
> No, we do not use XenServer HA.
>
> Remi
>
>
>
> On 7/25/14, 5:46 PM, "Tim Mackey"  wrote:
>
> >Remi,
> >
> >Did you have the native XenServer HA enabled at the same time?
> >
> >-tim
> >On Jul 25, 2014 8:31 AM, "Remi Bergsma" 
> >wrote:
> >
> >> Hi guys,
> >>
> >> We had some serious corruption today, described in CLOUDSTACK-7184
> >> (XenServer specific).
> >> https://issues.apache.org/jira/browse/CLOUDSTACK-7184
> >>
> >> Some discussion/help on how to prevent these issues is appreciated. Thx!
> >>
> >> Kind Regards,
> >> Remi Bergsma
> >>
> >>
>
>

Re: xenserver 6.5

[Tim Mackey]

I know that master had a bunch of cleanup work to make things work better
(commits were a month ago), but baring any significant issues, being able
to support a newer XenServer should be as simple as a database update.  So
net of this master *today* should work fine with 6.5 (and the various
pre-release builds since beta.2).

On Mon, Oct 20, 2014 at 12:45 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Someone correct me if I'm wrong but, if a previous XenServer resource class
> can handle the newer version of XenServer, then I don't think you need to
> make any changes to CloudStack files to use that newer version.
>
> If you do see some incompatibility with that version of XenServer, then
> someone would need to create a new resource class to handle the
> discrepancies.
>
> On Monday, October 20, 2014, Adrian Lewis 
> wrote:
>
> > Out of interest, on the assumption that there are no issues with using
> 6.5
> > when it's released and there are no backwards-compatibility problems,
> will
> > it then work with 4.4.1 or does CS need to be *explicitly* told that
> newer,
> > effectively unknown versions are 'acceptable' as a valid hypervisor?
> > Basically, If we deploy CS 4.4.1 and we like the look of XS 6.5 when it
> > comes out, will we need to make any changes to CS to start using it? If
> so,
> > are these simple edits to the contents of a file or would it require
> > rebuilding?
> >
> > -Original Message-
> > From: Stephen Turner [mailto:stephen.tur...@citrix.com ]
> > Sent: 20 October 2014 15:28
> > To: dev@cloudstack.apache.org 
> > Subject: RE: xenserver 6.5
> >
> > I think it should be minimal, because although there are large internal
> > changes (e.g., 3.x kernel, 64-bit dom0, new Xen, new storage datapath,
> > PVHVM
> > mode for RHEL/CentOS 7), the interface is essentially unchanged.
> >
> > --
> > Stephen Turner
> >
> >
> > -Original Message-
> > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com ]
> > Sent: 20 October 2014 14:32
> > To: dev
> > Subject: xenserver 6.5
> >
> > Does anybody (know of) work on supporting xenserver 6.5 or has an idea of
> > how much effort that is going to be?
> >
> > --
> > Daan
> >
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> *™*
>

Re: xenserver 6.5

[Tim Mackey]

Daan,

Here are the relevant commits:

https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commit;h=2be02d1f515d8d089b6596127614fe6b8030d723
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commit;h=b7f5e95c8f17cf42d35705872b4210db8c2def72
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commit;h=674af6e47313fa18c18536a2daed90d13b9a9a59

Mike,

Here's an example of the type of DB changes:
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blobdiff;f=setup/db/db/schema-441to450.sql;h=e6aae8e3d624744af9f19b132fa8f53b5a4cddb5;hp=34d5f8842005f8a2da4df8a9a838d919cc648831;hb=2be02d1f515d8d089b6596127614fe6b8030d723;hpb=f212aa57c32eb05d6a69730e37ac50bdb1f0a268

On Mon, Oct 20, 2014 at 1:51 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Yeah, Tim, I'm a little unclear of what you mean by requiring a DB update.
>
> Can you clarify that?
>
> Thanks!
>
> On Mon, Oct 20, 2014 at 11:29 AM, Daan Hoogland 
> wrote:
>
> > Tim, these changes are needed? so 4.4.1 will not work with db changes...
> Do
> > you have a commit id?
> >
> > On Mon, Oct 20, 2014 at 6:54 PM, Tim Mackey  wrote:
> >
> > > I know that master had a bunch of cleanup work to make things work
> better
> > > (commits were a month ago), but baring any significant issues, being
> able
> > > to support a newer XenServer should be as simple as a database update.
> > So
> > > net of this master *today* should work fine with 6.5 (and the various
> > > pre-release builds since beta.2).
> > >
> > > On Mon, Oct 20, 2014 at 12:45 PM, Mike Tutkowski <
> > > mike.tutkow...@solidfire.com> wrote:
> > >
> > > > Someone correct me if I'm wrong but, if a previous XenServer resource
> > > class
> > > > can handle the newer version of XenServer, then I don't think you
> need
> > to
> > > > make any changes to CloudStack files to use that newer version.
> > > >
> > > > If you do see some incompatibility with that version of XenServer,
> then
> > > > someone would need to create a new resource class to handle the
> > > > discrepancies.
> > > >
> > > > On Monday, October 20, 2014, Adrian Lewis <
> adr...@alsiconsulting.co.uk
> > >
> > > > wrote:
> > > >
> > > > > Out of interest, on the assumption that there are no issues with
> > using
> > > > 6.5
> > > > > when it's released and there are no backwards-compatibility
> problems,
> > > > will
> > > > > it then work with 4.4.1 or does CS need to be *explicitly* told
> that
> > > > newer,
> > > > > effectively unknown versions are 'acceptable' as a valid
> hypervisor?
> > > > > Basically, If we deploy CS 4.4.1 and we like the look of XS 6.5
> when
> > it
> > > > > comes out, will we need to make any changes to CS to start using
> it?
> > If
> > > > so,
> > > > > are these simple edits to the contents of a file or would it
> require
> > > > > rebuilding?
> > > > >
> > > > > -Original Message-
> > > > > From: Stephen Turner [mailto:stephen.tur...@citrix.com
> > ]
> > > > > Sent: 20 October 2014 15:28
> > > > > To: dev@cloudstack.apache.org 
> > > > > Subject: RE: xenserver 6.5
> > > > >
> > > > > I think it should be minimal, because although there are large
> > internal
> > > > > changes (e.g., 3.x kernel, 64-bit dom0, new Xen, new storage
> > datapath,
> > > > > PVHVM
> > > > > mode for RHEL/CentOS 7), the interface is essentially unchanged.
> > > > >
> > > > > --
> > > > > Stephen Turner
> > > > >
> > > > >
> > > > > -Original Message-
> > > > > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com
> ]
> > > > > Sent: 20 October 2014 14:32
> > > > > To: dev
> > > > > Subject: xenserver 6.5
> > > > >
> > > > > Does anybody (know of) work on supporting xenserver 6.5 or has an
> > idea
> > > of
> > > > > how much effort that is going to be?
> > > > >
> > > > > --
> > > > > Daan
> > > > >
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkow...@solidfire.com
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >
> > >
> >
> >
> >
> > --
> > Daan
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: xenserver 6.5

[Tim Mackey]

Correct on both counts

On Mon, Oct 20, 2014 at 2:55 PM, Daan Hoogland 
wrote:

> thanks Tim, from this I take that hypervisor versions are hardcoded still,
> and xenserver 6.5 is supported since 4.5. correct?
>
> On Mon, Oct 20, 2014 at 8:36 PM, Tim Mackey  wrote:
>
> > Daan,
> >
> > Here are the relevant commits:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commit;h=2be02d1f515d8d089b6596127614fe6b8030d723
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commit;h=b7f5e95c8f17cf42d35705872b4210db8c2def72
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commit;h=674af6e47313fa18c18536a2daed90d13b9a9a59
> >
> > Mike,
> >
> > Here's an example of the type of DB changes:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blobdiff;f=setup/db/db/schema-441to450.sql;h=e6aae8e3d624744af9f19b132fa8f53b5a4cddb5;hp=34d5f8842005f8a2da4df8a9a838d919cc648831;hb=2be02d1f515d8d089b6596127614fe6b8030d723;hpb=f212aa57c32eb05d6a69730e37ac50bdb1f0a268
> >
> > On Mon, Oct 20, 2014 at 1:51 PM, Mike Tutkowski <
> > mike.tutkow...@solidfire.com> wrote:
> >
> > > Yeah, Tim, I'm a little unclear of what you mean by requiring a DB
> > update.
> > >
> > > Can you clarify that?
> > >
> > > Thanks!
> > >
> > > On Mon, Oct 20, 2014 at 11:29 AM, Daan Hoogland <
> daan.hoogl...@gmail.com
> > >
> > > wrote:
> > >
> > > > Tim, these changes are needed? so 4.4.1 will not work with db
> > changes...
> > > Do
> > > > you have a commit id?
> > > >
> > > > On Mon, Oct 20, 2014 at 6:54 PM, Tim Mackey 
> wrote:
> > > >
> > > > > I know that master had a bunch of cleanup work to make things work
> > > better
> > > > > (commits were a month ago), but baring any significant issues,
> being
> > > able
> > > > > to support a newer XenServer should be as simple as a database
> > update.
> > > > So
> > > > > net of this master *today* should work fine with 6.5 (and the
> various
> > > > > pre-release builds since beta.2).
> > > > >
> > > > > On Mon, Oct 20, 2014 at 12:45 PM, Mike Tutkowski <
> > > > > mike.tutkow...@solidfire.com> wrote:
> > > > >
> > > > > > Someone correct me if I'm wrong but, if a previous XenServer
> > resource
> > > > > class
> > > > > > can handle the newer version of XenServer, then I don't think you
> > > need
> > > > to
> > > > > > make any changes to CloudStack files to use that newer version.
> > > > > >
> > > > > > If you do see some incompatibility with that version of
> XenServer,
> > > then
> > > > > > someone would need to create a new resource class to handle the
> > > > > > discrepancies.
> > > > > >
> > > > > > On Monday, October 20, 2014, Adrian Lewis <
> > > adr...@alsiconsulting.co.uk
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Out of interest, on the assumption that there are no issues
> with
> > > > using
> > > > > > 6.5
> > > > > > > when it's released and there are no backwards-compatibility
> > > problems,
> > > > > > will
> > > > > > > it then work with 4.4.1 or does CS need to be *explicitly* told
> > > that
> > > > > > newer,
> > > > > > > effectively unknown versions are 'acceptable' as a valid
> > > hypervisor?
> > > > > > > Basically, If we deploy CS 4.4.1 and we like the look of XS 6.5
> > > when
> > > > it
> > > > > > > comes out, will we need to make any changes to CS to start
> using
> > > it?
> > > > If
> > > > > > so,
> > > > > > > are these simple edits to the contents of a file or would it
> > > require
> > > > > > > rebuilding?
> > > > > > >
> > > > > > > -Original Message-
> > > > > > > From: Stephen Turner [mailto:stephen.tur...@citrix.com
> > > > ]
> > > > > > > Sent: 20 October 2014 15:28
> > > > > > > To: dev@cloudstack.apa

Re: [DISCUSS] Should we install XenServer tools in systemvmtemplates

[Tim Mackey]

>From the XenServer perspective, we have a small problem.  The XenServer
tools are backward compatible, but not guaranteed forward compatible.  What
that means is we'd need to include the XenServer 5.6 tools, and those have
a commercial license.  The XenServer 5.6 tools also have a few bugs which
were fixed in later versions.

I'm certain we could define a minimum version which would work, and could
solve any commercial license concerns, but what about having the system VMs
automatically download and apply the correct tools?  Since we know the
hypervisor version, could we even go so far as automatically update the
tools should the hypervisor be upgraded?

-tim

On Thu, Nov 20, 2014 at 10:19 AM, Rohit Yadav 
wrote:

> Hi,
>
> During the CCCEU conference, I’ve been in some discussions with few people
> on whether we should install/bundle xenserver tools (and vmware tools
> and/or virtio-modules).
>
> I think we can easily do this, but let’s discuss the following;
>
> - Which version of xs-tools etc. we should install? For the build system,
> is a publicly available source (iso etc)?
> - Will installing these tools cause any issue?
> - If we install all these tools on a single template, could that cause any
> issue?
>
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +91 88 262 30892 | rohit.ya...@shapeblue.com
> Blog: bhaisaab.org | Twitter: @_bhaisaab
>
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework
> CloudStack Consulting
> CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<
> http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape Blue
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
> a company registered by The Republic of South Africa and is traded under
> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>

Re: [DISCUSS] Should we install XenServer tools in systemvmtemplates

[Tim Mackey]

It's the other way around.  Newer XenServer works with older tools, not the
other way.
On Nov 20, 2014 4:57 PM, "Rohit Yadav"  wrote:

> If XenServer tools are backward compatible then we can perhaps install the
> latest xs-tools (6.2)? Will that cause issue if the underlying
> Xen/XenServer hypervisor version is not 6.2?
>
> > On 20-Nov-2014, at 9:12 pm, Tim Mackey  wrote:
> >
> > From the XenServer perspective, we have a small problem.  The XenServer
> > tools are backward compatible, but not guaranteed forward compatible.
> What
> > that means is we'd need to include the XenServer 5.6 tools, and those
> have
> > a commercial license.  The XenServer 5.6 tools also have a few bugs which
> > were fixed in later versions.
> >
> > I'm certain we could define a minimum version which would work, and could
> > solve any commercial license concerns, but what about having the system
> VMs
> > automatically download and apply the correct tools?  Since we know the
> > hypervisor version, could we even go so far as automatically update the
> > tools should the hypervisor be upgraded?
> >
> > -tim
> >
> > On Thu, Nov 20, 2014 at 10:19 AM, Rohit Yadav  >
> > wrote:
> >
> >> Hi,
> >>
> >> During the CCCEU conference, I’ve been in some discussions with few
> people
> >> on whether we should install/bundle xenserver tools (and vmware tools
> >> and/or virtio-modules).
> >>
> >> I think we can easily do this, but let’s discuss the following;
> >>
> >> - Which version of xs-tools etc. we should install? For the build
> system,
> >> is a publicly available source (iso etc)?
> >> - Will installing these tools cause any issue?
> >> - If we install all these tools on a single template, could that cause
> any
> >> issue?
> >>
> >> Regards,
> >> Rohit Yadav
> >> Software Architect, ShapeBlue
> >> M. +91 88 262 30892 | rohit.ya...@shapeblue.com
> >> Blog: bhaisaab.org | Twitter: @_bhaisaab
> >>
> >> Find out more about ShapeBlue and our range of CloudStack related
> services
> >>
> >> IaaS Cloud Design & Build<
> >> http://shapeblue.com/iaas-cloud-design-and-build//>
> >> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/
> >
> >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> >> CloudStack Software Engineering<
> >> http://shapeblue.com/cloudstack-software-engineering/>
> >> CloudStack Infrastructure Support<
> >> http://shapeblue.com/cloudstack-infrastructure-support/>
> >> CloudStack Bootcamp Training Courses<
> >> http://shapeblue.com/cloudstack-training/>
> >>
> >> This email and any attachments to it may be confidential and are
> intended
> >> solely for the use of the individual to whom it is addressed. Any views
> or
> >> opinions expressed are solely those of the author and do not necessarily
> >> represent those of Shape Blue Ltd or related companies. If you are not
> the
> >> intended recipient of this email, you must neither take any action based
> >> upon its contents, nor copy or show it to anyone. Please contact the
> sender
> >> if you believe you have received this email in error. Shape Blue Ltd is
> a
> >> company incorporated in England & Wales. ShapeBlue Services India LLP
> is a
> >> company incorporated in India and is operated under license from Shape
> Blue
> >> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> Brasil
> >> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd
> is
> >> a company registered by The Republic of South Africa and is traded under
> >> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
> >>
>
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +91 88 262 30892 | rohit.ya...@shapeblue.com
> Blog: bhaisaab.org | Twitter: @_bhaisaab
>
>
>
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/>
&

Re: [DISCUSS] Should we install XenServer tools in systemvmtemplates

[Tim Mackey]

The big problem I recall with the 5.6 Linux tools happened during live
migration.  I don't recall the specific circumstances, but if the VM was
migrated, it could cause either the guest or the host to crash.  I've never
paid attention to the specific Linux version in the system VMs, but the
tools themselves aren't supported for installation on a newer version of an
operating system than what was originally certified.  XenCenter when faced
with 5.6 tools on a 6.2 host would actually say that the tools are there
but need to be upgraded for optimal performance.

This also opens the question of XCP and the tools since XenServer tools
were never designed to work with XCP.  They probably do work just fine, but
if there was an issue we'd be challenged to get it sorted.

If the option of automatically having the tools installed isn't viable, the
next things I'd look at would be:

- removing support for XenServer versions prior to 6.0
- removing support for XCP (or at least only supporting 1.6)
- having the rpm/deb packages for each version of hypervisor pre-loaded
(assuming redist EULA) and when a systemVM starts having a first boot which
installs the correct package and then reboots the system VM

At least that way the tools would be there, and we'd have better system VM
performance while still locking ourselves to specific supported hypervisor
versions.  No idea what impact removal of support for older XenServer
versions might have on the install base, but I'd hope those on 5.6 would at
least be planning an upgrade given 5.6 went EOL from Citrix a while back.

-tim

On Fri, Nov 21, 2014 at 3:15 AM, Rohit Yadav 
wrote:

> Tim, during 4.0/4.1 we used to install a very old xstools 5.6, can we just
> use that and if we do will that cause any (potential) issue since you
> mentioned it has few bugs in it?
>
> Regards.
>
> > On 20-Nov-2014, at 10:23 pm, Tim Mackey  wrote:
> >
> > It's the other way around.  Newer XenServer works with older tools, not
> the
> > other way.
> > On Nov 20, 2014 4:57 PM, "Rohit Yadav" 
> wrote:
> >
> >> If XenServer tools are backward compatible then we can perhaps install
> the
> >> latest xs-tools (6.2)? Will that cause issue if the underlying
> >> Xen/XenServer hypervisor version is not 6.2?
> >>
> >>> On 20-Nov-2014, at 9:12 pm, Tim Mackey  wrote:
> >>>
> >>> From the XenServer perspective, we have a small problem.  The XenServer
> >>> tools are backward compatible, but not guaranteed forward compatible.
> >> What
> >>> that means is we'd need to include the XenServer 5.6 tools, and those
> >> have
> >>> a commercial license.  The XenServer 5.6 tools also have a few bugs
> which
> >>> were fixed in later versions.
> >>>
> >>> I'm certain we could define a minimum version which would work, and
> could
> >>> solve any commercial license concerns, but what about having the system
> >> VMs
> >>> automatically download and apply the correct tools?  Since we know the
> >>> hypervisor version, could we even go so far as automatically update the
> >>> tools should the hypervisor be upgraded?
> >>>
> >>> -tim
> >>>
> >>> On Thu, Nov 20, 2014 at 10:19 AM, Rohit Yadav <
> rohit.ya...@shapeblue.com
> >>>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> During the CCCEU conference, I’ve been in some discussions with few
> >> people
> >>>> on whether we should install/bundle xenserver tools (and vmware tools
> >>>> and/or virtio-modules).
> >>>>
> >>>> I think we can easily do this, but let’s discuss the following;
> >>>>
> >>>> - Which version of xs-tools etc. we should install? For the build
> >> system,
> >>>> is a publicly available source (iso etc)?
> >>>> - Will installing these tools cause any issue?
> >>>> - If we install all these tools on a single template, could that cause
> >> any
> >>>> issue?
> >>>>
> >>>> Regards,
> >>>> Rohit Yadav
> >>>> Software Architect, ShapeBlue
> >>>> M. +91 88 262 30892 | rohit.ya...@shapeblue.com
> >>>> Blog: bhaisaab.org | Twitter: @_bhaisaab
> >>>>
> >>>> Find out more about ShapeBlue and our range of CloudStack related
> >> services
> >>>>
> >>>> IaaS Cloud Design & Build<
> >>>> http://shapeblue.com/iaas-cloud-design-and-build//&

[DISCUSS] Issues with Ubuntu instance creation

[Tim Mackey]

I've been working through a series of issues getting Ubuntu 12.04 LTS
templates to provision correctly, and I *think* most are really doc issues,
but before I run off and update docs I wanted to confirm that I'm doing the
right thing.  Here's my list of issues, and what I did to get past my
"issue".  My documentation source is:
http://cloudstack-administration.readthedocs.org/en/latest/templates.html.
My CloudStack is 4.4.

1. The docs make no mention of an Ubuntu change password script, and Google
returns Shankar's GitHub scripts as option #4.  Unfortunately, that script
has a user of "ubuntu" hardcoded into it, so unless your template has an
"ubuntu" user, its not going to work.  I haven't tried to use the stock
CloudStack password change script in my template, but have found references
to it not working as expected.  For my purposes, I changed Shankar's script
to use a "root" user, but this leaves the following questions open:

- Does the current CloudStack script work with Ubuntu 12.04 and later?  If
so, I vote the docs be updated to reflect support for Ubuntu 12.04 and
later; with the objective of both clarifying the docs and helping boost our
docs to a higher rank than Shankar's GitHub.

- If the current CloudStack script doesn't work with Ubuntu 12.04 LTS,
should a JIRA ticket be entered to resolve this, or should we have multiple
scripts available and effectively incorporate Shankar's work more
officially?

2. The docs recommend setting the password to expire, but when the change
password script runs, that flag is cleared and the user isn't promoted to
reset the root password.  That leaves the following question in my mind.

- Is our password intended to be a one-time use password.  If so, then the
password change script should reset expiration forcing a new one to be
set.  If not, then should we not remove the "expire password"
recommendation from the docs?

3. The script in the docs covering clearing the logs (step 6) doesn't
include clearing syslog.  Recommend updating the script to include:  cat
/dev/null > /var/log/syslog 2>/dev/null

4. The script in the docs covering clearing of command history (step 9)
doesn't clear the in memory history.  Recommend updating the script to
become:  cat /dev/null > ~/.bash_history && history -c && unset HISTFILE &&
halt -p.  This would also remove the the shutdown step (step 10).

5. The script to set the hostname has a race condition which effectively
means it rarely sets the hostname correctly on initial boot.  I've attached
the script I used.  It doesn't depend upon the leases file being present,
and took care of some alternate "blank" hostname cases I encountered while
debugging.

I'm happy to update the docs, but want to make certain what I've
encountered as issues are things we care about updating.

-tim

Re: XenServer 6.5 RC and ACS

[Tim Mackey]

It's in master since September, and I think 4.5, but haven't explicitly
tried that.
On Dec 29, 2014 12:47 PM, "Pierre-Luc Dion"  wrote:

> Hi Andrei,
>
> I haven't tested it but it might work with ACS 4.4. do you have a lab to
> validate this ? their is reference to XS6.5 into the code which is why I'm
> expecting to have it working with 4.4.
>
> Cheers,
>
> PL
>
>
> On Mon, Dec 29, 2014 at 12:40 PM, Andrei Mikhailovsky 
> wrote:
>
> > Hello guys,
> >
> > Is there a way to connect XenServer 6.5 RC to ACS? I am using 4.2.1 at
> the
> > moment and I get an error when I try to add the host. Any unofficial tips
> > on how this could be done on 4.2.1? Or do I need to wait for acs 4.5 to
> be
> > out?
> >
> > Cheers
> >
> > Andrei
> >
>

Packer for XenServer VHDs

[Tim Mackey]

Everyone,

I've been extending a XenServer builder some of the XenServer engineering
team have been working on to produce a VHD which would be suitable for
upload as a template.   CentOS 7 is my guest OS type and have the XenServer
tools going in and guest password script running as a systemd service.

>From my perspective this is goodness, and I'll be showing at FOSDEM, but I
wanted to double check if there is more I should add in to make this as
usable as possible.  If anyone has experience with CentOS 7 templates and
can share any gotchas, I can see if I can take care of them in the packer
provisioners.  My plan is to also have samples for CentOS 6.5 and Ubuntu
14.04, and those look more straight forward, but any gotchas someone wants
to point out will be welcome.

Thanks

-tim

Re: Packer for XenServer VHDs

[Tim Mackey]

Thanks.  Once I regain access to my hosts I'll work on getting that in.
I'm assuming this is general to all OSes, not just CentOS7.

-tim

On Tue, Jan 27, 2015 at 2:31 AM, Nux!  wrote:

> Hi Tim,
>
> I think cloud-init is a must. I am maintaining CentOS building at
> http://dl.openvm.eu using virt-install+kickstarts (
> http://dl.openvm.eu/cloudstack/centos/ks/ ).
>
> HTH
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> - Original Message -
> > From: "Tim Mackey" 
> > To: dev@cloudstack.apache.org
> > Sent: Tuesday, 27 January, 2015 03:57:45
> > Subject: Packer for XenServer VHDs
>
> > Everyone,
> >
> > I've been extending a XenServer builder some of the XenServer engineering
> > team have been working on to produce a VHD which would be suitable for
> > upload as a template.   CentOS 7 is my guest OS type and have the
> XenServer
> > tools going in and guest password script running as a systemd service.
> >
> > From my perspective this is goodness, and I'll be showing at FOSDEM, but
> I
> > wanted to double check if there is more I should add in to make this as
> > usable as possible.  If anyone has experience with CentOS 7 templates and
> > can share any gotchas, I can see if I can take care of them in the packer
> > provisioners.  My plan is to also have samples for CentOS 6.5 and Ubuntu
> > 14.04, and those look more straight forward, but any gotchas someone
> wants
> > to point out will be welcome.
> >
> > Thanks
> >
> > -tim
>

Re: [DISCUSS] Important: XenServer labels reverted for backward compatibility

[Tim Mackey]

Rohit,  how does the issue manifest itself?  I ask because I thought I'd
taken care of every scenario during upgrade.
On Feb 2, 2015 10:52 AM, "Rohit Yadav"  wrote:

> Hi Tim,
>
> Geoff found a issue today that breaks backward compatibility for
> XenServer users.
>
> Until 4.4, XenServer traffic label was xennetworklabel but in 4.5/master
> it is xenservernetworklabel. To keep backward compatibility I'm
> reverting such changes introduced in
> a8212d9ef458dd7ac64b021e6fa33fcf64b3cce0 (xenserver plugin refactoring).
> When in future we'll have a separate xen project plugin (based on
> libvirt or what have you) we should add a new label like
> xenprojectnetworklabel but let's keep the old one for backward
> compatibility's sake.
>
> It could be valid change, if so please advise any other way to maintain
> backward compatibility? Please comment if this is okay?
>
> The issue has been fixed on 4.5/master now:
> https://issues.apache.org/jira/browse/CLOUDSTACK-8190
>
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +91 8826230892 | rohit.ya...@shapeblue.com
> Blog: bhaisaab.org | Twitter: @_bhaisaab
> PS. If you see any footer below, I did not add it :)
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework
> CloudStack Consulting
> CloudStack Software Engineering engineering/>
> CloudStack Infrastructure Support cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape Blue
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
> a company registered by The Republic of South Africa and is traded under
> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>

RE: [DISCUSS] Important: XenServer labels reverted for backward compatibility

[Tim Mackey]

Iirc, that was the only one.

I don't agree with the reversion, but let me look at supporting both once I
get to my desk.  Do we have any api commitment specs out there?  If so, I'd
like to mark the "Xen" one deprecated
On Feb 2, 2015 11:14 AM, "Geoff Higginbottom" <
geoff.higginbot...@shapeblue.com> wrote:

>  Hi Tim,
>
>
>
> The issue affects new deployments using existing tooling which references
> the original api parameter of xentrafficlabel.
>
>
>
> It will also affect any tooling which may update the labels, although this
> is a rare occurance.
>
>
>
> The main issue is backwards compatibility, but also a concern that there
> may be other areas where the api parameters have changed from xen to
> xenserver.
>
>
>
> Anyone who has developed a custom UI may run into issues.
>
>
>
> Is there a list of all the API changes that have been made?
>
>
>
> Regards
>
>
>
> Geoff Higginbottom
>
>
>
> D: +44 20 3603 0542 <+442036030542> | S: +44 20 3603 0540 <+442036030540>
> | M: +447968161581
>
>
>
> geoff.higginbot...@shapeblue.com
>
>
>
> *From:* Tim Mackey [mailto:tmac...@gmail.com]
> *Sent:* 02 February 2015 11:05
> *To:* Rohit Yadav
> *Cc:* Geoff Higginbottom; dev@cloudstack.apache.org
> *Subject:* Re: [DISCUSS] Important: XenServer labels reverted for
> backward compatibility
>
>
>
> Rohit,  how does the issue manifest itself?  I ask because I thought I'd
> taken care of every scenario during upgrade.
>
> On Feb 2, 2015 10:52 AM, "Rohit Yadav"  wrote:
>
> Hi Tim,
>
> Geoff found a issue today that breaks backward compatibility for
> XenServer users.
>
> Until 4.4, XenServer traffic label was xennetworklabel but in 4.5/master
> it is xenservernetworklabel. To keep backward compatibility I'm
> reverting such changes introduced in
> a8212d9ef458dd7ac64b021e6fa33fcf64b3cce0 (xenserver plugin refactoring).
> When in future we'll have a separate xen project plugin (based on
> libvirt or what have you) we should add a new label like
> xenprojectnetworklabel but let's keep the old one for backward
> compatibility's sake.
>
> It could be valid change, if so please advise any other way to maintain
> backward compatibility? Please comment if this is okay?
>
> The issue has been fixed on 4.5/master now:
> https://issues.apache.org/jira/browse/CLOUDSTACK-8190
>
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +91 8826230892 | rohit.ya...@shapeblue.com
> Blog: bhaisaab.org | Twitter: @_bhaisaab
> PS. If you see any footer below, I did not add it :)
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//
> <http://shapeblue.com/iaas-cloud-design-and-build/>>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<
> http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape Blue
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
> a company registered by The Republic of South Africa and is traded under
> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>
>  Find out more about ShapeBlue and our range of CloudStack related
> services
>
> IaaS Cloud Design & Build
> <http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework <http://shapeblue.com/csforge/>
> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering
> <http:/

Re: [DISCUSS] Important: XenServer labels reverted for backward compatibility

[Tim Mackey]

After poking around the code a bit, this is purely an API issue (was
worried there was more).  When I made the change I explicitly *didn't*
change the actual method names so for example there is still a
getXenLabel() call on the physical network traffic API calls.  That's
probably why we didn't see this sooner, and my fault for not thinking this
one through as well as I should've.

The simplest and fastest answer is to keep Rohit's revert and tackle it
when we get pure Xen in.  The second option would be to create an explicit
getXenServerLabel() call which is wrapped by getXenLabel() to do the name
translation.  Then we mark the getXenLabel() call as deprecated in the
docs, with backwards compat preserved and future work should use
getXenServerLabel().  The second option is obviously the
technically"correct" way to version the API, but will take a bit longer to
work through.

I'm OK with going back to "xennetworklabel" for 4.5 and putting in a ticket
in against master to do the more complete fix.

I also looked at Marvin, and found something which to my eye looks
incorrect, and might also explain why we didn't see this earlier.
 tools/marvin/marvin/deployDataCenter.py line 613/614 has a
traffictype.xen, but I *think* that should be traffictype.xenserver.  If
I'm reading things correctly, that code looks like it's been skipped for
XenServer for a while.  Is there any easy way to validate?

-tim



On Mon, Feb 2, 2015 at 2:52 PM, Rohit Yadav 
wrote:

> Hi Tim,
>
> You may be right on this one, we just want to make sure. Feel free to
> revert the fix is there is a better way around. I personally would like
> to avoid string changes because as Geoff mentions it changes the
> interface for 3rd party consumers such as CLI scripts, custom UI and
> possibly others.
>
> On Monday 02 February 2015 04:53 PM, Tim Mackey wrote:
>
>> Iirc, that was the only one.
>>
>> I don't agree with the reversion, but let me look at supporting both
>> once I get to my desk.  Do we have any api commitment specs out there?
>> If so, I'd like to mark the "Xen" one deprecated
>>
>> On Feb 2, 2015 11:14 AM, "Geoff Higginbottom"
>> > <mailto:geoff.higginbot...@shapeblue.com>> wrote:
>>
>> Hi Tim,
>>
>> __ __
>>
>> The issue affects new deployments using existing tooling which
>> references the original api parameter of xentrafficlabel.
>>
>> __ __
>>
>> It will also affect any tooling which may update the labels,
>> although this is a rare occurance.
>>
>> __ __
>>
>> The main issue is backwards compatibility, but also a concern that
>> there may be other areas where the api parameters have changed from
>> xen to xenserver.
>>
>> __ __
>>
>> Anyone who has developed a custom UI may run into issues.
>>
>> __ __
>>
>> Is there a list of all the API changes that have been made?
>>
>> __ __
>>
>> Regards
>>
>> __ __
>>
>> Geoff Higginbottom
>>
>> __ __
>>
>> D: +44 20 3603 0542 | S: +44 20 3603 0540
>> | M: +447968161581 
>>
>> __ __
>>
>> geoff.higginbot...@shapeblue.com
>> <mailto:geoff.higginbot...@shapeblue.com>
>>
>> __ __
>>
>> *From:*Tim Mackey [mailto:tmac...@gmail.com <mailto:tmac...@gmail.com
>> >]
>> *Sent:* 02 February 2015 11:05
>> *To:* Rohit Yadav
>> *Cc:* Geoff Higginbottom; dev@cloudstack.apache.org
>> <mailto:dev@cloudstack.apache.org>
>> *Subject:* Re: [DISCUSS] Important: XenServer labels reverted for
>> backward compatibility
>>
>> __ __
>>
>> Rohit,  how does the issue manifest itself?  I ask because I thought
>> I'd taken care of every scenario during upgrade.
>>
>> On Feb 2, 2015 10:52 AM, "Rohit Yadav" > <mailto:rohit.ya...@shapeblue.com>> wrote:
>>
>> Hi Tim,
>>
>> Geoff found a issue today that breaks backward compatibility for
>> XenServer users.
>>
>> Until 4.4, XenServer traffic label was xennetworklabel but in
>> 4.5/master
>> it is xenservernetworklabel. To keep backward compatibility I'm
>> reverting such changes introduced in
>> a8212d9ef458dd7ac64b021e6fa33fcf64b3cce0 (xenserver plugin
>> refactoring).
>> When in future we'll have a separate xen pro

Re: Any VxLan Support on Xenserver

[Tim Mackey]

The ovs contained within XenServer 6.5 is capable of supporting VxLAN, but
as Adrian stated the control plane is missing.  Prior XenServer versions
don't have an ovs capable of VxLAN.  I *think* Contrail also supports
VxLAN, but I don't know what its status is wrt XenServer 6.5

-tim

On Wed, Apr 1, 2015 at 9:25 AM, Keerthiraja SJ  wrote:

> But Nuage is commercial one right.
>
> On Tue, Mar 31, 2015 at 4:36 PM, Erik Weber  wrote:
>
> > On Tue, Mar 31, 2015 at 9:39 AM, Keerthiraja SJ 
> > wrote:
> >
> > > Hi All,
> > >
> > > Is there any plan to bring up VxLAN support for xenserver on future
> > release
> > > version.
> > >
> > >
> > Not sure if I remember correct or not, but I think Nuage is VXLAN-based
> and
> > works with XenServer.
> >
> >
> > --
> > Erik
> >
>

Re: Cloudstack and Xen Cloud Platform(XCP)

[Tim Mackey]

The most recent version of XCP is 1.6, and it approximately corresponds to
XenServer 6.1.  XCP was effectively discontinued when XenServer 6.2 was
released in June 2013 (
http://www.xenproject.org/downloads/xen-cloud-platform-archives.html).
Importantly, there are no longer any patches being created for it.  I can
give you guidance about migrating to XenServer 6.2 (or better yet 6.5), but
since this is a CloudStack question, the good news is that from a
CloudStack perspective you can deploy XenServer and everything will work
identically to XCP.  In other words, if you decide sticking with XCP is the
right answer for you, just follow the XenServer instructions, but do be
aware that since XCP is effectively discontinued, limited CloudStack
testing is done with it.

I know that's a bit of a long answer, but this is a bit of a messy area in
the XenServer world and I'd prefer the full answer be out there.

btw, if you want XenServer 6.1 (aka XCP 1.6), please use CloudStack 4.2.1.
Plain 4.2 only supports XenServer 6.0.2, or approximately 1.5 beta 2.

-tim

On Fri, Apr 10, 2015 at 6:18 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> When I was using 4.2, I used XenServer 6.1.
>
> If you really need to use XCP, perhaps someone else knows a compatible
> version number for it.
>
> On Fri, Apr 10, 2015 at 9:30 AM, ShajiA  wrote:
>
> >
> >
> > Hi
> >
> >
> >
> > Please recommend the compatible version of XCP for CloudStack Version 4.2
> >
> >
> >
> > Regards
> >
> >
> >
> > शाजी ए.
> >
> > Shaji A.
> > परियोजना सहायक-I
> >
> > Project Assistant - I
> >
> > स्वास्थ्य सूचना और सॉफ्टवेयर प्रौद्योगिकी समूह (एच.एस.टी.जी)
> >
> > Health Informatics & Software Technology Group ( HSTG)
> >
> > प्रगत संगणन विकास केंद्र ( सीडैक)
> >
> > Centre for Development of Advanced Computing (CDAC)
> >
> > तिरुवनंतपुरम -695033 / Thiruvanathapuram -695033
> >
> >
> >
> >
> >
> >
> >
> >
> ---
> > [ C-DAC is on Social-Media too. Kindly follow us at:
> > Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
> >
> > This e-mail is for the sole use of the intended recipient(s) and may
> > contain confidential and privileged information. If you are not the
> > intended recipient, please contact the sender by reply e-mail and destroy
> > all copies and the original message. Any unauthorized review, use,
> > disclosure, dissemination, forwarding, printing or copying of this email
> > is strictly prohibited and appropriate legal action will be taken.
> >
> >
> ---
> >
> >
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> *™*
>

Re: SG not working with 6.5 and PR review

[Tim Mackey]

Rohit,

I just added a comment to update line 457 to tick-quote the vmchain as
you've done elsewhere.  My main concern would be flushing the ipset while
the iptable entry still exists.

I am curious what in sm/util.py concerned you.  That's all storage
management code and should have nothing to do with security groups.  I also
diffed a 6.5 and 6.2 version which didn't show anything obvious to explain
a security group issue.

ipset definitely did change going from 4.5 to 6.11 to match our kernel
update.

-tim

On Tue, Apr 21, 2015 at 11:57 AM, Rohit Yadav 
wrote:

> Hi all,
>
> We discovered that Security Groups don’t work in ACS 4.5+ when used with
> XenServer 6.5 due to ipset, sm/util.py changes. I’ve opened the issue here
> which was found to be reproducible by my colleagues Geoff and Abhi:
> https://issues.apache.org/jira/browse/CLOUDSTACK-8395
>
> I’ve tried to fix it in a way such that vmops plugin would work on both XS
> 6.2 and 6.5 releases, here's the PR:
> https://github.com/apache/cloudstack/pull/186
>
> One of the major changes it introduces it to use “nethash” instead of
> “iphash” when storing CIDRs received as part of a ingress/egress rule. I’m
> not sure how it will affect users that will upgrade to ACS 4.5, as a
> precaution I’ve added a change to flush and remove old ipset entry before
> adding a new one. (Assuming all network rule addition/removals are
> idempotent, as everytime we add/remove a rule, all rules are sent to be
> applied by the XS vmops plugins).
>
> Tim - since you’re one of the Xen gurus can you help review it and suggest
> any other changes?
>
> I wanted to bring this issue on dev ML since it’s a potential blocker for
> 4.5. I’m not sure if we officially support XS 6.5 on 4.4 branch, but if
> needed once we have a reviewed commit it can be cherry-picked on 4.4 as
> well.
>
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +91 88 262 30892 | rohit.ya...@shapeblue.com
> Blog: bhaisaab.org | Twitter: @_bhaisaab
>
>
>
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework
> CloudStack Consulting
> CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<
> http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape Blue
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
> a company registered by The Republic of South Africa and is traded under
> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>

Re: SG not working with 6.5 and PR review

[Tim Mackey]

Rohit,

There are no code changes to util.pread2, so what you're observing would
have to be an artifact of either stdout changing, or how popen.communicate
handles the trailing newline from the actual command.  Regardless, your
solution should take care of both cases.

For the ## Discuss, I think we should separate the upgrade from the normal
operation.  For normal operation where we've sustained communication with
the host, then the optimistic case should allow for increased scalability.
In the case of upgrade of management server, host in maintenance mode,
upgrade of host, loss of communication with host, we should assume a very
pessimistic case and upon connection with the host validate the rules,
rebuilding those which don't agree with what we expect.  It's more work,
but validation would help ensure our view of the world is maintained.

-tim

On Tue, Apr 21, 2015 at 3:41 PM, Rohit Yadav 
wrote:

> Hi Tim,
>
> Thanks for your review. I’ve fixed the tick-quote issue on the mentioned
> line. Please find the various issues addressed in this patch;
>
> ## 1. Issue with sm/util.py
>
> My concern with sm/util.py was that, previously on XS 6.2 the output of a
> typical util.pread2().split(‘\n’) would return a list like
> [a,b,c,d,”"] with the last element as an empty string possibly because an
> extra newline in the command output. Due to this all the code in vmops
> plugin followed this kind of patten where the programmer coded it to do a
> list.pop() after calling pread2().split(‘\n’).
>
> But in case of XS 6.5, I found that no newline was at the end and we
> should not assume it and doing a list.pop() would remove the last element
> which can be a valid string and not a newline (or empty string after
> calling split() on it). I traced this issue when I saw following methods
> fail in the /var/log/cloud/cloud.log and looked at the code:
> delete_rules_for_vm_in_bridge_firewall_chain
> destroy_ebtables_rules
> destroy_arptables_rules
>
> The fix I tried was to filter out empty and None elements from the list
> using a "filter(util.pread2().split(‘\n’))” kind of pattern
> instead of doing a list.pop() which might remove a valid string. After this
> fix, I saw that SG rules applied correctly and the previous errors went
> away. I believe that this way of defensive coding would work on both XS 6.2
> and 6.5, irrespective of the output having empty lines or not.
>
>
> ## 2. Issue with large CIDRs and ipset overflow
>
> The other issue I found was when I added an ingress or egress CIDR that
> was in the block /8 or /4. I found that ipset failed in the
> /var/log/cloud/cloud.log. On tracing the issue I found that ipset data
> structure has been modified to use iphash instead of iptreemap (done in
> bd6f03aa954d4b3e7ead7e8010c5674d5d1f9513) because recent versions of ipset
> (as in XS 6.5) did not have that data-structure.
>
> The issue with “iphash” in this case was that when someone added a CIDR or
> CIDRs, it would add individual IPs instead of the CIDR itself. Which failed
> for /8 or /4 because the default max length of “iphash” entry was 65535.
> The fix for this was to use “nethash” since we receive CIDRs only and it
> becomes memory/space efficient. Only the ipset entry for the VM itself is
> “iphash” based now (to store a VM’s primary and secondary IPs).
>
>
> ## 3. Issue with removing Ingress/Egress rules
>
> I saw that when an ingress or egress rule was removed while the iptable
> rules were removed/fixed, the ipset entry stayed there and cidrs in it were
> not getting removed. Every time I added or removed an ingress or egress
> entry, CloudStack would send the vmops plugin all the ingress and egress
> rules. This assumed that all add/remove operations were sort of idempotent
> and all information was applied every time we made any change. Therefore,
> as a pessimistic approach I added the fix to flush/remove the ipset entries
> before adding them back again (that the mgmt server sent to vmops plugin).
> This also solves an upgrade error for users who would upgrade to XS 6.5 or
> ACS 4.5, as previous ipset entries were of type “iptreemap” or “iphash”,
> but new entries (the ingress and egress ones) is of “nethash” type.
>
>
> ## Discuss - should we flush/remove ipset rules before applying new rules?
>
> Can you advise if we should have this pessimistic approach (as it flushes
> ipset entries when rules may be still in place?) or use an optimistic
> approach to keep the entries. There are two issues with the current
> optimistic approach:
> 1. When egress/ingress rules are removed, for a brief period (until new
> ipset entry replace the old one) old entries still persist. But since
> iptables  rules are removed and no one is referencing them it is harmless.
>

Re: [DISCUSS] XenServer and HA: the way forward

[Tim Mackey]

Thanks for starting this thread Remi.

>From my perspective the pros of simply enabling XenServer HA are:

- automatic election of pool master in the event of hardware failure
- automatic fencing of a host in the event of dom0 corruption
- automatic fencing of a host in the event of heartbeat failure

The risks of simply enabling XenServer HA are:

- additional code to detect a newly elected pool master
- acceptance of the fact an admin can force a new pool master from
XenServer CLI
- requirement for pool size to be greater than 2 (pool size of 2 results in
semi-deterministic fencing which isn't user obvious)
- understanding that storage heartbeat can be shorter than storage timeout
(aggressive fencing)
- understanding that HA plans are computed even when no VMs are protected
(performance decrease)

One question we'll want to decide on is who is the primary actor when it
comes to creating the pool since that will define the first pool master.
During my demo build using 4.4 at CCCEU I expected to add pool members
through the CS UI, but found that adding them in XenServer was required.
This left me in an indeterminate state wrt pool members.

I vote that if a host is added to CS and it *is* already a member of a
pool, that the pool be imported as a cluster and any future membership
changes happen using CS APIs.  If a host is added which isn't a member of a
pool, then the user be asked if they wish to add it to an existing cluster
(and behind the scenes add it to a pool), or create a new cluster and add
it to that cluster.  This would be a change to the "add host" semantics.
Once the host is added, we can enable XenServer HA on the pool if it
satisfies the requirements for XenServer HA (has shared storage and three
or more pool members).

There are some details we'd want to take care of, but this flow makes sense
to me, and we could use it even with upgrades.

-tim

On Mon, May 4, 2015 at 6:04 AM, Remi Bergsma  wrote:

> Hi all,
>
> Since CloudStack 4.4 the implementation of HA in CloudStack was changed to
> use the XenHA feature of XenServer. As of 4.4, it is expected to have XenHA
> enabled for the pool (not for the VMs!) and so XenServer will be the one to
> elect a new pool master, whereas CloudStack did it before. Also, XenHA
> takes care of fencing the box instead of CloudStack should storage be
> unavailable. To be exact, they both try to fence but XenHA is usually
> faster.
>
> To be 100% clear: HA on VMs is in all cases done by CloudStack. It's just
> that without a pool master, no VMs will be recovered anyway. This brought
> some headaches to me, as first of all I didn't know. We probably need to
> document this somewhere. This is important, because without XenHA turned on
> you'll not get a new pool master (a behaviour change).
>
> Personally, I don't like the fact that we have "two captains" in case
> something goes wrong. But, some say they like this behaviour. I'm OK with
> both, as long as one can choose whatever suits their needs best.
>
> In Austin I talked to several people about this. We came up with the idea
> to have CloudStack check whether XenHA is on or not. If it is, it does the
> current 4.4+ behaviour (XenHA selects new pool master). When it is not, we
> do the CloudStack 4.3 behaviour where CloudStack is fully in control.
>
> I also talked to Tim Mackey and he wants to help implement this, but he
> doesn't have much time. The idea is to have someone else join in to code
> the change and then Tim will be able to help out on a regularly basis
> should we need in depth knowledge of XenServer or its implementation in
> CloudStack.
>
> Before we kick this off, I'd like to discuss and agree that this is the way
> forward. Also, if you're interested in joining this effort let me know and
> I'll kick it off.
>
> Regards,
> Remi
>

Re: Getting Started With Documentation

[Tim Mackey]

On Tue, Oct 20, 2015 at 7:08 PM, Eric  wrote:

> Thanks, Erik!
> Is there a discussion forum where contributors can discuss documentation?
> I believe that, like all things in technology, there should be a set of
> conventions that guide authors in the use of common terms. e.g., Xen? Or
> Xen Project? Or XenServer?
>

One thing we want to ensure is that the terms we use are accurate.  In
CloudStack, as with the general tech world, there is a tendency to make Xen
== XenServer when it isn't.  XenServer is supported supported by CloudStack
via XAPI where the Xen Project Hypervisor isn't currently present in native
form. Oracle VM is also supported by CloudStack, so we'd also want to
ensure any Xen discussion recognizes that Xen doesn't imply XenServer and
vice-versa.

In terms of other conventions, we've evolved to where we are and welcome
input for how the documentation can be both clearer and more usable.

-tim


>
> Eric PretoriousPortland, OR
>
>   From: Erik Weber 
>  To: dev ; Eric 
>  Sent: Sunday, October 18, 2015 11:50 PM
>  Subject: Re: Getting Started With Documentation
>
> On Sun, Oct 18, 2015 at 10:16 PM, Eric 
> wrote:
>
> Hello, All:
>
>
>
> Hi Eric!
>
>
>
> I'd like to contribute to the Cloudstack project documentation.
>
>
> That's excellent!
>
>
>
>
>
> What's the process for that? Is there a SIG? Are there guidelines or a
> style guide?
>
>
> We use ReStructuredText to write our documentation and and readthedocs to
> display them.
> Generally there are two ways to edit the documentation.
>
> 1) Edit it on GitHub. Go to any source file and press the 'Fork this
> project and edit the file'. This method gives easy access, but it is harder
> to preview the result, and you are stuck with the GitHub UI
> 2) Fork the project(s) and edit in your favorite editor. This requires
> that you have or learn some basic git knowledge.
>
> You can find the following repositories on GitHub:
>
> Administration guide: https://github.com/apache/cloudstack-docs-admin
> Installation guide: https://github.com/apache/cloudstack-docs-install
> Getting started: https://github.com/apache/cloudstack-docs
> Release notes: https://github.com/apache/cloudstack-docs-rn
>
> As for guidelines and/or style guides I'm not sure we have any, I guess we
> are open for suggestions :-)
>
>
> --
> Erik Weber
>
>
>
>
>

Re: Mentor

[Tim Mackey]

Prakash,

As you go through this, if you have questions related to how XenServer
works, do feel free to ask me. Another good avenue is the xs-devel list on
xenserver.org.
On Oct 27, 2015 6:27 PM, "Erik Weber"  wrote:

> On Tue, Oct 27, 2015 at 10:34 PM, David Willard 
> wrote:
>
> > Hi B. Prakash, Daan, Erik,
> >
> > I have emailed many time requesting a mentor and still no responses. I
> > have been following cloudstack for a few months. I am entry-level java
> > completed a Java class as I have earned my bachelor in IT from
> Northeastern
> > University. I earned a 4.0 for the Java course. Even if I have to start
> > troubleshooting code I am a quick learner and in a few months be able to
> > code. I am a member of the OSI and my ultimate goal is to obtain a job in
> > information security.
> >
> >
> Hi David,
>
> I am sorry that your efforts to get into the community hasn't given the
> wanted results yet.
>
> Coding CloudStack is beyond my skill set, so I can't really offer any
> mentoring, but if you are looking for simple tasks to carry out to get more
> familiar with the project I am sure we could come up with some issues for
> you :-)
>
> Your first matter of business should be to get a CloudStack cloud up and
> running so that you can test any changes you do.
> This can be done on a single machine if needed.
>
> --
> Erik
>

Re: Refactoring CitrixResourceBase

[Tim Mackey]

+1

When I went through this last time, not only was it hard to understand the
flows, but the XenServer version management was a pain. Would suggest
creating a base class which always works (i.e. is independent of XenServer
version) for core functions. Then add in that which exists for a specific
version. Should help greatly with testing IMO.

-tim

On Thu, May 19, 2016 at 2:37 PM, Syed Mushtaq 
wrote:

> Hi All,
>
> I would like to refactor CitrixResourceBase class which is responsible for
> communicating with Xenserver. It has grown too long (>5K lines) and has
> absolutely no testing.
>
> In my first pass I want to separate out the functionality buy the subsystem
> it targets (compute, storage, network etc) and will go on from there. What
> do you think? Is anyone working on this currently?
>
> Thanks,
> -Syed
>

Re: XenServer 7

[Tim Mackey]

Correct, all modern XenServer distros until 7 were centos 5.x. I doubt any
xapi changes would break cloudstack, but the plugins should be retested,
and btw it's also systemd time
On May 25, 2016 2:39 PM, "Remi Bergsma"  wrote:

As far as I know previous versions were based on Centos 5 even ;-)

Regards, Remi

> On 25 May 2016, at 16:55, Rafael Weingärtner 
wrote:
>
> Oh, I did know that, thanks for sharing Will.
>
> I was reading their Release notes; it seems that their “major changes”
> section does not reflect the experience you are getting.
>
> On Wed, May 25, 2016 at 11:42 AM, Will Stevens 
> wrote:
>
>> CentOS changes a LOT, especially around networking, so I would not be
>> surprised if it does not work. It still takes me like 20 minutes to
figure
>> out what the hell I am doing when I login to CentOS7.  No ifconfig, no
>> iptables, etc, etc, etc...  They pretty much did a wholesale change of
>> everything you expect to be there.
>>
>> Lots of testing will have to be done on this and I have not tried to do
>> that yet...
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> *CloudOps* *| *Cloud Solutions Experts
>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> w cloudops.com *|* tw @CloudOps_
>>
>> On Wed, May 25, 2016 at 9:35 AM, Rafael Weingärtner <
>> rafaelweingart...@gmail.com> wrote:
>>
>>> Considering those scripts you are right; but, my main concerns would be
>>> regarding some XAPI change. The scripts that we create and inject into
>> the
>>> Dom0 should not be that intrusive.
>>>
>>> I do not know about the differences between CentOS 6 and 7, but the only
>>> way I see a script stopping working is that if a program that the script
>>> uses does not come with the CentOS 7 that is bundled with XenServer or
if
>>> that program does not work with CentOS 7 anymore.
>>> I do not recall every single task that those scripts execute, at the top
>> of
>>> my memory I can say that they play with the systems' firewall. They also
>>> mount, copy, and move some files around. I believe they also remove some
>>> configurations from the XAPI sometimes.
>>>
>>> Looking at the release notes in [1], it seems that none of the scripts
>>> should stop working. However, to be sure about every single function and
>>> task that ACS execute on XenServer a batch of tests would be required.
>>>
>>> [1]
>>
https://wiki.centos.org/Manuals/ReleaseNotes/CentOS7#head-b02657cf2e223edb2d7946cbc45086d42c2bb41b
>>>
>>>
>>> On Wed, May 25, 2016 at 10:19 AM, Paul Angus 
>>> wrote:
>>>
 Rafael

 We don’t purely use XAPI with XenServer.
 XenServer 7 has a CentOS7 dom0 rather than CentOS6.  We do a whole load
>>> of
 stuff behind the scenes with python and shell scripts a chunk of which
 won't work anymore.


 Kind regards,

 Paul Angus

 paul.an...@shapeblue.com
 www.shapeblue.com
 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
 @shapeblue



 -Original Message-
 From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
 Sent: 25 May 2016 14:12
 To: dev@cloudstack.apache.org
 Subject: Re: XenServer 7

 Will there be any change on the XAPI? If not, it already works (just
>> the
 new functions, if any, that will not be used).

 On Wed, May 25, 2016 at 9:59 AM, Paul Angus 
 wrote:

> Is anyone here working on XenServer 7 support for CloudStack?
>
>
> Kind regards,
>
> Paul Angus
>
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue


 --
 Rafael Weingärtner
>>>
>>>
>>>
>>> --
>>> Rafael Weingärtner
>
>
>
> --
> Rafael Weingärtner

Re: Opportunity to contribute in Apache CloudStack

[Tim Mackey]

Jainesh, and by extension all members of TheAtom, welcome to the CloudStack
project. You'll want to join the development list (dev@cloudstack.apache.org),
and look at the contributing section here:
https://github.com/apache/cloudstack and here:
https://cloudstack.apache.org/developers.html.

A proposal was put to the dev list for the 2016-2017 roadmap earlier this
week (
http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201607.mbox/%3cc10c1a0c-4d57-4fb4-8812-b20aa1ea8...@shapeblue.com%3E
). That would probably be an excellent starting point to see the direction
of the project overall. As with most projects, its probably best to start
small and fix a few issues before starting in on something substantial.

For something more substantial, and depending upon your skill set, there
has been an open proposal to bring pure Xen in as fully supported
hypervisor. Knowing a bit of what's involved there, it might fit well with
your time frame, plus I know there are some who would welcome that work. I
also have a smaller project in the security space which might be
interesting, but haven't made the proposal to the list yet. In all cases,
the hope would be that you'd continue to be involved with the project upon
project completion.

(cross posted to dev@ to keep me honest - suspect further discussion should
occur on dev@ and not users@ )

-tim




On Wed, Jul 6, 2016 at 6:05 AM, Jainesh Patel  wrote:

> Hello,
>
> We are a group of students that are currently pursuing our undergraduate
> degree in Computer Science from Pune Insititute of Computer
> Technology(PICT), Maharashtra, India. We will be graduating in June 2017
> and are currently in our final year. For our B.E project, we have selected
> the domain as Cloud Computing and would be very interesting in working with
> open source cloud computing software, which is where we stumbled upon
> Apache CloudStack.
>
> It will be a great learning opportunity for us to work with Apache
> CloudStack and in turn work with you. We would appreciate if you could
> steer us towards the direction of choosing the right topic and working
> towards culminating a project in the same, which would be helpful for the
> community.
>
> Following are the few details which include information about us, which
> would help you in making an informed decision:
>
> 1) Group Name- TheAtom
>
> 2) Group Members:
> Shubham Mulay ( shubhammu...@gmail.com )
> Faizaan Shaikh ( faizaanshai...@gmail.com )
> Jainesh Patel ( jainesh...@gmail.com )
>
> 3) We have two mentors working with us, who will be guiding throughout the
> process,
> Dhruvesh Rathore ( dhruves...@hotmail.com )
> Prerit Auti ( prerita...@gmail.com )
>
> 4) Development time : 6 to 7 months from Aug '16 to Feb '17.
>
> We would love to hear from you about any ideas that you see fit for us to
> pursue and which are feasible in the specified time frame. Hoping to hear
> from you soon, and thanking you in anticipation.
>
> Regards,
> TheAtom
>

Re: Opportunity to contribute in Apache CloudStack

[Tim Mackey]

Jainesh,

As Syed mentioned, today Xen support in CloudStack is limited to XenServer
(or more precisely XAPI, but today XAPI really is only used with
XenServer). In 2014, I put forth a proposal to rectify that [1]. That
proposal contained two steps, first we needed to clean up the use of Xen
when XenServer (XAPI) was implied, and second was to implement a libvirt
interface for Xen in CloudStack. The cleanup was completed with ACS 4.5
[2], but the libvirt work required some upstream changes in multiple
projects. Those upstream changes were actually completed for OpenStack
Liberty last year [3][4], with the result being Xen became a "Group B"
hypervisor over there. I *feel* we're now in good shape to begin the second
part of the effort and have Xen + libvirt become fully supported with
CloudStack.  Syed and I spoke about this at the CloudStack Collaboration
Summit in Montreal last month.

I'll respond next week on the security proposal, in part as I'm gathering
the API references.

-tim


[1] http://markmail.org/thread/yrl3ii7gqlaaexij
[2]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Convert+Xen+usage+to+XenServer
[3]
http://docs.openstack.org/liberty/config-reference/content/xen_libvirt.html
[4]
https://blog.xenproject.org/2015/05/20/xen-project-now-in-openstack-nova-hypervisor-driver-quality-group-b/

On Fri, Jul 15, 2016 at 9:06 AM, Syed Ahmed  wrote:

> I can probably elaborate on this a bit. Currently Xen (XenServer) uses
> XAPI to interface with Cloudstack. This works for the most part but we
> would ideally want to use libvirt as communication interface. This will
> enable a lot of nice things like being able to support disk formats other
> than VHD and overall better integration with other providers.
>
> Now XenServer does not work with libvirt so we would have to go with
> opensource Xen. I have done some basic analysis around it. Feel free to
> reach out to me if you have any questions.
>
> -Syed
>
>
> On Mon, Jul 11, 2016 at 6:09 AM, Jainesh Patel 
> wrote:
>
>> Hello Sir,
>>
>> Can you please tell us more about the proposal to bring pure Xen in as
>> fully supported hypervisor and your project in security space?
>>
>> Regards,
>> TheAtom
>>
>> On Wed, Jul 6, 2016 at 4:15 PM, Tim Mackey  wrote:
>>
>> > Jainesh, and by extension all members of TheAtom, welcome to the
>> CloudStack
>> > project. You'll want to join the development list (
>> > dev@cloudstack.apache.org),
>> > and look at the contributing section here:
>> > https://github.com/apache/cloudstack and here:
>> > https://cloudstack.apache.org/developers.html.
>> >
>> > A proposal was put to the dev list for the 2016-2017 roadmap earlier
>> this
>> > week (
>> >
>> >
>> http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201607.mbox/%3cc10c1a0c-4d57-4fb4-8812-b20aa1ea8...@shapeblue.com%3E
>> > ). That would probably be an excellent starting point to see the
>> direction
>> > of the project overall. As with most projects, its probably best to
>> start
>> > small and fix a few issues before starting in on something substantial.
>> >
>> > For something more substantial, and depending upon your skill set, there
>> > has been an open proposal to bring pure Xen in as fully supported
>> > hypervisor. Knowing a bit of what's involved there, it might fit well
>> with
>> > your time frame, plus I know there are some who would welcome that
>> work. I
>> > also have a smaller project in the security space which might be
>> > interesting, but haven't made the proposal to the list yet. In all
>> cases,
>> > the hope would be that you'd continue to be involved with the project
>> upon
>> > project completion.
>> >
>> > (cross posted to dev@ to keep me honest - suspect further discussion
>> > should
>> > occur on dev@ and not users@ )
>> >
>> > -tim
>> >
>> >
>> >
>> >
>> > On Wed, Jul 6, 2016 at 6:05 AM, Jainesh Patel 
>> > wrote:
>> >
>> > > Hello,
>> > >
>> > > We are a group of students that are currently pursuing our
>> undergraduate
>> > > degree in Computer Science from Pune Insititute of Computer
>> > > Technology(PICT), Maharashtra, India. We will be graduating in June
>> 2017
>> > > and are currently in our final year. For our B.E project, we have
>> > selected
>> > > the domain as Cloud Computing and would be very interesting in working
>> > with
>> > > open source cloud comp

Re: [DISCUSS] XenServer and HA: the way forward

[Tim Mackey]

It's not just 6.0.2 which requires a license.  For 6.1 you also needed a
license to enable XenServer HA.  It's only 6.2 and higher (and legacy XCP)
which don't require a license for XenServer HA.  Thankfully you can detect
the version, and if a license is present, which means we can do the right
thing for older XenServer versions with a license.

The complication comes when a license existed and is about to expire.  When
a legacy license is used (6.1 and prior), if the license expires all
features keep running, but you can't create anything which requires an
operational license.  I don't know what that would mean for pool HA since
we're not talking about protecting VMs.

btw, in all versions of XenServer, you can just enable HA without requiring
VMs to also be protected.  The XenCenter UI makes it look like enabling HA
for the first time requires a VM be specified, but you don't.  The UI is
just making an assumption.

So for legacy, I can see modifying my flow to include:

- If legacy host and licensed follow 6.2 and higher, but if not licensed
degrade to 4.3 and prior but warn admin at time of host creation.
- If legacy host, monitor license state and add warning if license due to
expire. (Could also see doing this for vSphere as convenience)

-tim

On Tue, May 5, 2015 at 8:34 AM, Remi Bergsma  wrote:

> What if we follow the vendor here? Citrix supports XenServer 6.0.2 until
> June 2018:
> https://www.citrix.nl/support/product-lifecycle/product-matrix.html
>
> In my opinion we should support it. I don't want to leave people at 4.3.x
> because of this.
>
> Regards,
> Remi
>
>
> Op di 5 mei 2015 om 13:28 schreef S. Brüseke - proIO GmbH <
> s.brues...@proio.com>:
>
> > In my opinion the way we should go is "keep it simple". We really should
> > consider to drop support for XenServer 6.0.2 here and not making things
> > more complicated and provide more than 1 option.
> > Of course it depends on how many CS installations are still using
> > XenServer 6.0.2.!
> > Can somebody give more information on this?
> >
> > Mit freundlichen Grüßen / With kind regards,
> >
> > Swen Brüseke
> >
> > -Ursprüngliche Nachricht-
> > Von: Remi Bergsma [mailto:r...@remi.nl]
> > Gesendet: Dienstag, 5. Mai 2015 12:36
> > An: dev@cloudstack.apache.org
> > Betreff: Re: [DISCUSS] XenServer and HA: the way forward
> >
> > Hi all,
> >
> > Thanks for pointing me to the proposal, Koushik. Too bad no one responded
> > to such a major change.
> >
> > When I put my "Operations" hat on, I see several issues. First of all,
> > there was no mention of this change in the release notes. Not as a new
> > feature, nor as a bug that was fixed. How do we expect people that
> operate
> > CloudStack will know about this? It's not even in the recommended install
> > instructions for a new cloud today. In my experience, when I talk to
> people
> > about this, I find that almost no one knows. We as a community can do
> > better than this!
> >
> > For XenServer 6.5 and 6.2 one can enable XenHA for the pool only, but for
> > XenServer 6.0.2 this is a different story because as far as I know one
> can
> > only enable HA as a whole (HA on pool + HA on VMs). And this is only if
> you
> > have the paid version, which we happen to have. But I don't think it is a
> > solution, as this leads to corruption when XenServer and CloudStack try
> to
> > recover the same VM at the same time (Trust me, I've been there). Why do
> we
> > even "support" 6.0.2 one could ask?
> >
> > I still do have some XenServer 6.0.2 clusters running.. If the pool
> master
> > crashes I need to manually appoint a new one. I don't like manual work
> and
> > if I had known before I wouldn't have upgraded before this was resolved.
> Or
> > do I miss something here?
> >
> > If we want to drop support for older XenServer versions, then let's vote
> > for it and be very clear about it. Dropping XenServer 6.0.2 comes a bit
> too
> > early if  you ask me.
> >
> > Let's discuss how to proceed. I still feel the best solution is to add a
> > switch between both HA methods so one can choose which one suits best,
> and
> > for older XenServer versions we will restore the HA feature that way.
> >
> > Any thoughts?
> >
> > Regards,
> > Remi
> >
> > Op di 5 mei 2015 om 08:13 schreef Koushik Das :
> >
> > > The below is the proposal for switching to XenServer HA.
> > >
> > >
> > > http://mail-archives.apache.org/mod_mbox/cloudstack-dev

Re: XenServer SR Question

[Tim Mackey]

I'm assuming that's the value from XenCenter. What does the cli say? I
could see this being just a formatting question.
On Jan 30, 2014 5:59 PM, "Mike Tutkowski" 
wrote:

> Hi,
>
> Does anyone know how a XenServer SR could (correctly) say more space is
> being used than is allocated?
>
> This is what the Size field of one of my shared SRs says:
>
> Size: 20.1 GB used of 80 GB total (20 GB allocated)
>
> I would have expected used to be <= allocated at all times (typically I'd
> expect it to be < allocated most of the time).
>
> Any thoughts on this?
>
> Thanks!
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud
> *(tm)*
>

Re: XenServer SR Question

[Tim Mackey]

I'll need to confirm with engineering, but that makes sense. I'll also see
if there is a different format specifier in the XenCenter string
On Jan 30, 2014 6:27 PM, "Mike Tutkowski" 
wrote:

> Hi Tim,
>
> You are correct that I was looking at XenCenter. Below is the output from
> xe.
>
> It looks like physical-size (below) corresponds to total in XenCenter,
> physical-utilization (below) corresponds to used in XenCenter, and
> virtual-allocation (below) corresponds to allocated in XenCenter.
>
> Does that look right?
>
> Thanks!
>
> [root@XenServer-6 ~]# xe sr-list uuid=ec8db98c-8338-7bc7-75aa-f45207c32a83
> params=all
> uuid ( RO): ec8db98c-8338-7bc7-75aa-f45207c32a83
>   name-label ( RW): /iqn.2010-01.com.solidfire:3y8w.vol-1.126/0
> name-description ( RW): /iqn.2010-01.com.solidfire:3y8w.vol-1.126/0
> host ( RO): 
>   allowed-operations (SRO): VDI.create; VDI.snapshot; PBD.create;
> PBD.destroy; plug; update; VDI.destroy; scan; VDI.clone; VDI.resize; unplug
>   current-operations (SRO):
> VDIs (SRO): 8f14728f-938f-4e02-bc9c-8f67e86d5c86
> PBDs (SRO): c4766cdc-7975-9ae7-5953-6fd831a5c654;
> 7ef960f0-4074-ef18-b17b-1f9a1ae8a6ff
>   virtual-allocation ( RO): 21525168128
> physical-utilisation ( RO): 21529362432
>physical-size ( RO): 85886763008
> type ( RO): lvmoiscsi
> content-type ( RO): user
>   shared ( RW): true
>introduced-by ( RO): 
> other-config (MRW):
>sm-config (MRO): allocation: thick; use_vhd: true;
> multipathable: true; devserial: scsi-36f47acc133793877007e
>    blobs ( RO):
>  local-cache-enabled ( RO): false
> tags (SRW):
>
>
> On Thu, Jan 30, 2014 at 4:08 PM, Tim Mackey  wrote:
>
> > I'm assuming that's the value from XenCenter. What does the cli say? I
> > could see this being just a formatting question.
> > On Jan 30, 2014 5:59 PM, "Mike Tutkowski" 
> > wrote:
> >
> > > Hi,
> > >
> > > Does anyone know how a XenServer SR could (correctly) say more space is
> > > being used than is allocated?
> > >
> > > This is what the Size field of one of my shared SRs says:
> > >
> > > Size: 20.1 GB used of 80 GB total (20 GB allocated)
> > >
> > > I would have expected used to be <= allocated at all times (typically
> I'd
> > > expect it to be < allocated most of the time).
> > >
> > > Any thoughts on this?
> > >
> > > Thanks!
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkow...@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the
> > > cloud<http://solidfire.com/solution/overview/?video=play>
> > > *(tm)*
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *(tm)*
>

Re: XenServer SR Question

[Tim Mackey]

Mike, I've confirmed that the data items map as you suspected to XenCenter.
 In this case the values had a rounding effect just on either side of 20.05
respectively to create the observed numbers.


On Thu, Jan 30, 2014 at 6:37 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Great - thanks!
>
> I was just trying to make sense of the numbers. :)
>
>
> On Thu, Jan 30, 2014 at 4:36 PM, Tim Mackey  wrote:
>
> > I'll need to confirm with engineering, but that makes sense. I'll also
> see
> > if there is a different format specifier in the XenCenter string
> > On Jan 30, 2014 6:27 PM, "Mike Tutkowski" 
> > wrote:
> >
> > > Hi Tim,
> > >
> > > You are correct that I was looking at XenCenter. Below is the output
> from
> > > xe.
> > >
> > > It looks like physical-size (below) corresponds to total in XenCenter,
> > > physical-utilization (below) corresponds to used in XenCenter, and
> > > virtual-allocation (below) corresponds to allocated in XenCenter.
> > >
> > > Does that look right?
> > >
> > > Thanks!
> > >
> > > [root@XenServer-6 ~]# xe sr-list
> > uuid=ec8db98c-8338-7bc7-75aa-f45207c32a83
> > > params=all
> > > uuid ( RO): ec8db98c-8338-7bc7-75aa-f45207c32a83
> > >   name-label ( RW):
> > /iqn.2010-01.com.solidfire:3y8w.vol-1.126/0
> > > name-description ( RW):
> > /iqn.2010-01.com.solidfire:3y8w.vol-1.126/0
> > > host ( RO): 
> > >   allowed-operations (SRO): VDI.create; VDI.snapshot; PBD.create;
> > > PBD.destroy; plug; update; VDI.destroy; scan; VDI.clone; VDI.resize;
> > unplug
> > >   current-operations (SRO):
> > > VDIs (SRO): 8f14728f-938f-4e02-bc9c-8f67e86d5c86
> > > PBDs (SRO): c4766cdc-7975-9ae7-5953-6fd831a5c654;
> > > 7ef960f0-4074-ef18-b17b-1f9a1ae8a6ff
> > >   virtual-allocation ( RO): 21525168128
> > > physical-utilisation ( RO): 21529362432
> > >physical-size ( RO): 85886763008
> > > type ( RO): lvmoiscsi
> > > content-type ( RO): user
> > >   shared ( RW): true
> > >    introduced-by ( RO): 
> > > other-config (MRW):
> > >sm-config (MRO): allocation: thick; use_vhd: true;
> > > multipathable: true; devserial: scsi-36f47acc133793877007e
> > >blobs ( RO):
> > >  local-cache-enabled ( RO): false
> > > tags (SRW):
> > >
> > >
> > > On Thu, Jan 30, 2014 at 4:08 PM, Tim Mackey  wrote:
> > >
> > > > I'm assuming that's the value from XenCenter. What does the cli say?
> I
> > > > could see this being just a formatting question.
> > > > On Jan 30, 2014 5:59 PM, "Mike Tutkowski" <
> > mike.tutkow...@solidfire.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Does anyone know how a XenServer SR could (correctly) say more
> space
> > is
> > > > > being used than is allocated?
> > > > >
> > > > > This is what the Size field of one of my shared SRs says:
> > > > >
> > > > > Size: 20.1 GB used of 80 GB total (20 GB allocated)
> > > > >
> > > > > I would have expected used to be <= allocated at all times
> (typically
> > > I'd
> > > > > expect it to be < allocated most of the time).
> > > > >
> > > > > Any thoughts on this?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkow...@solidfire.com
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the
> > > > > cloud<http://solidfire.com/solution/overview/?video=play>
> > > > > *(tm)*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkow...@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the
> > > cloud<http://solidfire.com/solution/overview/?video=play>
> > > *(tm)*
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *(tm)*
>

Re: [PROPOSAL] Introduce API returning you an answer from CloudStack storage/host allocators whethere there is enough resources for vm deployment

[Tim Mackey]

How deep into dependencies would this go?  For example, if a new router VM
was required would that also be checked?

A chunk of the race conditions and state change issues should be resolvable
with ticket reservation system logic, but I'd be concerned the required
locks could create problems during infrastructure failures and with HA
enabled.  We ran into a ton of these types of problems implementing the
placement logic in XenServer's workload balancing with HA enabled.


On Sat, Feb 1, 2014 at 12:10 AM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> I like the idea in principal. We should fail as fast as is practical.
>
> I wonder, are you just checking if the resources are available? If so, it's
> still possible for deployVM to fail even if this check said all was good
> because the state of the system could change before the actual deployVM is
> started.
>
>
> On Fri, Jan 31, 2014 at 7:20 PM, Madan Ganesh Velayudham <
> madangan...@me.com
> > wrote:
>
> > Makes lot of sense. Would the same API allow the caller just to check
> (not
> > actually proceed to create) whether resources are present for a
> particular
> > payload? This may be useful for the client who want to proactively check
> > and avoid returning failure.
> >
> > Are there any race condition possibilities when multiple requests are
> > received?
> > 
> >
> > Madan Ganesh Velayudham P.M.P, C.S.M.,
> > madangan...@me.com
> >
> >
> >
> >
> > On 01-Feb-2014, at 5:56 am, Alena Prokharchyk <
> > alena.prokharc...@citrix.com> wrote:
> >
> > > Currently there is no way to know if there is enough resources for vm
> > deployment, before actual deployVm call is made. The sequence is the
> > following:
> > >
> > > 1) Deploy Vm is called
> > > 2) DB record is created for the Vm
> > > 3) Storage/Host allocators determine whethere there are enough
> resources
> > for vm to be deployed, and return deploy destination to the caller stack.
> > > 4) If allocator returns valid deploy destination, VM gets actually
> > created/started on the backend. If allocators don't return the
> destination,
> > the DB record created on step 2) gets destroyed, and ResourceAllocation
> > exception is thrown back to the API caller.
> > >
> > > The API I'm going to introduce, would help you to determine whether CS
> > physical resources - hosts, storages - can potentially accomodate vm
> > deployment (considering template/service/diskOffering) at a given time,
> w/o
> > actually calling the deploy vm. Some admins might find this call useful
> as
> > they can always make this check before submitting the deployVm, so in
> case
> > it returns NO, you can fail the deployment immediately, w/o calling
> > deployVm. Also you can make this call to determine what is lacking for
> > certain vm deployment, and expand your physical resources accordingly.
> > >
> > > Please let me know if see any pitfalls in the proposal, as well if you
> > see any other use cases that can be solved using this API.
> > >
> > > Prachi, can you please point me to an existing method (or interface)
> > defined in Allocators code serving this purpose?
> > >
> > > Thanks,
> > > -Alena.
> >
> >
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud
> *(tm)*
>

4.3 features with hypervisor differences

[Tim Mackey]

Hi everyone,

I'm updating my deck on hypervisor differences for 4.3, and I'd like
confirmation of what I understand made the cut.  Feedback welcome.

- Hyper-V 2012 R2 added (Donal, I'll be hitting you up to confirm all the
HV stuff)
- vSphere 5.5 added and supporting identical features to vSphere 5.1
- CPU sockets for KVM and vSphere all versions
- CPU sockets for XenServer 6.2 and higher only
- CPU sockets reports host count for LXC
- quiesced snapshots for XS don't use XS quiesce API and rely on storage
support
- quiesced snapshots for vSphere are natively ROOT only. DATA relies on
storage support
- Contrail supported on XS only
- Palo Alto VPN moved to 4.4

No hypervisors were dropped, and only Hyper-V was added.  No previously
supported features with hypervisor dependencies were dropped.

I'm going to be presenting this deck in a couple of weeks and would like to
give the right information ;)

Thanks

-tim