Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

[Marcus Sorensen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14381/
---

(Updated Sept. 28, 2013, 6:43 a.m.)


Review request for cloudstack, edison su and Mike Tutkowski.


Changes
---

Fixed the TODO.


Repository: cloudstack-git


Description (updated)
---

MigrateCommand: pass vmTO object so we can see which disks/storage pool types 
belong to the vm when migrating a VM. This facilitates being able to call 
disconnectPhysicalDisksViaVmSpec

VirtualMachineManagerImpl: pass VirtualMachineTO when migrating so that we can 
see which disks belong to the VM and what storage pools/adaptors should be used

LibvirtComputingResource: add calls KVMStoragePoolManager's 
connectPhysicalDiskViaVmSpec and disconnectPhysicalDiskViaVmSpec calls where 
appropriate (when starting a vm, migrating a vm). Ensure that we create 'raw' 
format XML disk definitions when the storage format is RAW. Move cleanupDisk 
logic to storage adaptors so that each adaptor type can clean up its disks in 
is own way.

KVMStoragePoolManager:  add connectPhysicalDisk, disconnectPhysicalDisk, 
connectPhysicalDiskViaVmSpec, disconnectPhysicalDiskViaVmSpec. These all call 
the specific StorageAdaptor's connectPhysicalDisk or disconnectPhysicalDisk 
calls.

KVMStorageProcessor: Call connectPhysicalDisk/disconnectPhysicalDisk on the 
storage adaptor. Whether or not this is implemented is up to the storage 
adaptor.

LibvirtStorageAdaptor: implement dummy 
connectPhysicalDisk/disconnectPhysicalDisk, move cleanupDisk logic from 
LibvirtComputingResource to disconnectPhysicalDiskByPath

StorageAdaptor: define 
connectPhysicalDisk/disconnectPhysicalDisk/disconnectPhysicalDiskByPath in the 
interface


Diffs (updated)
-

  core/src/com/cloud/agent/api/MigrateCommand.java 5042b8c 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
 3ee811f 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java
 e09c9ba 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java
 c69f9b0 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java
 123a9f1 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/StorageAdaptor.java
 4956d8d 
  server/src/com/cloud/vm/VirtualMachineManagerImpl.java d46bbb0 

Diff: https://reviews.apache.org/r/14381/diff/


Testing
---

Basic testing with my storage adaptor


Thanks,

Marcus Sorensen

Re: add connect method on KVM storage code

[Mike Tutkowski]

> I'd hate to have, say, 10,000 KVM hosts all stuck in the same VAG...each
> host having access (technically) to, say, 20,000 volumes.

Plus this modeling of VAGs does nothing to fence against rouge compute
nodes. If we did use VAGs, I'd like to use them in a way to protect against
rouge nodes.

CHAP won't protect this way either, but if we don't have a grantAccess and
revokeAccess in the storage plug-in, then CHAP would just be the easier
route to go.


On Sat, Sep 28, 2013 at 12:30 AM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> That works.
>
> Right now the cmd takes in CHAP info in the form of four strings
> (initiator username, initiator password, target username, and target
> password).
>
> I like your idea of a generic list better, though.
>
> We could remove the four strings from the cmd and I could use the map in
> not just KVM land, but Xen and VMware, as well.
>
>
> On Sat, Sep 28, 2013 at 12:28 AM, Marcus Sorensen wrote:
>
>> Yeah, maybe we just add a "Map connectDetails =
>> cmd.getConnectDetails()" into attachVolume, and pass that along to
>> connectPhysicalDisk. Then just set the details where you're currently
>> setting the chap info prior to the attach command.
>>
>> On Sat, Sep 28, 2013 at 12:25 AM, Mike Tutkowski
>>  wrote:
>> > CHAP credentials are per SolidFire/CloudStack account. A volume on the
>> > SolidFire SAN belongs to an account. I have a one-to-one mapping
>> between a
>> > CloudStack account and a SolidFire account.
>> >
>> > As far as Volume Access Groups (VAGs) go, I'm not sure what kind of
>> limits
>> > they have.
>> >
>> > I'd hate to have, say, 10,000 KVM hosts all stuck in the same VAG...each
>> > host having access (technically) to, say, 20,000 volumes.
>> >
>> > VAGs can be created as needed, though...and you can add and remove
>> > hosts/volumes from them as need be.
>> >
>> > On Sat, Sep 28, 2013 at 12:16 AM, Marcus Sorensen 
>> >  wrote:
>> >
>> >> On Sat, Sep 28, 2013 at 12:09 AM, Mike Tutkowski
>> >>  wrote:
>> >> > Yes, you are correct that our SAN requires CHAP or for the host and
>> >> volume
>> >> > to be in a Volume Access Group.
>> >> >
>> >> > If CHAP is provided, it is used. If it fails for whatever reason, the
>> >> > Volume Access Group is NOT leveraged to grant permissions.
>> >> >
>> >> > If CHAP is not provided, the host and volume must be in a common
>> Volume
>> >> > Access Group.
>> >> >
>> >> > As you say, CHAP only grants access...it will not save us from a
>> rouge
>> >> > compute node that has access to the volume.
>> >>
>> >> Ok, I figured the CHAP credentials were predetermined, but it sounds
>> >> like they're per-lun.
>> >>
>> >> >
>> >> > If we can't get grantAccess and revokeAccess working in the storage
>> >> > plug-ins for 4.3, I can just use CHAP. I'd prefer to have no SAN
>> calls
>> >> > coming in from the compute nodes.
>> >>
>> >> Yes, we only do it because we have to, but now with the new framework
>> >> I'm trying to refactor things as they add support for doing things via
>> >> the plugin.
>> >>
>> >> >
>> >> > Also, I don't think we can grant all hosts in the KVM cluster access
>> to
>> >> the
>> >> > volume ahead of time because the volume won't exist until the first
>> time
>> >> we
>> >> > try to attach it. We have to grant access programmatically.
>> >> >
>> >> > Plus you might want to disconnect the volume from a KVM host in one
>> >> cluster
>> >> > and connect it up in another KVM cluster (zone-wide storage).
>> >>
>> >> I was thinking more along the lines of some group that you'd add your
>> >> hosts to, regardless of which cluster they're in, as you deploy them.
>> >> But it sounds like those would need to be done dynamically? i.e. I
>> >> can't create a group, add a host, and then later add a lun and have
>> >> access to it?
>>
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the 
> cloud
> *™*
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud
*™*

Re: add connect method on KVM storage code

[Mike Tutkowski]

That works.

Right now the cmd takes in CHAP info in the form of four strings (initiator
username, initiator password, target username, and target password).

I like your idea of a generic list better, though.

We could remove the four strings from the cmd and I could use the map in
not just KVM land, but Xen and VMware, as well.


On Sat, Sep 28, 2013 at 12:28 AM, Marcus Sorensen wrote:

> Yeah, maybe we just add a "Map connectDetails =
> cmd.getConnectDetails()" into attachVolume, and pass that along to
> connectPhysicalDisk. Then just set the details where you're currently
> setting the chap info prior to the attach command.
>
> On Sat, Sep 28, 2013 at 12:25 AM, Mike Tutkowski
>  wrote:
> > CHAP credentials are per SolidFire/CloudStack account. A volume on the
> > SolidFire SAN belongs to an account. I have a one-to-one mapping between
> a
> > CloudStack account and a SolidFire account.
> >
> > As far as Volume Access Groups (VAGs) go, I'm not sure what kind of
> limits
> > they have.
> >
> > I'd hate to have, say, 10,000 KVM hosts all stuck in the same VAG...each
> > host having access (technically) to, say, 20,000 volumes.
> >
> > VAGs can be created as needed, though...and you can add and remove
> > hosts/volumes from them as need be.
> >
> > On Sat, Sep 28, 2013 at 12:16 AM, Marcus Sorensen 
> >  wrote:
> >
> >> On Sat, Sep 28, 2013 at 12:09 AM, Mike Tutkowski
> >>  wrote:
> >> > Yes, you are correct that our SAN requires CHAP or for the host and
> >> volume
> >> > to be in a Volume Access Group.
> >> >
> >> > If CHAP is provided, it is used. If it fails for whatever reason, the
> >> > Volume Access Group is NOT leveraged to grant permissions.
> >> >
> >> > If CHAP is not provided, the host and volume must be in a common
> Volume
> >> > Access Group.
> >> >
> >> > As you say, CHAP only grants access...it will not save us from a rouge
> >> > compute node that has access to the volume.
> >>
> >> Ok, I figured the CHAP credentials were predetermined, but it sounds
> >> like they're per-lun.
> >>
> >> >
> >> > If we can't get grantAccess and revokeAccess working in the storage
> >> > plug-ins for 4.3, I can just use CHAP. I'd prefer to have no SAN calls
> >> > coming in from the compute nodes.
> >>
> >> Yes, we only do it because we have to, but now with the new framework
> >> I'm trying to refactor things as they add support for doing things via
> >> the plugin.
> >>
> >> >
> >> > Also, I don't think we can grant all hosts in the KVM cluster access
> to
> >> the
> >> > volume ahead of time because the volume won't exist until the first
> time
> >> we
> >> > try to attach it. We have to grant access programmatically.
> >> >
> >> > Plus you might want to disconnect the volume from a KVM host in one
> >> cluster
> >> > and connect it up in another KVM cluster (zone-wide storage).
> >>
> >> I was thinking more along the lines of some group that you'd add your
> >> hosts to, regardless of which cluster they're in, as you deploy them.
> >> But it sounds like those would need to be done dynamically? i.e. I
> >> can't create a group, add a host, and then later add a lun and have
> >> access to it?
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud
*™*

Re: add connect method on KVM storage code

[Marcus Sorensen]

Yeah, maybe we just add a "Map connectDetails =
cmd.getConnectDetails()" into attachVolume, and pass that along to
connectPhysicalDisk. Then just set the details where you're currently
setting the chap info prior to the attach command.

On Sat, Sep 28, 2013 at 12:25 AM, Mike Tutkowski
 wrote:
> CHAP credentials are per SolidFire/CloudStack account. A volume on the
> SolidFire SAN belongs to an account. I have a one-to-one mapping between a
> CloudStack account and a SolidFire account.
>
> As far as Volume Access Groups (VAGs) go, I'm not sure what kind of limits
> they have.
>
> I'd hate to have, say, 10,000 KVM hosts all stuck in the same VAG...each
> host having access (technically) to, say, 20,000 volumes.
>
> VAGs can be created as needed, though...and you can add and remove
> hosts/volumes from them as need be.
>
> On Sat, Sep 28, 2013 at 12:16 AM, Marcus Sorensen 
>  wrote:
>
>> On Sat, Sep 28, 2013 at 12:09 AM, Mike Tutkowski
>>  wrote:
>> > Yes, you are correct that our SAN requires CHAP or for the host and
>> volume
>> > to be in a Volume Access Group.
>> >
>> > If CHAP is provided, it is used. If it fails for whatever reason, the
>> > Volume Access Group is NOT leveraged to grant permissions.
>> >
>> > If CHAP is not provided, the host and volume must be in a common Volume
>> > Access Group.
>> >
>> > As you say, CHAP only grants access...it will not save us from a rouge
>> > compute node that has access to the volume.
>>
>> Ok, I figured the CHAP credentials were predetermined, but it sounds
>> like they're per-lun.
>>
>> >
>> > If we can't get grantAccess and revokeAccess working in the storage
>> > plug-ins for 4.3, I can just use CHAP. I'd prefer to have no SAN calls
>> > coming in from the compute nodes.
>>
>> Yes, we only do it because we have to, but now with the new framework
>> I'm trying to refactor things as they add support for doing things via
>> the plugin.
>>
>> >
>> > Also, I don't think we can grant all hosts in the KVM cluster access to
>> the
>> > volume ahead of time because the volume won't exist until the first time
>> we
>> > try to attach it. We have to grant access programmatically.
>> >
>> > Plus you might want to disconnect the volume from a KVM host in one
>> cluster
>> > and connect it up in another KVM cluster (zone-wide storage).
>>
>> I was thinking more along the lines of some group that you'd add your
>> hosts to, regardless of which cluster they're in, as you deploy them.
>> But it sounds like those would need to be done dynamically? i.e. I
>> can't create a group, add a host, and then later add a lun and have
>> access to it?

Re: add connect method on KVM storage code

[Mike Tutkowski]

If grantAccess/revokeAccess worked in the storage plug-in, though, I could
create a VAG per KVM host and give it access to whatever volumes it needs
access to.

When we revoke access, I could remove the IQN of the volume from the
appropriate VAG. If the VAG is empty, I could delete it.

If you had, say, 10,000 KVM hosts each using at least one SolidFire volume,
you'd have 10,000 VAGs. That would be OK, though.


On Sat, Sep 28, 2013 at 12:25 AM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> CHAP credentials are per SolidFire/CloudStack account. A volume on the
> SolidFire SAN belongs to an account. I have a one-to-one mapping between a
> CloudStack account and a SolidFire account.
>
> As far as Volume Access Groups (VAGs) go, I'm not sure what kind of limits
> they have.
>
> I'd hate to have, say, 10,000 KVM hosts all stuck in the same VAG...each
> host having access (technically) to, say, 20,000 volumes.
>
> VAGs can be created as needed, though...and you can add and remove
> hosts/volumes from them as need be.
>
> On Sat, Sep 28, 2013 at 12:16 AM, Marcus Sorensen 
>  wrote:
>
> On Sat, Sep 28, 2013 at 12:09 AM, Mike Tutkowski
>>  wrote:
>> > Yes, you are correct that our SAN requires CHAP or for the host and
>> volume
>> > to be in a Volume Access Group.
>> >
>> > If CHAP is provided, it is used. If it fails for whatever reason, the
>> > Volume Access Group is NOT leveraged to grant permissions.
>> >
>> > If CHAP is not provided, the host and volume must be in a common Volume
>> > Access Group.
>> >
>> > As you say, CHAP only grants access...it will not save us from a rouge
>> > compute node that has access to the volume.
>>
>> Ok, I figured the CHAP credentials were predetermined, but it sounds
>> like they're per-lun.
>>
>> >
>> > If we can't get grantAccess and revokeAccess working in the storage
>> > plug-ins for 4.3, I can just use CHAP. I'd prefer to have no SAN calls
>> > coming in from the compute nodes.
>>
>> Yes, we only do it because we have to, but now with the new framework
>> I'm trying to refactor things as they add support for doing things via
>> the plugin.
>>
>> >
>> > Also, I don't think we can grant all hosts in the KVM cluster access to
>> the
>> > volume ahead of time because the volume won't exist until the first
>> time we
>> > try to attach it. We have to grant access programmatically.
>> >
>> > Plus you might want to disconnect the volume from a KVM host in one
>> cluster
>> > and connect it up in another KVM cluster (zone-wide storage).
>>
>> I was thinking more along the lines of some group that you'd add your
>> hosts to, regardless of which cluster they're in, as you deploy them.
>> But it sounds like those would need to be done dynamically? i.e. I
>> can't create a group, add a host, and then later add a lun and have
>> access to it?
>
>


-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud
*™*

Re: add connect method on KVM storage code

[Mike Tutkowski]

CHAP credentials are per SolidFire/CloudStack account. A volume on the
SolidFire SAN belongs to an account. I have a one-to-one mapping between a
CloudStack account and a SolidFire account.

As far as Volume Access Groups (VAGs) go, I'm not sure what kind of limits
they have.

I'd hate to have, say, 10,000 KVM hosts all stuck in the same VAG...each
host having access (technically) to, say, 20,000 volumes.

VAGs can be created as needed, though...and you can add and remove
hosts/volumes from them as need be.

On Sat, Sep 28, 2013 at 12:16 AM, Marcus Sorensen 
 wrote:

> On Sat, Sep 28, 2013 at 12:09 AM, Mike Tutkowski
>  wrote:
> > Yes, you are correct that our SAN requires CHAP or for the host and
> volume
> > to be in a Volume Access Group.
> >
> > If CHAP is provided, it is used. If it fails for whatever reason, the
> > Volume Access Group is NOT leveraged to grant permissions.
> >
> > If CHAP is not provided, the host and volume must be in a common Volume
> > Access Group.
> >
> > As you say, CHAP only grants access...it will not save us from a rouge
> > compute node that has access to the volume.
>
> Ok, I figured the CHAP credentials were predetermined, but it sounds
> like they're per-lun.
>
> >
> > If we can't get grantAccess and revokeAccess working in the storage
> > plug-ins for 4.3, I can just use CHAP. I'd prefer to have no SAN calls
> > coming in from the compute nodes.
>
> Yes, we only do it because we have to, but now with the new framework
> I'm trying to refactor things as they add support for doing things via
> the plugin.
>
> >
> > Also, I don't think we can grant all hosts in the KVM cluster access to
> the
> > volume ahead of time because the volume won't exist until the first time
> we
> > try to attach it. We have to grant access programmatically.
> >
> > Plus you might want to disconnect the volume from a KVM host in one
> cluster
> > and connect it up in another KVM cluster (zone-wide storage).
>
> I was thinking more along the lines of some group that you'd add your
> hosts to, regardless of which cluster they're in, as you deploy them.
> But it sounds like those would need to be done dynamically? i.e. I
> can't create a group, add a host, and then later add a lun and have
> access to it?

Re: [jira] [Created] (CLOUDSTACK-4753) Console proxy disk space fills up

[Wei ZHOU]

1.In 4.1,cloud service run before cloud-early-config,that could be an issue.
2.In 4.2,new template,the issue does not exist

2013/9/27, John Beredimas (JIRA) :
> John Beredimas created CLOUDSTACK-4753:
> --
>
>  Summary: Console proxy disk space fills up
>  Key: CLOUDSTACK-4753
>  URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4753
>  Project: CloudStack
>   Issue Type: Bug
>   Security Level: Public (Anyone can view this level - this is the
> default.)
>   Components: SystemVM
> Affects Versions: 4.1.0
> Reporter: John Beredimas
>
>
> I run into this when I restarted the console proxy as part of upgrading to
> Cloudstack 4.2. It seems that the console proxy is filling up the cloud.out
> files and can potentially exhaust disk space.
>
> root@v-1011-VM:~# df -h
> FilesystemSize  Used Avail Use% Mounted on
> /dev/xvda12.0G  1.1G  873M  55% /
> 
> root@v-1011-VM:~# du -sh /var/log/cloud/cloud.out*
> 3.2M/var/log/cloud/cloud.out
> 9.8M/var/log/cloud/cloud.out.1
> 502M/var/log/cloud/cloud.out.2
>
> This can cause the console proxy to become non-available on a reboot, due to
> no disk left on device.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

[Marcus Sorensen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14381/
---

(Updated Sept. 28, 2013, 4:51 a.m.)


Review request for cloudstack and Mike Tutkowski.


Repository: cloudstack-git


Description
---

MigrateCommand: pass vmTO object so we can see which disks/storage pool types 
belong to the vm when migrating a VM. This facilitates being able to call 
disconnectPhysicalDisksViaVmSpec

VirtualMachineManagerImpl: pass VirtualMachineTO when migrating so that we can 
see which disks belong to the VM and what storage pools/adaptors should be used

LibvirtComputingResource: add calls KVMStoragePoolManager's 
connectPhysicalDiskViaVmSpec and disconnectPhysicalDiskViaVmSpec calls where 
appropriate (when starting a vm, migrating a vm). Ensure that we create 'raw' 
format XML disk definitions when the storage format is RAW.

KVMStoragePoolManager:  add connectPhysicalDisk, disconnectPhysicalDisk, 
connectPhysicalDiskViaVmSpec, disconnectPhysicalDiskViaVmSpec. These all call 
the specific StorageAdaptor's connectPhysicalDisk or disconnectPhysicalDisk 
calls.

KVMStorageProcessor: Call connectPhysicalDisk/disconnectPhysicalDisk on the 
storage adaptor. Whether or not this is implemented is up to the storage 
adaptor.

LibvirtStorageAdaptor: need to implement dummy 
connectPhysicalDisk/disconnectPhysicalDisk

StorageAdaptor: define connectPhysicalDisk/disconnectPhysicalDisk in the 
interface

TODO: figure out how to disconnect disks on StopCommand. Currently cloudstack 
only passes us a VM name, which is sufficient to stop the vm, and we can see 
which disks are attached to the vm, but we don't really know which storage 
adaptors to call to disconnect, unless we can do fancy parsing of the disk path 
to determine adaptor type. We could maybe do a 'disconnectPhysicalDiskByPath in 
the KVMStoragePoolManager to determine adaptor by path and call adaptor. I 
looked around, and adding new info to StopCommand to be passed along seems less 
than trivial, like it was for migrate.


Diffs
-

  core/src/com/cloud/agent/api/MigrateCommand.java 5042b8c 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
 3ee811f 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java
 e09c9ba 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java
 c69f9b0 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java
 123a9f1 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/StorageAdaptor.java
 4956d8d 
  server/src/com/cloud/vm/VirtualMachineManagerImpl.java d46bbb0 

Diff: https://reviews.apache.org/r/14381/diff/


Testing
---

Basic testing with my storage adaptor


Thanks,

Marcus Sorensen

Re: add connect method on KVM storage code

[Marcus Sorensen]

On a side note, I'm actually a proponent of the storage plugin only talking
to its storage api. Best to be hypervisor agnostic. Its just in the case of
the default storage plugin the storage API is the cloudstack agent.

But enough about that, we can spin that off into another thread if we need
to. This is about adding connect calls to the StorageAdaptor interface so
custom storage pool types can prepare disks prior to use. I need to know
how it will look going forward so I can continue my work.
On Sep 27, 2013 8:00 PM, "Marcus Sorensen"  wrote:

> I actually think its irrelevant to this discussion, since its really only
> a contention point for the default KVM plugin. I did post this with the
> intention of getting feedback from anyone who cares about the KVM side and
> how it handles things.
>
> What you describe with Xen sounds identical to what we are doing here.
> That process is what I've been trying to explain. Your/my plugin create a
> volume on the SAN the first time its used, and then CS calls attach volume
> in KVMStorageProcessor or XenStorageProcessor to make the lun available to
> the hypervisor. On KVM this calls your storage adaptor to run iscsiadm
> commands (connectPhysicalDisk, the subject of this thread). I think we were
> good on all of it up to the point where you mentioned createPhysicalDisk,
> which isn't necessary to implement since your plugin did it way back when.
> On Sep 27, 2013 7:51 PM, "Mike Tutkowski" 
> wrote:
>
>> I think we should get John Burwell an Edison Su involved in these
>> discussions.
>>
>> John was a proponent for not having storage plug-ins talk to hypervisors
>> and Edison felt it was fine.
>>
>> Until we resolve that design issue, we might just be talking in circles
>> here. :)
>>
>>
>> On Fri, Sep 27, 2013 at 7:29 PM, Marcus Sorensen > >wrote:
>>
>> > The plugin itself doesn't talk to the hypervisor.
>> > On Sep 27, 2013 7:28 PM, "Mike Tutkowski" > >
>> > wrote:
>> >
>> > > Yeah, we should bring John Burwell into this conversation. He had a
>> > strong
>> > > opposition to storage plug-ins talking to hypervisors. Perhaps he was
>> > just
>> > > referring to shared storage, though (not local disks).
>> > >
>> > > Anyways, prior to 4.2 for XenServer and VMware, you had to preallocate
>> > your
>> > > SR/datastore (there was a minor exception for XenServer). CloudStack
>> > wasn't
>> > > involved in that process at all. It was a manual admin task.
>> > >
>> > > The storage plug-in framework didn't really solve that problem,
>> though.
>> > All
>> > > it did was allow you to create your storage volume and make it
>> available
>> > to
>> > > CloudStack. There was no way to get XenServer or VMware to actually
>> use
>> > it
>> > > because they didn't want an IQN or whatever, they wanted an
>> SR/datastore.
>> > >
>> > > That being the case, I had to write code to detect this "managed"
>> > situation
>> > > in XenServer and VMware (in the attach/detach methods).
>> > >
>> > > For example, for XenServer I had to first create an SR and a VDI
>> (which
>> > the
>> > > SolidFire plug-in probably should have no knowledge of). Then the
>> attach
>> > > logic would work.
>> > >
>> > > After a detach, I had to have XenServer "forget" the SR.
>> > >
>> > > What I was thinking is that the attach/detach methods aren't really
>> the
>> > > right place for this special logic.
>> > >
>> > > We could have VolumeApiServiceImpl know about the special needs of
>> > > "managed" storage.
>> > >
>> > > If you want to do an attach of a volume and you're managed storage,
>> > > VolumeApiService Imple could first tell the hypervisor to create the
>> > > necessary objects (XenServer = SR/VDI, VMware = datastore/VMDK, KVM =
>> > just
>> > > establish the connection to the iSCSI target).
>> > >
>> > > It true that there are commands like CreateObjectCommand in XenServer
>> and
>> > > VMware now, but they are only used when the storage is preallocated
>> (not
>> > > when the storage is managed by CloudStack).
>> > >
>> > > In the end, we kind of have two different storage models in
>> CloudStack:
>> > > Non-managed (preallocated storage by admin) and managed (plug-in
>> > > dynamically creates storage on a storage system).
>> > >
>> > >
>> > > On Fri, Sep 27, 2013 at 6:46 PM, Marcus Sorensen > > > >wrote:
>> > >
>> > > > On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen <
>> shadow...@gmail.com>
>> > > > wrote:
>> > > > > On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
>> > > > >  wrote:
>> > > > >> createAsync is just for creating the SAN (or whatever storage)
>> > volume.
>> > > > >> deleteAsync is the reverse.
>> > > > >
>> > > > > Exactly. It used to be that the hypervisor created the
>> disk/lun/file
>> > > > > volume via createPhysicalDisk. Now it's done on the SAN if the
>> plugin
>> > > > > supports it. Ideally, only calls that are required for utilizing
>> the
>> > > > > storage (or perhaps things like copy to NFS, where a server need
>> be
>> > > > > involved if your SAN can't do it

Re: add connect method on KVM storage code

[Marcus Sorensen]

I actually think its irrelevant to this discussion, since its really only a
contention point for the default KVM plugin. I did post this with the
intention of getting feedback from anyone who cares about the KVM side and
how it handles things.

What you describe with Xen sounds identical to what we are doing here. That
process is what I've been trying to explain. Your/my plugin create a volume
on the SAN the first time its used, and then CS calls attach volume in
KVMStorageProcessor or XenStorageProcessor to make the lun available to the
hypervisor. On KVM this calls your storage adaptor to run iscsiadm commands
(connectPhysicalDisk, the subject of this thread). I think we were good on
all of it up to the point where you mentioned createPhysicalDisk, which
isn't necessary to implement since your plugin did it way back when.
On Sep 27, 2013 7:51 PM, "Mike Tutkowski" 
wrote:

> I think we should get John Burwell an Edison Su involved in these
> discussions.
>
> John was a proponent for not having storage plug-ins talk to hypervisors
> and Edison felt it was fine.
>
> Until we resolve that design issue, we might just be talking in circles
> here. :)
>
>
> On Fri, Sep 27, 2013 at 7:29 PM, Marcus Sorensen  >wrote:
>
> > The plugin itself doesn't talk to the hypervisor.
> > On Sep 27, 2013 7:28 PM, "Mike Tutkowski" 
> > wrote:
> >
> > > Yeah, we should bring John Burwell into this conversation. He had a
> > strong
> > > opposition to storage plug-ins talking to hypervisors. Perhaps he was
> > just
> > > referring to shared storage, though (not local disks).
> > >
> > > Anyways, prior to 4.2 for XenServer and VMware, you had to preallocate
> > your
> > > SR/datastore (there was a minor exception for XenServer). CloudStack
> > wasn't
> > > involved in that process at all. It was a manual admin task.
> > >
> > > The storage plug-in framework didn't really solve that problem, though.
> > All
> > > it did was allow you to create your storage volume and make it
> available
> > to
> > > CloudStack. There was no way to get XenServer or VMware to actually use
> > it
> > > because they didn't want an IQN or whatever, they wanted an
> SR/datastore.
> > >
> > > That being the case, I had to write code to detect this "managed"
> > situation
> > > in XenServer and VMware (in the attach/detach methods).
> > >
> > > For example, for XenServer I had to first create an SR and a VDI (which
> > the
> > > SolidFire plug-in probably should have no knowledge of). Then the
> attach
> > > logic would work.
> > >
> > > After a detach, I had to have XenServer "forget" the SR.
> > >
> > > What I was thinking is that the attach/detach methods aren't really the
> > > right place for this special logic.
> > >
> > > We could have VolumeApiServiceImpl know about the special needs of
> > > "managed" storage.
> > >
> > > If you want to do an attach of a volume and you're managed storage,
> > > VolumeApiService Imple could first tell the hypervisor to create the
> > > necessary objects (XenServer = SR/VDI, VMware = datastore/VMDK, KVM =
> > just
> > > establish the connection to the iSCSI target).
> > >
> > > It true that there are commands like CreateObjectCommand in XenServer
> and
> > > VMware now, but they are only used when the storage is preallocated
> (not
> > > when the storage is managed by CloudStack).
> > >
> > > In the end, we kind of have two different storage models in CloudStack:
> > > Non-managed (preallocated storage by admin) and managed (plug-in
> > > dynamically creates storage on a storage system).
> > >
> > >
> > > On Fri, Sep 27, 2013 at 6:46 PM, Marcus Sorensen  > > >wrote:
> > >
> > > > On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen <
> shadow...@gmail.com>
> > > > wrote:
> > > > > On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
> > > > >  wrote:
> > > > >> createAsync is just for creating the SAN (or whatever storage)
> > volume.
> > > > >> deleteAsync is the reverse.
> > > > >
> > > > > Exactly. It used to be that the hypervisor created the
> disk/lun/file
> > > > > volume via createPhysicalDisk. Now it's done on the SAN if the
> plugin
> > > > > supports it. Ideally, only calls that are required for utilizing
> the
> > > > > storage (or perhaps things like copy to NFS, where a server need be
> > > > > involved if your SAN can't do it directly) will go to the
> hypervisor,
> > > > > for external plugins.
> > > > >
> > > > > So mgmt server creates the LUN on the SAN, then calls the
> hypervisor
> > > > > to attach it to the host, so that a VM can make use of it. The
> > > > > createAsync is hypervisor agnostic, it just creates a LUN, and then
> > > > > when you go to start up a VM or attach it to one it calls the
> > > > > hypervisor-specific code to make it available.
> > > >
> > > > I think this is the same with the other hypervisors as well. For
> > > > example with Xen, you create the LUN via your plugin, and then a call
> > > > is made to Xen to register it as an SR. It's basically the same, only
> > > > you're now

Re: add connect method on KVM storage code

[Mike Tutkowski]

I think we should get John Burwell an Edison Su involved in these
discussions.

John was a proponent for not having storage plug-ins talk to hypervisors
and Edison felt it was fine.

Until we resolve that design issue, we might just be talking in circles
here. :)


On Fri, Sep 27, 2013 at 7:29 PM, Marcus Sorensen wrote:

> The plugin itself doesn't talk to the hypervisor.
> On Sep 27, 2013 7:28 PM, "Mike Tutkowski" 
> wrote:
>
> > Yeah, we should bring John Burwell into this conversation. He had a
> strong
> > opposition to storage plug-ins talking to hypervisors. Perhaps he was
> just
> > referring to shared storage, though (not local disks).
> >
> > Anyways, prior to 4.2 for XenServer and VMware, you had to preallocate
> your
> > SR/datastore (there was a minor exception for XenServer). CloudStack
> wasn't
> > involved in that process at all. It was a manual admin task.
> >
> > The storage plug-in framework didn't really solve that problem, though.
> All
> > it did was allow you to create your storage volume and make it available
> to
> > CloudStack. There was no way to get XenServer or VMware to actually use
> it
> > because they didn't want an IQN or whatever, they wanted an SR/datastore.
> >
> > That being the case, I had to write code to detect this "managed"
> situation
> > in XenServer and VMware (in the attach/detach methods).
> >
> > For example, for XenServer I had to first create an SR and a VDI (which
> the
> > SolidFire plug-in probably should have no knowledge of). Then the attach
> > logic would work.
> >
> > After a detach, I had to have XenServer "forget" the SR.
> >
> > What I was thinking is that the attach/detach methods aren't really the
> > right place for this special logic.
> >
> > We could have VolumeApiServiceImpl know about the special needs of
> > "managed" storage.
> >
> > If you want to do an attach of a volume and you're managed storage,
> > VolumeApiService Imple could first tell the hypervisor to create the
> > necessary objects (XenServer = SR/VDI, VMware = datastore/VMDK, KVM =
> just
> > establish the connection to the iSCSI target).
> >
> > It true that there are commands like CreateObjectCommand in XenServer and
> > VMware now, but they are only used when the storage is preallocated (not
> > when the storage is managed by CloudStack).
> >
> > In the end, we kind of have two different storage models in CloudStack:
> > Non-managed (preallocated storage by admin) and managed (plug-in
> > dynamically creates storage on a storage system).
> >
> >
> > On Fri, Sep 27, 2013 at 6:46 PM, Marcus Sorensen  > >wrote:
> >
> > > On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen 
> > > wrote:
> > > > On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
> > > >  wrote:
> > > >> createAsync is just for creating the SAN (or whatever storage)
> volume.
> > > >> deleteAsync is the reverse.
> > > >
> > > > Exactly. It used to be that the hypervisor created the disk/lun/file
> > > > volume via createPhysicalDisk. Now it's done on the SAN if the plugin
> > > > supports it. Ideally, only calls that are required for utilizing the
> > > > storage (or perhaps things like copy to NFS, where a server need be
> > > > involved if your SAN can't do it directly) will go to the hypervisor,
> > > > for external plugins.
> > > >
> > > > So mgmt server creates the LUN on the SAN, then calls the hypervisor
> > > > to attach it to the host, so that a VM can make use of it. The
> > > > createAsync is hypervisor agnostic, it just creates a LUN, and then
> > > > when you go to start up a VM or attach it to one it calls the
> > > > hypervisor-specific code to make it available.
> > >
> > > I think this is the same with the other hypervisors as well. For
> > > example with Xen, you create the LUN via your plugin, and then a call
> > > is made to Xen to register it as an SR. It's basically the same, only
> > > you're now coding the 'register' part yourself (and it's ephemeral,
> > > it's re-registered whenever it's used). You basically take over some
> > > of the API coding that already exists on the other platforms, which
> > > means you can do whatever you want instead of just what's supported,
> > > but you have to do the work as well.
> > >
> > > >
> > > >>
> > > >> Technically even the default plug-in should not call into the
> > hypervisor
> > > >> layer.
> > > >
> > > > There's no way you can create a local storage file to use as a
> volume,
> > > > or CLVM volume, or other types of libvirt storage without calling a
> > > > service that runs on the hypervisor. Those things exist only on the
> > > > hypervisor, and are controlled by the hypervisor. For NFS, you could
> > > > create a separate API for your NFS server that creates qcow2 images
> on
> > > > your NFS primary, I suppose.
> > > >
> > > > One of the really nice things about KVM is that we can do whatever a
> > > > Linux box is capable of, it was one of the draws we had to it. We
> > > > wouldn't be able to do the storage we do with Xen or VMWare.
> > > 

Re: add connect method on KVM storage code

[Mike Tutkowski]

Let's see...to answer your question here:

With Xen, as with any hypervisor, the storage framework invokes createAsync
for you to create your volume just before it is to be attached to a VM for
the first time.

A call is not necessarily made from the storage plug-in to register it as
an SR, however. My storage plug-in simply creates the volume on the SAN and
updates the CS DB with this info. It is left to CloudStack to take it from
there.

This SR creation (and VDI creation) is taken care of when you attach the
volume (in XenServerStorageProcessor when the attachVolume method is
invoked).

Same idea for VMware.


On Fri, Sep 27, 2013 at 6:46 PM, Marcus Sorensen wrote:

> On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen 
> wrote:
> > On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
> >  wrote:
> >> createAsync is just for creating the SAN (or whatever storage) volume.
> >> deleteAsync is the reverse.
> >
> > Exactly. It used to be that the hypervisor created the disk/lun/file
> > volume via createPhysicalDisk. Now it's done on the SAN if the plugin
> > supports it. Ideally, only calls that are required for utilizing the
> > storage (or perhaps things like copy to NFS, where a server need be
> > involved if your SAN can't do it directly) will go to the hypervisor,
> > for external plugins.
> >
> > So mgmt server creates the LUN on the SAN, then calls the hypervisor
> > to attach it to the host, so that a VM can make use of it. The
> > createAsync is hypervisor agnostic, it just creates a LUN, and then
> > when you go to start up a VM or attach it to one it calls the
> > hypervisor-specific code to make it available.
>
> I think this is the same with the other hypervisors as well. For
> example with Xen, you create the LUN via your plugin, and then a call
> is made to Xen to register it as an SR. It's basically the same, only
> you're now coding the 'register' part yourself (and it's ephemeral,
> it's re-registered whenever it's used). You basically take over some
> of the API coding that already exists on the other platforms, which
> means you can do whatever you want instead of just what's supported,
> but you have to do the work as well.
>
> >
> >>
> >> Technically even the default plug-in should not call into the hypervisor
> >> layer.
> >
> > There's no way you can create a local storage file to use as a volume,
> > or CLVM volume, or other types of libvirt storage without calling a
> > service that runs on the hypervisor. Those things exist only on the
> > hypervisor, and are controlled by the hypervisor. For NFS, you could
> > create a separate API for your NFS server that creates qcow2 images on
> > your NFS primary, I suppose.
> >
> > One of the really nice things about KVM is that we can do whatever a
> > Linux box is capable of, it was one of the draws we had to it. We
> > wouldn't be able to do the storage we do with Xen or VMWare.
> >
> >>
> >> The storage layer should probably not be aware of the hypervisor layer.
> >
> > That's fine, but there's no reason why a storage plugin can't talk to
> > the agent that happens to be running on the hypervisor for
> > implementation, if that's what the plugin intends.  I don't see the
> > distinction between utilizing the kvm agent as you storage API or
> > talking to a custom SAN API, or some other concocted service. That's
> > sort of the point of the plugin, people can do whatever they want.
> >
> >>
> >> On Fri, Sep 27, 2013 at 5:14 PM, Mike Tutkowski <
> >> mike.tutkow...@solidfire.com> wrote:
> >>
> >>> Well, from what I saw with XenServer and VMware, that hypervisor
> logic's
> >>> attachVolume command also assumed a VDI/VMDK was created in advance.
> >>>
> >>> I had to put logic in those attachVolume methods to create the SR/VDI
> or
> >>> datastore/VMDK.
> >>>
> >>> However, thinking back on it, it might have made more sense for the
> >>> storage framework to detect if the storage in question was managed and
> -
> >>> before calling attach - call create.
> >>>
> >>> If that logic was in place, I could have left attach/detachVolume alone
> >>> and implemented create and delete in the hypervisor code to create my
> >>> SR/VDI or datastore/VMDK.
> >>>
> >>> That makes sense to me because it is CloudStack-managed storage (so
> >>> CloudStack is calling into the hypervisor to create and delete these
> types
> >>> of objects...it's managing them).
> >>>
> >>>
> >>> On Fri, Sep 27, 2013 at 5:00 PM, Marcus Sorensen  >wrote:
> >>>
>  On Fri, Sep 27, 2013 at 4:22 PM, Mike Tutkowski
>   wrote:
>  > Sure, sounds good - let me know when it's up on Review Board and I
> can
>  take
>  > a look.
>  >
>  > I made most of the changes you and I talked about:
>  >
>  >
> 
> https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1
>  >
>  > I have a new idea, though, that I think will simplify this:
>  >
>  > The main "weirdness" right now is when attachVolume is called

Re: add connect method on KVM storage code

[Marcus Sorensen]

Yeah, OK. That's kind of what I'm talking about. With KVM we have storage
adaptors, so they can just pass the VM info to the hypervisor and it can
select the right way to handle the storage on the fly, which is a bit
easier, even though ultimately you still have to write the hypervisor side
code.
On Sep 27, 2013 7:28 PM, "Mike Tutkowski" 
wrote:

> Yeah, we should bring John Burwell into this conversation. He had a strong
> opposition to storage plug-ins talking to hypervisors. Perhaps he was just
> referring to shared storage, though (not local disks).
>
> Anyways, prior to 4.2 for XenServer and VMware, you had to preallocate your
> SR/datastore (there was a minor exception for XenServer). CloudStack wasn't
> involved in that process at all. It was a manual admin task.
>
> The storage plug-in framework didn't really solve that problem, though. All
> it did was allow you to create your storage volume and make it available to
> CloudStack. There was no way to get XenServer or VMware to actually use it
> because they didn't want an IQN or whatever, they wanted an SR/datastore.
>
> That being the case, I had to write code to detect this "managed" situation
> in XenServer and VMware (in the attach/detach methods).
>
> For example, for XenServer I had to first create an SR and a VDI (which the
> SolidFire plug-in probably should have no knowledge of). Then the attach
> logic would work.
>
> After a detach, I had to have XenServer "forget" the SR.
>
> What I was thinking is that the attach/detach methods aren't really the
> right place for this special logic.
>
> We could have VolumeApiServiceImpl know about the special needs of
> "managed" storage.
>
> If you want to do an attach of a volume and you're managed storage,
> VolumeApiService Imple could first tell the hypervisor to create the
> necessary objects (XenServer = SR/VDI, VMware = datastore/VMDK, KVM = just
> establish the connection to the iSCSI target).
>
> It true that there are commands like CreateObjectCommand in XenServer and
> VMware now, but they are only used when the storage is preallocated (not
> when the storage is managed by CloudStack).
>
> In the end, we kind of have two different storage models in CloudStack:
> Non-managed (preallocated storage by admin) and managed (plug-in
> dynamically creates storage on a storage system).
>
>
> On Fri, Sep 27, 2013 at 6:46 PM, Marcus Sorensen  >wrote:
>
> > On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen 
> > wrote:
> > > On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
> > >  wrote:
> > >> createAsync is just for creating the SAN (or whatever storage) volume.
> > >> deleteAsync is the reverse.
> > >
> > > Exactly. It used to be that the hypervisor created the disk/lun/file
> > > volume via createPhysicalDisk. Now it's done on the SAN if the plugin
> > > supports it. Ideally, only calls that are required for utilizing the
> > > storage (or perhaps things like copy to NFS, where a server need be
> > > involved if your SAN can't do it directly) will go to the hypervisor,
> > > for external plugins.
> > >
> > > So mgmt server creates the LUN on the SAN, then calls the hypervisor
> > > to attach it to the host, so that a VM can make use of it. The
> > > createAsync is hypervisor agnostic, it just creates a LUN, and then
> > > when you go to start up a VM or attach it to one it calls the
> > > hypervisor-specific code to make it available.
> >
> > I think this is the same with the other hypervisors as well. For
> > example with Xen, you create the LUN via your plugin, and then a call
> > is made to Xen to register it as an SR. It's basically the same, only
> > you're now coding the 'register' part yourself (and it's ephemeral,
> > it's re-registered whenever it's used). You basically take over some
> > of the API coding that already exists on the other platforms, which
> > means you can do whatever you want instead of just what's supported,
> > but you have to do the work as well.
> >
> > >
> > >>
> > >> Technically even the default plug-in should not call into the
> hypervisor
> > >> layer.
> > >
> > > There's no way you can create a local storage file to use as a volume,
> > > or CLVM volume, or other types of libvirt storage without calling a
> > > service that runs on the hypervisor. Those things exist only on the
> > > hypervisor, and are controlled by the hypervisor. For NFS, you could
> > > create a separate API for your NFS server that creates qcow2 images on
> > > your NFS primary, I suppose.
> > >
> > > One of the really nice things about KVM is that we can do whatever a
> > > Linux box is capable of, it was one of the draws we had to it. We
> > > wouldn't be able to do the storage we do with Xen or VMWare.
> > >
> > >>
> > >> The storage layer should probably not be aware of the hypervisor
> layer.
> > >
> > > That's fine, but there's no reason why a storage plugin can't talk to
> > > the agent that happens to be running on the hypervisor for
> > > implementation, if that's what the plugin inten

Re: add connect method on KVM storage code

[Marcus Sorensen]

Well, the default KVM one does, but only because the hypervisor is playing
the part or storage API. My plugin certainly doesn't have to talk to the
hypervisor, and I don't think I've instructed you to.
On Sep 27, 2013 7:29 PM, "Marcus Sorensen"  wrote:

> The plugin itself doesn't talk to the hypervisor.
> On Sep 27, 2013 7:28 PM, "Mike Tutkowski" 
> wrote:
>
>> Yeah, we should bring John Burwell into this conversation. He had a strong
>> opposition to storage plug-ins talking to hypervisors. Perhaps he was just
>> referring to shared storage, though (not local disks).
>>
>> Anyways, prior to 4.2 for XenServer and VMware, you had to preallocate
>> your
>> SR/datastore (there was a minor exception for XenServer). CloudStack
>> wasn't
>> involved in that process at all. It was a manual admin task.
>>
>> The storage plug-in framework didn't really solve that problem, though.
>> All
>> it did was allow you to create your storage volume and make it available
>> to
>> CloudStack. There was no way to get XenServer or VMware to actually use it
>> because they didn't want an IQN or whatever, they wanted an SR/datastore.
>>
>> That being the case, I had to write code to detect this "managed"
>> situation
>> in XenServer and VMware (in the attach/detach methods).
>>
>> For example, for XenServer I had to first create an SR and a VDI (which
>> the
>> SolidFire plug-in probably should have no knowledge of). Then the attach
>> logic would work.
>>
>> After a detach, I had to have XenServer "forget" the SR.
>>
>> What I was thinking is that the attach/detach methods aren't really the
>> right place for this special logic.
>>
>> We could have VolumeApiServiceImpl know about the special needs of
>> "managed" storage.
>>
>> If you want to do an attach of a volume and you're managed storage,
>> VolumeApiService Imple could first tell the hypervisor to create the
>> necessary objects (XenServer = SR/VDI, VMware = datastore/VMDK, KVM = just
>> establish the connection to the iSCSI target).
>>
>> It true that there are commands like CreateObjectCommand in XenServer and
>> VMware now, but they are only used when the storage is preallocated (not
>> when the storage is managed by CloudStack).
>>
>> In the end, we kind of have two different storage models in CloudStack:
>> Non-managed (preallocated storage by admin) and managed (plug-in
>> dynamically creates storage on a storage system).
>>
>>
>> On Fri, Sep 27, 2013 at 6:46 PM, Marcus Sorensen > >wrote:
>>
>> > On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen 
>> > wrote:
>> > > On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
>> > >  wrote:
>> > >> createAsync is just for creating the SAN (or whatever storage)
>> volume.
>> > >> deleteAsync is the reverse.
>> > >
>> > > Exactly. It used to be that the hypervisor created the disk/lun/file
>> > > volume via createPhysicalDisk. Now it's done on the SAN if the plugin
>> > > supports it. Ideally, only calls that are required for utilizing the
>> > > storage (or perhaps things like copy to NFS, where a server need be
>> > > involved if your SAN can't do it directly) will go to the hypervisor,
>> > > for external plugins.
>> > >
>> > > So mgmt server creates the LUN on the SAN, then calls the hypervisor
>> > > to attach it to the host, so that a VM can make use of it. The
>> > > createAsync is hypervisor agnostic, it just creates a LUN, and then
>> > > when you go to start up a VM or attach it to one it calls the
>> > > hypervisor-specific code to make it available.
>> >
>> > I think this is the same with the other hypervisors as well. For
>> > example with Xen, you create the LUN via your plugin, and then a call
>> > is made to Xen to register it as an SR. It's basically the same, only
>> > you're now coding the 'register' part yourself (and it's ephemeral,
>> > it's re-registered whenever it's used). You basically take over some
>> > of the API coding that already exists on the other platforms, which
>> > means you can do whatever you want instead of just what's supported,
>> > but you have to do the work as well.
>> >
>> > >
>> > >>
>> > >> Technically even the default plug-in should not call into the
>> hypervisor
>> > >> layer.
>> > >
>> > > There's no way you can create a local storage file to use as a volume,
>> > > or CLVM volume, or other types of libvirt storage without calling a
>> > > service that runs on the hypervisor. Those things exist only on the
>> > > hypervisor, and are controlled by the hypervisor. For NFS, you could
>> > > create a separate API for your NFS server that creates qcow2 images on
>> > > your NFS primary, I suppose.
>> > >
>> > > One of the really nice things about KVM is that we can do whatever a
>> > > Linux box is capable of, it was one of the draws we had to it. We
>> > > wouldn't be able to do the storage we do with Xen or VMWare.
>> > >
>> > >>
>> > >> The storage layer should probably not be aware of the hypervisor
>> layer.
>> > >
>> > > That's fine, but there's no reason why a storage plugin ca

Re: add connect method on KVM storage code

[Marcus Sorensen]

The plugin itself doesn't talk to the hypervisor.
On Sep 27, 2013 7:28 PM, "Mike Tutkowski" 
wrote:

> Yeah, we should bring John Burwell into this conversation. He had a strong
> opposition to storage plug-ins talking to hypervisors. Perhaps he was just
> referring to shared storage, though (not local disks).
>
> Anyways, prior to 4.2 for XenServer and VMware, you had to preallocate your
> SR/datastore (there was a minor exception for XenServer). CloudStack wasn't
> involved in that process at all. It was a manual admin task.
>
> The storage plug-in framework didn't really solve that problem, though. All
> it did was allow you to create your storage volume and make it available to
> CloudStack. There was no way to get XenServer or VMware to actually use it
> because they didn't want an IQN or whatever, they wanted an SR/datastore.
>
> That being the case, I had to write code to detect this "managed" situation
> in XenServer and VMware (in the attach/detach methods).
>
> For example, for XenServer I had to first create an SR and a VDI (which the
> SolidFire plug-in probably should have no knowledge of). Then the attach
> logic would work.
>
> After a detach, I had to have XenServer "forget" the SR.
>
> What I was thinking is that the attach/detach methods aren't really the
> right place for this special logic.
>
> We could have VolumeApiServiceImpl know about the special needs of
> "managed" storage.
>
> If you want to do an attach of a volume and you're managed storage,
> VolumeApiService Imple could first tell the hypervisor to create the
> necessary objects (XenServer = SR/VDI, VMware = datastore/VMDK, KVM = just
> establish the connection to the iSCSI target).
>
> It true that there are commands like CreateObjectCommand in XenServer and
> VMware now, but they are only used when the storage is preallocated (not
> when the storage is managed by CloudStack).
>
> In the end, we kind of have two different storage models in CloudStack:
> Non-managed (preallocated storage by admin) and managed (plug-in
> dynamically creates storage on a storage system).
>
>
> On Fri, Sep 27, 2013 at 6:46 PM, Marcus Sorensen  >wrote:
>
> > On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen 
> > wrote:
> > > On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
> > >  wrote:
> > >> createAsync is just for creating the SAN (or whatever storage) volume.
> > >> deleteAsync is the reverse.
> > >
> > > Exactly. It used to be that the hypervisor created the disk/lun/file
> > > volume via createPhysicalDisk. Now it's done on the SAN if the plugin
> > > supports it. Ideally, only calls that are required for utilizing the
> > > storage (or perhaps things like copy to NFS, where a server need be
> > > involved if your SAN can't do it directly) will go to the hypervisor,
> > > for external plugins.
> > >
> > > So mgmt server creates the LUN on the SAN, then calls the hypervisor
> > > to attach it to the host, so that a VM can make use of it. The
> > > createAsync is hypervisor agnostic, it just creates a LUN, and then
> > > when you go to start up a VM or attach it to one it calls the
> > > hypervisor-specific code to make it available.
> >
> > I think this is the same with the other hypervisors as well. For
> > example with Xen, you create the LUN via your plugin, and then a call
> > is made to Xen to register it as an SR. It's basically the same, only
> > you're now coding the 'register' part yourself (and it's ephemeral,
> > it's re-registered whenever it's used). You basically take over some
> > of the API coding that already exists on the other platforms, which
> > means you can do whatever you want instead of just what's supported,
> > but you have to do the work as well.
> >
> > >
> > >>
> > >> Technically even the default plug-in should not call into the
> hypervisor
> > >> layer.
> > >
> > > There's no way you can create a local storage file to use as a volume,
> > > or CLVM volume, or other types of libvirt storage without calling a
> > > service that runs on the hypervisor. Those things exist only on the
> > > hypervisor, and are controlled by the hypervisor. For NFS, you could
> > > create a separate API for your NFS server that creates qcow2 images on
> > > your NFS primary, I suppose.
> > >
> > > One of the really nice things about KVM is that we can do whatever a
> > > Linux box is capable of, it was one of the draws we had to it. We
> > > wouldn't be able to do the storage we do with Xen or VMWare.
> > >
> > >>
> > >> The storage layer should probably not be aware of the hypervisor
> layer.
> > >
> > > That's fine, but there's no reason why a storage plugin can't talk to
> > > the agent that happens to be running on the hypervisor for
> > > implementation, if that's what the plugin intends.  I don't see the
> > > distinction between utilizing the kvm agent as you storage API or
> > > talking to a custom SAN API, or some other concocted service. That's
> > > sort of the point of the plugin, people can do whatever they want.
> > >
>

Re: add connect method on KVM storage code

[Mike Tutkowski]

Yeah, we should bring John Burwell into this conversation. He had a strong
opposition to storage plug-ins talking to hypervisors. Perhaps he was just
referring to shared storage, though (not local disks).

Anyways, prior to 4.2 for XenServer and VMware, you had to preallocate your
SR/datastore (there was a minor exception for XenServer). CloudStack wasn't
involved in that process at all. It was a manual admin task.

The storage plug-in framework didn't really solve that problem, though. All
it did was allow you to create your storage volume and make it available to
CloudStack. There was no way to get XenServer or VMware to actually use it
because they didn't want an IQN or whatever, they wanted an SR/datastore.

That being the case, I had to write code to detect this "managed" situation
in XenServer and VMware (in the attach/detach methods).

For example, for XenServer I had to first create an SR and a VDI (which the
SolidFire plug-in probably should have no knowledge of). Then the attach
logic would work.

After a detach, I had to have XenServer "forget" the SR.

What I was thinking is that the attach/detach methods aren't really the
right place for this special logic.

We could have VolumeApiServiceImpl know about the special needs of
"managed" storage.

If you want to do an attach of a volume and you're managed storage,
VolumeApiService Imple could first tell the hypervisor to create the
necessary objects (XenServer = SR/VDI, VMware = datastore/VMDK, KVM = just
establish the connection to the iSCSI target).

It true that there are commands like CreateObjectCommand in XenServer and
VMware now, but they are only used when the storage is preallocated (not
when the storage is managed by CloudStack).

In the end, we kind of have two different storage models in CloudStack:
Non-managed (preallocated storage by admin) and managed (plug-in
dynamically creates storage on a storage system).


On Fri, Sep 27, 2013 at 6:46 PM, Marcus Sorensen wrote:

> On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen 
> wrote:
> > On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
> >  wrote:
> >> createAsync is just for creating the SAN (or whatever storage) volume.
> >> deleteAsync is the reverse.
> >
> > Exactly. It used to be that the hypervisor created the disk/lun/file
> > volume via createPhysicalDisk. Now it's done on the SAN if the plugin
> > supports it. Ideally, only calls that are required for utilizing the
> > storage (or perhaps things like copy to NFS, where a server need be
> > involved if your SAN can't do it directly) will go to the hypervisor,
> > for external plugins.
> >
> > So mgmt server creates the LUN on the SAN, then calls the hypervisor
> > to attach it to the host, so that a VM can make use of it. The
> > createAsync is hypervisor agnostic, it just creates a LUN, and then
> > when you go to start up a VM or attach it to one it calls the
> > hypervisor-specific code to make it available.
>
> I think this is the same with the other hypervisors as well. For
> example with Xen, you create the LUN via your plugin, and then a call
> is made to Xen to register it as an SR. It's basically the same, only
> you're now coding the 'register' part yourself (and it's ephemeral,
> it's re-registered whenever it's used). You basically take over some
> of the API coding that already exists on the other platforms, which
> means you can do whatever you want instead of just what's supported,
> but you have to do the work as well.
>
> >
> >>
> >> Technically even the default plug-in should not call into the hypervisor
> >> layer.
> >
> > There's no way you can create a local storage file to use as a volume,
> > or CLVM volume, or other types of libvirt storage without calling a
> > service that runs on the hypervisor. Those things exist only on the
> > hypervisor, and are controlled by the hypervisor. For NFS, you could
> > create a separate API for your NFS server that creates qcow2 images on
> > your NFS primary, I suppose.
> >
> > One of the really nice things about KVM is that we can do whatever a
> > Linux box is capable of, it was one of the draws we had to it. We
> > wouldn't be able to do the storage we do with Xen or VMWare.
> >
> >>
> >> The storage layer should probably not be aware of the hypervisor layer.
> >
> > That's fine, but there's no reason why a storage plugin can't talk to
> > the agent that happens to be running on the hypervisor for
> > implementation, if that's what the plugin intends.  I don't see the
> > distinction between utilizing the kvm agent as you storage API or
> > talking to a custom SAN API, or some other concocted service. That's
> > sort of the point of the plugin, people can do whatever they want.
> >
> >>
> >> On Fri, Sep 27, 2013 at 5:14 PM, Mike Tutkowski <
> >> mike.tutkow...@solidfire.com> wrote:
> >>
> >>> Well, from what I saw with XenServer and VMware, that hypervisor
> logic's
> >>> attachVolume command also assumed a VDI/VMDK was created in advance.
> >>>
> >>> I had to put logic

Re: add connect method on KVM storage code

[Marcus Sorensen]

On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen  wrote:
> On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
>  wrote:
>> createAsync is just for creating the SAN (or whatever storage) volume.
>> deleteAsync is the reverse.
>
> Exactly. It used to be that the hypervisor created the disk/lun/file
> volume via createPhysicalDisk. Now it's done on the SAN if the plugin
> supports it. Ideally, only calls that are required for utilizing the
> storage (or perhaps things like copy to NFS, where a server need be
> involved if your SAN can't do it directly) will go to the hypervisor,
> for external plugins.
>
> So mgmt server creates the LUN on the SAN, then calls the hypervisor
> to attach it to the host, so that a VM can make use of it. The
> createAsync is hypervisor agnostic, it just creates a LUN, and then
> when you go to start up a VM or attach it to one it calls the
> hypervisor-specific code to make it available.

I think this is the same with the other hypervisors as well. For
example with Xen, you create the LUN via your plugin, and then a call
is made to Xen to register it as an SR. It's basically the same, only
you're now coding the 'register' part yourself (and it's ephemeral,
it's re-registered whenever it's used). You basically take over some
of the API coding that already exists on the other platforms, which
means you can do whatever you want instead of just what's supported,
but you have to do the work as well.

>
>>
>> Technically even the default plug-in should not call into the hypervisor
>> layer.
>
> There's no way you can create a local storage file to use as a volume,
> or CLVM volume, or other types of libvirt storage without calling a
> service that runs on the hypervisor. Those things exist only on the
> hypervisor, and are controlled by the hypervisor. For NFS, you could
> create a separate API for your NFS server that creates qcow2 images on
> your NFS primary, I suppose.
>
> One of the really nice things about KVM is that we can do whatever a
> Linux box is capable of, it was one of the draws we had to it. We
> wouldn't be able to do the storage we do with Xen or VMWare.
>
>>
>> The storage layer should probably not be aware of the hypervisor layer.
>
> That's fine, but there's no reason why a storage plugin can't talk to
> the agent that happens to be running on the hypervisor for
> implementation, if that's what the plugin intends.  I don't see the
> distinction between utilizing the kvm agent as you storage API or
> talking to a custom SAN API, or some other concocted service. That's
> sort of the point of the plugin, people can do whatever they want.
>
>>
>> On Fri, Sep 27, 2013 at 5:14 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> Well, from what I saw with XenServer and VMware, that hypervisor logic's
>>> attachVolume command also assumed a VDI/VMDK was created in advance.
>>>
>>> I had to put logic in those attachVolume methods to create the SR/VDI or
>>> datastore/VMDK.
>>>
>>> However, thinking back on it, it might have made more sense for the
>>> storage framework to detect if the storage in question was managed and -
>>> before calling attach - call create.
>>>
>>> If that logic was in place, I could have left attach/detachVolume alone
>>> and implemented create and delete in the hypervisor code to create my
>>> SR/VDI or datastore/VMDK.
>>>
>>> That makes sense to me because it is CloudStack-managed storage (so
>>> CloudStack is calling into the hypervisor to create and delete these types
>>> of objects...it's managing them).
>>>
>>>
>>> On Fri, Sep 27, 2013 at 5:00 PM, Marcus Sorensen wrote:
>>>
 On Fri, Sep 27, 2013 at 4:22 PM, Mike Tutkowski
  wrote:
 > Sure, sounds good - let me know when it's up on Review Board and I can
 take
 > a look.
 >
 > I made most of the changes you and I talked about:
 >
 >
 https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1
 >
 > I have a new idea, though, that I think will simplify this:
 >
 > The main "weirdness" right now is when attachVolume is called that the
 > original logic assumed createVolume had been called already.
 >
 > In my case, this doesn't apply, so we had to place extra logic in
 > attachVolume to essentially "create" a volume. We decided to make a
 connect
 > method, which establishes the iSCSI connection and creates a
 > KVMPhysicalDisk that can be returned when attachVolume calls
 > getPhysicalDisk.
 >
 > The "normal" place where you'd create a KVMPhysicalDisk, however, would
 be
 > in the createVolume method. Since I don't currently "create" a volume,
 my
 > only chance to note the size of the volume is in the connect method.

 I don't think createVolume applies to plugins. My impression wash that
 createAsync is called on the mgmt server side. If createVolume IS
 being called, that's weird. The idea here is that mgmt serv

Re: add connect method on KVM storage code

[Marcus Sorensen]

https://reviews.apache.org/r/14381/

On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen  wrote:
> On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
>  wrote:
>> createAsync is just for creating the SAN (or whatever storage) volume.
>> deleteAsync is the reverse.
>
> Exactly. It used to be that the hypervisor created the disk/lun/file
> volume via createPhysicalDisk. Now it's done on the SAN if the plugin
> supports it. Ideally, only calls that are required for utilizing the
> storage (or perhaps things like copy to NFS, where a server need be
> involved if your SAN can't do it directly) will go to the hypervisor,
> for external plugins.
>
> So mgmt server creates the LUN on the SAN, then calls the hypervisor
> to attach it to the host, so that a VM can make use of it. The
> createAsync is hypervisor agnostic, it just creates a LUN, and then
> when you go to start up a VM or attach it to one it calls the
> hypervisor-specific code to make it available.
>
>>
>> Technically even the default plug-in should not call into the hypervisor
>> layer.
>
> There's no way you can create a local storage file to use as a volume,
> or CLVM volume, or other types of libvirt storage without calling a
> service that runs on the hypervisor. Those things exist only on the
> hypervisor, and are controlled by the hypervisor. For NFS, you could
> create a separate API for your NFS server that creates qcow2 images on
> your NFS primary, I suppose.
>
> One of the really nice things about KVM is that we can do whatever a
> Linux box is capable of, it was one of the draws we had to it. We
> wouldn't be able to do the storage we do with Xen or VMWare.
>
>>
>> The storage layer should probably not be aware of the hypervisor layer.
>
> That's fine, but there's no reason why a storage plugin can't talk to
> the agent that happens to be running on the hypervisor for
> implementation, if that's what the plugin intends.  I don't see the
> distinction between utilizing the kvm agent as you storage API or
> talking to a custom SAN API, or some other concocted service. That's
> sort of the point of the plugin, people can do whatever they want.
>
>>
>> On Fri, Sep 27, 2013 at 5:14 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> Well, from what I saw with XenServer and VMware, that hypervisor logic's
>>> attachVolume command also assumed a VDI/VMDK was created in advance.
>>>
>>> I had to put logic in those attachVolume methods to create the SR/VDI or
>>> datastore/VMDK.
>>>
>>> However, thinking back on it, it might have made more sense for the
>>> storage framework to detect if the storage in question was managed and -
>>> before calling attach - call create.
>>>
>>> If that logic was in place, I could have left attach/detachVolume alone
>>> and implemented create and delete in the hypervisor code to create my
>>> SR/VDI or datastore/VMDK.
>>>
>>> That makes sense to me because it is CloudStack-managed storage (so
>>> CloudStack is calling into the hypervisor to create and delete these types
>>> of objects...it's managing them).
>>>
>>>
>>> On Fri, Sep 27, 2013 at 5:00 PM, Marcus Sorensen wrote:
>>>
 On Fri, Sep 27, 2013 at 4:22 PM, Mike Tutkowski
  wrote:
 > Sure, sounds good - let me know when it's up on Review Board and I can
 take
 > a look.
 >
 > I made most of the changes you and I talked about:
 >
 >
 https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1
 >
 > I have a new idea, though, that I think will simplify this:
 >
 > The main "weirdness" right now is when attachVolume is called that the
 > original logic assumed createVolume had been called already.
 >
 > In my case, this doesn't apply, so we had to place extra logic in
 > attachVolume to essentially "create" a volume. We decided to make a
 connect
 > method, which establishes the iSCSI connection and creates a
 > KVMPhysicalDisk that can be returned when attachVolume calls
 > getPhysicalDisk.
 >
 > The "normal" place where you'd create a KVMPhysicalDisk, however, would
 be
 > in the createVolume method. Since I don't currently "create" a volume,
 my
 > only chance to note the size of the volume is in the connect method.

 I don't think createVolume applies to plugins. My impression wash that
 createAsync is called on the mgmt server side. If createVolume IS
 being called, that's weird. The idea here is that mgmt server creates
 the LUN, and then on the KVM side attach is called (or StartCommand if
 it's a root volume and vm is being started), and it assumes that the
 LUN is already there, so we call connectPhysicalDisk to attach it to
 the KVM host.

 >
 > It ends up being kind of weird to pass a size into the connect method,
 as
 > you've noted.
 >
 > What if we essentially left the attachVolume and detachVolume methods
 alone
 > (as in

Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

[Marcus Sorensen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14381/
---

Review request for cloudstack and Mike Wang.


Repository: cloudstack-git


Description
---

MigrateCommand: pass vmTO object so we can see which disks/storage pool types 
belong to the vm when migrating a VM. This facilitates being able to call 
disconnectPhysicalDisksViaVmSpec

VirtualMachineManagerImpl: pass VirtualMachineTO when migrating so that we can 
see which disks belong to the VM and what storage pools/adaptors should be used

LibvirtComputingResource: add calls KVMStoragePoolManager's 
connectPhysicalDiskViaVmSpec and disconnectPhysicalDiskViaVmSpec calls where 
appropriate (when starting a vm, migrating a vm). Ensure that we create 'raw' 
format XML disk definitions when the storage format is RAW.

KVMStoragePoolManager:  add connectPhysicalDisk, disconnectPhysicalDisk, 
connectPhysicalDiskViaVmSpec, disconnectPhysicalDiskViaVmSpec. These all call 
the specific StorageAdaptor's connectPhysicalDisk or disconnectPhysicalDisk 
calls.

KVMStorageProcessor: Call connectPhysicalDisk/disconnectPhysicalDisk on the 
storage adaptor. Whether or not this is implemented is up to the storage 
adaptor.

LibvirtStorageAdaptor: need to implement dummy 
connectPhysicalDisk/disconnectPhysicalDisk

StorageAdaptor: define connectPhysicalDisk/disconnectPhysicalDisk in the 
interface

TODO: figure out how to disconnect disks on StopCommand. Currently cloudstack 
only passes us a VM name, which is sufficient to stop the vm, and we can see 
which disks are attached to the vm, but we don't really know which storage 
adaptors to call to disconnect, unless we can do fancy parsing of the disk path 
to determine adaptor type. We could maybe do a 'disconnectPhysicalDiskByPath in 
the KVMStoragePoolManager to determine adaptor by path and call adaptor. I 
looked around, and adding new info to StopCommand to be passed along seems less 
than trivial, like it was for migrate.


Diffs
-

  core/src/com/cloud/agent/api/MigrateCommand.java 5042b8c 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
 3ee811f 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java
 e09c9ba 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java
 c69f9b0 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java
 123a9f1 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/StorageAdaptor.java
 4956d8d 
  server/src/com/cloud/vm/VirtualMachineManagerImpl.java d46bbb0 

Diff: https://reviews.apache.org/r/14381/diff/


Testing
---

Basic testing with my storage adaptor


Thanks,

Marcus Sorensen

Re: add connect method on KVM storage code

[Marcus Sorensen]

On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
 wrote:
> createAsync is just for creating the SAN (or whatever storage) volume.
> deleteAsync is the reverse.

Exactly. It used to be that the hypervisor created the disk/lun/file
volume via createPhysicalDisk. Now it's done on the SAN if the plugin
supports it. Ideally, only calls that are required for utilizing the
storage (or perhaps things like copy to NFS, where a server need be
involved if your SAN can't do it directly) will go to the hypervisor,
for external plugins.

So mgmt server creates the LUN on the SAN, then calls the hypervisor
to attach it to the host, so that a VM can make use of it. The
createAsync is hypervisor agnostic, it just creates a LUN, and then
when you go to start up a VM or attach it to one it calls the
hypervisor-specific code to make it available.

>
> Technically even the default plug-in should not call into the hypervisor
> layer.

There's no way you can create a local storage file to use as a volume,
or CLVM volume, or other types of libvirt storage without calling a
service that runs on the hypervisor. Those things exist only on the
hypervisor, and are controlled by the hypervisor. For NFS, you could
create a separate API for your NFS server that creates qcow2 images on
your NFS primary, I suppose.

One of the really nice things about KVM is that we can do whatever a
Linux box is capable of, it was one of the draws we had to it. We
wouldn't be able to do the storage we do with Xen or VMWare.

>
> The storage layer should probably not be aware of the hypervisor layer.

That's fine, but there's no reason why a storage plugin can't talk to
the agent that happens to be running on the hypervisor for
implementation, if that's what the plugin intends.  I don't see the
distinction between utilizing the kvm agent as you storage API or
talking to a custom SAN API, or some other concocted service. That's
sort of the point of the plugin, people can do whatever they want.

>
> On Fri, Sep 27, 2013 at 5:14 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Well, from what I saw with XenServer and VMware, that hypervisor logic's
>> attachVolume command also assumed a VDI/VMDK was created in advance.
>>
>> I had to put logic in those attachVolume methods to create the SR/VDI or
>> datastore/VMDK.
>>
>> However, thinking back on it, it might have made more sense for the
>> storage framework to detect if the storage in question was managed and -
>> before calling attach - call create.
>>
>> If that logic was in place, I could have left attach/detachVolume alone
>> and implemented create and delete in the hypervisor code to create my
>> SR/VDI or datastore/VMDK.
>>
>> That makes sense to me because it is CloudStack-managed storage (so
>> CloudStack is calling into the hypervisor to create and delete these types
>> of objects...it's managing them).
>>
>>
>> On Fri, Sep 27, 2013 at 5:00 PM, Marcus Sorensen wrote:
>>
>>> On Fri, Sep 27, 2013 at 4:22 PM, Mike Tutkowski
>>>  wrote:
>>> > Sure, sounds good - let me know when it's up on Review Board and I can
>>> take
>>> > a look.
>>> >
>>> > I made most of the changes you and I talked about:
>>> >
>>> >
>>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1
>>> >
>>> > I have a new idea, though, that I think will simplify this:
>>> >
>>> > The main "weirdness" right now is when attachVolume is called that the
>>> > original logic assumed createVolume had been called already.
>>> >
>>> > In my case, this doesn't apply, so we had to place extra logic in
>>> > attachVolume to essentially "create" a volume. We decided to make a
>>> connect
>>> > method, which establishes the iSCSI connection and creates a
>>> > KVMPhysicalDisk that can be returned when attachVolume calls
>>> > getPhysicalDisk.
>>> >
>>> > The "normal" place where you'd create a KVMPhysicalDisk, however, would
>>> be
>>> > in the createVolume method. Since I don't currently "create" a volume,
>>> my
>>> > only chance to note the size of the volume is in the connect method.
>>>
>>> I don't think createVolume applies to plugins. My impression wash that
>>> createAsync is called on the mgmt server side. If createVolume IS
>>> being called, that's weird. The idea here is that mgmt server creates
>>> the LUN, and then on the KVM side attach is called (or StartCommand if
>>> it's a root volume and vm is being started), and it assumes that the
>>> LUN is already there, so we call connectPhysicalDisk to attach it to
>>> the KVM host.
>>>
>>> >
>>> > It ends up being kind of weird to pass a size into the connect method,
>>> as
>>> > you've noted.
>>> >
>>> > What if we essentially left the attachVolume and detachVolume methods
>>> alone
>>> > (as in how they were before my changes)? We could have
>>> > VolumeApiServiceImpl, before sending the AttachCommand, detect if the
>>> > storage in question is managed. If it is, VolumeApiServiceImpl could
>>> send a
>>> > CreateObjectComman

Re: Review Request 14285: Add multi-row action support to UI list views (CLOUDSTACK-1579)

[Brian Federle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14285/#review26456
---

Ship it!


Ship It!

- Brian Federle


On Sept. 27, 2013, 8:34 p.m., Chris Suich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14285/
> ---
> 
> (Updated Sept. 27, 2013, 8:34 p.m.)
> 
> 
> Review request for cloudstack.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> See https://issues.apache.org/jira/browse/CLOUDSTACK-1579
> 
> I have added the logic and hooks for allowing list views to indicate they 
> support multiselect rows and added hooks for list view actions to indicate 
> they are a multiselect action. The data for the selected rows is stored and 
> passed to callbacks in the same 'context' variable as with non-multiselect 
> callbacks, so implementing multiselect should remain consistent.
> 
> This is purely a functional update and there is cosmetic work to be done 
> shortly by someone other than myself (likely Brian Federle).
> 
> Some notes for those interested:
> - $.remove() vs $.detach(): remove will delete the DOM element while detach 
> simply removes it from view, allowing it to be added later and re-used
> - List views can indicate they support multi-select with 'multiSelect: true'
> - Actions can indicate they are a multi-select action with 
> 'isMultiSelectAction: true' and 'isHeader: true'
> - This patch also includes a new list view flag called 'noActionCol' which 
> intentional hides the 'Quickview' on rows which can be handy when the rows 
> become too wide and look weird
> - Currently, when one or more multi-select rows are selected, 
> non-multi-select actions (such as Add) are hidden and replaced with the 
> multi-select actions, however the idea was proposed to add another row of 
> buttons below the non-multi-select buttons when this is the case - this is 
> still open for discussion as there was no consensus
> 
> 
> Diffs
> -
> 
>   ui/plugins/netapp PRE-CREATION 
>   ui/scripts/ui/widgets/dataTable.js 66d7095 
>   ui/scripts/ui/widgets/listView.js 076b3ab 
> 
> Diff: https://reviews.apache.org/r/14285/diff/
> 
> 
> Testing
> ---
> 
> Testing has been done in conjunction with a UI plugin I am developing. 
> Additional testing will be performed as this is integrated into existing 
> parts of the UI.
> 
> 
> Thanks,
> 
> Chris Suich
> 
>

Re: Cloudstack doesn't seem to recognize we added 100GB of memory to the nodes.

[Nitin Mehta]

I guess you are referring to the same issue as discussed in the thread
http://www.mail-archive.com/dev@cloudstack.apache.org/msg15197.html


There is a bug created
https://issues.apache.org/jira/browse/CLOUDSTACK-4755
For this with the workaround.

On 27/09/13 1:48 PM, "Jeronimo Garcia"  wrote:

>Hi List.
>
>We've done some memory upgrades to our hosts but even tho we can see the
>new memory numbers and all that .. the cluster where this hosts belong to
>still showing full (in the dashboard).
>
>We've forced recalculation and still no luck.
>
>Do you guys happen to know what to do with this?
>
>Thanks

Re: bug? capacity not updated when adding memory to hypervisor

[Timothy Ehlers]

I created a bug here if a developer could describe the code problem. I have
merely added some mysql commands showing how to fix it.

https://issues.apache.org/jira/browse/CLOUDSTACK-4755


On Fri, Sep 27, 2013 at 4:54 PM, Jeronimo Garcia wrote:

> Has this bug been raised?  I have this issue as well .. we are on 4.0.2 and
> 4.1.1
>
>
> On Wed, Sep 25, 2013 at 4:43 PM, Prachi Damle  >wrote:
>
> > Yes, the main bug here is not updating the Total Capacity - this is
> > avoiding the total getting updated on host reconnects.
> >
> > Bug with the Used and Reserved Capacities gets masked since
> > CapacityManager also listens to the VM state changes and updates these
> > values.
> >
> > Prachi
> > -Original Message-
> > From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> > Sent: Wednesday, September 25, 2013 1:45 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: bug? capacity not updated when adding memory to hypervisor
> >
> > The main bug being the lack of
> > memCap.setTotalCapacity(host.getTotalMemory());
> >
> > Darren
> >
> > On Wed, Sep 25, 2013 at 1:13 PM, Daan Hoogland 
> > wrote:
> > > I agree Darren,
> > >
> > > if (cpuCap.getUsedCapacity() == usedCpu &&
> > > cpuCap.getReservedCapacity() == reservedCpu) { } else {
> > >if (cpuCap.getReservedCapacity() != reservedCpu) {
> > > cpuCap.setReservedCapacity(reservedCpu);
> > >}
> > >if (cpuCap.getUsedCapacity() != usedCpu) {
> > > cpuCap.setUsedCapacity(usedCpu);
> > > }
> > > }
> > >
> > > makes more sense. (same for mem)
> > >
> > >
> > > On Wed, Sep 25, 2013 at 9:59 PM, Darren Shepherd <
> > > darren.s.sheph...@gmail.com> wrote:
> > >
> > >> I added more memory to a hypervisor, and the host tables was updated
> > >> properly, but the op_host_capacity is not updated.  I tracked down
> > >> the code to CapacityManagerImpl.updateCapacityForHost.  The below
> > >> logic seems completely wrong.  Unless I'm reading it wrong, it seems
> > >> like it will only update EITHER used or reserved (not both) and never
> > >> update total.  I removed log statements to make it more readable.
> > >>
> > >> if (cpuCap.getUsedCapacity() == usedCpu &&
> > >> cpuCap.getReservedCapacity() == reservedCpu) {
> > >> } else if (cpuCap.getReservedCapacity() != reservedCpu) {
> > >> cpuCap.setReservedCapacity(reservedCpu);
> > >> } else if (cpuCap.getUsedCapacity() != usedCpu) {
> > >> cpuCap.setUsedCapacity(usedCpu);
> > >> }
> > >>
> > >> if (memCap.getUsedCapacity() == usedMemory &&
> > >> memCap.getReservedCapacity() == reservedMemory) {
> > >> } else if (memCap.getReservedCapacity() !=
> reservedMemory) {
> > >> memCap.setReservedCapacity(reservedMemory);
> > >> } else if (memCap.getUsedCapacity() != usedMemory) {
> > >> memCap.setUsedCapacity(usedMemory);
> > >> }
> > >>
> >
>



-- 
Tim Ehlers

Re: add connect method on KVM storage code

[Mike Tutkowski]

createAsync is just for creating the SAN (or whatever storage) volume.
deleteAsync is the reverse.

Technically even the default plug-in should not call into the hypervisor
layer.

The storage layer should probably not be aware of the hypervisor layer.


On Fri, Sep 27, 2013 at 5:14 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Well, from what I saw with XenServer and VMware, that hypervisor logic's
> attachVolume command also assumed a VDI/VMDK was created in advance.
>
> I had to put logic in those attachVolume methods to create the SR/VDI or
> datastore/VMDK.
>
> However, thinking back on it, it might have made more sense for the
> storage framework to detect if the storage in question was managed and -
> before calling attach - call create.
>
> If that logic was in place, I could have left attach/detachVolume alone
> and implemented create and delete in the hypervisor code to create my
> SR/VDI or datastore/VMDK.
>
> That makes sense to me because it is CloudStack-managed storage (so
> CloudStack is calling into the hypervisor to create and delete these types
> of objects...it's managing them).
>
>
> On Fri, Sep 27, 2013 at 5:00 PM, Marcus Sorensen wrote:
>
>> On Fri, Sep 27, 2013 at 4:22 PM, Mike Tutkowski
>>  wrote:
>> > Sure, sounds good - let me know when it's up on Review Board and I can
>> take
>> > a look.
>> >
>> > I made most of the changes you and I talked about:
>> >
>> >
>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1
>> >
>> > I have a new idea, though, that I think will simplify this:
>> >
>> > The main "weirdness" right now is when attachVolume is called that the
>> > original logic assumed createVolume had been called already.
>> >
>> > In my case, this doesn't apply, so we had to place extra logic in
>> > attachVolume to essentially "create" a volume. We decided to make a
>> connect
>> > method, which establishes the iSCSI connection and creates a
>> > KVMPhysicalDisk that can be returned when attachVolume calls
>> > getPhysicalDisk.
>> >
>> > The "normal" place where you'd create a KVMPhysicalDisk, however, would
>> be
>> > in the createVolume method. Since I don't currently "create" a volume,
>> my
>> > only chance to note the size of the volume is in the connect method.
>>
>> I don't think createVolume applies to plugins. My impression wash that
>> createAsync is called on the mgmt server side. If createVolume IS
>> being called, that's weird. The idea here is that mgmt server creates
>> the LUN, and then on the KVM side attach is called (or StartCommand if
>> it's a root volume and vm is being started), and it assumes that the
>> LUN is already there, so we call connectPhysicalDisk to attach it to
>> the KVM host.
>>
>> >
>> > It ends up being kind of weird to pass a size into the connect method,
>> as
>> > you've noted.
>> >
>> > What if we essentially left the attachVolume and detachVolume methods
>> alone
>> > (as in how they were before my changes)? We could have
>> > VolumeApiServiceImpl, before sending the AttachCommand, detect if the
>> > storage in question is managed. If it is, VolumeApiServiceImpl could
>> send a
>> > CreateObjectCommand. I would then implement createPhysicalDisk to
>> connect
>> > my iSCSI target and create a KVMPhysicalDisk.
>> >
>> > On the reverse side, VolumeApiServiceImpl, after sending the
>> DetachCommand,
>> > could detect if the storage in question is managed. If it is,
>> > VolumeApiServiceImpl could send a DeleteCommand. I would then implement
>> the
>> > deletePhysicalDisk method to disconnect my iSCSI session.
>> >
>> > What do you think?
>>
>> Maybe I'm just confused, but I thought the create and delete on the
>> KVM side only apply to the default storage plugin, which has to pass
>> everything on the agent. I thought the creation/deletion of LUNs
>> occured via createAsync and deleteAsync in your plugin.
>>
>> >
>> >
>> > On Fri, Sep 27, 2013 at 3:21 PM, Marcus Sorensen > >wrote:
>> >
>> >> Ok, I've got our plugin working against 4.2. Tested start vm, stop vm,
>> >> migrate vm, attach volume, detach volume.  Other functions that we
>> >> already had in our StorageAdaptor implementation, such as copying
>> >> templates to primary storage, just worked without any modification
>> >> from our 4.1 version.
>> >>
>> >> I'll post a patch to reviewboard with the applicable changes. I was
>> >> correct that attachVolume and dettachVolume only apply to
>> >> adding/removing disks from running VMs, so there were some more
>> >> changes to LibvirtComputingResource. I don't intend for this patch to
>> >> be applied (for one it's against 4.2), but I want you to take a look
>> >> and see if it will work for you as well. If it does, then it's a good
>> >> indicator that it should work for other plugins too, or if it needs to
>> >> be tweaked we can work it out.
>> >>
>> >> The gist is that we needed a connectPhysicalDisk call that can accept
>> >> the pool/volume info (which we've di

Re: add connect method on KVM storage code

[Mike Tutkowski]

Well, from what I saw with XenServer and VMware, that hypervisor logic's
attachVolume command also assumed a VDI/VMDK was created in advance.

I had to put logic in those attachVolume methods to create the SR/VDI or
datastore/VMDK.

However, thinking back on it, it might have made more sense for the storage
framework to detect if the storage in question was managed and - before
calling attach - call create.

If that logic was in place, I could have left attach/detachVolume alone and
implemented create and delete in the hypervisor code to create my SR/VDI or
datastore/VMDK.

That makes sense to me because it is CloudStack-managed storage (so
CloudStack is calling into the hypervisor to create and delete these types
of objects...it's managing them).


On Fri, Sep 27, 2013 at 5:00 PM, Marcus Sorensen wrote:

> On Fri, Sep 27, 2013 at 4:22 PM, Mike Tutkowski
>  wrote:
> > Sure, sounds good - let me know when it's up on Review Board and I can
> take
> > a look.
> >
> > I made most of the changes you and I talked about:
> >
> >
> https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1
> >
> > I have a new idea, though, that I think will simplify this:
> >
> > The main "weirdness" right now is when attachVolume is called that the
> > original logic assumed createVolume had been called already.
> >
> > In my case, this doesn't apply, so we had to place extra logic in
> > attachVolume to essentially "create" a volume. We decided to make a
> connect
> > method, which establishes the iSCSI connection and creates a
> > KVMPhysicalDisk that can be returned when attachVolume calls
> > getPhysicalDisk.
> >
> > The "normal" place where you'd create a KVMPhysicalDisk, however, would
> be
> > in the createVolume method. Since I don't currently "create" a volume, my
> > only chance to note the size of the volume is in the connect method.
>
> I don't think createVolume applies to plugins. My impression wash that
> createAsync is called on the mgmt server side. If createVolume IS
> being called, that's weird. The idea here is that mgmt server creates
> the LUN, and then on the KVM side attach is called (or StartCommand if
> it's a root volume and vm is being started), and it assumes that the
> LUN is already there, so we call connectPhysicalDisk to attach it to
> the KVM host.
>
> >
> > It ends up being kind of weird to pass a size into the connect method, as
> > you've noted.
> >
> > What if we essentially left the attachVolume and detachVolume methods
> alone
> > (as in how they were before my changes)? We could have
> > VolumeApiServiceImpl, before sending the AttachCommand, detect if the
> > storage in question is managed. If it is, VolumeApiServiceImpl could
> send a
> > CreateObjectCommand. I would then implement createPhysicalDisk to connect
> > my iSCSI target and create a KVMPhysicalDisk.
> >
> > On the reverse side, VolumeApiServiceImpl, after sending the
> DetachCommand,
> > could detect if the storage in question is managed. If it is,
> > VolumeApiServiceImpl could send a DeleteCommand. I would then implement
> the
> > deletePhysicalDisk method to disconnect my iSCSI session.
> >
> > What do you think?
>
> Maybe I'm just confused, but I thought the create and delete on the
> KVM side only apply to the default storage plugin, which has to pass
> everything on the agent. I thought the creation/deletion of LUNs
> occured via createAsync and deleteAsync in your plugin.
>
> >
> >
> > On Fri, Sep 27, 2013 at 3:21 PM, Marcus Sorensen  >wrote:
> >
> >> Ok, I've got our plugin working against 4.2. Tested start vm, stop vm,
> >> migrate vm, attach volume, detach volume.  Other functions that we
> >> already had in our StorageAdaptor implementation, such as copying
> >> templates to primary storage, just worked without any modification
> >> from our 4.1 version.
> >>
> >> I'll post a patch to reviewboard with the applicable changes. I was
> >> correct that attachVolume and dettachVolume only apply to
> >> adding/removing disks from running VMs, so there were some more
> >> changes to LibvirtComputingResource. I don't intend for this patch to
> >> be applied (for one it's against 4.2), but I want you to take a look
> >> and see if it will work for you as well. If it does, then it's a good
> >> indicator that it should work for other plugins too, or if it needs to
> >> be tweaked we can work it out.
> >>
> >> The gist is that we needed a connectPhysicalDisk call that can accept
> >> the pool/volume info (which we've discussed), but also a version of
> >> connectPhysicalDisk that can take a vm specification
> >> (VirtualMachineTO) and figure out which pools/disks are needed and
> >> attach them. I largely copied the code we had custom inserted into our
> >> 4.1 and put it into KVMStoragePoolManager so that it will be adaptor
> >> agnostic.
> >>
> >> Same goes for disconnectPhysicalDisk.
> >>
> >> We also needed to pass the VirtualMachineTO in a few other places like
> >> MigrateComm

Re: add connect method on KVM storage code

[Marcus Sorensen]

Yeah, I can verify that my createPhysicalDisk method is not being
called at all in 4.2. The create happens on the mgmt server in your
driver (createAsync), then you pass the lun info on to KVM in
attachVolume (or StartCommand for starting VMs), and it assumes the
LUN will be there on the SAN when it calls your connectPhysicalDisk
implementation to log into the target and do your stuff.

On Fri, Sep 27, 2013 at 5:00 PM, Marcus Sorensen  wrote:
> On Fri, Sep 27, 2013 at 4:22 PM, Mike Tutkowski
>  wrote:
>> Sure, sounds good - let me know when it's up on Review Board and I can take
>> a look.
>>
>> I made most of the changes you and I talked about:
>>
>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1
>>
>> I have a new idea, though, that I think will simplify this:
>>
>> The main "weirdness" right now is when attachVolume is called that the
>> original logic assumed createVolume had been called already.
>>
>> In my case, this doesn't apply, so we had to place extra logic in
>> attachVolume to essentially "create" a volume. We decided to make a connect
>> method, which establishes the iSCSI connection and creates a
>> KVMPhysicalDisk that can be returned when attachVolume calls
>> getPhysicalDisk.
>>
>> The "normal" place where you'd create a KVMPhysicalDisk, however, would be
>> in the createVolume method. Since I don't currently "create" a volume, my
>> only chance to note the size of the volume is in the connect method.
>
> I don't think createVolume applies to plugins. My impression wash that
> createAsync is called on the mgmt server side. If createVolume IS
> being called, that's weird. The idea here is that mgmt server creates
> the LUN, and then on the KVM side attach is called (or StartCommand if
> it's a root volume and vm is being started), and it assumes that the
> LUN is already there, so we call connectPhysicalDisk to attach it to
> the KVM host.
>
>>
>> It ends up being kind of weird to pass a size into the connect method, as
>> you've noted.
>>
>> What if we essentially left the attachVolume and detachVolume methods alone
>> (as in how they were before my changes)? We could have
>> VolumeApiServiceImpl, before sending the AttachCommand, detect if the
>> storage in question is managed. If it is, VolumeApiServiceImpl could send a
>> CreateObjectCommand. I would then implement createPhysicalDisk to connect
>> my iSCSI target and create a KVMPhysicalDisk.
>>
>> On the reverse side, VolumeApiServiceImpl, after sending the DetachCommand,
>> could detect if the storage in question is managed. If it is,
>> VolumeApiServiceImpl could send a DeleteCommand. I would then implement the
>> deletePhysicalDisk method to disconnect my iSCSI session.
>>
>> What do you think?
>
> Maybe I'm just confused, but I thought the create and delete on the
> KVM side only apply to the default storage plugin, which has to pass
> everything on the agent. I thought the creation/deletion of LUNs
> occured via createAsync and deleteAsync in your plugin.
>
>>
>>
>> On Fri, Sep 27, 2013 at 3:21 PM, Marcus Sorensen wrote:
>>
>>> Ok, I've got our plugin working against 4.2. Tested start vm, stop vm,
>>> migrate vm, attach volume, detach volume.  Other functions that we
>>> already had in our StorageAdaptor implementation, such as copying
>>> templates to primary storage, just worked without any modification
>>> from our 4.1 version.
>>>
>>> I'll post a patch to reviewboard with the applicable changes. I was
>>> correct that attachVolume and dettachVolume only apply to
>>> adding/removing disks from running VMs, so there were some more
>>> changes to LibvirtComputingResource. I don't intend for this patch to
>>> be applied (for one it's against 4.2), but I want you to take a look
>>> and see if it will work for you as well. If it does, then it's a good
>>> indicator that it should work for other plugins too, or if it needs to
>>> be tweaked we can work it out.
>>>
>>> The gist is that we needed a connectPhysicalDisk call that can accept
>>> the pool/volume info (which we've discussed), but also a version of
>>> connectPhysicalDisk that can take a vm specification
>>> (VirtualMachineTO) and figure out which pools/disks are needed and
>>> attach them. I largely copied the code we had custom inserted into our
>>> 4.1 and put it into KVMStoragePoolManager so that it will be adaptor
>>> agnostic.
>>>
>>> Same goes for disconnectPhysicalDisk.
>>>
>>> We also needed to pass the VirtualMachineTO in a few other places like
>>> MigrateCommand and StopCommand, it's otherwise hard to know which
>>> storage adaptors we need to deal with when all we have is a vm name or
>>> something like that.
>>>
>>> On Fri, Sep 27, 2013 at 12:56 AM, Mike Tutkowski
>>>  wrote:
>>> > Thanks for the clarification on how that works.
>>> >
>>> > Also, yeah, I think CHAP only grants you access to a volume. If multiple
>>> > hosts are using the CHAP credentials for a single volume, it's up to
>>> those
>>> > hos

Re: add connect method on KVM storage code

[Marcus Sorensen]

On Fri, Sep 27, 2013 at 4:22 PM, Mike Tutkowski
 wrote:
> Sure, sounds good - let me know when it's up on Review Board and I can take
> a look.
>
> I made most of the changes you and I talked about:
>
> https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1
>
> I have a new idea, though, that I think will simplify this:
>
> The main "weirdness" right now is when attachVolume is called that the
> original logic assumed createVolume had been called already.
>
> In my case, this doesn't apply, so we had to place extra logic in
> attachVolume to essentially "create" a volume. We decided to make a connect
> method, which establishes the iSCSI connection and creates a
> KVMPhysicalDisk that can be returned when attachVolume calls
> getPhysicalDisk.
>
> The "normal" place where you'd create a KVMPhysicalDisk, however, would be
> in the createVolume method. Since I don't currently "create" a volume, my
> only chance to note the size of the volume is in the connect method.

I don't think createVolume applies to plugins. My impression wash that
createAsync is called on the mgmt server side. If createVolume IS
being called, that's weird. The idea here is that mgmt server creates
the LUN, and then on the KVM side attach is called (or StartCommand if
it's a root volume and vm is being started), and it assumes that the
LUN is already there, so we call connectPhysicalDisk to attach it to
the KVM host.

>
> It ends up being kind of weird to pass a size into the connect method, as
> you've noted.
>
> What if we essentially left the attachVolume and detachVolume methods alone
> (as in how they were before my changes)? We could have
> VolumeApiServiceImpl, before sending the AttachCommand, detect if the
> storage in question is managed. If it is, VolumeApiServiceImpl could send a
> CreateObjectCommand. I would then implement createPhysicalDisk to connect
> my iSCSI target and create a KVMPhysicalDisk.
>
> On the reverse side, VolumeApiServiceImpl, after sending the DetachCommand,
> could detect if the storage in question is managed. If it is,
> VolumeApiServiceImpl could send a DeleteCommand. I would then implement the
> deletePhysicalDisk method to disconnect my iSCSI session.
>
> What do you think?

Maybe I'm just confused, but I thought the create and delete on the
KVM side only apply to the default storage plugin, which has to pass
everything on the agent. I thought the creation/deletion of LUNs
occured via createAsync and deleteAsync in your plugin.

>
>
> On Fri, Sep 27, 2013 at 3:21 PM, Marcus Sorensen wrote:
>
>> Ok, I've got our plugin working against 4.2. Tested start vm, stop vm,
>> migrate vm, attach volume, detach volume.  Other functions that we
>> already had in our StorageAdaptor implementation, such as copying
>> templates to primary storage, just worked without any modification
>> from our 4.1 version.
>>
>> I'll post a patch to reviewboard with the applicable changes. I was
>> correct that attachVolume and dettachVolume only apply to
>> adding/removing disks from running VMs, so there were some more
>> changes to LibvirtComputingResource. I don't intend for this patch to
>> be applied (for one it's against 4.2), but I want you to take a look
>> and see if it will work for you as well. If it does, then it's a good
>> indicator that it should work for other plugins too, or if it needs to
>> be tweaked we can work it out.
>>
>> The gist is that we needed a connectPhysicalDisk call that can accept
>> the pool/volume info (which we've discussed), but also a version of
>> connectPhysicalDisk that can take a vm specification
>> (VirtualMachineTO) and figure out which pools/disks are needed and
>> attach them. I largely copied the code we had custom inserted into our
>> 4.1 and put it into KVMStoragePoolManager so that it will be adaptor
>> agnostic.
>>
>> Same goes for disconnectPhysicalDisk.
>>
>> We also needed to pass the VirtualMachineTO in a few other places like
>> MigrateCommand and StopCommand, it's otherwise hard to know which
>> storage adaptors we need to deal with when all we have is a vm name or
>> something like that.
>>
>> On Fri, Sep 27, 2013 at 12:56 AM, Mike Tutkowski
>>  wrote:
>> > Thanks for the clarification on how that works.
>> >
>> > Also, yeah, I think CHAP only grants you access to a volume. If multiple
>> > hosts are using the CHAP credentials for a single volume, it's up to
>> those
>> > hosts to make sure they don't step on each other's toes (and this is - to
>> > my understanding - how it works with XenServer and VMware).
>> >
>> >
>> > On Fri, Sep 27, 2013 at 12:45 AM, Marcus Sorensen > >wrote:
>> >
>> >> On Fri, Sep 27, 2013 at 12:21 AM, Mike Tutkowski
>> >>  wrote:
>> >> > Maybe I should seek a little clarification as to how live migration
>> works
>> >> > in CS with KVM.
>> >> >
>> >> > Before we do a live migration of VM 1 from Host 1 to Host 2, do we
>> detach
>> >> > all disks from VM1?
>> >> >
>> >> > If so, then we're good to 

Re: add connect method on KVM storage code

[Mike Tutkowski]

URL has changed to this:

https://github.com/mike-tutkowski/incubator-cloudstack/commit/636cf78bcd9d32ae9f20c0ccd631fcf41b829d43

I've just been squashing my commits and sending them to GitHub as one
commit. It should make it easier to see what's changed when comparing
against what's fairly recent on master.


On Fri, Sep 27, 2013 at 4:22 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Sure, sounds good - let me know when it's up on Review Board and I can
> take a look.
>
> I made most of the changes you and I talked about:
>
>
> https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1
>
> I have a new idea, though, that I think will simplify this:
>
> The main "weirdness" right now is when attachVolume is called that the
> original logic assumed createVolume had been called already.
>
> In my case, this doesn't apply, so we had to place extra logic in
> attachVolume to essentially "create" a volume. We decided to make a connect
> method, which establishes the iSCSI connection and creates a
> KVMPhysicalDisk that can be returned when attachVolume calls
> getPhysicalDisk.
>
> The "normal" place where you'd create a KVMPhysicalDisk, however, would be
> in the createVolume method. Since I don't currently "create" a volume, my
> only chance to note the size of the volume is in the connect method.
>
> It ends up being kind of weird to pass a size into the connect method, as
> you've noted.
>
> What if we essentially left the attachVolume and detachVolume methods
> alone (as in how they were before my changes)? We could have
> VolumeApiServiceImpl, before sending the AttachCommand, detect if the
> storage in question is managed. If it is, VolumeApiServiceImpl could send a
> CreateObjectCommand. I would then implement createPhysicalDisk to connect
> my iSCSI target and create a KVMPhysicalDisk.
>
> On the reverse side, VolumeApiServiceImpl, after sending the
> DetachCommand, could detect if the storage in question is managed. If it
> is, VolumeApiServiceImpl could send a DeleteCommand. I would then implement
> the deletePhysicalDisk method to disconnect my iSCSI session.
>
> What do you think?
>
>
> On Fri, Sep 27, 2013 at 3:21 PM, Marcus Sorensen wrote:
>
>> Ok, I've got our plugin working against 4.2. Tested start vm, stop vm,
>> migrate vm, attach volume, detach volume.  Other functions that we
>> already had in our StorageAdaptor implementation, such as copying
>> templates to primary storage, just worked without any modification
>> from our 4.1 version.
>>
>> I'll post a patch to reviewboard with the applicable changes. I was
>> correct that attachVolume and dettachVolume only apply to
>> adding/removing disks from running VMs, so there were some more
>> changes to LibvirtComputingResource. I don't intend for this patch to
>> be applied (for one it's against 4.2), but I want you to take a look
>> and see if it will work for you as well. If it does, then it's a good
>> indicator that it should work for other plugins too, or if it needs to
>> be tweaked we can work it out.
>>
>> The gist is that we needed a connectPhysicalDisk call that can accept
>> the pool/volume info (which we've discussed), but also a version of
>> connectPhysicalDisk that can take a vm specification
>> (VirtualMachineTO) and figure out which pools/disks are needed and
>> attach them. I largely copied the code we had custom inserted into our
>> 4.1 and put it into KVMStoragePoolManager so that it will be adaptor
>> agnostic.
>>
>> Same goes for disconnectPhysicalDisk.
>>
>> We also needed to pass the VirtualMachineTO in a few other places like
>> MigrateCommand and StopCommand, it's otherwise hard to know which
>> storage adaptors we need to deal with when all we have is a vm name or
>> something like that.
>>
>> On Fri, Sep 27, 2013 at 12:56 AM, Mike Tutkowski
>>  wrote:
>> > Thanks for the clarification on how that works.
>> >
>> > Also, yeah, I think CHAP only grants you access to a volume. If multiple
>> > hosts are using the CHAP credentials for a single volume, it's up to
>> those
>> > hosts to make sure they don't step on each other's toes (and this is -
>> to
>> > my understanding - how it works with XenServer and VMware).
>> >
>> >
>> > On Fri, Sep 27, 2013 at 12:45 AM, Marcus Sorensen > >wrote:
>> >
>> >> On Fri, Sep 27, 2013 at 12:21 AM, Mike Tutkowski
>> >>  wrote:
>> >> > Maybe I should seek a little clarification as to how live migration
>> works
>> >> > in CS with KVM.
>> >> >
>> >> > Before we do a live migration of VM 1 from Host 1 to Host 2, do we
>> detach
>> >> > all disks from VM1?
>> >> >
>> >> > If so, then we're good to go there.
>> >> >
>> >> > I'm not as clear with HA.
>> >>
>> >> During live migration this is what we currently do in our modified
>> >> 4.1, I'm not sure if the new framework is set up for this, but it
>> >> should be made to do this if not.
>> >>
>> >> PrepareForMigrationCommand is called on destination host. In
>> >> PrepareForMigrationCo

Re: Scalable Backup and Recovery

[Chiradeep Vittal]

Ah I see. You mean a "scalable user experience".

The actual scalability of the snapshot process itself is limited by
available disk and network bandwidth.
Also certain hypervisors have various quirks which stand in the way of an
efficient solution.

On 9/27/13 10:27 AM, "SuichII, Christopher"  wrote:

>I'd like to start a discussion around the direction of scalable backup
>and recovery in CloudStack. Currently, the only want to backup and
>recover vms is by setting up a schedule or manually snapshotting up
>individual vm disks or manually snapshotting vms. Unfortunately, I don't
>believe this is a very scalable solution. What if a user wants all of
>their vm disks to be backed up on the same schedule? What if a domain
>administrator wants all of the vms in their domain to be backed up on the
>same schedule or to manually backup every vm in their domain?
>
>Here are some use cases I see for helping to scale things up:
>-Scheduled and manual backup of 1 to all of a user's vms and vm disks
>-Scheduled and manual backup of 1 to all of a domain's vms and vm disks
>(by a domain admin)
>-Scheduled and manual backup of 1 to all vms and vm disks on primary
>storage (by a cloud admin) - this one is tougher to find a valid use case
>for
>-Backup schedules attached to service offerings
>
>I know I previously started a discussion about backing up multiple vm
>disks at once, but I think these use cases, broken down by user type
>(user, domain admin and admin), should help clear things up and show the
>utility of being able to backup multiple objects at once.
>
>Thanks!
>Chris
>-- 
>Chris Suich
>chris.su...@netapp.com
>NetApp Software Engineer
>Data Center Platforms ­ Cloud Solutions
>Citrix, Cisco & Red Hat
>

Re: add connect method on KVM storage code

[Mike Tutkowski]

Sure, sounds good - let me know when it's up on Review Board and I can take
a look.

I made most of the changes you and I talked about:

https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1

I have a new idea, though, that I think will simplify this:

The main "weirdness" right now is when attachVolume is called that the
original logic assumed createVolume had been called already.

In my case, this doesn't apply, so we had to place extra logic in
attachVolume to essentially "create" a volume. We decided to make a connect
method, which establishes the iSCSI connection and creates a
KVMPhysicalDisk that can be returned when attachVolume calls
getPhysicalDisk.

The "normal" place where you'd create a KVMPhysicalDisk, however, would be
in the createVolume method. Since I don't currently "create" a volume, my
only chance to note the size of the volume is in the connect method.

It ends up being kind of weird to pass a size into the connect method, as
you've noted.

What if we essentially left the attachVolume and detachVolume methods alone
(as in how they were before my changes)? We could have
VolumeApiServiceImpl, before sending the AttachCommand, detect if the
storage in question is managed. If it is, VolumeApiServiceImpl could send a
CreateObjectCommand. I would then implement createPhysicalDisk to connect
my iSCSI target and create a KVMPhysicalDisk.

On the reverse side, VolumeApiServiceImpl, after sending the DetachCommand,
could detect if the storage in question is managed. If it is,
VolumeApiServiceImpl could send a DeleteCommand. I would then implement the
deletePhysicalDisk method to disconnect my iSCSI session.

What do you think?


On Fri, Sep 27, 2013 at 3:21 PM, Marcus Sorensen wrote:

> Ok, I've got our plugin working against 4.2. Tested start vm, stop vm,
> migrate vm, attach volume, detach volume.  Other functions that we
> already had in our StorageAdaptor implementation, such as copying
> templates to primary storage, just worked without any modification
> from our 4.1 version.
>
> I'll post a patch to reviewboard with the applicable changes. I was
> correct that attachVolume and dettachVolume only apply to
> adding/removing disks from running VMs, so there were some more
> changes to LibvirtComputingResource. I don't intend for this patch to
> be applied (for one it's against 4.2), but I want you to take a look
> and see if it will work for you as well. If it does, then it's a good
> indicator that it should work for other plugins too, or if it needs to
> be tweaked we can work it out.
>
> The gist is that we needed a connectPhysicalDisk call that can accept
> the pool/volume info (which we've discussed), but also a version of
> connectPhysicalDisk that can take a vm specification
> (VirtualMachineTO) and figure out which pools/disks are needed and
> attach them. I largely copied the code we had custom inserted into our
> 4.1 and put it into KVMStoragePoolManager so that it will be adaptor
> agnostic.
>
> Same goes for disconnectPhysicalDisk.
>
> We also needed to pass the VirtualMachineTO in a few other places like
> MigrateCommand and StopCommand, it's otherwise hard to know which
> storage adaptors we need to deal with when all we have is a vm name or
> something like that.
>
> On Fri, Sep 27, 2013 at 12:56 AM, Mike Tutkowski
>  wrote:
> > Thanks for the clarification on how that works.
> >
> > Also, yeah, I think CHAP only grants you access to a volume. If multiple
> > hosts are using the CHAP credentials for a single volume, it's up to
> those
> > hosts to make sure they don't step on each other's toes (and this is - to
> > my understanding - how it works with XenServer and VMware).
> >
> >
> > On Fri, Sep 27, 2013 at 12:45 AM, Marcus Sorensen  >wrote:
> >
> >> On Fri, Sep 27, 2013 at 12:21 AM, Mike Tutkowski
> >>  wrote:
> >> > Maybe I should seek a little clarification as to how live migration
> works
> >> > in CS with KVM.
> >> >
> >> > Before we do a live migration of VM 1 from Host 1 to Host 2, do we
> detach
> >> > all disks from VM1?
> >> >
> >> > If so, then we're good to go there.
> >> >
> >> > I'm not as clear with HA.
> >>
> >> During live migration this is what we currently do in our modified
> >> 4.1, I'm not sure if the new framework is set up for this, but it
> >> should be made to do this if not.
> >>
> >> PrepareForMigrationCommand is called on destination host. In
> >> PrepareForMigrationCommand we added a few lines to call
> >> connectPhysicalDisk. This host connects the SAN disks to this new
> >> host, then creates a paused VM.
> >>
> >> MigrateCommand is called on the source host. This sends the proper
> >> command to transfer VM memory, then atomically cuts over to the
> >> destination host. During this time, the disks are attached on both
> >> sides, but the VM is still the only thing that is using them, and it
> >> atomically cuts over. There's no caching on the host (qemu is using
> >> directio), so this is safe.

Re: bug? capacity not updated when adding memory to hypervisor

[Jeronimo Garcia]

Has this bug been raised?  I have this issue as well .. we are on 4.0.2 and
4.1.1


On Wed, Sep 25, 2013 at 4:43 PM, Prachi Damle wrote:

> Yes, the main bug here is not updating the Total Capacity - this is
> avoiding the total getting updated on host reconnects.
>
> Bug with the Used and Reserved Capacities gets masked since
> CapacityManager also listens to the VM state changes and updates these
> values.
>
> Prachi
> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Wednesday, September 25, 2013 1:45 PM
> To: dev@cloudstack.apache.org
> Subject: Re: bug? capacity not updated when adding memory to hypervisor
>
> The main bug being the lack of
> memCap.setTotalCapacity(host.getTotalMemory());
>
> Darren
>
> On Wed, Sep 25, 2013 at 1:13 PM, Daan Hoogland 
> wrote:
> > I agree Darren,
> >
> > if (cpuCap.getUsedCapacity() == usedCpu &&
> > cpuCap.getReservedCapacity() == reservedCpu) { } else {
> >if (cpuCap.getReservedCapacity() != reservedCpu) {
> > cpuCap.setReservedCapacity(reservedCpu);
> >}
> >if (cpuCap.getUsedCapacity() != usedCpu) {
> > cpuCap.setUsedCapacity(usedCpu);
> > }
> > }
> >
> > makes more sense. (same for mem)
> >
> >
> > On Wed, Sep 25, 2013 at 9:59 PM, Darren Shepherd <
> > darren.s.sheph...@gmail.com> wrote:
> >
> >> I added more memory to a hypervisor, and the host tables was updated
> >> properly, but the op_host_capacity is not updated.  I tracked down
> >> the code to CapacityManagerImpl.updateCapacityForHost.  The below
> >> logic seems completely wrong.  Unless I'm reading it wrong, it seems
> >> like it will only update EITHER used or reserved (not both) and never
> >> update total.  I removed log statements to make it more readable.
> >>
> >> if (cpuCap.getUsedCapacity() == usedCpu &&
> >> cpuCap.getReservedCapacity() == reservedCpu) {
> >> } else if (cpuCap.getReservedCapacity() != reservedCpu) {
> >> cpuCap.setReservedCapacity(reservedCpu);
> >> } else if (cpuCap.getUsedCapacity() != usedCpu) {
> >> cpuCap.setUsedCapacity(usedCpu);
> >> }
> >>
> >> if (memCap.getUsedCapacity() == usedMemory &&
> >> memCap.getReservedCapacity() == reservedMemory) {
> >> } else if (memCap.getReservedCapacity() != reservedMemory) {
> >> memCap.setReservedCapacity(reservedMemory);
> >> } else if (memCap.getUsedCapacity() != usedMemory) {
> >> memCap.setUsedCapacity(usedMemory);
> >> }
> >>
>

RE: Problem creating VirtualRouter on XenServer 6.2 due to problem with copy_vhd_from_secondarystorage.sh

[Soheil Eizadi]

I followed the steps here: 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/How+to+build+CloudStack

Under Install the SystemVM Template, I put the steps there as there was not 
enough detail for developers. I think it is OK but you can review. 

As I said I am not sure of the steps that led to this problem as I merged from 
4.2 to 4.3 and upgraded my systemVM image on secondary store, may be somewhere 
in that transition things broke.
-Soheil

From: Nitin Mehta [nitin.me...@citrix.com]
Sent: Friday, September 27, 2013 12:00 PM
To: dev@cloudstack.apache.org
Subject: Re: Problem creating VirtualRouter on XenServer 6.2 due to problem 
with copy_vhd_from_secondarystorage.sh

ListTemplateCommand() just reads the content present on secondary storage
through template.properties. How did you put the new template ? Was
template.properties and DB updated accordingly ?


On 27/09/13 11:30 AM, "Soheil Eizadi"  wrote:

>I traced this to a problem with ListTemplateCommand() that is returning
>what I assumed is a cached value
>10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-
>082b387b51c5.vhd, the new secondary storage has a new template now:
>template/tmpl/1/1/170c3425-6078-479e-8843-3f8fdc02aaf3.vhd.
>
>I am not sure at what stage this happened, and rather than debug this
>further, I just reinstalled it from scratch and fixed the issue.
>-Soheil
>
>From: Soheil Eizadi [seiz...@infoblox.com]
>Sent: Tuesday, September 24, 2013 6:10 PM
>To: dev@cloudstack.apache.org
>Subject: RE: Problem creating VirtualRouter on XenServer 6.2 due to
>problem with copy_vhd_from_secondarystorage.sh
>
>The error with copy_vhd_from_secondarystorage.sh is due to a problem with
>secondary storage, vhd_util returning a negative number to a query as the
>file does not exist. I am not sure why
>34b74283-916e-4562-82fd-082b387b51c5.vhd versus
>170c3425-6078-479e-8843-3f8fdc02aaf3.vhd is referenced, that is the next
>step to chase down.
>-Soheil
>
>[root@cloudstackxenserver1 bin]# ./vhd-util query -v -n
>10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-
>082b387b51c5.vhd
>error opening
>10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-
>082b387b51c5.vhd: -2
>
>root@nfs-server-10-48-15-21:/opt/storage/secondary# find
>.
>./template
>./template/tmpl
>./template/tmpl/1
>./template/tmpl/1/1
>./template/tmpl/1/1/170c3425-6078-479e-8843-3f8fdc02aaf3.vhd
>./template/tmpl/1/1/template.properties
>./template/tmpl/1/5
>./template/tmpl/1/5/0911ae22-350f-38f5-ad3e-423e7e5d6539.vhd
>./template/tmpl/1/5/template.properties
>./template/tmpl/2
>./volumes
>./snapshots
>
>
>From: Soheil Eizadi [seiz...@infoblox.com]
>Sent: Tuesday, September 24, 2013 5:11 PM
>To: dev@cloudstack.apache.org
>Subject: Problem creating VirtualRouter on XenServer 6.2 due to problem
>with copy_vhd_from_secondarystorage.sh
>
>Environment:
>- 4.3 master (synced with master 08/28/2013 08:43 AM)
>- XenServer 6.2
>
>I have a problem creating a Virtual Router on XenServer 6.2, here are the
>symptoms, vmops.log and SMlog don't tell you much, but when I run the
>copy command native I get an error like the
>copy_vhd_from_secondarystorage.sh shell is not compatible with XenServer
>6.2. I cleared the XenServer host tag to make sure the latest scripts
>were downloaded from my build. If someone thinks this problem is fixed on
>the top of the trunk, I can do another pull, otherwise I guess I need
>chase this some more in my enviroment?
>-Soheil
>
>[root@cloudstackxenserver1 log]#
>/opt/xensource/bin/copy_vhd_from_secondarystorage.sh
>10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-
>082b387b51c5.vhd 1cca28ee-886a-6d52-5461-662b33f1d8d8
>cloud-0230b187-29c2-4060-826c-7f697cceea35
>Syntax error: Unknown switch: -22MiB
>For usage run: 'xe help'
>9#can not create vdi in sr 1cca28ee-886a-6d52-5461-662b33f1d8d8
>
>vmops.log:
>
>INFO  [o.a.c.n.e.InfobloxElement] (Job-Executor-5:ctx-e6414bc5)
>InfobloxDeviceElement Host Record created for VM r-9-VM
>WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-382:null)
>destoryVDIbyNameLabel failed due to there are 0 VDIs with name
>cloud-0230b187-29c2-4060-826c-7f697cceea35
>WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-382:null) can
>not create vdi in sr 1cca28ee-886a-6d52-5461-662b33f1d8d8
>.
>
>SMlog:
>Sep 24 16:39:41 cloudstackxenserver1 SM: [28919]  VMOPS enter
>copy_vhd_from_secondarystorage 
>Sep 24 16:39:41 cloudstackxenserver1 SM: [28919] ['bash',
>'/opt/xensource/bin/copy_vhd_from_secondarystorage.sh',
>'10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd
>-082b387b51c5.vhd', '1cca28ee-886a-6d52-5461-662b33f1d8d8',
>'cloud-0230b187-29c2-4060-826c-7f697cceea35']
>Sep 24 16:39:41 cloudstackxenserver1 SM: [28919]   pread SUCCESS
>Sep 24 16:39:41 cloudstackxenserver1 SM: [28919]  VMO

Re: add connect method on KVM storage code

[Marcus Sorensen]

Ok, I've got our plugin working against 4.2. Tested start vm, stop vm,
migrate vm, attach volume, detach volume.  Other functions that we
already had in our StorageAdaptor implementation, such as copying
templates to primary storage, just worked without any modification
from our 4.1 version.

I'll post a patch to reviewboard with the applicable changes. I was
correct that attachVolume and dettachVolume only apply to
adding/removing disks from running VMs, so there were some more
changes to LibvirtComputingResource. I don't intend for this patch to
be applied (for one it's against 4.2), but I want you to take a look
and see if it will work for you as well. If it does, then it's a good
indicator that it should work for other plugins too, or if it needs to
be tweaked we can work it out.

The gist is that we needed a connectPhysicalDisk call that can accept
the pool/volume info (which we've discussed), but also a version of
connectPhysicalDisk that can take a vm specification
(VirtualMachineTO) and figure out which pools/disks are needed and
attach them. I largely copied the code we had custom inserted into our
4.1 and put it into KVMStoragePoolManager so that it will be adaptor
agnostic.

Same goes for disconnectPhysicalDisk.

We also needed to pass the VirtualMachineTO in a few other places like
MigrateCommand and StopCommand, it's otherwise hard to know which
storage adaptors we need to deal with when all we have is a vm name or
something like that.

On Fri, Sep 27, 2013 at 12:56 AM, Mike Tutkowski
 wrote:
> Thanks for the clarification on how that works.
>
> Also, yeah, I think CHAP only grants you access to a volume. If multiple
> hosts are using the CHAP credentials for a single volume, it's up to those
> hosts to make sure they don't step on each other's toes (and this is - to
> my understanding - how it works with XenServer and VMware).
>
>
> On Fri, Sep 27, 2013 at 12:45 AM, Marcus Sorensen wrote:
>
>> On Fri, Sep 27, 2013 at 12:21 AM, Mike Tutkowski
>>  wrote:
>> > Maybe I should seek a little clarification as to how live migration works
>> > in CS with KVM.
>> >
>> > Before we do a live migration of VM 1 from Host 1 to Host 2, do we detach
>> > all disks from VM1?
>> >
>> > If so, then we're good to go there.
>> >
>> > I'm not as clear with HA.
>>
>> During live migration this is what we currently do in our modified
>> 4.1, I'm not sure if the new framework is set up for this, but it
>> should be made to do this if not.
>>
>> PrepareForMigrationCommand is called on destination host. In
>> PrepareForMigrationCommand we added a few lines to call
>> connectPhysicalDisk. This host connects the SAN disks to this new
>> host, then creates a paused VM.
>>
>> MigrateCommand is called on the source host. This sends the proper
>> command to transfer VM memory, then atomically cuts over to the
>> destination host. During this time, the disks are attached on both
>> sides, but the VM is still the only thing that is using them, and it
>> atomically cuts over. There's no caching on the host (qemu is using
>> directio), so this is safe.
>>
>> After MigrateCommand completes it's VM passoff, we detach the disks
>> before returning.
>>
>> >
>> > If VM 1 goes down because Host 1 crashes, is the attach-volume command
>> > invoked as many times as need be (depending on how many volumes need to
>> be
>> > attached) when VM 1 is restarted on Host 2?
>>
>> From what I can tell, the attachVolume and dettachVolume seemed to
>> only be for attaching disks to existing, running VMs (i.e. inserting
>> new XML into an existing domain definition).  Normally when starting a
>> vm from scratch the vm definition, along with any currently attached
>> disks, is passed in to StartCommand (which would also be called during
>> HA restart of a VM). In our 4.1 branch we also have a call to
>> connectPhysicalDisk here, where we loop through the disk definitions
>> that were passed.
>>
>> Again, I should be able to flesh out the differences in 4.2 and how to
>> go about making this suitable for everyone in the coming days, so long
>> as you and anyone else writing plugins agree with the changes.
>>
>> These processes would make sure the disks are available on the hosts
>> they need to be, but they don't really provide locking or ensure that
>> only the necessary hosts can write to or see the disks at any given
>> time. I don't think CHAP does that either. We currently generate ACLs
>> via our SAN api during connectPhysicalDisk as a safety measure, but if
>> CloudStack is working properly it will be in charge of controlling
>> that the disks are only being used where they should be. The ACLs just
>> ensure that if the VM somehow gets started in two different places
>> (e.g. HA malfunction), only one of them will have access to the disks.
>>
>> >
>> >
>> > On Fri, Sep 27, 2013 at 12:17 AM, Mike Tutkowski <
>> > mike.tutkow...@solidfire.com> wrote:
>> >
>> >> Let me clarify this line a bit:
>> >>
>> >> "We get away without this with XenServer an

Cloudstack doesn't seem to recognize we added 100GB of memory to the nodes.

[Jeronimo Garcia]

Hi List.

We've done some memory upgrades to our hosts but even tho we can see the
new memory numbers and all that .. the cluster where this hosts belong to
still showing full (in the dashboard).

We've forced recalculation and still no luck.

Do you guys happen to know what to do with this?

Thanks

Re: Review Request 14058: Including tests for VPC VM Lifecycle on Tagged hosts

[Sheng Yang]

> On Sept. 26, 2013, 1:30 a.m., Sheng Yang wrote:
> > Please test it before submit.
> 
> Ashutosh Kelkar wrote:
> The tests have been tested and run correctly. There is a manual step 
> needed in setup for these tests - Adding host tags to the hosts which is why 
> the tests are skipped when committing.

No, looks like the test case added storage tag instead of host tag. Don't know 
how it could run.


- Sheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14058/#review26402
---


On Sept. 18, 2013, 2:28 p.m., Ashutosh Kelkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14058/
> ---
> 
> (Updated Sept. 18, 2013, 2:28 p.m.)
> 
> 
> Review request for cloudstack, Girish Shilamkar, venkata swamy babu  
> budumuru, and Sheng Yang.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> Added 10 tests for VPC VM liftcycle on tagged hosts
> 
> New class added  : TestVMLifeCycleDiffHosts
> 
> def test_01_deploy_instance_in_network(self):
> def test_02_stop_instance_in_network(self):
> def test_03_start_instance_in_network(self):
> def test_04_reboot_instance_in_network(self):
> def test_05_destroy_instance_in_network(self):
> def test_06_recover_instance_in_network(self):
> def test_07_migrate_instance_in_network(self):
> def test_08_user_data(self):
> def test_09_meta_data(self):
> def   test_10_expunge_instance_in_network(self):
> 
> This set of tests requires a multi host tagged setup (3 hosts - 2 hosts with 
> tag 'host1' and  1 host with tag 'host2')
> 
> 
> Diffs
> -
> 
>   test/integration/component/test_vpc_vm_life_cycle.py 9844c1f 
> 
> Diff: https://reviews.apache.org/r/14058/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ashutosh Kelkar
> 
>

Re: Review Request 14285: Add multi-row action support to UI list views (CLOUDSTACK-1579)

[Chris Suich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14285/
---

(Updated Sept. 27, 2013, 8:34 p.m.)


Review request for cloudstack.


Changes
---

Uploaded a .patch instead of a .diff so that it has the git commit header.


Repository: cloudstack-git


Description
---

See https://issues.apache.org/jira/browse/CLOUDSTACK-1579

I have added the logic and hooks for allowing list views to indicate they 
support multiselect rows and added hooks for list view actions to indicate they 
are a multiselect action. The data for the selected rows is stored and passed 
to callbacks in the same 'context' variable as with non-multiselect callbacks, 
so implementing multiselect should remain consistent.

This is purely a functional update and there is cosmetic work to be done 
shortly by someone other than myself (likely Brian Federle).

Some notes for those interested:
- $.remove() vs $.detach(): remove will delete the DOM element while detach 
simply removes it from view, allowing it to be added later and re-used
- List views can indicate they support multi-select with 'multiSelect: true'
- Actions can indicate they are a multi-select action with 
'isMultiSelectAction: true' and 'isHeader: true'
- This patch also includes a new list view flag called 'noActionCol' which 
intentional hides the 'Quickview' on rows which can be handy when the rows 
become too wide and look weird
- Currently, when one or more multi-select rows are selected, non-multi-select 
actions (such as Add) are hidden and replaced with the multi-select actions, 
however the idea was proposed to add another row of buttons below the 
non-multi-select buttons when this is the case - this is still open for 
discussion as there was no consensus


Diffs (updated)
-

  ui/plugins/netapp PRE-CREATION 
  ui/scripts/ui/widgets/dataTable.js 66d7095 
  ui/scripts/ui/widgets/listView.js 076b3ab 

Diff: https://reviews.apache.org/r/14285/diff/


Testing
---

Testing has been done in conjunction with a UI plugin I am developing. 
Additional testing will be performed as this is integrated into existing parts 
of the UI.


Thanks,

Chris Suich

Re: Review Request 14360: Make getName() match the constant names for Network Capability, and Service

[Alena Prokharchyk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14360/#review26447
---


Darren, changes in Network.java are going to break API compatibility as 
capabilities parameters are used in createNetworkOffering API call 
(serviceCapabilitystList param). CS UI and third party UI will get impacted as 
well.
If we have to change the name, its gotta be the capability parameter name, not 
its name value.

- Alena Prokharchyk


On Sept. 27, 2013, 5:43 p.m., Darren Shepherd wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14360/
> ---
> 
> (Updated Sept. 27, 2013, 5:43 p.m.)
> 
> 
> Review request for cloudstack, Alena Prokharchyk and Chiradeep Vittal.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> VPCVirtualRouter is still left as inconsitent as changing that name will be 
> impactful.  The unit test checked in will still ensure that future providers 
> match though
> 
> 
> Diffs
> -
> 
>   api/src/com/cloud/network/Network.java 49f380b 
>   api/test/com/cloud/network/NetworkTest.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/14360/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Darren Shepherd
> 
>

Advanced Networking Shared Network Deletion Fail due to VRouter

[Soheil Eizadi]

Here is problem I just encoutered on 4.3 Master:

1. Created a Shared Advanced network
2. Created an instance
3. Removed the instance
4. After the GUI showed the Instance was removed I tried to delete the network
The network deletion fails, due to VRouter Instance being present:
>> WARN  [c.c.n.NetworkManagerImpl] (Job-Executor-2:ctx-820db89a) Unable to 
>> complete destroy of the network due to element: VirtualRouter

Really the problem is that VRouter is already deleted on XenServer but looks 
like it is still in the MS database! I cleaned up the database and removed the 
VRouter instance information from the vm_instance table, and was able to 
successfully remove the network.

Is this a known issue?
-Soheil

[root@xenserver1-cloudstack ~]# xe vm-list
uuid ( RO)   : 58a9612b-7a73-8a14-4b29-58f043acec7c
 name-label ( RW): s-1-VM
power-state ( RO): running


uuid ( RO)   : 9e5503d1-16a5-40ab-a0dc-788bad583c0d
 name-label ( RW): Control domain on host: xenserver1-cloudstack
power-state ( RO): running


uuid ( RO)   : 91950b00-584b-a6b8-e3f9-27a75f071e6d
 name-label ( RW): v-2-VM
power-state ( RO): running

The exception is from the call to destroy VM that calls advanceExpunge() and 
fails when it finally called the Hypervisor:

s_logger.debug("Cleaning up NICS");

>>List nicExpungeCommands = hvGuru.finalizeExpungeNics(vm, 
>> profile.getNics());


Here is what the database looks like for the instances:


mysql> select * from vm_instance;

+++--+---+---++-+-++++-+--+--+-+--++---+--+-+-+-++++---+-+--+-+--+--+--+---+---+---+--+---+--++--+-+--++

| id | name   | uuid | instance_name | state
 | vm_template_id | guest_os_id | private_mac_address | private_ip_address | 
pod_id | data_center_id | host_id | last_host_id | proxy_id | proxy_assign_time 
  | vnc_password | ha_enabled | limit_cpu_use | update_count | update_time  
   | created | removed | type   | 
vm_type| account_id | domain_id | service_offering_id | 
reservation_id   | hypervisor_type | disk_offering_id | cpu 
 | ram  | owner | speed | host_name | display_name | desired_state | 
dynamically_scalable | display_vm | power_state  | power_state_update_time | 
power_state_update_count | power_host |

+++--+---+---++-+-++++-+--+--+-+--++---+--+-+-+-++++---+-+--+-+--+--+--+---+---+---+--+---+--++--+-+--++

|  1 | s-1-VM | 4cd7eee7-b420-4fce-8aec-90fae16ff3a0 | s-1-VM| Running  
 |  1 | 183 | 06:f2:48:00:00:0c   | 10.48.15.41|
  1 |  1 |   1 |1 | NULL | NULL
| 3e0934c4827e7b71 |  0 | 0 |3 | 2013-09-27 
05:27:32 | 2013-09-27 05:24:44 | NULL| SecondaryStorageVm | 
SecondaryStorageVm |  1 | 1 |   9 | 
002fd3d7-a347-4b21-8ee5-0c4418b8cce6 | XenServer   | NULL | 
NULL | NULL | NULL  |  NULL | NULL  | NULL | NULL  |
0 |  1 | PowerUnknown | NULL|   
 0 |   NULL |

|  2 | v-2-VM | d89219d1-e741-4242-ac20-215dd3d325e3 | v-2-VM| Running  
 |  1 | 183 | 06:55:34:00:00:05   | 10.48.15.34|
  1 |  1 |   1 |1 | NULL | NULL
| 735aa2bf84c2b6e  |  0 | 0 |3 | 2013-09-27 
05:27:32 | 2013-09-27 05:24:44 | NULL| ConsoleProxy   | 
ConsoleProxy   |  1 | 1 |  11 | 
7fce9eb6-37e3-4005-992d-2b5f6697455d | XenServer   | NULL | 
NULL | NU

Re: Libvirt-java 0.5.0 has been released

[Mike Tutkowski]

Looks like listNetworks() is called a couple times in
cloudstack-agent-upgrade:

if __name__ == '__main__':
   netlib = networkConfig()
   bridges = netlib.listNetworks()
   bridges = filter(isOldStyleBridge, bridges)
   for br in bridges:
   enslavedDev = netlib.getEnslavedDev(br, 1)
   if enslavedDev is not None:
   upgradeBridgeName(br, enslavedDev)

   bridges = netlib.listNetworks()
   bridges = filter(isOldStyleBridge, bridges)
   if len(bridges) > 0:
   print("Warning: upgrade is not finished, still some bridges have the
old style name:" + str(bridges))
   else:
   print("Upgrade succeed")


On Fri, Sep 27, 2013 at 1:43 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Kind of made progress here. :) Looks like we have another problem. Is it
> possible another item is missing from the scripts?
>
> mtutkowski@ubuntu:~$ sudo apt-get install cloudstack-agent
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> The following NEW packages will be installed:
>   cloudstack-agent
> 0 upgraded, 1 newly installed, 0 to remove and 468 not upgraded.
> Need to get 39.1 MB of archives.
> After this operation, 43.6 MB of additional disk space will be used.
> WARNING: The following packages cannot be authenticated!
>   cloudstack-agent
> Install these packages without verification [y/N]? y
> Get:1 http://localhost/cloudstack/repo/ binary/ cloudstack-agent 4.3.0
> [39.1 MB]
> Fetched 39.1 MB in 0s (50.5 MB/s)
> Selecting previously unselected package cloudstack-agent.
> (Reading database ... 168800 files and directories currently installed.)
> Unpacking cloudstack-agent (from .../cloudstack-agent_4.3.0_all.deb) ...
> Processing triggers for ureadahead ...
> Setting up cloudstack-agent (4.3.0) ...
> Traceback (most recent call last):
>   File "/usr/bin/cloudstack-agent-upgrade", line 51, in 
> bridges = netlib.listNetworks()
> AttributeError: networkConfig instance has no attribute 'listNetworks'
> dpkg: error processing cloudstack-agent (--configure):
>  subprocess installed post-installation script returned error exit status 1
> Errors were encountered while processing:
>  cloudstack-agent
> E: Sub-process /usr/bin/dpkg returned an error code (1)
>
>
> On Fri, Sep 27, 2013 at 8:26 AM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Great - thanks, Wei!
>>
>>
>> On Fri, Sep 27, 2013 at 2:34 AM, Wei ZHOU  wrote:
>>
>>> Mike,
>>> cloudstack-agent-upgrade is missing in cloudstack-agent installation.
>>> I added it in commit a6bfd9602129d8ae308ba58f36623c04826e15ca
>>>
>>> [root@weizhou-centos master(4.2)]# git show
>>> a6bfd9602129d8ae308ba58f36623c04826e15ca
>>> commit a6bfd9602129d8ae308ba58f36623c04826e15ca
>>> Author: Wei Zhou 
>>> Date:   Fri Sep 27 10:31:54 2013 +0200
>>> add missing cloudstack-agent-upgrade in cloudstack-agent debian
>>> packages
>>> diff --git a/debian/cloudstack-agent.install
>>> b/debian/cloudstack-agent.install
>>> index a3cc869..d708514 100644
>>> --- a/debian/cloudstack-agent.install
>>> +++ b/debian/cloudstack-agent.install
>>> @@ -21,6 +21,7 @@
>>>  /etc/init.d/cloudstack-agent
>>>  /usr/bin/cloudstack-setup-agent
>>>  /usr/bin/cloudstack-ssh
>>> +/usr/bin/cloudstack-agent-upgrade
>>>  /var/log/cloudstack/agent
>>>  /usr/share/cloudstack-agent/lib/*
>>>  /usr/share/cloudstack-agent/plugins
>>>
>>>
>>> 2013/9/27 Mike Tutkowski 
>>>
>>> > Thanks for investigating this!
>>> >
>>> > Talk to you soon!
>>> >
>>> >
>>> > On Fri, Sep 27, 2013 at 12:54 AM, Wei ZHOU 
>>> wrote:
>>> >
>>> > > good night
>>> > >
>>> > >
>>> > > 2013/9/27 Mike Tutkowski 
>>> > >
>>> > > > Sounds good
>>> > > >
>>> > > > Might have to get back to you tomorrow, though. I have to get up
>>> early.
>>> > > :)
>>> > > >
>>> > > >
>>> > > > On Fri, Sep 27, 2013 at 12:43 AM, Wei ZHOU 
>>> > > wrote:
>>> > > >
>>> > > > > ok. Thanks for your reply!
>>> > > > > The last question, could you try to download the
>>> cloudstack-agent and
>>> > > > > cloudstack-common deb packages, and "dpkg -i" to install it?
>>> > > > >
>>> > > > > I will test it on my local machine.
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > 2013/9/27 Mike Tutkowski 
>>> > > > >
>>> > > > > > Before re-installing the DEBs I run the following:
>>> > > > > >
>>> > > > > > #sudo apt-get remove --purge cloudstack-agent
>>> > > > > >
>>> > > > > > #sudo apt-get clean
>>> > > > > > Would that be sufficient with regards to what you were asking?
>>> > > > > >
>>> > > > > >
>>> > > > > > On Fri, Sep 27, 2013 at 12:36 AM, Wei ZHOU <
>>> ustcweiz...@gmail.com>
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > > What if you apt-get remove and apt-get install again?
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > 2013/9/27 Mike Tutkowski 
>>> > > > > > >
>>> > > > > > > > Yeah, I had cleaned, rebuilt the codebase, regenerated the
>>> > DEBs,
>>> > > > then
>>> > > > > > > > apt-get update and apt-get install cloudstack-agent.
>>> > > > > > > >
>>> > > > > > > > I can try it

Re: Libvirt-java 0.5.0 has been released

[Mike Tutkowski]

Kind of made progress here. :) Looks like we have another problem. Is it
possible another item is missing from the scripts?

mtutkowski@ubuntu:~$ sudo apt-get install cloudstack-agent
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  cloudstack-agent
0 upgraded, 1 newly installed, 0 to remove and 468 not upgraded.
Need to get 39.1 MB of archives.
After this operation, 43.6 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  cloudstack-agent
Install these packages without verification [y/N]? y
Get:1 http://localhost/cloudstack/repo/ binary/ cloudstack-agent 4.3.0
[39.1 MB]
Fetched 39.1 MB in 0s (50.5 MB/s)
Selecting previously unselected package cloudstack-agent.
(Reading database ... 168800 files and directories currently installed.)
Unpacking cloudstack-agent (from .../cloudstack-agent_4.3.0_all.deb) ...
Processing triggers for ureadahead ...
Setting up cloudstack-agent (4.3.0) ...
Traceback (most recent call last):
  File "/usr/bin/cloudstack-agent-upgrade", line 51, in 
bridges = netlib.listNetworks()
AttributeError: networkConfig instance has no attribute 'listNetworks'
dpkg: error processing cloudstack-agent (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 cloudstack-agent
E: Sub-process /usr/bin/dpkg returned an error code (1)


On Fri, Sep 27, 2013 at 8:26 AM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Great - thanks, Wei!
>
>
> On Fri, Sep 27, 2013 at 2:34 AM, Wei ZHOU  wrote:
>
>> Mike,
>> cloudstack-agent-upgrade is missing in cloudstack-agent installation.
>> I added it in commit a6bfd9602129d8ae308ba58f36623c04826e15ca
>>
>> [root@weizhou-centos master(4.2)]# git show
>> a6bfd9602129d8ae308ba58f36623c04826e15ca
>> commit a6bfd9602129d8ae308ba58f36623c04826e15ca
>> Author: Wei Zhou 
>> Date:   Fri Sep 27 10:31:54 2013 +0200
>> add missing cloudstack-agent-upgrade in cloudstack-agent debian
>> packages
>> diff --git a/debian/cloudstack-agent.install
>> b/debian/cloudstack-agent.install
>> index a3cc869..d708514 100644
>> --- a/debian/cloudstack-agent.install
>> +++ b/debian/cloudstack-agent.install
>> @@ -21,6 +21,7 @@
>>  /etc/init.d/cloudstack-agent
>>  /usr/bin/cloudstack-setup-agent
>>  /usr/bin/cloudstack-ssh
>> +/usr/bin/cloudstack-agent-upgrade
>>  /var/log/cloudstack/agent
>>  /usr/share/cloudstack-agent/lib/*
>>  /usr/share/cloudstack-agent/plugins
>>
>>
>> 2013/9/27 Mike Tutkowski 
>>
>> > Thanks for investigating this!
>> >
>> > Talk to you soon!
>> >
>> >
>> > On Fri, Sep 27, 2013 at 12:54 AM, Wei ZHOU 
>> wrote:
>> >
>> > > good night
>> > >
>> > >
>> > > 2013/9/27 Mike Tutkowski 
>> > >
>> > > > Sounds good
>> > > >
>> > > > Might have to get back to you tomorrow, though. I have to get up
>> early.
>> > > :)
>> > > >
>> > > >
>> > > > On Fri, Sep 27, 2013 at 12:43 AM, Wei ZHOU 
>> > > wrote:
>> > > >
>> > > > > ok. Thanks for your reply!
>> > > > > The last question, could you try to download the cloudstack-agent
>> and
>> > > > > cloudstack-common deb packages, and "dpkg -i" to install it?
>> > > > >
>> > > > > I will test it on my local machine.
>> > > > >
>> > > > >
>> > > > >
>> > > > > 2013/9/27 Mike Tutkowski 
>> > > > >
>> > > > > > Before re-installing the DEBs I run the following:
>> > > > > >
>> > > > > > #sudo apt-get remove --purge cloudstack-agent
>> > > > > >
>> > > > > > #sudo apt-get clean
>> > > > > > Would that be sufficient with regards to what you were asking?
>> > > > > >
>> > > > > >
>> > > > > > On Fri, Sep 27, 2013 at 12:36 AM, Wei ZHOU <
>> ustcweiz...@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > > What if you apt-get remove and apt-get install again?
>> > > > > > >
>> > > > > > >
>> > > > > > > 2013/9/27 Mike Tutkowski 
>> > > > > > >
>> > > > > > > > Yeah, I had cleaned, rebuilt the codebase, regenerated the
>> > DEBs,
>> > > > then
>> > > > > > > > apt-get update and apt-get install cloudstack-agent.
>> > > > > > > >
>> > > > > > > > I can try it again and see what happens. I thought I tried
>> the
>> > > > > process
>> > > > > > > > twice and got the same results.
>> > > > > > > >
>> > > > > > > > I did a search for cloudstack-agent-upgrade on my file
>> system
>> > and
>> > > > > only
>> > > > > > > > found references in the source directory.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Sep 27, 2013 at 12:30 AM, Wei ZHOU <
>> > > ustcweiz...@gmail.com>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > It is correct.
>> > > > > > > > > Have you re-created the cloudstack-agent deb files and
>> > uploaded
>> > > > to
>> > > > > > your
>> > > > > > > > > local apt repository?
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > 2013/9/27 Mike Tutkowski 
>> > > > > > > > >
>> > > > > > > > > > Here you go:
>> > > > > > > > > >
>> > > > > > > > > > mtutkowski@ubuntu:~/cloudstack$ grep

Re: Review Request 12228: static resource compression

[Laszlo Hornyak]

> On Sept. 27, 2013, 6:40 p.m., Brian Federle wrote:
> > For the next release I'm planning to refactor the stylesheets to use the 
> > SASS compiler. Will this effect integration of a SASS compiler in any way?
> > 
> > FYI The one I'm considering is https://github.com/Jasig/sass-maven-plugin. 
> > This hasn't been integrated yet and I am looking for someone with mvn build 
> > experience to work on it, though you are welcome to give this a try as well 
> > if you want :)

Hi Brian,

I think it can work together, I will give it a try and send you a prototype.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12228/#review26435
---


On Sept. 24, 2013, 6:57 a.m., Laszlo Hornyak wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12228/
> ---
> 
> (Updated Sept. 24, 2013, 6:57 a.m.)
> 
> 
> Review request for cloudstack, Brian Federle and Prasanna Santhanam.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> CloudStack at first use downloads some 3.5 MB of css and javascript to the 
> client. With a weak internet connection, this might take a long time. With 
> gzip compression content can be compressed to 850 KB.
> 
> This version of the patch uses a custom plugin to compress static resources, 
> so that no dynamic compression is needed at runtime. When the static resource 
> servlet notices that there is gzipped version of the resource and the client 
> accepts gzipped content, then it is going to send the gziped version, while 
> still respects http caching.
> 
> 
> Diffs
> -
> 
>   client/WEB-INF/web.xml e5c05d3 
>   client/pom.xml 119c96e 
>   server/src/com/cloud/servlet/StaticResourceServlet.java PRE-CREATION 
>   server/test/com/cloud/servlet/StaticResourceServletTest.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/12228/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Hornyak
> 
>

Re: Problem creating VirtualRouter on XenServer 6.2 due to problem with copy_vhd_from_secondarystorage.sh

[Nitin Mehta]

ListTemplateCommand() just reads the content present on secondary storage
through template.properties. How did you put the new template ? Was
template.properties and DB updated accordingly ?


On 27/09/13 11:30 AM, "Soheil Eizadi"  wrote:

>I traced this to a problem with ListTemplateCommand() that is returning
>what I assumed is a cached value
>10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-
>082b387b51c5.vhd, the new secondary storage has a new template now:
>template/tmpl/1/1/170c3425-6078-479e-8843-3f8fdc02aaf3.vhd.
>
>I am not sure at what stage this happened, and rather than debug this
>further, I just reinstalled it from scratch and fixed the issue.
>-Soheil
>
>From: Soheil Eizadi [seiz...@infoblox.com]
>Sent: Tuesday, September 24, 2013 6:10 PM
>To: dev@cloudstack.apache.org
>Subject: RE: Problem creating VirtualRouter on XenServer 6.2 due to
>problem with copy_vhd_from_secondarystorage.sh
>
>The error with copy_vhd_from_secondarystorage.sh is due to a problem with
>secondary storage, vhd_util returning a negative number to a query as the
>file does not exist. I am not sure why
>34b74283-916e-4562-82fd-082b387b51c5.vhd versus
>170c3425-6078-479e-8843-3f8fdc02aaf3.vhd is referenced, that is the next
>step to chase down.
>-Soheil
>
>[root@cloudstackxenserver1 bin]# ./vhd-util query -v -n
>10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-
>082b387b51c5.vhd
>error opening 
>10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-
>082b387b51c5.vhd: -2
>
>root@nfs-server-10-48-15-21:/opt/storage/secondary# find
>.
>./template
>./template/tmpl
>./template/tmpl/1
>./template/tmpl/1/1
>./template/tmpl/1/1/170c3425-6078-479e-8843-3f8fdc02aaf3.vhd
>./template/tmpl/1/1/template.properties
>./template/tmpl/1/5
>./template/tmpl/1/5/0911ae22-350f-38f5-ad3e-423e7e5d6539.vhd
>./template/tmpl/1/5/template.properties
>./template/tmpl/2
>./volumes
>./snapshots
>
>
>From: Soheil Eizadi [seiz...@infoblox.com]
>Sent: Tuesday, September 24, 2013 5:11 PM
>To: dev@cloudstack.apache.org
>Subject: Problem creating VirtualRouter on XenServer 6.2 due to problem
>with copy_vhd_from_secondarystorage.sh
>
>Environment:
>- 4.3 master (synced with master 08/28/2013 08:43 AM)
>- XenServer 6.2
>
>I have a problem creating a Virtual Router on XenServer 6.2, here are the
>symptoms, vmops.log and SMlog don't tell you much, but when I run the
>copy command native I get an error like the
>copy_vhd_from_secondarystorage.sh shell is not compatible with XenServer
>6.2. I cleared the XenServer host tag to make sure the latest scripts
>were downloaded from my build. If someone thinks this problem is fixed on
>the top of the trunk, I can do another pull, otherwise I guess I need
>chase this some more in my enviroment?
>-Soheil
>
>[root@cloudstackxenserver1 log]#
>/opt/xensource/bin/copy_vhd_from_secondarystorage.sh
>10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-
>082b387b51c5.vhd 1cca28ee-886a-6d52-5461-662b33f1d8d8
>cloud-0230b187-29c2-4060-826c-7f697cceea35
>Syntax error: Unknown switch: -22MiB
>For usage run: 'xe help'
>9#can not create vdi in sr 1cca28ee-886a-6d52-5461-662b33f1d8d8
>
>vmops.log:
>
>INFO  [o.a.c.n.e.InfobloxElement] (Job-Executor-5:ctx-e6414bc5)
>InfobloxDeviceElement Host Record created for VM r-9-VM
>WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-382:null)
>destoryVDIbyNameLabel failed due to there are 0 VDIs with name
>cloud-0230b187-29c2-4060-826c-7f697cceea35
>WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-382:null) can
>not create vdi in sr 1cca28ee-886a-6d52-5461-662b33f1d8d8
>.
>
>SMlog:
>Sep 24 16:39:41 cloudstackxenserver1 SM: [28919]  VMOPS enter
>copy_vhd_from_secondarystorage 
>Sep 24 16:39:41 cloudstackxenserver1 SM: [28919] ['bash',
>'/opt/xensource/bin/copy_vhd_from_secondarystorage.sh',
>'10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd
>-082b387b51c5.vhd', '1cca28ee-886a-6d52-5461-662b33f1d8d8',
>'cloud-0230b187-29c2-4060-826c-7f697cceea35']
>Sep 24 16:39:41 cloudstackxenserver1 SM: [28919]   pread SUCCESS
>Sep 24 16:39:41 cloudstackxenserver1 SM: [28919]  VMOPS exit
>copy_vhd_from_secondarystorage 
>Sep 24 16:39:42 cloudstackxenserver1 SM: [28955]  VMOPS enter
>kill_copy_process 
>Sep 24 16:39:42 cloudstackxenserver1 SM: [28955] ['bash',
>'/opt/xensource/bin/kill_copy_process.sh',
>'34b74283-916e-4562-82fd-082b387b51c5.vhd']
>Sep 24 16:39:42 cloudstackxenserver1 SM: [28955]   pread SUCCESS
>Sep 24 16:39:42 cloudstackxenserver1 SM: [28955]  VMOPS exit
>kill_copy_process 

Re: Review Request 13770: CLOUDSTACK-3363 UI > fix list Events/Alerts screen is in processing state forever with Delete Events/Alerts request having no filter

[ASF Subversion and Git Services]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13770/#review26442
---


Commit 38b94e934c38556ed3974ab16a35d5be33f4edc4 in branch refs/heads/master 
from brito wang
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=38b94e9 ]

CLOUDSTACK-3363 UI > fix list Events/Alerts screen is in processing state 
forever with Delete Events/Alerts request having no filter


- ASF Subversion and Git Services


On Aug. 28, 2013, 7:46 a.m., bo wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13770/
> ---
> 
> (Updated Aug. 28, 2013, 7:46 a.m.)
> 
> 
> Review request for cloudstack, Brian Federle and bharat kumar.
> 
> 
> Bugs: CLOUDSTACK-3363
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> 1.The loading screen won't disappear when delete the events or alerts with 
> incorrect parameters ,Because there is no handling for exception in the js 
> code.
> 2.Error messages are not completely correct, it should be "Either type or 
> enddate must be specified".
> 
> 
> Diffs
> -
> 
>   ui/scripts/events.js 46d45c0 
> 
> Diff: https://reviews.apache.org/r/13770/diff/
> 
> 
> Testing
> ---
> 
> 1.The loading screen will disappear when close the error message dialog.
> 2.The error message will be "Either type or enddate must be specified".
> 
> 
> Thanks,
> 
> bo wang
> 
>

Re: Review Request 13770: CLOUDSTACK-3363 UI > fix list Events/Alerts screen is in processing state forever with Delete Events/Alerts request having no filter

[Brian Federle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13770/#review26441
---

Ship it!


Ship It!

- Brian Federle


On Aug. 28, 2013, 7:46 a.m., bo wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13770/
> ---
> 
> (Updated Aug. 28, 2013, 7:46 a.m.)
> 
> 
> Review request for cloudstack, Brian Federle and bharat kumar.
> 
> 
> Bugs: CLOUDSTACK-3363
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> 1.The loading screen won't disappear when delete the events or alerts with 
> incorrect parameters ,Because there is no handling for exception in the js 
> code.
> 2.Error messages are not completely correct, it should be "Either type or 
> enddate must be specified".
> 
> 
> Diffs
> -
> 
>   ui/scripts/events.js 46d45c0 
> 
> Diff: https://reviews.apache.org/r/13770/diff/
> 
> 
> Testing
> ---
> 
> 1.The loading screen will disappear when close the error message dialog.
> 2.The error message will be "Either type or enddate must be specified".
> 
> 
> Thanks,
> 
> bo wang
> 
>

Re: Review Request 12228: static resource compression

[Brian Federle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12228/#review26435
---


For the next release I'm planning to refactor the stylesheets to use the SASS 
compiler. Will this effect integration of a SASS compiler in any way?

FYI The one I'm considering is https://github.com/Jasig/sass-maven-plugin. This 
hasn't been integrated yet and I am looking for someone with mvn build 
experience to work on it, though you are welcome to give this a try as well if 
you want :)

- Brian Federle


On Sept. 24, 2013, 6:57 a.m., Laszlo Hornyak wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12228/
> ---
> 
> (Updated Sept. 24, 2013, 6:57 a.m.)
> 
> 
> Review request for cloudstack, Brian Federle and Prasanna Santhanam.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> CloudStack at first use downloads some 3.5 MB of css and javascript to the 
> client. With a weak internet connection, this might take a long time. With 
> gzip compression content can be compressed to 850 KB.
> 
> This version of the patch uses a custom plugin to compress static resources, 
> so that no dynamic compression is needed at runtime. When the static resource 
> servlet notices that there is gzipped version of the resource and the client 
> accepts gzipped content, then it is going to send the gziped version, while 
> still respects http caching.
> 
> 
> Diffs
> -
> 
>   client/WEB-INF/web.xml e5c05d3 
>   client/pom.xml 119c96e 
>   server/src/com/cloud/servlet/StaticResourceServlet.java PRE-CREATION 
>   server/test/com/cloud/servlet/StaticResourceServletTest.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/12228/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Hornyak
> 
>

Re: [DISCUSS] UI: New look and feel

[Chip Childers]

On Fri, Sep 27, 2013 at 06:31:10PM +, Brian Federle wrote:
> Hey Chip, the BG is basically a layered background with a gradient and 
> overlay pattern on top. I can send along the PNGs I used for those and a 
> snippet of CSS, and you can apply them to the site.
> 
> I'm also using Noto Sans for the font (ASF 2.0 licensed), if you want to use 
> that as well.
> 
> What I can do is either attach the bitmap assets I'm using (BG images, logos, 
> font, etc.) to the wiki page, so anyone can grab them there for use in other 
> materials, or else they are all in the ui-restyle git branch, under 
> /ui/stylesheets/csui/
> 
> -Brian

Yep - I actually found it all.

I wrapped up the redesign (without the above stuff), so that we can have
it ready for 4.2.0's announcement on Tuesday.  I'll (or anyone else can
feel free) probably look at more refinements to align with what you're
up to in the project's UI at a later point.

FWIW: http://cloudstack.staging.apache.org/index.html

-chip

RE: [DISCUSS] UI: New look and feel

[Edison Su]

Is there anybody like Win8 style UI: especially for the icons: 
http://aozora.github.io/bootmetro/docs/docs-advanced-components.html

> -Original Message-
> From: Brian Federle [mailto:brian.fede...@citrix.com]
> Sent: Friday, September 27, 2013 10:53 AM
> To: dev@cloudstack.apache.org; Sonny Chhen
> Subject: RE: [DISCUSS] UI: New look and feel
> 
> Right now the plan was to remove the icons, though if people think that they
> are important to usability then we can definitely put them back in. I'm
> thinking flat icons though, which would look better with the new design. I'll
> play around with it and maybe post a screenshot with icons included.
> 
> The action icons on the detail pages will still be there, and of course if 
> plugins
> supply their own icons they will be displayed.
> 
> -Brian
> 
> 
> From: SuichII, Christopher [chris.su...@netapp.com]
> Sent: Friday, September 27, 2013 5:14 AM
> To: 
> Subject: Re: [DISCUSS] UI: New look and feel
> 
> Brian - The new style looks great, but I'd like to repeat someone else's
> question: Are we getting rid of the icons on the nav bar? As a plugin dev, it
> would be really nice to keep our company logo by our UI plugin.
> 
> Shiva & Sebastien - What impact would this angular.js project have on UI
> plugins?
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms - Cloud Solutions
> Citrix, Cisco & Red Hat
> 
> On Sep 27, 2013, at 2:44 AM, sebgoa  wrote:
> 
> >
> > On Sep 27, 2013, at 6:52 AM, Shiva Teja  wrote:
> >
> >> On Fri, Sep 27, 2013 at 9:28 AM, Ian Duffy  wrote:
> >>
> >>> I think so
> >>> implementation of AngularJS like the way Shiva did it for his GSoC
> >>> project would be good.
> >>>
> >>
> >> I'm trying to setup a demo for my project. This should give an idea
> >> about the code.
> >>
> >> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static
> >> /js/common/resources/virtualmachines.js
> >> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static
> >> /js/app/instances/instances.js
> >> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static
> >> /js/app/instances/instances.tpl.html
> >>
> >> Thanks,
> >> Shiva Teja
> >
> > Thanks Shiva, I was going to mention it.
> >
> > Shiva has worked on an angular.js app for a cloudstack frontend.
> > All the code has been contributed in tools/ngui
> >
> > This could easily be used with Brian new "CSS" and it would clean up all the
> javascript.
> >
> > -Sebastien
> >

RE: Problem creating VirtualRouter on XenServer 6.2 due to problem with copy_vhd_from_secondarystorage.sh

[Soheil Eizadi]

I traced this to a problem with ListTemplateCommand() that is returning what I 
assumed is a cached value 
10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-082b387b51c5.vhd,
 the new secondary storage has a new template now: 
template/tmpl/1/1/170c3425-6078-479e-8843-3f8fdc02aaf3.vhd.

I am not sure at what stage this happened, and rather than debug this further, 
I just reinstalled it from scratch and fixed the issue.
-Soheil

From: Soheil Eizadi [seiz...@infoblox.com]
Sent: Tuesday, September 24, 2013 6:10 PM
To: dev@cloudstack.apache.org
Subject: RE: Problem creating VirtualRouter on XenServer 6.2 due to problem 
with copy_vhd_from_secondarystorage.sh

The error with copy_vhd_from_secondarystorage.sh is due to a problem with 
secondary storage, vhd_util returning a negative number to a query as the file 
does not exist. I am not sure why 34b74283-916e-4562-82fd-082b387b51c5.vhd 
versus 170c3425-6078-479e-8843-3f8fdc02aaf3.vhd is referenced, that is the next 
step to chase down.
-Soheil

[root@cloudstackxenserver1 bin]# ./vhd-util query -v -n 
10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-082b387b51c5.vhd
error opening 
10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-082b387b51c5.vhd:
 -2

root@nfs-server-10-48-15-21:/opt/storage/secondary# find
.
./template
./template/tmpl
./template/tmpl/1
./template/tmpl/1/1
./template/tmpl/1/1/170c3425-6078-479e-8843-3f8fdc02aaf3.vhd
./template/tmpl/1/1/template.properties
./template/tmpl/1/5
./template/tmpl/1/5/0911ae22-350f-38f5-ad3e-423e7e5d6539.vhd
./template/tmpl/1/5/template.properties
./template/tmpl/2
./volumes
./snapshots


From: Soheil Eizadi [seiz...@infoblox.com]
Sent: Tuesday, September 24, 2013 5:11 PM
To: dev@cloudstack.apache.org
Subject: Problem creating VirtualRouter on XenServer 6.2 due to problem with 
copy_vhd_from_secondarystorage.sh

Environment:
- 4.3 master (synced with master 08/28/2013 08:43 AM)
- XenServer 6.2

I have a problem creating a Virtual Router on XenServer 6.2, here are the 
symptoms, vmops.log and SMlog don't tell you much, but when I run the copy 
command native I get an error like the copy_vhd_from_secondarystorage.sh shell 
is not compatible with XenServer 6.2. I cleared the XenServer host tag to make 
sure the latest scripts were downloaded from my build. If someone thinks this 
problem is fixed on the top of the trunk, I can do another pull, otherwise I 
guess I need chase this some more in my enviroment?
-Soheil

[root@cloudstackxenserver1 log]# 
/opt/xensource/bin/copy_vhd_from_secondarystorage.sh 
10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-082b387b51c5.vhd
 1cca28ee-886a-6d52-5461-662b33f1d8d8 cloud-0230b187-29c2-4060-826c-7f697cceea35
Syntax error: Unknown switch: -22MiB
For usage run: 'xe help'
9#can not create vdi in sr 1cca28ee-886a-6d52-5461-662b33f1d8d8

vmops.log:

INFO  [o.a.c.n.e.InfobloxElement] (Job-Executor-5:ctx-e6414bc5) 
InfobloxDeviceElement Host Record created for VM r-9-VM
WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-382:null) 
destoryVDIbyNameLabel failed due to there are 0 VDIs with name 
cloud-0230b187-29c2-4060-826c-7f697cceea35
WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-382:null) can not 
create vdi in sr 1cca28ee-886a-6d52-5461-662b33f1d8d8
.

SMlog:
Sep 24 16:39:41 cloudstackxenserver1 SM: [28919]  VMOPS enter  
copy_vhd_from_secondarystorage 
Sep 24 16:39:41 cloudstackxenserver1 SM: [28919] ['bash', 
'/opt/xensource/bin/copy_vhd_from_secondarystorage.sh', 
'10.48.15.22:/home/nfs/secondary/template/tmpl/1/1/34b74283-916e-4562-82fd-082b387b51c5.vhd',
 '1cca28ee-886a-6d52-5461-662b33f1d8d8', 
'cloud-0230b187-29c2-4060-826c-7f697cceea35']
Sep 24 16:39:41 cloudstackxenserver1 SM: [28919]   pread SUCCESS
Sep 24 16:39:41 cloudstackxenserver1 SM: [28919]  VMOPS exit  
copy_vhd_from_secondarystorage 
Sep 24 16:39:42 cloudstackxenserver1 SM: [28955]  VMOPS enter  
kill_copy_process 
Sep 24 16:39:42 cloudstackxenserver1 SM: [28955] ['bash', 
'/opt/xensource/bin/kill_copy_process.sh', 
'34b74283-916e-4562-82fd-082b387b51c5.vhd']
Sep 24 16:39:42 cloudstackxenserver1 SM: [28955]   pread SUCCESS
Sep 24 16:39:42 cloudstackxenserver1 SM: [28955]  VMOPS exit  
kill_copy_process 

RE: [DISCUSS] UI: New look and feel

[Brian Federle]

Hey Chip, the BG is basically a layered background with a gradient and overlay 
pattern on top. I can send along the PNGs I used for those and a snippet of 
CSS, and you can apply them to the site.

I'm also using Noto Sans for the font (ASF 2.0 licensed), if you want to use 
that as well.

What I can do is either attach the bitmap assets I'm using (BG images, logos, 
font, etc.) to the wiki page, so anyone can grab them there for use in other 
materials, or else they are all in the ui-restyle git branch, under 
/ui/stylesheets/csui/

-Brian

From: Chip Childers [chip.child...@sungard.com]
Sent: Thursday, September 26, 2013 6:09 PM
To: dev@cloudstack.apache.org
Cc: Sonny Chhen; Animesh Chaturvedi; Jessica Wang
Subject: Re: [DISCUSS] UI: New look and feel

Looks great Brian!

One request actually.  I'm working on a proposed cloudstack.apache.org
site redesign right now, and I'd actually love to get the relevant CSS
you used for the header area (faded blue).

It might be nice to have some relationship between the cs.a.o branding
and the product itself. ;-)

(watch the marketing list for my site redesign proposal shortly)


On Thu, Sep 26, 2013 at 10:11:07PM +, Brian Federle wrote:
> Hi all,
>
> In addition to the CSS code cleanup I'm working on, I am planning a 'reskin' 
> of the current UI to give a better user experience and look and feel. This 
> will utilize SASS and a grid system as I have discussed in the previous 
> thread.
>
> I created a task in JIRA and wiki functional spec, which has screenshots of 
> what I've done so far and what I plan to do:
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Update+UI+visual+appearance
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-4748
>
> You can also checkout the ui-restyle branch on git, if you are able to 
> manually compile the cloudstack.scss file via SASS (that will be automated in 
> the future). I can send over instructions on how to compile SASS manually; 
> it's pretty easy to setup.
>
> Let me know what you think so far :)  I'll of course up and post more 
> screenshots as I get more done. This reskin is only changing the CSS mainly, 
> with minimal changes to actual usage or JS code, so it is basically a drop-in 
> replacement for the current styling. I'm hoping to get this in by 4.3, so 
> please give me feedback on anything from the current UI you don't like or 
> want changed, and I can see about improving it.
>
> Thanks!
> Brian
>

RE: NetworkElement and other *ServiceProviders often have useless return values

[Alex Huang]

Shouldn't be a problem.  They were Boolean previously to mean this 
networkelement does not know how to apply the rule, move on to the next one 
because previously it was really more like a broadcast to all networkelements.  
Whichever networkelment believes they should apply the rule, then applies.  
Now, it is more prescriptive as the service providers are actually specified by 
the admin when they create the zone so the Boolean is no longer needed.  I 
doubt it's checked anywhere.

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Friday, September 27, 2013 11:07 AM
> To: dev@cloudstack.apache.org; Chiradeep Vittal; Alex Huang
> Subject: NetworkElement and other *ServiceProviders often have useless
> return values
> 
> On NetworkElement and many other *ServiceProvider interfaces (ie
> DhcpServiceProvider), the return value for the methods are a boolean, but
> those booleans are never checked.  I would like to clean up all these
> interfaces and put the majority of the methods to return "void."
>  Would there be any objection to this?  I find it quite confusing when
> implementing an interface when I have to look at calling code to determine if
> I should return false or throw an exception in some case.
> 
> Cleaning this up would mean I'd have to update all the implementations of
> the interfaces.
> 
> Darren

RE: [DISCUSS] UI: New look and feel

[Brian Federle]

I can definitely look into that...and it wouldn't require Photoshopping a large 
sprite sheet, so much easier for maintenance :)

-Brian

From: Tracy Phillips [tracp...@mantoso.com]
Sent: Friday, September 27, 2013 11:01 AM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] UI: New look and feel

If you could use font icons, that would be really nice..

(MIT License)
http://fortawesome.github.io/Font-Awesome/icons/

(Apache License)
http://getbootstrap.com/components/#glyphicons



On Fri, Sep 27, 2013 at 1:53 PM, Brian Federle wrote:

> Right now the plan was to remove the icons, though if people think that
> they are important to usability then we can definitely put them back in.
> I'm thinking flat icons though, which would look better with the new
> design. I'll play around with it and maybe post a screenshot with icons
> included.
>
> The action icons on the detail pages will still be there, and of course if
> plugins supply their own icons they will be displayed.
>
> -Brian
>
> 
> From: SuichII, Christopher [chris.su...@netapp.com]
> Sent: Friday, September 27, 2013 5:14 AM
> To: 
> Subject: Re: [DISCUSS] UI: New look and feel
>
> Brian - The new style looks great, but I'd like to repeat someone else's
> question: Are we getting rid of the icons on the nav bar? As a plugin dev,
> it would be really nice to keep our company logo by our UI plugin.
>
> Shiva & Sebastien - What impact would this angular.js project have on UI
> plugins?
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Sep 27, 2013, at 2:44 AM, sebgoa  wrote:
>
> >
> > On Sep 27, 2013, at 6:52 AM, Shiva Teja  wrote:
> >
> >> On Fri, Sep 27, 2013 at 9:28 AM, Ian Duffy  wrote:
> >>
> >>> I think so
> >>> implementation of AngularJS like the way Shiva did it for his GSoC
> >>> project would be good.
> >>>
> >>
> >> I'm trying to setup a demo for my project. This should give an idea
> about
> >> the code.
> >>
> >>
> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/common/resources/virtualmachines.js
> >>
> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.js
> >>
> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.tpl.html
> >>
> >> Thanks,
> >> Shiva Teja
> >
> > Thanks Shiva, I was going to mention it.
> >
> > Shiva has worked on an angular.js app for a cloudstack frontend.
> > All the code has been contributed in tools/ngui
> >
> > This could easily be used with Brian new "CSS" and it would clean up all
> the javascript.
> >
> > -Sebastien
> >
>
>

NetworkElement and other *ServiceProviders often have useless return values

[Darren Shepherd]

On NetworkElement and many other *ServiceProvider interfaces (ie
DhcpServiceProvider), the return value for the methods are a boolean,
but those booleans are never checked.  I would like to clean up all
these interfaces and put the majority of the methods to return "void."
 Would there be any objection to this?  I find it quite confusing when
implementing an interface when I have to look at calling code to
determine if I should return false or throw an exception in some case.

Cleaning this up would mean I'd have to update all the implementations
of the interfaces.

Darren

Re: [DISCUSS] UI: New look and feel

[Tracy Phillips]

If you could use font icons, that would be really nice..

(MIT License)
http://fortawesome.github.io/Font-Awesome/icons/

(Apache License)
http://getbootstrap.com/components/#glyphicons



On Fri, Sep 27, 2013 at 1:53 PM, Brian Federle wrote:

> Right now the plan was to remove the icons, though if people think that
> they are important to usability then we can definitely put them back in.
> I'm thinking flat icons though, which would look better with the new
> design. I'll play around with it and maybe post a screenshot with icons
> included.
>
> The action icons on the detail pages will still be there, and of course if
> plugins supply their own icons they will be displayed.
>
> -Brian
>
> 
> From: SuichII, Christopher [chris.su...@netapp.com]
> Sent: Friday, September 27, 2013 5:14 AM
> To: 
> Subject: Re: [DISCUSS] UI: New look and feel
>
> Brian - The new style looks great, but I'd like to repeat someone else's
> question: Are we getting rid of the icons on the nav bar? As a plugin dev,
> it would be really nice to keep our company logo by our UI plugin.
>
> Shiva & Sebastien - What impact would this angular.js project have on UI
> plugins?
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Sep 27, 2013, at 2:44 AM, sebgoa  wrote:
>
> >
> > On Sep 27, 2013, at 6:52 AM, Shiva Teja  wrote:
> >
> >> On Fri, Sep 27, 2013 at 9:28 AM, Ian Duffy  wrote:
> >>
> >>> I think so
> >>> implementation of AngularJS like the way Shiva did it for his GSoC
> >>> project would be good.
> >>>
> >>
> >> I'm trying to setup a demo for my project. This should give an idea
> about
> >> the code.
> >>
> >>
> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/common/resources/virtualmachines.js
> >>
> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.js
> >>
> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.tpl.html
> >>
> >> Thanks,
> >> Shiva Teja
> >
> > Thanks Shiva, I was going to mention it.
> >
> > Shiva has worked on an angular.js app for a cloudstack frontend.
> > All the code has been contributed in tools/ngui
> >
> > This could easily be used with Brian new "CSS" and it would clean up all
> the javascript.
> >
> > -Sebastien
> >
>
>

RE: [DISCUSS] UI: New look and feel

[Brian Federle]

Right now the plan was to remove the icons, though if people think that they 
are important to usability then we can definitely put them back in. I'm 
thinking flat icons though, which would look better with the new design. I'll 
play around with it and maybe post a screenshot with icons included.

The action icons on the detail pages will still be there, and of course if 
plugins supply their own icons they will be displayed.

-Brian


From: SuichII, Christopher [chris.su...@netapp.com]
Sent: Friday, September 27, 2013 5:14 AM
To: 
Subject: Re: [DISCUSS] UI: New look and feel

Brian - The new style looks great, but I'd like to repeat someone else's 
question: Are we getting rid of the icons on the nav bar? As a plugin dev, it 
would be really nice to keep our company logo by our UI plugin.

Shiva & Sebastien - What impact would this angular.js project have on UI 
plugins?
--
Chris Suich
chris.su...@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

On Sep 27, 2013, at 2:44 AM, sebgoa  wrote:

>
> On Sep 27, 2013, at 6:52 AM, Shiva Teja  wrote:
>
>> On Fri, Sep 27, 2013 at 9:28 AM, Ian Duffy  wrote:
>>
>>> I think so
>>> implementation of AngularJS like the way Shiva did it for his GSoC
>>> project would be good.
>>>
>>
>> I'm trying to setup a demo for my project. This should give an idea about
>> the code.
>>
>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/common/resources/virtualmachines.js
>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.js
>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.tpl.html
>>
>> Thanks,
>> Shiva Teja
>
> Thanks Shiva, I was going to mention it.
>
> Shiva has worked on an angular.js app for a cloudstack frontend.
> All the code has been contributed in tools/ngui
>
> This could easily be used with Brian new "CSS" and it would clean up all the 
> javascript.
>
> -Sebastien
>

RE: [DISCUSS] UI: New look and feel

[Brian Federle]

Yeah, I am definitely planning on JS cleanup, though not for 4.3. That will be 
a much more complex project, and needs to be a more iterative refactor since we 
have innumerable lines of complex business logic to take account for in there.

I've been learning about AngularJS as well, and it would definitely solve a lot 
of problems in the architecture. I wish a library like that existed 2 1/2 years 
ago when I started writing the UI :)

For JS cleanup, I'm thinking of it in this order right now:

1.) Add unit tests for existing widget functionality
2.) Add full documentation for UI 'API' (i.e., all options supported for 
widget). This is to cater to both plugin writers and CloudStack UI devs
3.) Start refactoring widget code (in /ui/scripts/ui/widgets). Ideally I would 
like to use AngularJS for this, but it depends on how well it will integrate 
with our existing jQuery-based interface. It seems flexible enough, though I'll 
need to get much more adept at the library than I am right now.
4.) Refactor implementation code in /ui/scripts. This wouldn't be migrated to 
AngluarJS nor be rewritten since there is simply too much code, but a good 
start would be reducing redundancy by centralizing common API calls into 
reusable libraries, etc.

-Brian


From: Ian Duffy [i...@ianduffy.ie]
Sent: Thursday, September 26, 2013 8:58 PM
To: CloudStack Dev
Cc: Sonny Chhen; Animesh Chaturvedi; Jessica Wang
Subject: Re: [DISCUSS] UI: New look and feel

Looks good.

Moving away from design and more into front-end functionality. Is
there any talks about cleaning up the javascript behind the UI? I
recall it being messy and hard to follow from what I recall of looking
at it. The idea single page app with no URL defining a view and a
broken browser back button is quite annoying. I think so
implementation of AngularJS like the way Shiva did it for his GSoC
project would be good.

On 27 September 2013 02:09, Chip Childers  wrote:
> Looks great Brian!
>
> One request actually.  I'm working on a proposed cloudstack.apache.org
> site redesign right now, and I'd actually love to get the relevant CSS
> you used for the header area (faded blue).
>
> It might be nice to have some relationship between the cs.a.o branding
> and the product itself. ;-)
>
> (watch the marketing list for my site redesign proposal shortly)
>
>
> On Thu, Sep 26, 2013 at 10:11:07PM +, Brian Federle wrote:
>> Hi all,
>>
>> In addition to the CSS code cleanup I'm working on, I am planning a 'reskin' 
>> of the current UI to give a better user experience and look and feel. This 
>> will utilize SASS and a grid system as I have discussed in the previous 
>> thread.
>>
>> I created a task in JIRA and wiki functional spec, which has screenshots of 
>> what I've done so far and what I plan to do:
>>
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Update+UI+visual+appearance
>>
>> https://issues.apache.org/jira/browse/CLOUDSTACK-4748
>>
>> You can also checkout the ui-restyle branch on git, if you are able to 
>> manually compile the cloudstack.scss file via SASS (that will be automated 
>> in the future). I can send over instructions on how to compile SASS 
>> manually; it's pretty easy to setup.
>>
>> Let me know what you think so far :)  I'll of course up and post more 
>> screenshots as I get more done. This reskin is only changing the CSS mainly, 
>> with minimal changes to actual usage or JS code, so it is basically a 
>> drop-in replacement for the current styling. I'm hoping to get this in by 
>> 4.3, so please give me feedback on anything from the current UI you don't 
>> like or want changed, and I can see about improving it.
>>
>> Thanks!
>> Brian
>>

Re: Network.{Provider,Service,Capability} string names don't match constant names

[Darren Shepherd]

I created https://reviews.apache.org/r/14360/ for this.  I found that
changing the provider names would be rather impactful, but changing
the Capability and Service didn't seem to matter.  I can't find any
reference to the string constant names anywhere.  The unit test I
added will ensure that future additions to Capability, Service, and
Provider match the constant names.

Darren

On Fri, Sep 27, 2013 at 10:22 AM, Darren Shepherd
 wrote:
> For Network.{Provider,Service,Capability} the getName() does not
> always match the constant names.  For example
> Capability.SupportedVpnProtocols, the getName() is
> "SupportedVpnTypes".  For Capability.SupportedLBAlgorithms the
> getName() is "SupportedLbAlgorithms".
>
> Can we just clean up all those constants, meaning make the string
> match what the constant name is?  Is that going to have some impact?
> I can submit a patch that has a unit test to ensure the static's
> getName always matches the static name.
>
> Darren

Review Request 14360: Make getName() match the constant names for Network Capability, and Service

[Darren Shepherd]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14360/
---

Review request for cloudstack, Alena Prokharchyk and Chiradeep Vittal.


Repository: cloudstack-git


Description
---

VPCVirtualRouter is still left as inconsitent as changing that name will be 
impactful.  The unit test checked in will still ensure that future providers 
match though


Diffs
-

  api/src/com/cloud/network/Network.java 49f380b 
  api/test/com/cloud/network/NetworkTest.java PRE-CREATION 

Diff: https://reviews.apache.org/r/14360/diff/


Testing
---


Thanks,

Darren Shepherd

Re: Scalable Backup and Recovery

[Kelcey Jamison Damage]

I agree this needs to be approached. I find it quite frustrating to have a VM 
with 4-5 volumes and not be able to backup all 5 volumes as a set. Plus the 
restore operation gets messy when you have to restore 5 volumes and re-attach 
them to the VM.

The user and domain sets you discuss below I can also clearly see the use for.

Thanks for starting this discussion.

-Kelcey

- Original Message -
From: "Christopher SuichII" 
To: "" 
Sent: Friday, September 27, 2013 10:27:29 AM
Subject: Scalable Backup and Recovery

I'd like to start a discussion around the direction of scalable backup and 
recovery in CloudStack. Currently, the only want to backup and recover vms is 
by setting up a schedule or manually snapshotting up individual vm disks or 
manually snapshotting vms. Unfortunately, I don't believe this is a very 
scalable solution. What if a user wants all of their vm disks to be backed up 
on the same schedule? What if a domain administrator wants all of the vms in 
their domain to be backed up on the same schedule or to manually backup every 
vm in their domain?

Here are some use cases I see for helping to scale things up:
-Scheduled and manual backup of 1 to all of a user's vms and vm disks
-Scheduled and manual backup of 1 to all of a domain's vms and vm disks (by a 
domain admin)
-Scheduled and manual backup of 1 to all vms and vm disks on primary storage 
(by a cloud admin) - this one is tougher to find a valid use case for
-Backup schedules attached to service offerings

I know I previously started a discussion about backing up multiple vm disks at 
once, but I think these use cases, broken down by user type (user, domain admin 
and admin), should help clear things up and show the utility of being able to 
backup multiple objects at once.

Thanks!
Chris
-- 
Chris Suich
chris.su...@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

Re: [DISCUSS] UI: New look and feel

[sebgoa]

On Sep 27, 2013, at 2:14 PM, "SuichII, Christopher"  
wrote:

> Brian - The new style looks great, but I'd like to repeat someone else's 
> question: Are we getting rid of the icons on the nav bar? As a plugin dev, it 
> would be really nice to keep our company logo by our UI plugin.
> 
> Shiva & Sebastien - What impact would this angular.js project have on UI 
> plugins?

honestly no idea, 

> -- 
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
> 
> On Sep 27, 2013, at 2:44 AM, sebgoa  wrote:
> 
>> 
>> On Sep 27, 2013, at 6:52 AM, Shiva Teja  wrote:
>> 
>>> On Fri, Sep 27, 2013 at 9:28 AM, Ian Duffy  wrote:
>>> 
 I think so
 implementation of AngularJS like the way Shiva did it for his GSoC
 project would be good.
 
>>> 
>>> I'm trying to setup a demo for my project. This should give an idea about
>>> the code.
>>> 
>>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/common/resources/virtualmachines.js
>>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.js
>>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.tpl.html
>>> 
>>> Thanks,
>>> Shiva Teja
>> 
>> Thanks Shiva, I was going to mention it. 
>> 
>> Shiva has worked on an angular.js app for a cloudstack frontend.
>> All the code has been contributed in tools/ngui
>> 
>> This could easily be used with Brian new "CSS" and it would clean up all the 
>> javascript.
>> 
>> -Sebastien
>> 
> 

RE: [DISCUSS] UI: New look and feel

[Edison Su]

That's nice! Do you need help to setup a demo, or coding? Just finished read < 
Mastering Web Application Development with AngularJS >, so trying to mastering 
something:)

> -Original Message-
> From: Shiva Teja [mailto:shivate...@gmail.com]
> Sent: Thursday, September 26, 2013 9:53 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] UI: New look and feel
> 
> On Fri, Sep 27, 2013 at 9:28 AM, Ian Duffy  wrote:
> 
> > I think so
> > implementation of AngularJS like the way Shiva did it for his GSoC
> > project would be good.
> >
> 
> I'm trying to setup a demo for my project. This should give an idea about the
> code.
> 
> https://github.com/shivateja/cloudstack-ui/blob/angular-
> rawapi/static/js/common/resources/virtualmachines.js
> https://github.com/shivateja/cloudstack-ui/blob/angular-
> rawapi/static/js/app/instances/instances.js
> https://github.com/shivateja/cloudstack-ui/blob/angular-
> rawapi/static/js/app/instances/instances.tpl.html
> 
> Thanks,
> Shiva Teja

Scalable Backup and Recovery

[SuichII, Christopher]

I'd like to start a discussion around the direction of scalable backup and 
recovery in CloudStack. Currently, the only want to backup and recover vms is 
by setting up a schedule or manually snapshotting up individual vm disks or 
manually snapshotting vms. Unfortunately, I don't believe this is a very 
scalable solution. What if a user wants all of their vm disks to be backed up 
on the same schedule? What if a domain administrator wants all of the vms in 
their domain to be backed up on the same schedule or to manually backup every 
vm in their domain?

Here are some use cases I see for helping to scale things up:
-Scheduled and manual backup of 1 to all of a user's vms and vm disks
-Scheduled and manual backup of 1 to all of a domain's vms and vm disks (by a 
domain admin)
-Scheduled and manual backup of 1 to all vms and vm disks on primary storage 
(by a cloud admin) - this one is tougher to find a valid use case for
-Backup schedules attached to service offerings

I know I previously started a discussion about backing up multiple vm disks at 
once, but I think these use cases, broken down by user type (user, domain admin 
and admin), should help clear things up and show the utility of being able to 
backup multiple objects at once.

Thanks!
Chris
-- 
Chris Suich
chris.su...@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

Re: [DISCUSS] UI: New look and feel

[Kelcey Jamison Damage]

We should make a CSS skinning bin/repo so people can easily share their skins. 
This could even just be a Wiki section. I have been playing with a dark(Greys) 
themed flatter skin that is designed for comfortable 1920 x 1080 usage. 

I think us having a fairly easily skin-able UI will attract a lot more people.

I always enjoy seeing these re-skin projects. I am most excited to setup and 
use Shiva's angular.js project with the re-skins.

-Kelcey

- Original Message -
From: "Christopher SuichII" 
To: "" 
Sent: Friday, September 27, 2013 5:14:10 AM
Subject: Re: [DISCUSS] UI: New look and feel

Brian - The new style looks great, but I'd like to repeat someone else's 
question: Are we getting rid of the icons on the nav bar? As a plugin dev, it 
would be really nice to keep our company logo by our UI plugin.

Shiva & Sebastien - What impact would this angular.js project have on UI 
plugins?
-- 
Chris Suich
chris.su...@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

On Sep 27, 2013, at 2:44 AM, sebgoa  wrote:

> 
> On Sep 27, 2013, at 6:52 AM, Shiva Teja  wrote:
> 
>> On Fri, Sep 27, 2013 at 9:28 AM, Ian Duffy  wrote:
>> 
>>> I think so
>>> implementation of AngularJS like the way Shiva did it for his GSoC
>>> project would be good.
>>> 
>> 
>> I'm trying to setup a demo for my project. This should give an idea about
>> the code.
>> 
>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/common/resources/virtualmachines.js
>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.js
>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.tpl.html
>> 
>> Thanks,
>> Shiva Teja
> 
> Thanks Shiva, I was going to mention it. 
> 
> Shiva has worked on an angular.js app for a cloudstack frontend.
> All the code has been contributed in tools/ngui
> 
> This could easily be used with Brian new "CSS" and it would clean up all the 
> javascript.
> 
> -Sebastien
> 

Network.{Provider,Service,Capability} string names don't match constant names

[Darren Shepherd]

For Network.{Provider,Service,Capability} the getName() does not
always match the constant names.  For example
Capability.SupportedVpnProtocols, the getName() is
"SupportedVpnTypes".  For Capability.SupportedLBAlgorithms the
getName() is "SupportedLbAlgorithms".

Can we just clean up all those constants, meaning make the string
match what the constant name is?  Is that going to have some impact?
I can submit a patch that has a unit test to ensure the static's
getName always matches the static name.

Darren

Re: LocalHostEndPoint seems to get called

[Min Chen]

Can you file a bug and attach your db dump and ms log with it?

Thanks
-min

On 9/27/13 3:14 AM, "Daan Hoogland"  wrote:

>Min, I started the MS with the hosts and the ssvm running and while it
>was registered as running in the db. It still keeps reporting the ssvm
>as not running:
>
>INFO  [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-1:ctx-e0e62e7a)
>Host: mccdxen2 connected with hypervisor type: XenServer. Checking
>CIDR...
>INFO  [c.c.a.m.DirectAgentAttache] (AgentTaskPool-1:ctx-e0e62e7a)
>StartupAnswer received 1 Interval = 60
>INFO  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-2:null) Host
>10.200.23.42 OpaqueRef:78780f36-0f5a-10d8-b037-2e9e26e1d29c: Host
>10.200.23.42 is already setup.
>INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-1:ctx-e0e62e7a)
>Received a host startup notification
>INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-1:ctx-e0e62e7a)
>Scheduled network rules cleanup, interval=2334
>INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-1:ctx-e0e62e7a)
>Received a host startup notification
>INFO  [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-2:ctx-a3209637)
>Host: mccdxen1 connected with hypervisor type: XenServer. Checking
>CIDR...
>INFO  [c.c.a.m.DirectAgentAttache] (AgentTaskPool-2:ctx-a3209637)
>StartupAnswer received 2 Interval = 60
>INFO  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-1:null) Host
>10.200.23.41 OpaqueRef:04b11bd1-ff1b-d2e2-f0f7-c70d7f8dc58e: Host
>10.200.23.41 is already setup.
>INFO  [o.a.c.s.d.p.DefaultHostListener] (AgentTaskPool-1:ctx-e0e62e7a)
>Connection established between
>org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@6fd511e2
>host + 1
>INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-2:ctx-a3209637)
>Received a host startup notification
>INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-2:ctx-a3209637)
>Scheduled network rules cleanup, interval=2080
>INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-2:ctx-a3209637)
>Received a host startup notification
>INFO  [o.a.c.s.d.p.DefaultHostListener] (AgentTaskPool-2:ctx-a3209637)
>Connection established between
>org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@479eb52a
>host + 2
>INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
>running ssvm is found, so command will be sent to LocalHostEndPoint
>INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
>running ssvm is found, so command will be sent to LocalHostEndPoint
>INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
>running ssvm is found, so command will be sent to LocalHostEndPoint
>INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
>running ssvm is found, so command will be sent to LocalHostEndPoint
>INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
>running ssvm is found, so command will be sent to LocalHostEndPoint
>INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
>running ssvm is found, so command will be sent to LocalHostEndPoint
>INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
>running ssvm is found, so command will be sent to LocalHostEndPoint
>INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
>running ssvm is found, so command will be sent to LocalHostEndPoint
>INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
>running ssvm is found, so command will be sent to LocalHostEndPoint
>INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
>running ssvm is found, so command will be sent to LocalHostEndPoint
>
>is this a bug or am i missing some architectural quirk?
>
>regards,
>Daan
>
>On Thu, Sep 26, 2013 at 6:36 PM, Daan Hoogland 
>wrote:
>> The state is 'RUNNING' not 'Up' . Maybe it is me out racing the
>> computer. so don't deepdive. I just thought it was weird I got this
>> message while the ssvm was up.
>>
>> once again, no hinder, Min. Thanks
>>
>> On Thu, Sep 26, 2013 at 6:27 PM, Min Chen  wrote:
>>> I am only aware that before ssvm is up, you will see such message in
>>> StatCollector, but that is no hurt. I don't understand why you
>>>encountered
>>> this even if your ssvm is up? Is your s-1-VM entry in host db table is
>>>in
>>> Up state? Are you using devcloud or real hypervisor in your setup? We
>>>may
>>> need detailed log to figure out what is going on here in your case.
>>>
>>> Thanks
>>> -min
>>>
>>> On 9/26/13 6:39 AM, "Daan Hoogland"  wrote:
>>>
I get "No running ssvm is found, so command will be sent to
LocalHostEndPoint" while I can see and log into a machine called
s-1-VM which has state 'RUNNING' in the database. Is there some
relation? I never though much of it as it does not seem to be the
first thing stopping me usually.

On Wed, Sep 25, 2013 at 11:20 PM, Edison Su 
wrote:
>
>
>> -Original Message-
>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>> Sent: Wednesday, September 25, 2013 12:38 PM
>> To: dev@cloudstack.apache.org

Re: ovirt/RHEV-M support

[Laszlo Hornyak]

Ok, so you want to keep your rhevm but you want to use
(start/stop/create/delete/etc) its resources (CS, storage, networks, vms)
in CS?


On Fri, Sep 27, 2013 at 6:13 PM, Jithin Raju  wrote:

> Hi,
>
> I meant managing ovirt managed resources using cs.
>
> Thanks,
> Jithin
>
>
>
> On Fri, Sep 27, 2013 at 8:51 PM, Laszlo Hornyak  >wrote:
>
> > Hi,
> >
> > Exactly what support do you mean? Migration from rhevm to CS? :-)
> >
> >
> > On Fri, Sep 27, 2013 at 3:14 PM, Jithin Raju  wrote:
> >
> > > Hi,
> > >
> > > Is there any plan to support ovirt/RHEV-M ?
> > >
> > > Thanks,
> > > Jithin
> > >
> >
> >
> >
> > --
> >
> > EOF
> >
>



-- 

EOF

Re: ovirt/RHEV-M support

[Jithin Raju]

Hi,

I meant managing ovirt managed resources using cs.

Thanks,
Jithin



On Fri, Sep 27, 2013 at 8:51 PM, Laszlo Hornyak wrote:

> Hi,
>
> Exactly what support do you mean? Migration from rhevm to CS? :-)
>
>
> On Fri, Sep 27, 2013 at 3:14 PM, Jithin Raju  wrote:
>
> > Hi,
> >
> > Is there any plan to support ovirt/RHEV-M ?
> >
> > Thanks,
> > Jithin
> >
>
>
>
> --
>
> EOF
>

Re: Review Request 13901: fix long cast to double

[Laszlo Hornyak]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13901/#review26432
---


ping?

- Laszlo Hornyak


On Aug. 29, 2013, 12:26 p.m., Laszlo Hornyak wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13901/
> ---
> 
> (Updated Aug. 29, 2013, 12:26 p.m.)
> 
> 
> Review request for cloudstack, Alena Prokharchyk and Wei Zhou.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> - the result of dividing long with long resulted in loss of precision both 
> for network and IO
> - unit tests included
> 
> 
> Diffs
> -
> 
>   
> plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
>  a156ae6 
>   
> plugins/hypervisors/kvm/test/com/cloud/hypervisor/kvm/resource/LibvirtComputingResourceTest.java
>  3640030 
> 
> Diff: https://reviews.apache.org/r/13901/diff/
> 
> 
> Testing
> ---
> 
> unit test included
> 
> 
> Thanks,
> 
> Laszlo Hornyak
> 
>

Re: ovirt/RHEV-M support

[Laszlo Hornyak]

Hi,

Exactly what support do you mean? Migration from rhevm to CS? :-)


On Fri, Sep 27, 2013 at 3:14 PM, Jithin Raju  wrote:

> Hi,
>
> Is there any plan to support ovirt/RHEV-M ?
>
> Thanks,
> Jithin
>



-- 

EOF

Re: ovirt/RHEV-M support

[David Nalley]

I don't think that anyone is currently working on it, but we'd love to
see it if you are interested in working on it.

--David

On Fri, Sep 27, 2013 at 6:14 AM, Jithin Raju  wrote:
> Hi,
>
> Is there any plan to support ovirt/RHEV-M ?
>
> Thanks,
> Jithin

Re: Libvirt-java 0.5.0 has been released

[Mike Tutkowski]

Great - thanks, Wei!


On Fri, Sep 27, 2013 at 2:34 AM, Wei ZHOU  wrote:

> Mike,
> cloudstack-agent-upgrade is missing in cloudstack-agent installation.
> I added it in commit a6bfd9602129d8ae308ba58f36623c04826e15ca
>
> [root@weizhou-centos master(4.2)]# git show
> a6bfd9602129d8ae308ba58f36623c04826e15ca
> commit a6bfd9602129d8ae308ba58f36623c04826e15ca
> Author: Wei Zhou 
> Date:   Fri Sep 27 10:31:54 2013 +0200
> add missing cloudstack-agent-upgrade in cloudstack-agent debian
> packages
> diff --git a/debian/cloudstack-agent.install
> b/debian/cloudstack-agent.install
> index a3cc869..d708514 100644
> --- a/debian/cloudstack-agent.install
> +++ b/debian/cloudstack-agent.install
> @@ -21,6 +21,7 @@
>  /etc/init.d/cloudstack-agent
>  /usr/bin/cloudstack-setup-agent
>  /usr/bin/cloudstack-ssh
> +/usr/bin/cloudstack-agent-upgrade
>  /var/log/cloudstack/agent
>  /usr/share/cloudstack-agent/lib/*
>  /usr/share/cloudstack-agent/plugins
>
>
> 2013/9/27 Mike Tutkowski 
>
> > Thanks for investigating this!
> >
> > Talk to you soon!
> >
> >
> > On Fri, Sep 27, 2013 at 12:54 AM, Wei ZHOU 
> wrote:
> >
> > > good night
> > >
> > >
> > > 2013/9/27 Mike Tutkowski 
> > >
> > > > Sounds good
> > > >
> > > > Might have to get back to you tomorrow, though. I have to get up
> early.
> > > :)
> > > >
> > > >
> > > > On Fri, Sep 27, 2013 at 12:43 AM, Wei ZHOU 
> > > wrote:
> > > >
> > > > > ok. Thanks for your reply!
> > > > > The last question, could you try to download the cloudstack-agent
> and
> > > > > cloudstack-common deb packages, and "dpkg -i" to install it?
> > > > >
> > > > > I will test it on my local machine.
> > > > >
> > > > >
> > > > >
> > > > > 2013/9/27 Mike Tutkowski 
> > > > >
> > > > > > Before re-installing the DEBs I run the following:
> > > > > >
> > > > > > #sudo apt-get remove --purge cloudstack-agent
> > > > > >
> > > > > > #sudo apt-get clean
> > > > > > Would that be sufficient with regards to what you were asking?
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 27, 2013 at 12:36 AM, Wei ZHOU <
> ustcweiz...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > What if you apt-get remove and apt-get install again?
> > > > > > >
> > > > > > >
> > > > > > > 2013/9/27 Mike Tutkowski 
> > > > > > >
> > > > > > > > Yeah, I had cleaned, rebuilt the codebase, regenerated the
> > DEBs,
> > > > then
> > > > > > > > apt-get update and apt-get install cloudstack-agent.
> > > > > > > >
> > > > > > > > I can try it again and see what happens. I thought I tried
> the
> > > > > process
> > > > > > > > twice and got the same results.
> > > > > > > >
> > > > > > > > I did a search for cloudstack-agent-upgrade on my file system
> > and
> > > > > only
> > > > > > > > found references in the source directory.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Sep 27, 2013 at 12:30 AM, Wei ZHOU <
> > > ustcweiz...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > It is correct.
> > > > > > > > > Have you re-created the cloudstack-agent deb files and
> > uploaded
> > > > to
> > > > > > your
> > > > > > > > > local apt repository?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2013/9/27 Mike Tutkowski 
> > > > > > > > >
> > > > > > > > > > Here you go:
> > > > > > > > > >
> > > > > > > > > > mtutkowski@ubuntu:~/cloudstack$ grep
> > > cloudstack-agent-upgrade
> > > > > > > > > debian/rules
> > > > > > > > > > install -D
> > agent/target/transformed/cloudstack-agent-upgrade
> > > > > > > > > > $(DESTDIR)/usr/bin/cloudstack-agent-upgrade
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 27, 2013 at 12:20 AM, Wei ZHOU <
> > > > > ustcweiz...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Did you build the latest source?
> > > > > > > > > > > Could you paste the result of the following command in
> > your
> > > > > > source
> > > > > > > > > > > directory?
> > > > > > > > > > > "grep cloudstack-agent-upgrade debian/rules"
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *Mike Tutkowski*
> > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > e: mike.tutkow...@solidfire.com
> > > > > > > > > > o: 303.746.7302
> > > > > > > > > > Advancing the way the world uses the
> > > > > > > > > > cloud >
> > > > > > > > > > *™*
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Mike Tutkowski*
> > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > e: mike.tutkow...@solidfire.com
> > > > > > > > o: 303.746.7302
> > > > > > > > Advancing the way the world uses the
> > > > > > > > cloud
> > > > > > > > *™*
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Mike Tutkowski*
> > > > > > *Senior CloudStack Developer,

ovirt/RHEV-M support

[Jithin Raju]

Hi,

Is there any plan to support ovirt/RHEV-M ?

Thanks,
Jithin

Re: [DISCUSS] UI: New look and feel

[SuichII, Christopher]

Brian - The new style looks great, but I'd like to repeat someone else's 
question: Are we getting rid of the icons on the nav bar? As a plugin dev, it 
would be really nice to keep our company logo by our UI plugin.

Shiva & Sebastien - What impact would this angular.js project have on UI 
plugins?
-- 
Chris Suich
chris.su...@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

On Sep 27, 2013, at 2:44 AM, sebgoa  wrote:

> 
> On Sep 27, 2013, at 6:52 AM, Shiva Teja  wrote:
> 
>> On Fri, Sep 27, 2013 at 9:28 AM, Ian Duffy  wrote:
>> 
>>> I think so
>>> implementation of AngularJS like the way Shiva did it for his GSoC
>>> project would be good.
>>> 
>> 
>> I'm trying to setup a demo for my project. This should give an idea about
>> the code.
>> 
>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/common/resources/virtualmachines.js
>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.js
>> https://github.com/shivateja/cloudstack-ui/blob/angular-rawapi/static/js/app/instances/instances.tpl.html
>> 
>> Thanks,
>> Shiva Teja
> 
> Thanks Shiva, I was going to mention it. 
> 
> Shiva has worked on an angular.js app for a cloudstack frontend.
> All the code has been contributed in tools/ngui
> 
> This could easily be used with Brian new "CSS" and it would clean up all the 
> javascript.
> 
> -Sebastien
> 

Re: LocalHostEndPoint seems to get called

[Daan Hoogland]

Min, I started the MS with the hosts and the ssvm running and while it
was registered as running in the db. It still keeps reporting the ssvm
as not running:

INFO  [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-1:ctx-e0e62e7a)
Host: mccdxen2 connected with hypervisor type: XenServer. Checking
CIDR...
INFO  [c.c.a.m.DirectAgentAttache] (AgentTaskPool-1:ctx-e0e62e7a)
StartupAnswer received 1 Interval = 60
INFO  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-2:null) Host
10.200.23.42 OpaqueRef:78780f36-0f5a-10d8-b037-2e9e26e1d29c: Host
10.200.23.42 is already setup.
INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-1:ctx-e0e62e7a)
Received a host startup notification
INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-1:ctx-e0e62e7a)
Scheduled network rules cleanup, interval=2334
INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-1:ctx-e0e62e7a)
Received a host startup notification
INFO  [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-2:ctx-a3209637)
Host: mccdxen1 connected with hypervisor type: XenServer. Checking
CIDR...
INFO  [c.c.a.m.DirectAgentAttache] (AgentTaskPool-2:ctx-a3209637)
StartupAnswer received 2 Interval = 60
INFO  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-1:null) Host
10.200.23.41 OpaqueRef:04b11bd1-ff1b-d2e2-f0f7-c70d7f8dc58e: Host
10.200.23.41 is already setup.
INFO  [o.a.c.s.d.p.DefaultHostListener] (AgentTaskPool-1:ctx-e0e62e7a)
Connection established between
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@6fd511e2
host + 1
INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-2:ctx-a3209637)
Received a host startup notification
INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-2:ctx-a3209637)
Scheduled network rules cleanup, interval=2080
INFO  [c.c.n.s.SecurityGroupListener] (AgentTaskPool-2:ctx-a3209637)
Received a host startup notification
INFO  [o.a.c.s.d.p.DefaultHostListener] (AgentTaskPool-2:ctx-a3209637)
Connection established between
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@479eb52a
host + 2
INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [o.a.c.s.e.DefaultEndPointSelector] (StatsCollector-1:null) No
running ssvm is found, so command will be sent to LocalHostEndPoint

is this a bug or am i missing some architectural quirk?

regards,
Daan

On Thu, Sep 26, 2013 at 6:36 PM, Daan Hoogland  wrote:
> The state is 'RUNNING' not 'Up' . Maybe it is me out racing the
> computer. so don't deepdive. I just thought it was weird I got this
> message while the ssvm was up.
>
> once again, no hinder, Min. Thanks
>
> On Thu, Sep 26, 2013 at 6:27 PM, Min Chen  wrote:
>> I am only aware that before ssvm is up, you will see such message in
>> StatCollector, but that is no hurt. I don't understand why you encountered
>> this even if your ssvm is up? Is your s-1-VM entry in host db table is in
>> Up state? Are you using devcloud or real hypervisor in your setup? We may
>> need detailed log to figure out what is going on here in your case.
>>
>> Thanks
>> -min
>>
>> On 9/26/13 6:39 AM, "Daan Hoogland"  wrote:
>>
>>>I get "No running ssvm is found, so command will be sent to
>>>LocalHostEndPoint" while I can see and log into a machine called
>>>s-1-VM which has state 'RUNNING' in the database. Is there some
>>>relation? I never though much of it as it does not seem to be the
>>>first thing stopping me usually.
>>>
>>>On Wed, Sep 25, 2013 at 11:20 PM, Edison Su  wrote:


> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Wednesday, September 25, 2013 12:38 PM
> To: dev@cloudstack.apache.org
> Subject: LocalHostEndPoint seems to get called
>
> While I'm doing development and restarting things and what not, it
>seems
> often storage commands get routed to LocalHostEndPoint.  This seems
>bad.  I
> do

Re: Libvirt-java 0.5.0 has been released

[Wei ZHOU]

Mike,
cloudstack-agent-upgrade is missing in cloudstack-agent installation.
I added it in commit a6bfd9602129d8ae308ba58f36623c04826e15ca

[root@weizhou-centos master(4.2)]# git show
a6bfd9602129d8ae308ba58f36623c04826e15ca
commit a6bfd9602129d8ae308ba58f36623c04826e15ca
Author: Wei Zhou 
Date:   Fri Sep 27 10:31:54 2013 +0200
add missing cloudstack-agent-upgrade in cloudstack-agent debian packages
diff --git a/debian/cloudstack-agent.install
b/debian/cloudstack-agent.install
index a3cc869..d708514 100644
--- a/debian/cloudstack-agent.install
+++ b/debian/cloudstack-agent.install
@@ -21,6 +21,7 @@
 /etc/init.d/cloudstack-agent
 /usr/bin/cloudstack-setup-agent
 /usr/bin/cloudstack-ssh
+/usr/bin/cloudstack-agent-upgrade
 /var/log/cloudstack/agent
 /usr/share/cloudstack-agent/lib/*
 /usr/share/cloudstack-agent/plugins


2013/9/27 Mike Tutkowski 

> Thanks for investigating this!
>
> Talk to you soon!
>
>
> On Fri, Sep 27, 2013 at 12:54 AM, Wei ZHOU  wrote:
>
> > good night
> >
> >
> > 2013/9/27 Mike Tutkowski 
> >
> > > Sounds good
> > >
> > > Might have to get back to you tomorrow, though. I have to get up early.
> > :)
> > >
> > >
> > > On Fri, Sep 27, 2013 at 12:43 AM, Wei ZHOU 
> > wrote:
> > >
> > > > ok. Thanks for your reply!
> > > > The last question, could you try to download the cloudstack-agent and
> > > > cloudstack-common deb packages, and "dpkg -i" to install it?
> > > >
> > > > I will test it on my local machine.
> > > >
> > > >
> > > >
> > > > 2013/9/27 Mike Tutkowski 
> > > >
> > > > > Before re-installing the DEBs I run the following:
> > > > >
> > > > > #sudo apt-get remove --purge cloudstack-agent
> > > > >
> > > > > #sudo apt-get clean
> > > > > Would that be sufficient with regards to what you were asking?
> > > > >
> > > > >
> > > > > On Fri, Sep 27, 2013 at 12:36 AM, Wei ZHOU 
> > > > wrote:
> > > > >
> > > > > > What if you apt-get remove and apt-get install again?
> > > > > >
> > > > > >
> > > > > > 2013/9/27 Mike Tutkowski 
> > > > > >
> > > > > > > Yeah, I had cleaned, rebuilt the codebase, regenerated the
> DEBs,
> > > then
> > > > > > > apt-get update and apt-get install cloudstack-agent.
> > > > > > >
> > > > > > > I can try it again and see what happens. I thought I tried the
> > > > process
> > > > > > > twice and got the same results.
> > > > > > >
> > > > > > > I did a search for cloudstack-agent-upgrade on my file system
> and
> > > > only
> > > > > > > found references in the source directory.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Sep 27, 2013 at 12:30 AM, Wei ZHOU <
> > ustcweiz...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > It is correct.
> > > > > > > > Have you re-created the cloudstack-agent deb files and
> uploaded
> > > to
> > > > > your
> > > > > > > > local apt repository?
> > > > > > > >
> > > > > > > >
> > > > > > > > 2013/9/27 Mike Tutkowski 
> > > > > > > >
> > > > > > > > > Here you go:
> > > > > > > > >
> > > > > > > > > mtutkowski@ubuntu:~/cloudstack$ grep
> > cloudstack-agent-upgrade
> > > > > > > > debian/rules
> > > > > > > > > install -D
> agent/target/transformed/cloudstack-agent-upgrade
> > > > > > > > > $(DESTDIR)/usr/bin/cloudstack-agent-upgrade
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Sep 27, 2013 at 12:20 AM, Wei ZHOU <
> > > > ustcweiz...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Did you build the latest source?
> > > > > > > > > > Could you paste the result of the following command in
> your
> > > > > source
> > > > > > > > > > directory?
> > > > > > > > > > "grep cloudstack-agent-upgrade debian/rules"
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Mike Tutkowski*
> > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > e: mike.tutkow...@solidfire.com
> > > > > > > > > o: 303.746.7302
> > > > > > > > > Advancing the way the world uses the
> > > > > > > > > cloud
> > > > > > > > > *™*
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Mike Tutkowski*
> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > e: mike.tutkow...@solidfire.com
> > > > > > > o: 303.746.7302
> > > > > > > Advancing the way the world uses the
> > > > > > > cloud
> > > > > > > *™*
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkow...@solidfire.com
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the
> > > > > cloud
> > > > > *™*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkow...@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the

RE: CloudStack Server Memory Requirements

[Geoff Higginbottom]

All,

Thanks for all the replies, now a simple question, what do I need to change to 
test it?

Regards

Geoff Higginbottom

D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581

geoff.higginbot...@shapeblue.com

-Original Message-
From: Marcus Sorensen [mailto:shadow...@gmail.com]
Sent: 26 September 2013 20:29
To: dev@cloudstack.apache.org
Subject: Re: CloudStack Server Memory Requirements

If I recall, we were able to start running it in devcloud again with only 1G of 
memory allocated to dom0 just a few weeks after the initial spring merge. I 
just think the default was never set back.

On Thu, Sep 26, 2013 at 11:29 AM, Chiradeep Vittal 
 wrote:
> I believe Darren's proposed Spring refactor will help greatly.
>
> On 9/26/13 7:41 AM, "Marcus Sorensen"  wrote:
>
>>I think its an artifact from the Spring stuff six months ago. We can
>>probably decrease that in the default tomcat conf now.
>>On Sep 26, 2013 6:11 AM, "Geoff Higginbottom" <
>>geoff.higginbot...@shapeblue.com> wrote:
>>
>>>  I¹ve been testing the 4.2 release of CloudStack using Virtual Box
>>> and have noticed a need to allocate significantly more memory to the VM.
>>> Previously I would use a CentOS VM with 1 GB of RAM for the
>>> installation but then drop the memory to 512MB, leaving plenty of
>>> RAM on the host machine to then stand up a XenServer VM or a KVM VM etc.
>>>
>>>
>>>
>>> I initially had problems logging into 4.2 after a clean install, and
>>>discovered that only by increasing the memory to 2GB could I get the
>>>system  to function.
>>>
>>>
>>>
>>> I am quite shocked that the memory footprint has increased 400%
>>>between  releases.  Obviously for a real production system,
>>>allocating more than 2GB  or RAM to CloudStack is not an issue, but
>>>it does make standing up a simple  test environment in Virtual Box
>>>more difficult.
>>>
>>>
>>>
>>> Does anyone have ideas why this has increased and is it something
>>> that should be looked at.
>>>
>>>
>>>
>>> Regards
>>>
>>>
>>>
>>> Geoff Higginbottom
>>>
>>> *CTO / Cloud Architect*
>>>
>>>
>>>
>>> [image: Description: Mail Logo Bottom Align]
>>>
>>>
>>>
>>> D: +44 20 3603 0542 <+442036030542> | S: +44 20 3603 0540
>>><+442036030540>| M:
>>> +447968161581
>>>
>>>
>>>
>>> geoff.higginbot...@shapeblue.com | www.shapeblue.com |
>>>Twitter:@shapeblue
>>>
>>>
>>>
>>> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS
>>>
>>>
>>>
>>> Apache CloudStack Bootcamp training courses
>>>
>>> 18/19 September,
>>>Bangalore>>e/>
>>>
>>> 02/03 October,
>>>London
>>>
>>> 13/14 November,
>>>London
>>>
>>> 27/28 November,
>>>Bangalore>>e/>
>>>
>>> 08/09 January 2014,
>>>London
>>> **
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>  This email and any attachments to it may be confidential and are
>>>intended  solely for the use of the individual to whom it is
>>>addressed. Any views or  opinions expressed are solely those of the
>>>author and do not necessarily  represent those of Shape Blue Ltd or
>>>related companies. If you are not the  intended recipient of this
>>>email, you must neither take any action based  upon its contents, nor
>>>copy or show it to anyone. Please contact the sender  if you believe
>>>you have received this email in error. Shape Blue Ltd is a  company
>>>incorporated in England & Wales. ShapeBlue Services India LLP is a
>>>company incorporated in India and is operated under license from
>>>Shape Blue  Ltd. Shape Blue Brasil Consultoria Ltda is a company
>>>incorporated in Brasil  and is operated under license from Shape Blue
>>>Ltd. ShapeBlue is a  registered trademark.
>>>
>
This email and any attachments to it may be confidential and are intended 
solely for the use of the individual to whom it is addressed. Any views or 
opinions expressed are solely those of the author and do not necessarily 
represent those of Shape Blue Ltd or related companies. If you are not the 
intended recipient of this email, you must neither take any action based upon 
its contents, nor copy or show it to anyone. Please contact the sender if you 
believe you have received this email in error. Shape Blue Ltd is a company 
incorporated in England & Wales. ShapeBlue Services India LLP is a company 
incorporated in India and is operated under license from Shape Blue Ltd. Shape 
Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is 
operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.