[jira] [Commented] (JCLOUDS-1230) azurecompute-arm: VM deletion takes very long time (many extra api calls?)

2017-01-26 Thread Andrea Turli (JIRA) 

[ 
https://issues.apache.org/jira/browse/JCLOUDS-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839695#comment-15839695
 ] 

Andrea Turli commented on JCLOUDS-1230:
---

Thanks [~nacx] for heads-up! you are right but that call returns the wrong 
object, I think it always returns a VirtualMachineImage object 

Shame that 
https://docs.microsoft.com/en-us/rest/api/compute/virtualmachineimages#VirtualMachineImages_Get
 returns more details than 
https://docs.microsoft.com/en-us/rest/api/compute/virtualmachineimages#VirtualMachineImages_List
 so we need to call the former for each image only to get VersionProperties 
versionProperties



> azurecompute-arm: VM deletion takes very long time (many extra api calls?)
> --
>
> Key: JCLOUDS-1230
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1230
> Project: jclouds
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aled Sage
>Assignee: Andrea Turli
>
> Deleting a VM in azurecompute-arm took 5mins 55seconds. Looking at the 
> logging for that thread, I suspect there are a lot of optimisations that 
> could be done.
> Here's a rough breakdown of the destroy steps, and how long they take:
> * Get details of the VM and other pieces (e.g. network interface, etc  
>   3 seconds
> * List all the images  
>   11 seconds
> * List all the storageaccounts  
>   3mins 39secs
> * List providers/locations  
>   1 second
> * Delete the VM, and poll for completion . 
>   1min 51secs
> * Delete nic  
>   3 seconds
> * Delete VM disk storage  
>   1 second
> * delete storage account  
>   4 seconds
> Can we avoid listing all the images and all the storage?
> Below are some log snippets that illustrate this (all taken from a single 
> thread):
> {noformat}  
> 2017-01-25 16:01:44,266 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> node(southeastasia/qa-entity-id-ca)
> ... 
> 2017-01-25 16:01:44,267 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking GetVirtualMachine
> ... 
> 2017-01-25 16:01:47,859 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking sku:list
> ... 
> 2017-01-25 16:01:58,598 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking resourcegroup:list
> ... 
> 2017-01-25 16:01:58,628 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:list
> ... 
> 2017-01-25 16:05:37,543 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking providers:get
> ...
> 2017-01-25 16:05:37,592 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking location:list
> ...
> ...
> 2017-01-25 16:05:38,916 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> southeastasia/qa-entity-id-ca ...
> 2017-01-25 16:05:38,917 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteVirtualMachine
> ...
> 2017-01-25 16:07:29,170 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> nic jc-nic-qa-entity-id-ca...
> 2017-01-25 16:07:29,171 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking networkinterfacecard:delete
> ...
> 2017-01-25 16:07:32,091 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> public ip nic public-address-qa-entity-id-ca...
> 2017-01-25 16:07:32,091 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking publicipaddress:delete
> ...
> 2017-01-25 16:07:34,107 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> virtual machine disk storage...
> 2017-01-25 16:07:34,121 DEBUG 101 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteContainer
> ...
> 2017-01-25 16:07:35,141 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> storage account qaentityid524b...
> 2017-01-25 16:07:35,142 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:delete
> ...
> 2017-01-25 16:07:39,855 DEBUG 106 j.compute [ger-M9fVKLUG-247] << destroyed 
> node(southeastasia/qa-entity-id-ca) success(true)
> {noformat}
> Also, combined with https://issues.apache.org/jira/browse/JCLOUDS-1229 this 
> could be very dangerous: we might well be rate-limited while trying to list 
> all the images and thus the VM deletion / cleanup would be aborted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JCLOUDS-1230) azurecompute-arm: VM deletion takes very long time (many extra api calls?)

2017-01-26 Thread Ignasi Barrera (JIRA) 

[ 
https://issues.apache.org/jira/browse/JCLOUDS-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839630#comment-15839630
 ] 

Ignasi Barrera commented on JCLOUDS-1230:
-

That call [is already implemented and in 
use|https://github.com/jclouds/jclouds-labs/blob/master/azurecompute-arm/src/main/java/org/jclouds/azurecompute/arm/features/OSImageApi.java#L74-L82].
 Are you sure it provides all needed information to build an Image object?

> azurecompute-arm: VM deletion takes very long time (many extra api calls?)
> --
>
> Key: JCLOUDS-1230
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1230
> Project: jclouds
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aled Sage
>Assignee: Andrea Turli
>
> Deleting a VM in azurecompute-arm took 5mins 55seconds. Looking at the 
> logging for that thread, I suspect there are a lot of optimisations that 
> could be done.
> Here's a rough breakdown of the destroy steps, and how long they take:
> * Get details of the VM and other pieces (e.g. network interface, etc  
>   3 seconds
> * List all the images  
>   11 seconds
> * List all the storageaccounts  
>   3mins 39secs
> * List providers/locations  
>   1 second
> * Delete the VM, and poll for completion . 
>   1min 51secs
> * Delete nic  
>   3 seconds
> * Delete VM disk storage  
>   1 second
> * delete storage account  
>   4 seconds
> Can we avoid listing all the images and all the storage?
> Below are some log snippets that illustrate this (all taken from a single 
> thread):
> {noformat}  
> 2017-01-25 16:01:44,266 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> node(southeastasia/qa-entity-id-ca)
> ... 
> 2017-01-25 16:01:44,267 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking GetVirtualMachine
> ... 
> 2017-01-25 16:01:47,859 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking sku:list
> ... 
> 2017-01-25 16:01:58,598 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking resourcegroup:list
> ... 
> 2017-01-25 16:01:58,628 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:list
> ... 
> 2017-01-25 16:05:37,543 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking providers:get
> ...
> 2017-01-25 16:05:37,592 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking location:list
> ...
> ...
> 2017-01-25 16:05:38,916 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> southeastasia/qa-entity-id-ca ...
> 2017-01-25 16:05:38,917 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteVirtualMachine
> ...
> 2017-01-25 16:07:29,170 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> nic jc-nic-qa-entity-id-ca...
> 2017-01-25 16:07:29,171 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking networkinterfacecard:delete
> ...
> 2017-01-25 16:07:32,091 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> public ip nic public-address-qa-entity-id-ca...
> 2017-01-25 16:07:32,091 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking publicipaddress:delete
> ...
> 2017-01-25 16:07:34,107 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> virtual machine disk storage...
> 2017-01-25 16:07:34,121 DEBUG 101 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteContainer
> ...
> 2017-01-25 16:07:35,141 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> storage account qaentityid524b...
> 2017-01-25 16:07:35,142 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:delete
> ...
> 2017-01-25 16:07:39,855 DEBUG 106 j.compute [ger-M9fVKLUG-247] << destroyed 
> node(southeastasia/qa-entity-id-ca) success(true)
> {noformat}
> Also, combined with https://issues.apache.org/jira/browse/JCLOUDS-1229 this 
> could be very dangerous: we might well be rate-limited while trying to list 
> all the images and thus the VM deletion / cleanup would be aborted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JCLOUDS-1230) azurecompute-arm: VM deletion takes very long time (many extra api calls?)

2017-01-26 Thread Ignasi Barrera (JIRA) 

[ 
https://issues.apache.org/jira/browse/JCLOUDS-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839595#comment-15839595
 ] 

Ignasi Barrera commented on JCLOUDS-1230:
-

I will add the commits in there to the SecurityGroupExtension PR.

One of the caches I've created is to get a resource group for a given location, 
and using it avoids many errors when the resource group does not exist. For 
example if you use the SG extension to create firewalls before you create any 
node; in that case the resource group could not exist and the current code of 
the extension would fail. The same happens in many places in the code where we 
assume the resource group always exists.

I'm running the tests now and will push all the changes to that PR branch. it 
will contain lots of changes, because I've done a significant cleanup there 
(remove unused custom template options, removed useless constants class that 
duplicated timeout and polling properties, etc).

I'll ping you once everything is uploaded so you can review and test it too.

> azurecompute-arm: VM deletion takes very long time (many extra api calls?)
> --
>
> Key: JCLOUDS-1230
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1230
> Project: jclouds
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aled Sage
>Assignee: Andrea Turli
>
> Deleting a VM in azurecompute-arm took 5mins 55seconds. Looking at the 
> logging for that thread, I suspect there are a lot of optimisations that 
> could be done.
> Here's a rough breakdown of the destroy steps, and how long they take:
> * Get details of the VM and other pieces (e.g. network interface, etc  
>   3 seconds
> * List all the images  
>   11 seconds
> * List all the storageaccounts  
>   3mins 39secs
> * List providers/locations  
>   1 second
> * Delete the VM, and poll for completion . 
>   1min 51secs
> * Delete nic  
>   3 seconds
> * Delete VM disk storage  
>   1 second
> * delete storage account  
>   4 seconds
> Can we avoid listing all the images and all the storage?
> Below are some log snippets that illustrate this (all taken from a single 
> thread):
> {noformat}  
> 2017-01-25 16:01:44,266 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> node(southeastasia/qa-entity-id-ca)
> ... 
> 2017-01-25 16:01:44,267 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking GetVirtualMachine
> ... 
> 2017-01-25 16:01:47,859 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking sku:list
> ... 
> 2017-01-25 16:01:58,598 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking resourcegroup:list
> ... 
> 2017-01-25 16:01:58,628 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:list
> ... 
> 2017-01-25 16:05:37,543 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking providers:get
> ...
> 2017-01-25 16:05:37,592 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking location:list
> ...
> ...
> 2017-01-25 16:05:38,916 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> southeastasia/qa-entity-id-ca ...
> 2017-01-25 16:05:38,917 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteVirtualMachine
> ...
> 2017-01-25 16:07:29,170 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> nic jc-nic-qa-entity-id-ca...
> 2017-01-25 16:07:29,171 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking networkinterfacecard:delete
> ...
> 2017-01-25 16:07:32,091 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> public ip nic public-address-qa-entity-id-ca...
> 2017-01-25 16:07:32,091 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking publicipaddress:delete
> ...
> 2017-01-25 16:07:34,107 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> virtual machine disk storage...
> 2017-01-25 16:07:34,121 DEBUG 101 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteContainer
> ...
> 2017-01-25 16:07:35,141 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> storage account qaentityid524b...
> 2017-01-25 16:07:35,142 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:delete
> ...
> 2017-01-25 16:07:39,855 DEBUG 106 j.compute [ger-M9fVKLUG-247] << destroyed 
> node(southeastasia/qa-entity-id-ca) success(true)
> {noformat}
> Also, combined with https://issues.apache.org/jira/browse/JCLOUDS-1229 this 
> could be very dangerous: we might well be rate-limited while trying to list 
> all the images and thus the VM deletion / cleanup would be aborted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JCLOUDS-1230) azurecompute-arm: VM deletion takes very long time (many extra api calls?)

2017-01-26 Thread Andrea Turli (JIRA) 

[ 
https://issues.apache.org/jira/browse/JCLOUDS-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839582#comment-15839582
 ] 

Andrea Turli commented on JCLOUDS-1230:
---

thanks [~nacx], please open a PR for that as I want to improve CleanupResources 
as described above.


> azurecompute-arm: VM deletion takes very long time (many extra api calls?)
> --
>
> Key: JCLOUDS-1230
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1230
> Project: jclouds
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aled Sage
>Assignee: Andrea Turli
>
> Deleting a VM in azurecompute-arm took 5mins 55seconds. Looking at the 
> logging for that thread, I suspect there are a lot of optimisations that 
> could be done.
> Here's a rough breakdown of the destroy steps, and how long they take:
> * Get details of the VM and other pieces (e.g. network interface, etc  
>   3 seconds
> * List all the images  
>   11 seconds
> * List all the storageaccounts  
>   3mins 39secs
> * List providers/locations  
>   1 second
> * Delete the VM, and poll for completion . 
>   1min 51secs
> * Delete nic  
>   3 seconds
> * Delete VM disk storage  
>   1 second
> * delete storage account  
>   4 seconds
> Can we avoid listing all the images and all the storage?
> Below are some log snippets that illustrate this (all taken from a single 
> thread):
> {noformat}  
> 2017-01-25 16:01:44,266 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> node(southeastasia/qa-entity-id-ca)
> ... 
> 2017-01-25 16:01:44,267 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking GetVirtualMachine
> ... 
> 2017-01-25 16:01:47,859 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking sku:list
> ... 
> 2017-01-25 16:01:58,598 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking resourcegroup:list
> ... 
> 2017-01-25 16:01:58,628 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:list
> ... 
> 2017-01-25 16:05:37,543 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking providers:get
> ...
> 2017-01-25 16:05:37,592 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking location:list
> ...
> ...
> 2017-01-25 16:05:38,916 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> southeastasia/qa-entity-id-ca ...
> 2017-01-25 16:05:38,917 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteVirtualMachine
> ...
> 2017-01-25 16:07:29,170 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> nic jc-nic-qa-entity-id-ca...
> 2017-01-25 16:07:29,171 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking networkinterfacecard:delete
> ...
> 2017-01-25 16:07:32,091 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> public ip nic public-address-qa-entity-id-ca...
> 2017-01-25 16:07:32,091 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking publicipaddress:delete
> ...
> 2017-01-25 16:07:34,107 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> virtual machine disk storage...
> 2017-01-25 16:07:34,121 DEBUG 101 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteContainer
> ...
> 2017-01-25 16:07:35,141 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> storage account qaentityid524b...
> 2017-01-25 16:07:35,142 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:delete
> ...
> 2017-01-25 16:07:39,855 DEBUG 106 j.compute [ger-M9fVKLUG-247] << destroyed 
> node(southeastasia/qa-entity-id-ca) success(true)
> {noformat}
> Also, combined with https://issues.apache.org/jira/browse/JCLOUDS-1229 this 
> could be very dangerous: we might well be rate-limited while trying to list 
> all the images and thus the VM deletion / cleanup would be aborted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JCLOUDS-1230) azurecompute-arm: VM deletion takes very long time (many extra api calls?)

2017-01-26 Thread Ignasi Barrera (JIRA) 

[ 
https://issues.apache.org/jira/browse/JCLOUDS-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839506#comment-15839506
 ] 

Ignasi Barrera commented on JCLOUDS-1230:
-

BTW: https://github.com/nacx/jclouds-labs/tree/cleanup-old-code

> azurecompute-arm: VM deletion takes very long time (many extra api calls?)
> --
>
> Key: JCLOUDS-1230
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1230
> Project: jclouds
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aled Sage
>Assignee: Andrea Turli
>
> Deleting a VM in azurecompute-arm took 5mins 55seconds. Looking at the 
> logging for that thread, I suspect there are a lot of optimisations that 
> could be done.
> Here's a rough breakdown of the destroy steps, and how long they take:
> * Get details of the VM and other pieces (e.g. network interface, etc  
>   3 seconds
> * List all the images  
>   11 seconds
> * List all the storageaccounts  
>   3mins 39secs
> * List providers/locations  
>   1 second
> * Delete the VM, and poll for completion . 
>   1min 51secs
> * Delete nic  
>   3 seconds
> * Delete VM disk storage  
>   1 second
> * delete storage account  
>   4 seconds
> Can we avoid listing all the images and all the storage?
> Below are some log snippets that illustrate this (all taken from a single 
> thread):
> {noformat}  
> 2017-01-25 16:01:44,266 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> node(southeastasia/qa-entity-id-ca)
> ... 
> 2017-01-25 16:01:44,267 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking GetVirtualMachine
> ... 
> 2017-01-25 16:01:47,859 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking sku:list
> ... 
> 2017-01-25 16:01:58,598 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking resourcegroup:list
> ... 
> 2017-01-25 16:01:58,628 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:list
> ... 
> 2017-01-25 16:05:37,543 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking providers:get
> ...
> 2017-01-25 16:05:37,592 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking location:list
> ...
> ...
> 2017-01-25 16:05:38,916 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> southeastasia/qa-entity-id-ca ...
> 2017-01-25 16:05:38,917 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteVirtualMachine
> ...
> 2017-01-25 16:07:29,170 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> nic jc-nic-qa-entity-id-ca...
> 2017-01-25 16:07:29,171 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking networkinterfacecard:delete
> ...
> 2017-01-25 16:07:32,091 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> public ip nic public-address-qa-entity-id-ca...
> 2017-01-25 16:07:32,091 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking publicipaddress:delete
> ...
> 2017-01-25 16:07:34,107 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> virtual machine disk storage...
> 2017-01-25 16:07:34,121 DEBUG 101 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteContainer
> ...
> 2017-01-25 16:07:35,141 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> storage account qaentityid524b...
> 2017-01-25 16:07:35,142 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:delete
> ...
> 2017-01-25 16:07:39,855 DEBUG 106 j.compute [ger-M9fVKLUG-247] << destroyed 
> node(southeastasia/qa-entity-id-ca) success(true)
> {noformat}
> Also, combined with https://issues.apache.org/jira/browse/JCLOUDS-1229 this 
> could be very dangerous: we might well be rate-limited while trying to list 
> all the images and thus the VM deletion / cleanup would be aborted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JCLOUDS-1230) azurecompute-arm: VM deletion takes very long time (many extra api calls?)

2017-01-26 Thread Ignasi Barrera (JIRA) 

[ 
https://issues.apache.org/jira/browse/JCLOUDS-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839505#comment-15839505
 ] 

Ignasi Barrera commented on JCLOUDS-1230:
-

I have a WIP branch (rebased to the security group extension branch) where I'm 
removing lots of legacy code from the previous implementation, and I'm also 
reducing the calls that are made by introducing caches for the common 
"createIfMIssing" operations that are done when provisioning nodes.
[~andreaturli] Would it make sense to implement the call you mention in that 
branch?

> azurecompute-arm: VM deletion takes very long time (many extra api calls?)
> --
>
> Key: JCLOUDS-1230
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1230
> Project: jclouds
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aled Sage
>Assignee: Andrea Turli
>
> Deleting a VM in azurecompute-arm took 5mins 55seconds. Looking at the 
> logging for that thread, I suspect there are a lot of optimisations that 
> could be done.
> Here's a rough breakdown of the destroy steps, and how long they take:
> * Get details of the VM and other pieces (e.g. network interface, etc  
>   3 seconds
> * List all the images  
>   11 seconds
> * List all the storageaccounts  
>   3mins 39secs
> * List providers/locations  
>   1 second
> * Delete the VM, and poll for completion . 
>   1min 51secs
> * Delete nic  
>   3 seconds
> * Delete VM disk storage  
>   1 second
> * delete storage account  
>   4 seconds
> Can we avoid listing all the images and all the storage?
> Below are some log snippets that illustrate this (all taken from a single 
> thread):
> {noformat}  
> 2017-01-25 16:01:44,266 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> node(southeastasia/qa-entity-id-ca)
> ... 
> 2017-01-25 16:01:44,267 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking GetVirtualMachine
> ... 
> 2017-01-25 16:01:47,859 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking sku:list
> ... 
> 2017-01-25 16:01:58,598 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking resourcegroup:list
> ... 
> 2017-01-25 16:01:58,628 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:list
> ... 
> 2017-01-25 16:05:37,543 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking providers:get
> ...
> 2017-01-25 16:05:37,592 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking location:list
> ...
> ...
> 2017-01-25 16:05:38,916 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> southeastasia/qa-entity-id-ca ...
> 2017-01-25 16:05:38,917 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteVirtualMachine
> ...
> 2017-01-25 16:07:29,170 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> nic jc-nic-qa-entity-id-ca...
> 2017-01-25 16:07:29,171 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking networkinterfacecard:delete
> ...
> 2017-01-25 16:07:32,091 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> public ip nic public-address-qa-entity-id-ca...
> 2017-01-25 16:07:32,091 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking publicipaddress:delete
> ...
> 2017-01-25 16:07:34,107 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> virtual machine disk storage...
> 2017-01-25 16:07:34,121 DEBUG 101 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteContainer
> ...
> 2017-01-25 16:07:35,141 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> storage account qaentityid524b...
> 2017-01-25 16:07:35,142 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:delete
> ...
> 2017-01-25 16:07:39,855 DEBUG 106 j.compute [ger-M9fVKLUG-247] << destroyed 
> node(southeastasia/qa-entity-id-ca) success(true)
> {noformat}
> Also, combined with https://issues.apache.org/jira/browse/JCLOUDS-1229 this 
> could be very dangerous: we might well be rate-limited while trying to list 
> all the images and thus the VM deletion / cleanup would be aborted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JCLOUDS-1230) azurecompute-arm: VM deletion takes very long time (many extra api calls?)

2017-01-26 Thread Andrea Turli (JIRA) 

[ 
https://issues.apache.org/jira/browse/JCLOUDS-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839486#comment-15839486
 ] 

Andrea Turli commented on JCLOUDS-1230:
---

[~aledsage], looking at adding 
https://docs.microsoft.com/en-us/rest/api/resources/resourcegroups?redirectedfrom=MSDN#ResourceGroups_ListResources
this will eliminate some other heavy rest call used currently to check if the 
resourceGroup is empty and will avoid at least, namely
{code}
api.getVirtualMachineApi(group).list().isEmpty() 
api.getStorageAccountApi(group).list().isEmpty()
api.getNetworkInterfaceCardApi(group).list().isEmpty()
api.getPublicIPAddressApi(group).list().isEmpty()) 
{code}



> azurecompute-arm: VM deletion takes very long time (many extra api calls?)
> --
>
> Key: JCLOUDS-1230
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1230
> Project: jclouds
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aled Sage
>Assignee: Andrea Turli
>
> Deleting a VM in azurecompute-arm took 5mins 55seconds. Looking at the 
> logging for that thread, I suspect there are a lot of optimisations that 
> could be done.
> Here's a rough breakdown of the destroy steps, and how long they take:
> * Get details of the VM and other pieces (e.g. network interface, etc  
>   3 seconds
> * List all the images  
>   11 seconds
> * List all the storageaccounts  
>   3mins 39secs
> * List providers/locations  
>   1 second
> * Delete the VM, and poll for completion . 
>   1min 51secs
> * Delete nic  
>   3 seconds
> * Delete VM disk storage  
>   1 second
> * delete storage account  
>   4 seconds
> Can we avoid listing all the images and all the storage?
> Below are some log snippets that illustrate this (all taken from a single 
> thread):
> {noformat}  
> 2017-01-25 16:01:44,266 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> node(southeastasia/qa-entity-id-ca)
> ... 
> 2017-01-25 16:01:44,267 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking GetVirtualMachine
> ... 
> 2017-01-25 16:01:47,859 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking sku:list
> ... 
> 2017-01-25 16:01:58,598 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking resourcegroup:list
> ... 
> 2017-01-25 16:01:58,628 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:list
> ... 
> 2017-01-25 16:05:37,543 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking providers:get
> ...
> 2017-01-25 16:05:37,592 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking location:list
> ...
> ...
> 2017-01-25 16:05:38,916 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> southeastasia/qa-entity-id-ca ...
> 2017-01-25 16:05:38,917 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteVirtualMachine
> ...
> 2017-01-25 16:07:29,170 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> nic jc-nic-qa-entity-id-ca...
> 2017-01-25 16:07:29,171 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking networkinterfacecard:delete
> ...
> 2017-01-25 16:07:32,091 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> public ip nic public-address-qa-entity-id-ca...
> 2017-01-25 16:07:32,091 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking publicipaddress:delete
> ...
> 2017-01-25 16:07:34,107 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> virtual machine disk storage...
> 2017-01-25 16:07:34,121 DEBUG 101 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteContainer
> ...
> 2017-01-25 16:07:35,141 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> storage account qaentityid524b...
> 2017-01-25 16:07:35,142 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:delete
> ...
> 2017-01-25 16:07:39,855 DEBUG 106 j.compute [ger-M9fVKLUG-247] << destroyed 
> node(southeastasia/qa-entity-id-ca) success(true)
> {noformat}
> Also, combined with https://issues.apache.org/jira/browse/JCLOUDS-1229 this 
> could be very dangerous: we might well be rate-limited while trying to list 
> all the images and thus the VM deletion / cleanup would be aborted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JCLOUDS-1230) azurecompute-arm: VM deletion takes very long time (many extra api calls?)

2017-01-25 Thread Andrea Turli (JIRA) 

[ 
https://issues.apache.org/jira/browse/JCLOUDS-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838419#comment-15838419
 ] 

Andrea Turli commented on JCLOUDS-1230:
---

Thanks @aledsage I'll have a look

> azurecompute-arm: VM deletion takes very long time (many extra api calls?)
> --
>
> Key: JCLOUDS-1230
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1230
> Project: jclouds
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aled Sage
>Assignee: Andrea Turli
>
> Deleting a VM in azurecompute-arm took 5mins 55seconds. Looking at the 
> logging for that thread, I suspect there are a lot of optimisations that 
> could be done.
> Here's a rough breakdown of the destroy steps, and how long they take:
> * Get details of the VM and other pieces (e.g. network interface, etc  
>   3 seconds
> * List all the images  
>   11 seconds
> * List all the storageaccounts  
>   3mins 39secs
> * List providers/locations  
>   1 second
> * Delete the VM, and poll for completion . 
>   1min 51secs
> * Delete nic  
>   3 seconds
> * Delete VM disk storage  
>   1 second
> * delete storage account  
>   4 seconds
> Can we avoid listing all the images and all the storage?
> Below are some log snippets that illustrate this (all taken from a single 
> thread):
> {noformat}  
> 2017-01-25 16:01:44,266 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> node(southeastasia/qa-entity-id-ca)
> ... 
> 2017-01-25 16:01:44,267 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking GetVirtualMachine
> ... 
> 2017-01-25 16:01:47,859 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking sku:list
> ... 
> 2017-01-25 16:01:58,598 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking resourcegroup:list
> ... 
> 2017-01-25 16:01:58,628 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:list
> ... 
> 2017-01-25 16:05:37,543 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking providers:get
> ...
> 2017-01-25 16:05:37,592 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking location:list
> ...
> ...
> 2017-01-25 16:05:38,916 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> southeastasia/qa-entity-id-ca ...
> 2017-01-25 16:05:38,917 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteVirtualMachine
> ...
> 2017-01-25 16:07:29,170 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> destroying 
> nic jc-nic-qa-entity-id-ca...
> 2017-01-25 16:07:29,171 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking networkinterfacecard:delete
> ...
> 2017-01-25 16:07:32,091 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> public ip nic public-address-qa-entity-id-ca...
> 2017-01-25 16:07:32,091 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking publicipaddress:delete
> ...
> 2017-01-25 16:07:34,107 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> virtual machine disk storage...
> 2017-01-25 16:07:34,121 DEBUG 101 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking DeleteContainer
> ...
> 2017-01-25 16:07:35,141 DEBUG 106 j.compute [ger-M9fVKLUG-247] >> deleting 
> storage account qaentityid524b...
> 2017-01-25 16:07:35,142 DEBUG 106 o.j.r.i.InvokeHttpMethod [ger-M9fVKLUG-247] 
> >> invoking storageaccount:delete
> ...
> 2017-01-25 16:07:39,855 DEBUG 106 j.compute [ger-M9fVKLUG-247] << destroyed 
> node(southeastasia/qa-entity-id-ca) success(true)
> {noformat}
> Also, combined with https://issues.apache.org/jira/browse/JCLOUDS-1229 this 
> could be very dangerous: we might well be rate-limited while trying to list 
> all the images and thus the VM deletion / cleanup would be aborted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)