[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976441#comment-15976441
]
ASF subversion and git services commented on CLOUDSTACK-9764:
-
Commit 6e93ca1f7dcedb39c2d683c701462c48b1a9d951 in cloudstack's branch
refs/heads/master from [~rajanik]
[ https://gitbox.apache.org/repos/asf?p=cloudstack.git;h=6e93ca1 ]
Merge pull request #1935 from nvazquez/deleteDomainFix
CLOUDSTACK-9764: Delete domain failure due to Account Cleanup task
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
>
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976439#comment-15976439
]
ASF subversion and git services commented on CLOUDSTACK-9764:
-
Commit 60ff09d4f7cadfebafd84e3cbc651369cae116d9 in cloudstack's branch
refs/heads/master from [~nicolas.vazquez]
[ https://gitbox.apache.org/repos/asf?p=cloudstack.git;h=60ff09d ]
CLOUDSTACK-9764: Delete domain failure due to Account Cleanup task
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961010#comment-15961010
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@nvazquez now everything seems to be ok
LGTM.
Great job!
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960997#comment-15960997
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@rafaelweingartner thanks for reviewing again! Minor refactor pushed
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960993#comment-15960993
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r110421451
--- Diff: server/test/com/cloud/user/DomainManagerImplTest.java ---
@@ -134,4 +164,69 @@ public void testFindDomainByIdOrPathValidId() {
Assert.assertEquals(domain, domainManager.findDomainByIdOrPath(1L,
"/validDomain/"));
}
+@Test(expected=InvalidParameterValueException.class)
+public void testDeleteDomainNullDomain() {
+Mockito.when(_domainDao.findById(DOMAIN_ID)).thenReturn(null);
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+}
+
+@Test(expected=PermissionDeniedException.class)
+public void testDeleteDomainRootDomain() {
+
Mockito.when(_domainDao.findById(Domain.ROOT_DOMAIN)).thenReturn(domain);
+domainManager.deleteDomain(Domain.ROOT_DOMAIN, testDomainCleanup);
+}
+
+@Test
+public void testDeleteDomainNoCleanup() {
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+Mockito.verify(domainManager).deleteDomain(domain,
testDomainCleanup);
+
Mockito.verify(domainManager).removeDomainWithNoAccountsForCleanupNetworksOrDedicatedResources(domain);
+Mockito.verify(domainManager).cleanupDomainOfferings(DOMAIN_ID);
+Mockito.verify(lock).unlock();
+}
+
+@Test
+public void
testRemoveDomainWithNoAccountsForCleanupNetworksOrDedicatedResourcesRemoveDomain()
{
+
domainManager.removeDomainWithNoAccountsForCleanupNetworksOrDedicatedResources(domain);
+
Mockito.verify(domainManager).publishRemoveEventsAndRemoveDomain(domain);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void
testRemoveDomainWithNoAccountsForCleanupNetworksOrDedicatedResourcesDontRemoveDomain()
{
+domainNetworkIds.add(2l);
+
domainManager.removeDomainWithNoAccountsForCleanupNetworksOrDedicatedResources(domain);
+Mockito.verify(domainManager).failRemoveOperation(domain,
domainAccountsForCleanup, domainNetworkIds, false);
+}
+
+@Test
+public void testPublishRemoveEventsAndRemoveDomainSuccessfulDelete() {
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void testPublishRemoveEventsAndRemoveDomainExceptionDelete() {
+Mockito.when(_domainDao.remove(DOMAIN_ID)).thenReturn(false);
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus,
Mockito.never()).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test
+public void testFailRemoveOperation() {
+try {
+domainManager.failRemoveOperation(domain,
domainAccountsForCleanup, domainNetworkIds, true);
--- End diff --
Great, thanks!
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
>
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960990#comment-15960990
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r110421376
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,82 +284,145 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = getGlobalLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+
+boolean rollBackState = false;
+
+try {
+long ownerId = domain.getAccountId();
+if (BooleanUtils.toBoolean(cleanup)) {
+tryCleanupDomain(domain, ownerId);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg = networkIds.size() + " non-removed networks";
-} else if (hasDedicatedResources) {
-msg = "dedicated resources.";
-}
+
removeDomainWithNoAccountsForCleanupNetworksOrDedicatedResources(domain);
+}
-
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[ https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959500#comment-15959500 ] ASF GitHub Bot commented on CLOUDSTACK-9764: Github user blueorangutan commented on the issue: https://github.com/apache/cloudstack/pull/1935 Trillian test result (tid-973) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 29259 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1935-t973-kvm-centos7.zip Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py Test completed. 48 look ok, 1 have error(s) Test | Result | Time (s) | Test File --- | --- | --- | --- test_04_rvpc_privategw_static_routes | `Failure` | 335.65 | test_privategw_acl.py test_01_vpc_site2site_vpn | Success | 150.05 | test_vpc_vpn.py test_01_vpc_remote_access_vpn | Success | 61.16 | test_vpc_vpn.py test_01_redundant_vpc_site2site_vpn | Success | 245.99 | test_vpc_vpn.py test_02_VPC_default_routes | Success | 264.01 | test_vpc_router_nics.py test_01_VPC_nics_after_destroy | Success | 553.63 | test_vpc_router_nics.py test_05_rvpc_multi_tiers | Success | 512.64 | test_vpc_redundant.py test_04_rvpc_network_garbage_collector_nics | Success | 1410.93 | test_vpc_redundant.py test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | Success | 548.06 | test_vpc_redundant.py test_02_redundant_VPC_default_routes | Success | 745.11 | test_vpc_redundant.py test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Success | 1306.39 | test_vpc_redundant.py test_09_delete_detached_volume | Success | 156.51 | test_volumes.py test_08_resize_volume | Success | 156.45 | test_volumes.py test_07_resize_fail | Success | 161.44 | test_volumes.py test_06_download_detached_volume | Success | 156.43 | test_volumes.py test_05_detach_volume | Success | 150.80 | test_volumes.py test_04_delete_attached_volume | Success | 151.22 | test_volumes.py test_03_download_attached_volume | Success | 156.38 | test_volumes.py test_02_attach_volume | Success | 89.17 | test_volumes.py test_01_create_volume | Success | 622.16 | test_volumes.py test_03_delete_vm_snapshots | Success | 275.25 | test_vm_snapshots.py test_02_revert_vm_snapshots | Success | 100.71 | test_vm_snapshots.py test_01_create_vm_snapshots | Success | 168.82 | test_vm_snapshots.py test_deploy_vm_multiple | Success | 272.66 | test_vm_life_cycle.py test_deploy_vm | Success | 0.03 | test_vm_life_cycle.py test_advZoneVirtualRouter | Success | 0.02 | test_vm_life_cycle.py test_10_attachAndDetach_iso | Success | 26.55 | test_vm_life_cycle.py test_09_expunge_vm | Success | 125.24 | test_vm_life_cycle.py test_08_migrate_vm | Success | 30.88 | test_vm_life_cycle.py test_07_restore_vm | Success | 0.13 | test_vm_life_cycle.py test_06_destroy_vm | Success | 125.81 | test_vm_life_cycle.py test_03_reboot_vm | Success | 125.84 | test_vm_life_cycle.py test_02_start_vm | Success | 10.17 | test_vm_life_cycle.py test_01_stop_vm | Success | 35.29 | test_vm_life_cycle.py test_CreateTemplateWithDuplicateName | Success | 161.14 | test_templates.py test_08_list_system_templates | Success | 0.04 | test_templates.py test_07_list_public_templates | Success | 0.04 | test_templates.py test_05_template_permissions | Success | 0.05 | test_templates.py test_04_extract_template | Success | 5.17 | test_templates.py test_03_delete_template | Success | 5.11 | test_templates.py test_02_edit_template | Success | 90.13 | test_templates.py test_01_create_template | Success | 35.41 | test_templates.py test_10_destroy_cpvm | Success | 161.64 | test_ssvm.py test_09_destroy_ssvm | Success | 133.24 | test_ssvm.py test_08_reboot_cpvm | Success | 131.61 | test_ssvm.py test_07_reboot_ssvm | Success | 133.65 | test_ssvm.py test_06_stop_cpvm | Success | 131.74 | test_ssvm.py test_05_stop_ssvm | Success | 163.77 | test_ssvm.py test_04_cpvm_internals | Success | 1.24 | test_ssvm.py test_03_ssvm_internals | Success | 3.37 | test_ssvm.py test_02_list_cpvm_vm | Success | 0.12 | test_ssvm.py test_01_list_sec_storage_vm | Success | 0.13 | test_ssvm.py test_02_list_snapshots_with_removed_data_store | Success | 86.81 | test_snapshots.py test_01_snapshot_root_disk | Success | 16.33 | test_snapshots.py test_04_change_offering_small | Success | 209.54 | test_service_offerings.py test_03_delete_service_offering | Success | 0.04 | test_service_offerings.py test_02_edit_service_offering | Success | 0.06 | test_service_offerings.py test_01_create_service_offering | Success | 0.11 | test_service_offerings.py test_02_sys_template_ready | Success | 0.14 | test_secondary_storage.py
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955355#comment-15955355
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user borisstoyanov commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@blueorangutan test
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is already removed
> 2017-01-26
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955356#comment-15955356
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has
been kicked to run smoke tests
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955335#comment-15955335
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1935
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-614
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955306#comment-15955306
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r103529571
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,82 +284,145 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = getGlobalLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+
+boolean rollBackState = false;
+
+try {
+long ownerId = domain.getAccountId();
+if (BooleanUtils.toBoolean(cleanup)) {
+tryCleanupDomain(domain, ownerId);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg = networkIds.size() + " non-removed networks";
-} else if (hasDedicatedResources) {
-msg = "dedicated resources.";
-}
+
removeDomainWithNoAccountsForCleanupNetworksOrDedicatedResources(domain);
+}
-
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955307#comment-15955307
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r103529869
--- Diff: server/test/com/cloud/user/DomainManagerImplTest.java ---
@@ -134,4 +164,69 @@ public void testFindDomainByIdOrPathValidId() {
Assert.assertEquals(domain, domainManager.findDomainByIdOrPath(1L,
"/validDomain/"));
}
+@Test(expected=InvalidParameterValueException.class)
+public void testDeleteDomainNullDomain() {
+Mockito.when(_domainDao.findById(DOMAIN_ID)).thenReturn(null);
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+}
+
+@Test(expected=PermissionDeniedException.class)
+public void testDeleteDomainRootDomain() {
+
Mockito.when(_domainDao.findById(Domain.ROOT_DOMAIN)).thenReturn(domain);
+domainManager.deleteDomain(Domain.ROOT_DOMAIN, testDomainCleanup);
+}
+
+@Test
+public void testDeleteDomainNoCleanup() {
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+Mockito.verify(domainManager).deleteDomain(domain,
testDomainCleanup);
+
Mockito.verify(domainManager).removeDomainWithNoAccountsForCleanupNetworksOrDedicatedResources(domain);
+Mockito.verify(domainManager).cleanupDomainOfferings(DOMAIN_ID);
+Mockito.verify(lock).unlock();
+}
+
+@Test
+public void
testRemoveDomainWithNoAccountsForCleanupNetworksOrDedicatedResourcesRemoveDomain()
{
+
domainManager.removeDomainWithNoAccountsForCleanupNetworksOrDedicatedResources(domain);
+
Mockito.verify(domainManager).publishRemoveEventsAndRemoveDomain(domain);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void
testRemoveDomainWithNoAccountsForCleanupNetworksOrDedicatedResourcesDontRemoveDomain()
{
+domainNetworkIds.add(2l);
+
domainManager.removeDomainWithNoAccountsForCleanupNetworksOrDedicatedResources(domain);
+Mockito.verify(domainManager).failRemoveOperation(domain,
domainAccountsForCleanup, domainNetworkIds, false);
+}
+
+@Test
+public void testPublishRemoveEventsAndRemoveDomainSuccessfulDelete() {
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void testPublishRemoveEventsAndRemoveDomainExceptionDelete() {
+Mockito.when(_domainDao.remove(DOMAIN_ID)).thenReturn(false);
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus,
Mockito.never()).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test
+public void testFailRemoveOperation() {
+try {
+domainManager.failRemoveOperation(domain,
domainAccountsForCleanup, domainNetworkIds, true);
--- End diff --
Now that you removed the use of `rollBackState`, made the method last
problematic to test. Therefore, you can use the `@Test(expected=...)`, instead
of this very unusual construction here.
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955305#comment-15955305
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102538324
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -109,6 +112,20 @@
@Inject
MessageBus _messageBus;
+static boolean rollBackState = false;
--- End diff --
@nvazquez I have been thinking about this variable you introduced here. I
think it can cause problems (concurrency problems). The `DomainManagerImpl` is
a singleton. Therefore, it should not have state variables. The `rollBackState
` is acting as a state variable for requests that use
`com.cloud.user.DomainManagerImpl.deleteDomain(DomainVO, Boolean)`. The problem
is that every call should have its own context/state for `rollBackState`.
However, this will not happen with the current implementation.
I think we should re-work the use of that variable. What do you think?
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
>
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955300#comment-15955300
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep
you posted as I make progress.
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955297#comment-15955297
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user borisstoyanov commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@blueorangutan package
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is already removed
> 2017-01-26
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955177#comment-15955177
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@borisstoyanov I've rebased master branch, can we re-run tests on this PR?
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888687#comment-15888687
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on the issue:
https://github.com/apache/cloudstack/pull/1935
Hi @rafaelweingartner, I've refactored the code instead of using
`rollBackState` as static. I think that using static variable could lead to a
problem if new methods are invoked from another method different than
`deleteDomain` method. Instead of declaring it as static, we reduced the scope
again and only set it true when `CloudRuntimeException` is thrown. What do you
think about this refactor? I tryied not to introduce major changes in original
code
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
>
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885257#comment-15885257
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user serg38 commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@rhtyd @karuturi To fix errors in B.O in
test_02_list_snapshots_with_removed_data_store we need to merge PR1961 and then
adjust test_data on B.O side to have correct mapping for nfs2 label
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885234#comment-15885234
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rhtyd commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@nvazquez can you check if the failures in above test results are related
to your changes, thanks.
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
>
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[ https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883595#comment-15883595 ] ASF GitHub Bot commented on CLOUDSTACK-9764: Github user blueorangutan commented on the issue: https://github.com/apache/cloudstack/pull/1935 Trillian test result (tid-896) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 32396 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1935-t896-kvm-centos7.zip Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py Intermitten failure detected: /marvin/tests/smoke/test_snapshots.py Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py Test completed. 46 look ok, 3 have error(s) Test | Result | Time (s) | Test File --- | --- | --- | --- test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Failure` | 363.77 | test_vpc_redundant.py test_04_rvpc_privategw_static_routes | `Failure` | 345.94 | test_privategw_acl.py test_02_list_snapshots_with_removed_data_store | `Error` | 0.04 | test_snapshots.py test_01_vpc_site2site_vpn | Success | 141.33 | test_vpc_vpn.py test_01_vpc_remote_access_vpn | Success | 56.08 | test_vpc_vpn.py test_01_redundant_vpc_site2site_vpn | Success | 231.05 | test_vpc_vpn.py test_02_VPC_default_routes | Success | 258.91 | test_vpc_router_nics.py test_01_VPC_nics_after_destroy | Success | 531.86 | test_vpc_router_nics.py test_05_rvpc_multi_tiers | Success | 521.44 | test_vpc_redundant.py test_04_rvpc_network_garbage_collector_nics | Success | 1407.93 | test_vpc_redundant.py test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | Success | 537.92 | test_vpc_redundant.py test_02_redundant_VPC_default_routes | Success | 755.43 | test_vpc_redundant.py test_09_delete_detached_volume | Success | 151.68 | test_volumes.py test_08_resize_volume | Success | 156.42 | test_volumes.py test_07_resize_fail | Success | 161.56 | test_volumes.py test_06_download_detached_volume | Success | 156.35 | test_volumes.py test_05_detach_volume | Success | 145.71 | test_volumes.py test_04_delete_attached_volume | Success | 146.22 | test_volumes.py test_03_download_attached_volume | Success | 156.37 | test_volumes.py test_02_attach_volume | Success | 84.13 | test_volumes.py test_01_create_volume | Success | 621.25 | test_volumes.py test_03_delete_vm_snapshots | Success | 275.20 | test_vm_snapshots.py test_02_revert_vm_snapshots | Success | 95.69 | test_vm_snapshots.py test_01_create_vm_snapshots | Success | 163.72 | test_vm_snapshots.py test_deploy_vm_multiple | Success | 232.62 | test_vm_life_cycle.py test_deploy_vm | Success | 0.04 | test_vm_life_cycle.py test_advZoneVirtualRouter | Success | 0.03 | test_vm_life_cycle.py test_10_attachAndDetach_iso | Success | 26.52 | test_vm_life_cycle.py test_09_expunge_vm | Success | 125.25 | test_vm_life_cycle.py test_08_migrate_vm | Success | 35.90 | test_vm_life_cycle.py test_07_restore_vm | Success | 0.13 | test_vm_life_cycle.py test_06_destroy_vm | Success | 125.94 | test_vm_life_cycle.py test_03_reboot_vm | Success | 125.90 | test_vm_life_cycle.py test_02_start_vm | Success | 10.17 | test_vm_life_cycle.py test_01_stop_vm | Success | 35.30 | test_vm_life_cycle.py test_CreateTemplateWithDuplicateName | Success | 50.55 | test_templates.py test_08_list_system_templates | Success | 0.03 | test_templates.py test_07_list_public_templates | Success | 0.06 | test_templates.py test_05_template_permissions | Success | 0.06 | test_templates.py test_04_extract_template | Success | 5.14 | test_templates.py test_03_delete_template | Success | 5.13 | test_templates.py test_02_edit_template | Success | 90.13 | test_templates.py test_01_create_template | Success | 30.36 | test_templates.py test_10_destroy_cpvm | Success | 161.39 | test_ssvm.py test_09_destroy_ssvm | Success | 138.56 | test_ssvm.py test_08_reboot_cpvm | Success | 101.35 | test_ssvm.py test_07_reboot_ssvm | Success | 133.54 | test_ssvm.py test_06_stop_cpvm | Success | 131.57 | test_ssvm.py test_05_stop_ssvm | Success | 163.76 | test_ssvm.py test_04_cpvm_internals | Success | 0.97 | test_ssvm.py test_03_ssvm_internals | Success | 3.30 | test_ssvm.py test_02_list_cpvm_vm | Success | 0.13 | test_ssvm.py test_01_list_sec_storage_vm | Success | 0.15 | test_ssvm.py test_01_snapshot_root_disk | Success | 11.27 | test_snapshots.py test_04_change_offering_small | Success | 234.58 | test_service_offerings.py test_03_delete_service_offering | Success | 0.04 | test_service_offerings.py test_02_edit_service_offering | Success | 0.06 | test_service_offerings.py
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882620#comment-15882620
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has
been kicked to run smoke tests
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882619#comment-15882619
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user borisstoyanov commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@blueorangutan test
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is already removed
> 2017-01-26
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882566#comment-15882566
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1935
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-537
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882535#comment-15882535
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user borisstoyanov commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@blueorangutan package
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is already removed
> 2017-01-26
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881000#comment-15881000
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@serg38 I did, the code is great as always. However, I have a concern about
that `rollBackState` variable being static there. Because the
`DomainManagerImpl ` is a singleton, using that variable may create problems.
Instead of using the variable to indicate when the rollback has to be executed,
we could do the same using an exception (the exception is already thrown when
the variable is set to `true`).
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
>
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880924#comment-15880924
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user serg38 commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@rafaelweingartner Can you review latest updates from @nvazquez . Since
tests are passing this PR will be ready for merging to 4.10
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879095#comment-15879095
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102556528
--- Diff: server/test/com/cloud/user/DomainManagerImplTest.java ---
@@ -134,4 +164,67 @@ public void testFindDomainByIdOrPathValidId() {
Assert.assertEquals(domain, domainManager.findDomainByIdOrPath(1L,
"/validDomain/"));
}
+@Test(expected=InvalidParameterValueException.class)
+public void testDeleteDomainNullDomain() {
+Mockito.when(_domainDao.findById(DOMAIN_ID)).thenReturn(null);
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+}
+
+@Test(expected=PermissionDeniedException.class)
+public void testDeleteDomainRootDomain() {
+
Mockito.when(_domainDao.findById(Domain.ROOT_DOMAIN)).thenReturn(domain);
+domainManager.deleteDomain(Domain.ROOT_DOMAIN, testDomainCleanup);
+}
+
+//TODO testDeleteDomainCleanup
+
+@Test
+public void testDeleteDomainNoCleanup() {
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+Mockito.verify(domainManager).deleteDomain(domain,
testDomainCleanup);
+
Mockito.verify(domainManager).checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+Mockito.verify(domainManager).cleanupDomainOfferings(DOMAIN_ID);
+Mockito.verify(lock).unlock();
+Assert.assertEquals(false, domainManager.getRollBackState());
+}
+
+@Test
+public void
testCheckDomainAccountsNetworksAndResourcesBeforeRemovingRemoveDomain() {
+
domainManager.checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+
Mockito.verify(domainManager).publishRemoveEventsAndRemoveDomain(domain);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void
testCheckDomainAccountsNetworksAndResourcesBeforeRemovingDontRemoveDomain() {
+domainNetworkIds.add(2l);
+
domainManager.checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+Mockito.verify(domainManager).failRemoveOperation(domain,
domainAccountsForCleanup, domainNetworkIds, false);
+}
+
+@Test
+public void testPublishRemoveEventsAndRemoveDomainSuccessfulDelete() {
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void testPublishRemoveEventsAndRemoveDomainExceptionDelete() {
+Mockito.when(_domainDao.remove(DOMAIN_ID)).thenReturn(false);
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus,
Mockito.never()).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void testFailRemoveOperation() {
--- End diff --
Thanks! I removed `expected` from `@Test` annotation and added a catch
block to assert exception class. It is a nasty method to test indeed :)
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878931#comment-15878931
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102536773
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,82 +289,133 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = getGlobalLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+
+try {
+long ownerId = domain.getAccountId();
+if (BooleanUtils.toBoolean(cleanup)) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg = networkIds.size() + " non-removed networks";
-} else if (hasDedicatedResources) {
-msg =
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878923#comment-15878923
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102535697
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,82 +289,133 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = getGlobalLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+
+try {
+long ownerId = domain.getAccountId();
+if (BooleanUtils.toBoolean(cleanup)) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
--- End diff --
Ah, the naming of methods, I also find very hard sometimes; I think most
programmers have this problem. I think the name you proposed is ok. I am not
creative today.
I also liked the way the code became cleaner and easier to read. It also
facilitates future changes.
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878916#comment-15878916
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102534985
--- Diff: server/test/com/cloud/user/DomainManagerImplTest.java ---
@@ -134,4 +164,67 @@ public void testFindDomainByIdOrPathValidId() {
Assert.assertEquals(domain, domainManager.findDomainByIdOrPath(1L,
"/validDomain/"));
}
+@Test(expected=InvalidParameterValueException.class)
+public void testDeleteDomainNullDomain() {
+Mockito.when(_domainDao.findById(DOMAIN_ID)).thenReturn(null);
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+}
+
+@Test(expected=PermissionDeniedException.class)
+public void testDeleteDomainRootDomain() {
+
Mockito.when(_domainDao.findById(Domain.ROOT_DOMAIN)).thenReturn(domain);
+domainManager.deleteDomain(Domain.ROOT_DOMAIN, testDomainCleanup);
+}
+
+//TODO testDeleteDomainCleanup
+
+@Test
+public void testDeleteDomainNoCleanup() {
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+Mockito.verify(domainManager).deleteDomain(domain,
testDomainCleanup);
+
Mockito.verify(domainManager).checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+Mockito.verify(domainManager).cleanupDomainOfferings(DOMAIN_ID);
+Mockito.verify(lock).unlock();
+Assert.assertEquals(false, domainManager.getRollBackState());
+}
+
+@Test
+public void
testCheckDomainAccountsNetworksAndResourcesBeforeRemovingRemoveDomain() {
+
domainManager.checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+
Mockito.verify(domainManager).publishRemoveEventsAndRemoveDomain(domain);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void
testCheckDomainAccountsNetworksAndResourcesBeforeRemovingDontRemoveDomain() {
+domainNetworkIds.add(2l);
+
domainManager.checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+Mockito.verify(domainManager).failRemoveOperation(domain,
domainAccountsForCleanup, domainNetworkIds, false);
+}
+
+@Test
+public void testPublishRemoveEventsAndRemoveDomainSuccessfulDelete() {
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void testPublishRemoveEventsAndRemoveDomainExceptionDelete() {
+Mockito.when(_domainDao.remove(DOMAIN_ID)).thenReturn(false);
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus,
Mockito.never()).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void testFailRemoveOperation() {
--- End diff --
Here we still have a problem.
the exception is thrown at line 227 and the code at line 228 is never
executed.
That is why I called nasty to write a test here. It is very tricky
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878889#comment-15878889
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102531047
--- Diff: server/test/com/cloud/user/DomainManagerImplTest.java ---
@@ -134,4 +164,67 @@ public void testFindDomainByIdOrPathValidId() {
Assert.assertEquals(domain, domainManager.findDomainByIdOrPath(1L,
"/validDomain/"));
}
+@Test(expected=InvalidParameterValueException.class)
+public void testDeleteDomainNullDomain() {
+Mockito.when(_domainDao.findById(DOMAIN_ID)).thenReturn(null);
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+}
+
+@Test(expected=PermissionDeniedException.class)
+public void testDeleteDomainRootDomain() {
+
Mockito.when(_domainDao.findById(Domain.ROOT_DOMAIN)).thenReturn(domain);
+domainManager.deleteDomain(Domain.ROOT_DOMAIN, testDomainCleanup);
+}
+
+//TODO testDeleteDomainCleanup
+
+@Test
+public void testDeleteDomainNoCleanup() {
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+Mockito.verify(domainManager).deleteDomain(domain,
testDomainCleanup);
+
Mockito.verify(domainManager).checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+Mockito.verify(domainManager).cleanupDomainOfferings(DOMAIN_ID);
+Mockito.verify(lock).unlock();
+Assert.assertEquals(false, domainManager.getRollBackState());
+}
+
+@Test
+public void
testCheckDomainAccountsNetworksAndResourcesBeforeRemovingRemoveDomain() {
+
domainManager.checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+
Mockito.verify(domainManager).publishRemoveEventsAndRemoveDomain(domain);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void
testCheckDomainAccountsNetworksAndResourcesBeforeRemovingDontRemoveDomain() {
+domainNetworkIds.add(2l);
+
domainManager.checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+Mockito.verify(domainManager).failRemoveOperation(domain,
domainAccountsForCleanup, domainNetworkIds, false);
+}
+
+@Test
+public void testPublishRemoveEventsAndRemoveDomainSuccessfulDelete() {
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void testPublishRemoveEventsAndRemoveDomainExceptionDelete() {
+Mockito.when(_domainDao.remove(DOMAIN_ID)).thenReturn(false);
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus,
Mockito.never()).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void testFailRemoveOperation() {
--- End diff --
Great, I've added verification, I have missed that out, thanks!
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878890#comment-15878890
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102531132
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,82 +289,133 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = getGlobalLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+
+try {
+long ownerId = domain.getAccountId();
+if (BooleanUtils.toBoolean(cleanup)) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg = networkIds.size() + " non-removed networks";
-} else if (hasDedicatedResources) {
-msg = "dedicated
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1587#comment-1587
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102530895
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,82 +289,133 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = getGlobalLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+
+try {
+long ownerId = domain.getAccountId();
+if (BooleanUtils.toBoolean(cleanup)) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg = networkIds.size() + " non-removed networks";
-} else if (hasDedicatedResources) {
-msg = "dedicated
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878875#comment-15878875
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102529801
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,82 +289,133 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = getGlobalLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+
+try {
+long ownerId = domain.getAccountId();
+if (BooleanUtils.toBoolean(cleanup)) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
--- End diff --
Done, I liked it how it simplified the code inside those blocks. However, I
still find it difficult to name methods, do you agree with name
`tryCleanupDomain`?
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains.
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878729#comment-15878729
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@rafaelweingartner no problem, I should have mentioned about changing the
variable to static. I'll work on your last comments :)
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878464#comment-15878464
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102487341
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,82 +289,133 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = getGlobalLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+
+try {
+long ownerId = domain.getAccountId();
+if (BooleanUtils.toBoolean(cleanup)) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
--- End diff --
@nvazquez you have already improve greatly this method.
How do you feel about another code extraction? lines 312-319 can be easily
extracted to a method, enabling the reduction of the lines here in these
try/try/finally/finally blocks.
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878462#comment-15878462
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102486318
--- Diff: server/test/com/cloud/user/DomainManagerImplTest.java ---
@@ -134,4 +164,67 @@ public void testFindDomainByIdOrPathValidId() {
Assert.assertEquals(domain, domainManager.findDomainByIdOrPath(1L,
"/validDomain/"));
}
+@Test(expected=InvalidParameterValueException.class)
+public void testDeleteDomainNullDomain() {
+Mockito.when(_domainDao.findById(DOMAIN_ID)).thenReturn(null);
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+}
+
+@Test(expected=PermissionDeniedException.class)
+public void testDeleteDomainRootDomain() {
+
Mockito.when(_domainDao.findById(Domain.ROOT_DOMAIN)).thenReturn(domain);
+domainManager.deleteDomain(Domain.ROOT_DOMAIN, testDomainCleanup);
+}
+
+//TODO testDeleteDomainCleanup
+
+@Test
+public void testDeleteDomainNoCleanup() {
+domainManager.deleteDomain(DOMAIN_ID, testDomainCleanup);
+Mockito.verify(domainManager).deleteDomain(domain,
testDomainCleanup);
+
Mockito.verify(domainManager).checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+Mockito.verify(domainManager).cleanupDomainOfferings(DOMAIN_ID);
+Mockito.verify(lock).unlock();
+Assert.assertEquals(false, domainManager.getRollBackState());
+}
+
+@Test
+public void
testCheckDomainAccountsNetworksAndResourcesBeforeRemovingRemoveDomain() {
+
domainManager.checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+
Mockito.verify(domainManager).publishRemoveEventsAndRemoveDomain(domain);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void
testCheckDomainAccountsNetworksAndResourcesBeforeRemovingDontRemoveDomain() {
+domainNetworkIds.add(2l);
+
domainManager.checkDomainAccountsNetworksAndResourcesBeforeRemoving(domain);
+Mockito.verify(domainManager).failRemoveOperation(domain,
domainAccountsForCleanup, domainNetworkIds, false);
+}
+
+@Test
+public void testPublishRemoveEventsAndRemoveDomainSuccessfulDelete() {
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void testPublishRemoveEventsAndRemoveDomainExceptionDelete() {
+Mockito.when(_domainDao.remove(DOMAIN_ID)).thenReturn(false);
+domainManager.publishRemoveEventsAndRemoveDomain(domain);
+Mockito.verify(_messageBus).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_PRE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_messageBus,
Mockito.never()).publish(Mockito.anyString(),
Matchers.eq(DomainManager.MESSAGE_REMOVE_DOMAIN_EVENT),
+Matchers.eq(PublishScope.LOCAL), Matchers.eq(domain));
+Mockito.verify(_domainDao).remove(DOMAIN_ID);
+}
+
+@Test(expected=CloudRuntimeException.class)
+public void testFailRemoveOperation() {
--- End diff --
Here is a nasty method to test.
when an exception happens, the variable `rollBackState ` is changed to
`true`. should not we verify that?
What do you think about this?
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878466#comment-15878466
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102323551
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,82 +289,133 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = getGlobalLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+
+try {
+long ownerId = domain.getAccountId();
+if (BooleanUtils.toBoolean(cleanup)) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg = networkIds.size() + " non-removed networks";
-} else if (hasDedicatedResources) {
-msg =
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878465#comment-15878465
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102322603
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,82 +289,133 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = getGlobalLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+
+try {
+long ownerId = domain.getAccountId();
+if (BooleanUtils.toBoolean(cleanup)) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg = networkIds.size() + " non-removed networks";
-} else if (hasDedicatedResources) {
-msg =
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878432#comment-15878432
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@nvazquez was the static declaration there before?! I am so sorry I missed
it.
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> //
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[ https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877585#comment-15877585 ] ASF GitHub Bot commented on CLOUDSTACK-9764: Github user blueorangutan commented on the issue: https://github.com/apache/cloudstack/pull/1935 Trillian test result (tid-876) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 35807 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1935-t876-kvm-centos7.zip Intermitten failure detected: /marvin/tests/smoke/test_network.py Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py Intermitten failure detected: /marvin/tests/smoke/test_snapshots.py Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py Test completed. 46 look ok, 3 have error(s) Test | Result | Time (s) | Test File --- | --- | --- | --- test_02_redundant_VPC_default_routes | `Failure` | 864.13 | test_vpc_redundant.py test_04_rvpc_privategw_static_routes | `Failure` | 320.45 | test_privategw_acl.py test_02_list_snapshots_with_removed_data_store | `Error` | 0.04 | test_snapshots.py test_01_vpc_site2site_vpn | Success | 160.52 | test_vpc_vpn.py test_01_vpc_remote_access_vpn | Success | 61.11 | test_vpc_vpn.py test_01_redundant_vpc_site2site_vpn | Success | 250.72 | test_vpc_vpn.py test_02_VPC_default_routes | Success | 287.25 | test_vpc_router_nics.py test_01_VPC_nics_after_destroy | Success | 545.04 | test_vpc_router_nics.py test_05_rvpc_multi_tiers | Success | 512.25 | test_vpc_redundant.py test_04_rvpc_network_garbage_collector_nics | Success | 1414.74 | test_vpc_redundant.py test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | Success | 548.99 | test_vpc_redundant.py test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Success | 1297.58 | test_vpc_redundant.py test_09_delete_detached_volume | Success | 151.41 | test_volumes.py test_08_resize_volume | Success | 156.44 | test_volumes.py test_07_resize_fail | Success | 156.52 | test_volumes.py test_06_download_detached_volume | Success | 156.34 | test_volumes.py test_05_detach_volume | Success | 155.91 | test_volumes.py test_04_delete_attached_volume | Success | 151.44 | test_volumes.py test_03_download_attached_volume | Success | 151.32 | test_volumes.py test_02_attach_volume | Success | 95.17 | test_volumes.py test_01_create_volume | Success | 711.28 | test_volumes.py test_03_delete_vm_snapshots | Success | 275.17 | test_vm_snapshots.py test_02_revert_vm_snapshots | Success | 95.78 | test_vm_snapshots.py test_01_create_vm_snapshots | Success | 163.76 | test_vm_snapshots.py test_deploy_vm_multiple | Success | 247.75 | test_vm_life_cycle.py test_deploy_vm | Success | 0.04 | test_vm_life_cycle.py test_advZoneVirtualRouter | Success | 0.03 | test_vm_life_cycle.py test_10_attachAndDetach_iso | Success | 26.64 | test_vm_life_cycle.py test_09_expunge_vm | Success | 125.25 | test_vm_life_cycle.py test_08_migrate_vm | Success | 40.94 | test_vm_life_cycle.py test_07_restore_vm | Success | 0.13 | test_vm_life_cycle.py test_06_destroy_vm | Success | 125.84 | test_vm_life_cycle.py test_03_reboot_vm | Success | 125.87 | test_vm_life_cycle.py test_02_start_vm | Success | 10.17 | test_vm_life_cycle.py test_01_stop_vm | Success | 40.33 | test_vm_life_cycle.py test_CreateTemplateWithDuplicateName | Success | 40.46 | test_templates.py test_08_list_system_templates | Success | 0.03 | test_templates.py test_07_list_public_templates | Success | 0.04 | test_templates.py test_05_template_permissions | Success | 0.06 | test_templates.py test_04_extract_template | Success | 5.16 | test_templates.py test_03_delete_template | Success | 5.11 | test_templates.py test_02_edit_template | Success | 90.18 | test_templates.py test_01_create_template | Success | 40.43 | test_templates.py test_10_destroy_cpvm | Success | 166.69 | test_ssvm.py test_09_destroy_ssvm | Success | 163.56 | test_ssvm.py test_08_reboot_cpvm | Success | 101.57 | test_ssvm.py test_07_reboot_ssvm | Success | 163.59 | test_ssvm.py test_06_stop_cpvm | Success | 132.19 | test_ssvm.py test_05_stop_ssvm | Success | 164.02 | test_ssvm.py test_04_cpvm_internals | Success | 1.22 | test_ssvm.py test_03_ssvm_internals | Success | 3.42 | test_ssvm.py test_02_list_cpvm_vm | Success | 0.12 | test_ssvm.py test_01_list_sec_storage_vm | Success | 0.13 | test_ssvm.py test_01_snapshot_root_disk | Success | 11.11 | test_snapshots.py test_04_change_offering_small | Success | 210.27 | test_service_offerings.py test_03_delete_service_offering | Success | 0.04 | test_service_offerings.py
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877481#comment-15877481
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@rafaelweingartner I think I got your point, I tried to keep code as
similar as it was before, by declaring `rollBackState` as static class variable
on line 114. This way inner `finally` block would work the same as before when
one of new methods set `rollBackState = true.` On outter `finally` block,
`rollBackState` is set to false (line 345), this way each time `deleteDomain`
is invoked it would start on false (maybe it would be easier to move it at the
beggining of `deleteDomain`). Do you agree with this approach?
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
>
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876803#comment-15876803
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@nvazquez great work.
However, there is a catch there that I think you might have overlooked.
This problem is caused by the method extraction I suggested.
If you take a look at the code before the extraction, every time that an
exception is thrown, the code was setting the variable `rollBackState = true`.
This happens at lines 287, 305, and 313. Now that the code was extracted,
setting those variables to `true` does not work anymore, because of the context
those variables are declared change.
In my opinion, this code was kind of weird before. It was throwing an
exception that is caught right away and setting a control variable to be
executed on `finally` block. The only reason I see for this is that if other
exceptions that are not the ones generated at lines 292, 310, and 325 happen,
and we do not want to execute the rollback for them. However, this seems error
prone, leading to database inconsistencies.
I would change the "rollback" code (lines 342-345) to the catch block.
I do not know if I have been clear, we can discuss this further. I may have
overlooked some bits of it as well (it is a quite complicated bit of code).
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
>
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876750#comment-15876750
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@rafaelweingartner thanks for reviewing! I extracted code to new methods
and also added unit tests for them
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876682#comment-15876682
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has
been kicked to run smoke tests
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876675#comment-15876675
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user borisstoyanov commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@blueorangutan test
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is already removed
> 2017-01-26
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876593#comment-15876593
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1935
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-520
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876542#comment-15876542
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user borisstoyanov commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@blueorangutan package
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is already removed
> 2017-01-26
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876546#comment-15876546
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep
you posted as I make progress.
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876318#comment-15876318
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102265306
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,79 +274,97 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = GlobalLock.getInternLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+boolean rollBackState = false;
+boolean hasDedicatedResources = false;
+
+try {
+long ownerId = domain.getAccountId();
+if ((cleanup != null) && cleanup.booleanValue()) {
--- End diff --
Done, thanks @rafaelweingartner
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876315#comment-15876315
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102265150
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,79 +274,97 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = GlobalLock.getInternLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+boolean rollBackState = false;
+boolean hasDedicatedResources = false;
+
+try {
+long ownerId = domain.getAccountId();
+if ((cleanup != null) && cleanup.booleanValue()) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg = networkIds.size() + "
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876312#comment-15876312
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user nvazquez commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r102264841
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,79 +274,97 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = GlobalLock.getInternLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+boolean rollBackState = false;
+boolean hasDedicatedResources = false;
+
+try {
+long ownerId = domain.getAccountId();
+if ((cleanup != null) && cleanup.booleanValue()) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg = networkIds.size() + "
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861532#comment-15861532
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r100575787
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,79 +274,97 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = GlobalLock.getInternLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+boolean rollBackState = false;
+boolean hasDedicatedResources = false;
+
+try {
+long ownerId = domain.getAccountId();
+if ((cleanup != null) && cleanup.booleanValue()) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg =
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861529#comment-15861529
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r100574299
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,79 +274,97 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = GlobalLock.getInternLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+boolean rollBackState = false;
+boolean hasDedicatedResources = false;
+
+try {
+long ownerId = domain.getAccountId();
+if ((cleanup != null) && cleanup.booleanValue()) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
--- End diff --
What about extracting this "else block" to a method?
Lines 308-340
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project:
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861530#comment-15861530
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r100574038
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,79 +274,97 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = GlobalLock.getInternLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+boolean rollBackState = false;
+boolean hasDedicatedResources = false;
+
+try {
+long ownerId = domain.getAccountId();
+if ((cleanup != null) && cleanup.booleanValue()) {
--- End diff --
What about using "org.apache.commons.lang.BooleanUtils.toBoolean(Boolean)"
here?
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861531#comment-15861531
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r100575967
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,79 +274,97 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = GlobalLock.getInternLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+boolean rollBackState = false;
+boolean hasDedicatedResources = false;
+
+try {
+long ownerId = domain.getAccountId();
+if ((cleanup != null) && cleanup.booleanValue()) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg =
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861192#comment-15861192
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has
been kicked to run smoke tests
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861191#comment-15861191
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user borisstoyanov commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@blueorangutan test
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is already removed
> 2017-01-26
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861153#comment-15861153
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1935
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-478
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861125#comment-15861125
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep
you posted as I make progress.
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861121#comment-15861121
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user borisstoyanov commented on the issue:
https://github.com/apache/cloudstack/pull/1935
@blueorangutan package
> Delete domain failure due to Account Cleanup task
> -
>
> Key: CLOUDSTACK-9764
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9764
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
>Affects Versions: 4.10.0.0
>Reporter: Nicolas Vazquez
>Assignee: Nicolas Vazquez
> Fix For: 4.10.0.0
>
>
> It was noticed in production environments that {{deleteDomain}} task failed
> for domains with multiple accounts and resources. Examining logs it was found
> out that if Account Cleanup Task got executed after domain (and all of its
> subchilds) got marked as Inactive; and before delete domain task finishes, it
> produces a failure.
> {{AccountCleanupTask}} gets executed every {{account.cleanup.interval}}
> seconds looking for:
> * Removed accounts
> * Disabled accounts
> * Inactive domains
> As {{deleteDomain}} marks domain to delete (and its subchilds) as Inactive
> before deleting them, when {{AccountCleanupTask}} is executed, it removes
> marked domains. When there are resources to cleanup on domain accounts,
> domain is not found throwing exception:
> {{com.cloud.exception.InvalidParameterValueException: Please specify a valid
> domain ID}}
> h3. Example
> {{account.cleanup.interval}} = 100
> {noformat}
> 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET
> command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475
> ...
> // Domain and its subchilds marked as Inactive
> 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27
> as Inactive before actually deleting it
> 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=27
> 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=28
> 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl]
> (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain
> id=29
> ...
> // AccountCleanupTask removes Inactive domain id=29, no rollback for it
> 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup
> 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup
> 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup
> 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27
> 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28
> 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction]
> (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name =
> AccountChecker-1; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29
> 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl]
> (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup
> ...
> // Failure due to domain is already removed
> 2017-01-26
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860824#comment-15860824
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user koushik-das commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r100478202
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,79 +274,97 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = GlobalLock.getInternLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+boolean rollBackState = false;
+boolean hasDedicatedResources = false;
+
+try {
+long ownerId = domain.getAccountId();
+if ((cleanup != null) && cleanup.booleanValue()) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg = networkIds.size() + "
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860817#comment-15860817
]
ASF GitHub Bot commented on CLOUDSTACK-9764:
Github user koushik-das commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1935#discussion_r100477949
--- Diff: server/src/com/cloud/user/DomainManagerImpl.java ---
@@ -273,79 +274,97 @@ public boolean deleteDomain(long domainId, Boolean
cleanup) {
@Override
public boolean deleteDomain(DomainVO domain, Boolean cleanup) {
-// mark domain as inactive
-s_logger.debug("Marking domain id=" + domain.getId() + " as " +
Domain.State.Inactive + " before actually deleting it");
-domain.setState(Domain.State.Inactive);
-_domainDao.update(domain.getId(), domain);
-boolean rollBackState = false;
-boolean hasDedicatedResources = false;
+GlobalLock lock = GlobalLock.getInternLock("AccountCleanup");
+if (lock == null) {
+s_logger.debug("Couldn't get the global lock");
+return false;
+}
+
+if (!lock.lock(30)) {
+s_logger.debug("Couldn't lock the db");
+return false;
+}
try {
-long ownerId = domain.getAccountId();
-if ((cleanup != null) && cleanup.booleanValue()) {
-if (!cleanupDomain(domain.getId(), ownerId)) {
-rollBackState = true;
-CloudRuntimeException e =
-new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
-domain.getId() + ").");
-e.addProxyObject(domain.getUuid(), "domainId");
-throw e;
-}
-} else {
-//don't delete the domain if there are accounts set for
cleanup, or non-removed networks exist, or domain has dedicated resources
-List networkIds =
_networkDomainDao.listNetworkIdsByDomain(domain.getId());
-List accountsForCleanup =
_accountDao.findCleanupsForRemovedAccounts(domain.getId());
-List dedicatedResources =
_dedicatedDao.listByDomainId(domain.getId());
-if (dedicatedResources != null &&
!dedicatedResources.isEmpty()) {
-s_logger.error("There are dedicated resources for the
domain " + domain.getId());
-hasDedicatedResources = true;
-}
-if (accountsForCleanup.isEmpty() && networkIds.isEmpty()
&& !hasDedicatedResources) {
-_messageBus.publish(_name,
MESSAGE_PRE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
-if (!_domainDao.remove(domain.getId())) {
+// mark domain as inactive
+s_logger.debug("Marking domain id=" + domain.getId() + " as "
+ Domain.State.Inactive + " before actually deleting it");
+domain.setState(Domain.State.Inactive);
+_domainDao.update(domain.getId(), domain);
+boolean rollBackState = false;
+boolean hasDedicatedResources = false;
+
+try {
+long ownerId = domain.getAccountId();
+if ((cleanup != null) && cleanup.booleanValue()) {
+if (!cleanupDomain(domain.getId(), ownerId)) {
rollBackState = true;
CloudRuntimeException e =
-new CloudRuntimeException("Delete failed on
domain " + domain.getName() + " (id: " + domain.getId() +
-"); Please make sure all users and sub
domains have been removed from the domain before deleting");
+new CloudRuntimeException("Failed to clean up
domain resources and sub domains, delete failed on domain " + domain.getName()
+ " (id: " +
+domain.getId() + ").");
e.addProxyObject(domain.getUuid(), "domainId");
throw e;
}
-_messageBus.publish(_name,
MESSAGE_REMOVE_DOMAIN_EVENT, PublishScope.LOCAL, domain);
} else {
-rollBackState = true;
-String msg = null;
-if (!accountsForCleanup.isEmpty()) {
-msg = accountsForCleanup.size() + " accounts to
cleanup";
-} else if (!networkIds.isEmpty()) {
-msg = networkIds.size() + "
[jira] [Commented] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task
[ https://issues.apache.org/jira/browse/CLOUDSTACK-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854181#comment-15854181 ] ASF GitHub Bot commented on CLOUDSTACK-9764: GitHub user nvazquez opened a pull request: https://github.com/apache/cloudstack/pull/1935 CLOUDSTACK-9764: Delete domain failure due to Account Cleanup task It was noticed in production environments that `deleteDomain` task failed for domains with multiple accounts and resources. Examining logs it was found out that if Account Cleanup Task got executed after domain (and all of its subchilds) got marked as Inactive; and before delete domain task finishes, it produces a failure. `AccountCleanupTask` gets executed every `account.cleanup.interval` seconds looking for: * Removed accounts * Disabled accounts * Inactive domains As `deleteDomain` marks domain to delete (and its subchilds) as Inactive before deleting them, when `AccountCleanupTask` is executed, it removes marked domains. When there are resources to cleanup on domain accounts, domain is not found throwing exception: `com.cloud.exception.InvalidParameterValueException: Please specify a valid domain ID` ### Example `account.cleanup.interval` = 100 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet] (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET command=deleteDomain=1910a3dc-6fa6-457b-ab3a-602b0cfb6686=true=json&_=1485439623475 ... // Domain and its subchilds marked as Inactive 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl] (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27 as Inactive before actually deleting it 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl] (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain id=27 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl] (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain id=28 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl] (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain id=29 ... // AccountCleanupTask removes Inactive domain id=29, no rollback for it 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction] (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name = AccountChecker-1; called by -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction] (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name = AccountChecker-1; called by -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup ... // Failure due to domain is already removed 2017-01-26 06:07:46,369 WARN [cloud.user.AccountManagerImpl] (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Failed to cleanup account Acct[6a6e63ad-d89b-4a53-b3ae-1c06ea3d1899-ac2] due to com.cloud.exception.InvalidParameterValueException: Please specify a valid domain ID. at com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateResourceCount(ResourceLimitManagerImpl.java:752) at
