Hi, It seems Jenkins build is back to normal now. I've triggered another build just to make sure. Created a JIRA to track the registry exception handling issue at [1]. Appreciate if someone can go through the commit [2] and review the changes.
This will only make sure registry exceptions are properly handled but it does not explain the root cause of this issue. After going through the configuration changes caused by assembly descriptor issue, it was noticed that <versionResourcesOnChange> parameter in registry.xml is set to true in default registry.xml (coming from the p2 profile) and this is set to false in Stratos maintained registry.xml. This resource versioning feature is not needed in Stratos and there are some issues around it. It is possible that this issue occurred due to that. [1] https://issues.apache.org/jira/browse/STRATOS-1565 [2] https://github.com/apache/stratos/commit/0b913ebdcb40d173d2fa4548c7e8e4217dfa4e6a Thanks. On Wed, Sep 16, 2015 at 11:07 AM, Gayan Gunarathne <gay...@wso2.com> wrote: > Great. That registry locks without releasing may cause this intermittent > issue. Will see with the new build. As there are no such a big functional > changes from 4.1.2 to 4.1.3, we can check this integration test against > 4.1.2 release as we don't experience this issue with 4.1.2 > > Thanks, > Gayan > > > On Wed, Sep 16, 2015 at 10:41 AM, Akila Ravihansa Perera < > raviha...@wso2.com> wrote: > >> Hi Gayan, >> >> Yes, this was my mistake. Sorry for the inconvenience. I've fixed it now. >> >> While going through Carbon registry transactions one by one, I noticed >> that some transactions are not handled properly. There were cases in which >> registry exception could permanently leave a lock without releasing. I've >> fixed those and now we can get the actual stacktrace when a registry >> exception occurs. I've triggered a new build. Let's see if it fixes the >> problem. >> >> Thanks. >> >> On Wed, Sep 16, 2015 at 10:36 AM, Gayan Gunarathne <gay...@wso2.com> >> wrote: >> >>> yeah.Seems we are catching the wrong exception. >>> >>> StratosApiV41Utils.java >>> try { >>> >>> autoscalerServiceClient.undeployApplication(applicationId, force); >>> >>> try { >>> clearMetadata(applicationId); >>> } catch (RegistryException e) { >>> String message = "Could not remove application >>> metadata: [application-id] " + applicationId; >>> log.error(message, e); >>> throw new RestAPIException(message, e); >>> } >>> } >>> >>> We need to catch the RestAPIException >>> >>> Thanks, >>> Gayan >>> >>> On Wed, Sep 16, 2015 at 9:44 AM, Imesh Gunaratne <im...@apache.org> >>> wrote: >>> >>>> I'm now seeing below error in 4.1.x branch: >>>> >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [INFO] BUILD FAILURE >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [INFO] Total time: 1:48.133s >>>> [INFO] Finished at: Wed Sep 16 09:42:32 IST 2015 >>>> [INFO] Final Memory: 438M/1073M >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [ERROR] Failed to execute goal >>>> org.apache.maven.plugins:maven-compiler-plugin:2.3.1:compile >>>> (default-compile) on project org.apache.stratos.rest.endpoint: Compilation >>>> failure >>>> [ERROR] >>>> /Users/imesh/dev/apache/stratos/stratos/components/org.apache.stratos.rest.endpoint/src/main/java/org/apache/stratos/rest/endpoint/api/StratosApiV41Utils.java:[1906,18] >>>> error: exception RegistryException is never thrown in body of corresponding >>>> try statement >>>> [ERROR] -> [Help 1] >>>> >>>> On Wed, Sep 16, 2015 at 12:23 AM, Akila Ravihansa Perera < >>>> raviha...@wso2.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm still not able to reproduce this issue locally. I've run >>>>> integration tests multiple times and they are successful at every time. I >>>>> compared the p2-profile and and configuration files between Stratos 4.1.2 >>>>> and Stratos 4.1.3-SNAPSHOT, and there are no differences in p2-profile. >>>>> Same dependency jar files exist in repository/plugins directory. But I >>>>> found some minor changes to configuration files (packed inside >>>>> repository/conf). This was due to some issues in distribution descriptor >>>>> file (assembly.xml used by maven-assembly plugin). I've fixed that and now >>>>> the configuration matches that of Stratos 4.1.2. >>>>> >>>>> Can someone please pull the latest from Stratos 4.1.x branch and try >>>>> to build with tests? >>>>> >>>>> Thanks. >>>>> >>>>> >>>>> On Tue, Sep 15, 2015 at 12:13 PM, Akila Ravihansa Perera < >>>>> raviha...@wso2.com> wrote: >>>>> >>>>>> Hi Gayan, >>>>>> >>>>>> lock timeout is already set to 60s which is not the problem I think. >>>>>> It seems this is a known issue in Carbon registry [1]. There is a fix >>>>>> suggested in wso2-dev mailing list which is to set the mvcc option. I'm >>>>>> currently trying that option locally. >>>>>> >>>>>> [1] https://wso2.org/jira/browse/REGISTRY-2715 >>>>>> >>>>>> Thanks. >>>>>> >>>>>> >>>>>> On Tue, Sep 15, 2015 at 12:04 PM, Gayan Gunarathne <gay...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Seems this is a intermittent issue.Seems to be that another process >>>>>>> has locked the table, and I think we may need to try with increasing the >>>>>>> lock timeout and multi-version concurrency option[1][2] >>>>>>> >>>>>>> [1]http://h2database.com/html/grammar.html#set_lock_timeout >>>>>>> [2]http://www.h2database.com/html/advanced.html#mvcc >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Gayan >>>>>>> >>>>>>> On Tue, Sep 15, 2015 at 7:22 AM, Dinithi De Silva <dinit...@wso2.com >>>>>>> > wrote: >>>>>>> >>>>>>>> It looks like this is the same issue which Udara has reported >>>>>>>> sometime back [1]. >>>>>>>> >>>>>>>> [1] - Registry issue when persisting topology >>>>>>>> >>>>>>>> On Tue, Sep 15, 2015 at 7:20 AM, Dinithi De Silva < >>>>>>>> dinit...@wso2.com> wrote: >>>>>>>> >>>>>>>>> It looks like this is the same issue which Udara has reported >>>>>>>>> sometime back [1]. >>>>>>>>> >>>>>>>>> [1] - Registry issue when persisting topology >>>>>>>>> >>>>>>>>> On Mon, Sep 14, 2015 at 6:38 PM, Imesh Gunaratne <im...@apache.org >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Hi Devs, >>>>>>>>>> >>>>>>>>>> I see the following issue in 4.1.x branch: >>>>>>>>>> >>>>>>>>>> INFO - [2015-09-14 06:39:49,522] INFO >>>>>>>>>> {org.apache.stratos.mock.iaas.services.impl.MockIaasServiceImpl} - >>>>>>>>>> Terminating instance: [instance-id] >>>>>>>>>> sample-applications-test-1.c1-1x0-sample-applications-test.c1-sample-applications-test.domaina2d8a0c8-a1dd-4e3e-84d5-cadc56b0e58e >>>>>>>>>> INFO - [2015-09-14 06:39:49,568] INFO >>>>>>>>>> {org.apache.stratos.mock.iaas.services.impl.MockInstance} - Mock >>>>>>>>>> member >>>>>>>>>> terminated: [member-id] >>>>>>>>>> sample-applications-test-1.c1-1x0-sample-applications-test.c1-sample-applications-test.domaina2d8a0c8-a1dd-4e3e-84d5-cadc56b0e58e >>>>>>>>>> INFO - [2015-09-14 06:40:08,147] INFO >>>>>>>>>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - >>>>>>>>>> Faulty >>>>>>>>>> member detected [member-id] >>>>>>>>>> partition-round-robin-test.my-c7.c7-partition-round-robin-test.domain5379abc6-b698-4a11-962b-7581d12cc99f >>>>>>>>>> with [last time-stamp] 1442192853747 [time-out] 60000 milliseconds >>>>>>>>>> INFO - [2015-09-14 06:40:08,147] WARN >>>>>>>>>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - >>>>>>>>>> Failed >>>>>>>>>> to publish member fault event. Member having [member-id] >>>>>>>>>> partition-round-robin-test.my-c7.c7-partition-round-robin-test.domain5379abc6-b698-4a11-962b-7581d12cc99f >>>>>>>>>> does not exist in topology >>>>>>>>>> INFO - [2015-09-14 06:40:08,147] INFO >>>>>>>>>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - >>>>>>>>>> Faulty >>>>>>>>>> member detected [member-id] >>>>>>>>>> partition-round-robin-test.my-c7.c7-partition-round-robin-test.domaincc424d16-4aaa-4321-bd53-9d0721f8cdad >>>>>>>>>> with [last time-stamp] 1442192778658 [time-out] 60000 milliseconds >>>>>>>>>> INFO - [2015-09-14 06:40:08,147] WARN >>>>>>>>>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - >>>>>>>>>> Failed >>>>>>>>>> to publish member fault event. Member having [member-id] >>>>>>>>>> partition-round-robin-test.my-c7.c7-partition-round-robin-test.domaincc424d16-4aaa-4321-bd53-9d0721f8cdad >>>>>>>>>> does not exist in topology >>>>>>>>>> INFO - [2015-09-14 06:40:49,535] ERROR >>>>>>>>>> {org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} - Failed >>>>>>>>>> to check >>>>>>>>>> the existence of the resource >>>>>>>>>> /_system/governance/mock.iaas/instances. >>>>>>>>>> Timeout trying to lock table "REG_RESOURCE"; SQL statement: >>>>>>>>>> INFO - SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? >>>>>>>>>> AND REG_NAME=? AND REG_TENANT_ID=? [50200-140] >>>>>>>>>> INFO - org.h2.jdbc.JdbcSQLException: Timeout trying to lock table >>>>>>>>>> "REG_RESOURCE"; SQL statement: >>>>>>>>>> INFO - SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? >>>>>>>>>> AND REG_NAME=? AND REG_TENANT_ID=? [50200-140] >>>>>>>>>> INFO - at >>>>>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327) >>>>>>>>>> INFO - at org.h2.message.DbException.get(DbException.java:167) >>>>>>>>>> INFO - at org.h2.message.DbException.get(DbException.java:144) >>>>>>>>>> INFO - at org.h2.table.RegularTable.doLock(RegularTable.java:466) >>>>>>>>>> INFO - at org.h2.table.RegularTable.lock(RegularTable.java:404) >>>>>>>>>> INFO - at org.h2.table.TableFilter.lock(TableFilter.java:139) >>>>>>>>>> INFO - at >>>>>>>>>> org.h2.command.dml.Select.queryWithoutCache(Select.java:554) >>>>>>>>>> INFO - at org.h2.command.dml.Query.query(Query.java:241) >>>>>>>>>> INFO - at >>>>>>>>>> org.h2.command.CommandContainer.query(CommandContainer.java:80) >>>>>>>>>> INFO - at org.h2.command.Command.executeQuery(Command.java:132) >>>>>>>>>> INFO - at >>>>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96) >>>>>>>>>> INFO - at >>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197) >>>>>>>>>> INFO - at >>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147) >>>>>>>>>> INFO - at >>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156) >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Imesh Gunaratne >>>>>>>>>> >>>>>>>>>> Senior Technical Lead, WSO2 >>>>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Dinithi De Silva* >>>>>>>>> Associate Software Engineer, WSO2 Inc. >>>>>>>>> m:+94716667655 | e:dinit...@wso2.com | w: www.wso2.com >>>>>>>>> | a: #20, Palm Grove, Colombo 03 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Dinithi De Silva* >>>>>>>> Associate Software Engineer, WSO2 Inc. >>>>>>>> m:+94716667655 | e:dinit...@wso2.com | w: www.wso2.com >>>>>>>> | a: #20, Palm Grove, Colombo 03 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Gayan Gunarathne >>>>>>> Technical Lead, WSO2 Inc. (http://wso2.com) >>>>>>> Committer & PMC Member, Apache Stratos >>>>>>> email : gay...@wso2.com | mobile : +94 775030545 >>>>>>> <%2B94%20766819985> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Akila Ravihansa Perera >>>>>> WSO2 Inc.; http://wso2.com/ >>>>>> >>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Akila Ravihansa Perera >>>>> WSO2 Inc.; http://wso2.com/ >>>>> >>>>> Blog: http://ravihansa3000.blogspot.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Imesh Gunaratne >>>> >>>> Senior Technical Lead, WSO2 >>>> Committer & PMC Member, Apache Stratos >>>> >>> >>> >>> >>> -- >>> >>> Gayan Gunarathne >>> Technical Lead, WSO2 Inc. (http://wso2.com) >>> Committer & PMC Member, Apache Stratos >>> email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985> >>> >>> >>> >> >> >> >> -- >> Akila Ravihansa Perera >> WSO2 Inc.; http://wso2.com/ >> >> Blog: http://ravihansa3000.blogspot.com >> > > > > -- > > Gayan Gunarathne > Technical Lead, WSO2 Inc. (http://wso2.com) > Committer & PMC Member, Apache Stratos > email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985> > > > -- Akila Ravihansa Perera WSO2 Inc.; http://wso2.com/ Blog: http://ravihansa3000.blogspot.com
