Hi Gayan,
Yes, this was my mistake. Sorry for the inconvenience. I've fixed it now.
While going through Carbon registry transactions one by one, I noticed that
some transactions are not handled properly. There were cases in which
registry exception could permanently leave a lock without releasing. I've
fixed those and now we can get the actual stacktrace when a registry
exception occurs. I've triggered a new build. Let's see if it fixes the
problem.
Thanks.
On Wed, Sep 16, 2015 at 10:36 AM, Gayan Gunarathne <gay...@wso2.com> wrote:
> yeah.Seems we are catching the wrong exception.
>
> StratosApiV41Utils.java
> try {
> autoscalerServiceClient.undeployApplication(applicationId,
> force);
>
> try {
> clearMetadata(applicationId);
> } catch (RegistryException e) {
> String message = "Could not remove application
> metadata: [application-id] " + applicationId;
> log.error(message, e);
> throw new RestAPIException(message, e);
> }
> }
>
> We need to catch the RestAPIException
>
> Thanks,
> Gayan
>
> On Wed, Sep 16, 2015 at 9:44 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> I'm now seeing below error in 4.1.x branch:
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 1:48.133s
>> [INFO] Finished at: Wed Sep 16 09:42:32 IST 2015
>> [INFO] Final Memory: 438M/1073M
>> [INFO]
>> ------------------------------------------------------------------------
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-compiler-plugin:2.3.1:compile
>> (default-compile) on project org.apache.stratos.rest.endpoint: Compilation
>> failure
>> [ERROR]
>> /Users/imesh/dev/apache/stratos/stratos/components/org.apache.stratos.rest.endpoint/src/main/java/org/apache/stratos/rest/endpoint/api/StratosApiV41Utils.java:[1906,18]
>> error: exception RegistryException is never thrown in body of corresponding
>> try statement
>> [ERROR] -> [Help 1]
>>
>> On Wed, Sep 16, 2015 at 12:23 AM, Akila Ravihansa Perera <
>> raviha...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> I'm still not able to reproduce this issue locally. I've run integration
>>> tests multiple times and they are successful at every time. I compared the
>>> p2-profile and and configuration files between Stratos 4.1.2 and Stratos
>>> 4.1.3-SNAPSHOT, and there are no differences in p2-profile. Same dependency
>>> jar files exist in repository/plugins directory. But I found some minor
>>> changes to configuration files (packed inside repository/conf). This was
>>> due to some issues in distribution descriptor file (assembly.xml used by
>>> maven-assembly plugin). I've fixed that and now the configuration matches
>>> that of Stratos 4.1.2.
>>>
>>> Can someone please pull the latest from Stratos 4.1.x branch and try to
>>> build with tests?
>>>
>>> Thanks.
>>>
>>>
>>> On Tue, Sep 15, 2015 at 12:13 PM, Akila Ravihansa Perera <
>>> raviha...@wso2.com> wrote:
>>>
>>>> Hi Gayan,
>>>>
>>>> lock timeout is already set to 60s which is not the problem I think. It
>>>> seems this is a known issue in Carbon registry [1]. There is a fix
>>>> suggested in wso2-dev mailing list which is to set the mvcc option. I'm
>>>> currently trying that option locally.
>>>>
>>>> [1] https://wso2.org/jira/browse/REGISTRY-2715
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Tue, Sep 15, 2015 at 12:04 PM, Gayan Gunarathne <gay...@wso2.com>
>>>> wrote:
>>>>
>>>>> Seems this is a intermittent issue.Seems to be that another process
>>>>> has locked the table, and I think we may need to try with increasing the
>>>>> lock timeout and multi-version concurrency option[1][2]
>>>>>
>>>>> [1]http://h2database.com/html/grammar.html#set_lock_timeout
>>>>> [2]http://www.h2database.com/html/advanced.html#mvcc
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Gayan
>>>>>
>>>>> On Tue, Sep 15, 2015 at 7:22 AM, Dinithi De Silva <dinit...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> It looks like this is the same issue which Udara has reported
>>>>>> sometime back [1].
>>>>>>
>>>>>> [1] - Registry issue when persisting topology
>>>>>>
>>>>>> On Tue, Sep 15, 2015 at 7:20 AM, Dinithi De Silva <dinit...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> It looks like this is the same issue which Udara has reported
>>>>>>> sometime back [1].
>>>>>>>
>>>>>>> [1] - Registry issue when persisting topology
>>>>>>>
>>>>>>> On Mon, Sep 14, 2015 at 6:38 PM, Imesh Gunaratne <im...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Devs,
>>>>>>>>
>>>>>>>> I see the following issue in 4.1.x branch:
>>>>>>>>
>>>>>>>> INFO - [2015-09-14 06:39:49,522] INFO
>>>>>>>> {org.apache.stratos.mock.iaas.services.impl.MockIaasServiceImpl} -
>>>>>>>> Terminating instance: [instance-id]
>>>>>>>> sample-applications-test-1.c1-1x0-sample-applications-test.c1-sample-applications-test.domaina2d8a0c8-a1dd-4e3e-84d5-cadc56b0e58e
>>>>>>>> INFO - [2015-09-14 06:39:49,568] INFO
>>>>>>>> {org.apache.stratos.mock.iaas.services.impl.MockInstance} - Mock
>>>>>>>> member
>>>>>>>> terminated: [member-id]
>>>>>>>> sample-applications-test-1.c1-1x0-sample-applications-test.c1-sample-applications-test.domaina2d8a0c8-a1dd-4e3e-84d5-cadc56b0e58e
>>>>>>>> INFO - [2015-09-14 06:40:08,147] INFO
>>>>>>>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} -
>>>>>>>> Faulty
>>>>>>>> member detected [member-id]
>>>>>>>> partition-round-robin-test.my-c7.c7-partition-round-robin-test.domain5379abc6-b698-4a11-962b-7581d12cc99f
>>>>>>>> with [last time-stamp] 1442192853747 [time-out] 60000 milliseconds
>>>>>>>> INFO - [2015-09-14 06:40:08,147] WARN
>>>>>>>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} -
>>>>>>>> Failed
>>>>>>>> to publish member fault event. Member having [member-id]
>>>>>>>> partition-round-robin-test.my-c7.c7-partition-round-robin-test.domain5379abc6-b698-4a11-962b-7581d12cc99f
>>>>>>>> does not exist in topology
>>>>>>>> INFO - [2015-09-14 06:40:08,147] INFO
>>>>>>>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} -
>>>>>>>> Faulty
>>>>>>>> member detected [member-id]
>>>>>>>> partition-round-robin-test.my-c7.c7-partition-round-robin-test.domaincc424d16-4aaa-4321-bd53-9d0721f8cdad
>>>>>>>> with [last time-stamp] 1442192778658 [time-out] 60000 milliseconds
>>>>>>>> INFO - [2015-09-14 06:40:08,147] WARN
>>>>>>>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} -
>>>>>>>> Failed
>>>>>>>> to publish member fault event. Member having [member-id]
>>>>>>>> partition-round-robin-test.my-c7.c7-partition-round-robin-test.domaincc424d16-4aaa-4321-bd53-9d0721f8cdad
>>>>>>>> does not exist in topology
>>>>>>>> INFO - [2015-09-14 06:40:49,535] ERROR
>>>>>>>> {org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} - Failed to
>>>>>>>> check
>>>>>>>> the existence of the resource /_system/governance/mock.iaas/instances.
>>>>>>>> Timeout trying to lock table "REG_RESOURCE"; SQL statement:
>>>>>>>> INFO - SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>> INFO - org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
>>>>>>>> "REG_RESOURCE"; SQL statement:
>>>>>>>> INFO - SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>> INFO - at
>>>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>>>>>> INFO - at org.h2.message.DbException.get(DbException.java:167)
>>>>>>>> INFO - at org.h2.message.DbException.get(DbException.java:144)
>>>>>>>> INFO - at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>>>>>> INFO - at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>>>>>> INFO - at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>>>>>> INFO - at
>>>>>>>> org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>>>>>> INFO - at org.h2.command.dml.Query.query(Query.java:241)
>>>>>>>> INFO - at
>>>>>>>> org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>>>>>> INFO - at org.h2.command.Command.executeQuery(Command.java:132)
>>>>>>>> INFO - at
>>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>>>>>> INFO - at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>>>>>> INFO - at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>>>>>> INFO - at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> --
>>>>>>>> Imesh Gunaratne
>>>>>>>>
>>>>>>>> Senior Technical Lead, WSO2
>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Dinithi De Silva*
>>>>>>> Associate Software Engineer, WSO2 Inc.
>>>>>>> m:+94716667655 | e:dinit...@wso2.com | w: www.wso2.com
>>>>>>> | a: #20, Palm Grove, Colombo 03
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Dinithi De Silva*
>>>>>> Associate Software Engineer, WSO2 Inc.
>>>>>> m:+94716667655 | e:dinit...@wso2.com | w: www.wso2.com
>>>>>> | a: #20, Palm Grove, Colombo 03
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Gayan Gunarathne
>>>>> Technical Lead, WSO2 Inc. (http://wso2.com)
>>>>> Committer & PMC Member, Apache Stratos
>>>>> email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Akila Ravihansa Perera
>>>> WSO2 Inc.; http://wso2.com/
>>>>
>>>> Blog: http://ravihansa3000.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> Akila Ravihansa Perera
>>> WSO2 Inc.; http://wso2.com/
>>>
>>> Blog: http://ravihansa3000.blogspot.com
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Senior Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead, WSO2 Inc. (http://wso2.com)
> Committer & PMC Member, Apache Stratos
> email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
>
>
>
--
Akila Ravihansa Perera
WSO2 Inc.; http://wso2.com/
Blog: http://ravihansa3000.blogspot.com
