Re: [Discuss]Business exception cannot be rollbackedinShardingSpherewith saga transaction.
If there is SQL execution error before the commit. ShardingSphere will throw the exception to user if policy is backward. User catches the exception and call rollback generally. And if policy is forward, it means user think SQL should be executed successfully. So ShardingSphere will catch the SQLException and go ahead. After user committing, the failed SQL will retry. Currently, my plan is ShardingSphere should get snapshot and do SQL user given, and saga-actuator only is responsible for control recovery. -- Yi Yang (Sion) Apache ShardingSphere contributor -- Original -- From: "Willem Jiang"; Date: Fri, Jan 25, 2019 02:25 PM To: "dev"; Cc: "dev"; Subject: Re: [Discuss]Business exception cannot be rollbackedinShardingSpherewith saga transaction. It looks like the ShardingSphere hijack the SQL execution before the commit or the rollback call?? what if there is SQL execution error before the commit. What's your plan to let saga-actuator interact with ShardingSphere. Willem Jiang Twitter: willemjiang Weibo: willem On Fri, Jan 25, 2019 at 10:06 AM tsubasaotl wrote: > > When users execute SQL with saga transaction, ShardingSphere will execute the > SQL immediately. > After `commit` or `rollback` called, ShardingSphere starts to generate the > saga execution graph. > At this time, all SQL and their order in transaction has be known by > ShardingSphere, > so the saga execution graph can be generated. > > > Specially, for `rollback`, ShardingSphere add a rollback request at the end > of graph. > > > > -- > Yi Yang (Sion) > Apache ShardingSphere contributor > > > > > > -- Original -- > From: "Willem Jiang"; > Date: Fri, Jan 25, 2019 08:50 AM > To: "dev"; > Cc: "dev"; > Subject: Re: [Discuss]Business exception cannot be rollbacked > inShardingSpherewith saga transaction. > > > > Hi Sion, > > My question is how can we generate the saga execution graph with the > rollback request at the first place? > > Willem Jiang > > Twitter: willemjiang > Weibo: willem > > On Thu, Jan 24, 2019 at 6:04 PM tsubasaotl wrote: > > > > I think we think it too complicated. In fact, no matter what the exception > > is, As long as the rollback is called, a rollback request is added at the > > end of the call graph of saga. The actuator reads the rollback request and > > starts to reverse the compensation. The saga definition should be > > > > > > { > > "policy": "BackwardRecovery", > > "requests": [{ > > "id": "xxx", > > "datasource": "ds", > > "type": "sql", > > "transaction": { > > ... > > }, > > "compensation": { > > ... > > }, > > "parents": [], > > }, > > ... > > ,{ > > "id": "yyy", > > "datasource": "ds", > > "type": "sql", > > "transaction": { > > "sql": "ROLLBACK_EVENT", > > ... > > }, > > "compensation": { > > ... > > }, > > "parents": [the last ids of actual requests], > > }] > > } > > > > > > > > -- > > Yi Yang (Sion) > > Apache ShardingSphere contributor > > > > > > > > > > > > -- Original -- > > From: "Zheng Feng"; > > Date: Fri, Jan 18, 2019 10:26 AM > > To: "dev"; > > Cc: "dev"; > > Subject: Re: [Discuss]Business exception cannot be rollbacked > > inShardingSphere with saga transaction. > > > > > > > > I think it could be be resolved in ShardingSphere and catch the business > > exception and mark the cached result "ROLLBACK_ONLY". > > Anyway, this should be done before submitting to the saga actor. > > > > tsubasaotl ??2019??1??16?? 8:06?? > > > > > Hi, everyone. > > > > > > > > > In ShardingSphere, SQL and their execution result will be cached in saga > > > transaction manager. > > > When users call `commit` or `rollback` method, the cached SQL will > > > generate `SagaDefinition` > > > and submit it to the saga actuator. > > > > > > > > > Saga actuator do `Transaction` first and get execution result from cached. > > > When the result is successful, the actuator will directly judge that the > > > Transaction is successful > > > and execute the next Transaction. > > > When the result not found in cached or the result is failed, the actuator > > > will do retry or compensation > > > according to configuration. > > > > > > > > > But there is a problem that saga actuator cannot rollback when business > > > exception happened. > > > The situation will happen in following workflow. > > > > > > > > >---multiple times--- > > > begin transaction --> | exec
Re: [Discuss]Business exception cannot be rollbacked inShardingSpherewith saga transaction.
It looks like the ShardingSphere hijack the SQL execution before the commit or the rollback call, what if there is SQL execution error before the commit. What's your plan to let saga-actuator interact with ShardingSphere. Willem Jiang Twitter: willemjiang Weibo: 姜宁willem On Fri, Jan 25, 2019 at 10:06 AM tsubasaotl wrote: > > When users execute SQL with saga transaction, ShardingSphere will execute the > SQL immediately. > After `commit` or `rollback` called, ShardingSphere starts to generate the > saga execution graph. > At this time, all SQL and their order in transaction has be known by > ShardingSphere, > so the saga execution graph can be generated. > > > Specially, for `rollback`, ShardingSphere add a rollback request at the end > of graph. > > > > -- > Yi Yang (Sion) > Apache ShardingSphere contributor > > > > > > -- Original -- > From: "Willem Jiang"; > Date: Fri, Jan 25, 2019 08:50 AM > To: "dev"; > Cc: "dev"; > Subject: Re: [Discuss]Business exception cannot be rollbacked > inShardingSpherewith saga transaction. > > > > Hi Sion, > > My question is how can we generate the saga execution graph with the > rollback request at the first place? > > Willem Jiang > > Twitter: willemjiang > Weibo: 姜宁willem > > On Thu, Jan 24, 2019 at 6:04 PM tsubasaotl wrote: > > > > I think we think it too complicated. In fact, no matter what the exception > > is, As long as the rollback is called, a rollback request is added at the > > end of the call graph of saga. The actuator reads the rollback request and > > starts to reverse the compensation. The saga definition should be > > > > > > { > > "policy": "BackwardRecovery", > > "requests": [{ > > "id": "xxx", > > "datasource": "ds", > > "type": "sql", > > "transaction": { > > ... > > }, > > "compensation": { > > ... > > }, > > "parents": [], > > }, > > ... > > ,{ > > "id": "yyy", > > "datasource": "ds", > > "type": "sql", > > "transaction": { > > "sql": "ROLLBACK_EVENT", > > ... > > }, > > "compensation": { > > ... > > }, > > "parents": [the last ids of actual requests], > > }] > > } > > > > > > > > -- > > Yi Yang (Sion) > > Apache ShardingSphere contributor > > > > > > > > > > > > -- Original -- > > From: "Zheng Feng"; > > Date: Fri, Jan 18, 2019 10:26 AM > > To: "dev"; > > Cc: "dev"; > > Subject: Re: [Discuss]Business exception cannot be rollbacked > > inShardingSphere with saga transaction. > > > > > > > > I think it could be be resolved in ShardingSphere and catch the business > > exception and mark the cached result "ROLLBACK_ONLY". > > Anyway, this should be done before submitting to the saga actor. > > > > tsubasaotl 于2019年1月16日周三 下午8:06写道: > > > > > Hi, everyone. > > > > > > > > > In ShardingSphere, SQL and their execution result will be cached in saga > > > transaction manager. > > > When users call `commit` or `rollback` method, the cached SQL will > > > generate `SagaDefinition` > > > and submit it to the saga actuator. > > > > > > > > > Saga actuator do `Transaction` first and get execution result from cached. > > > When the result is successful, the actuator will directly judge that the > > > Transaction is successful > > > and execute the next Transaction. > > > When the result not found in cached or the result is failed, the actuator > > > will do retry or compensation > > > according to configuration. > > > > > > > > > But there is a problem that saga actuator cannot rollback when business > > > exception happened. > > > The situation will happen in following workflow. > > > > > > > > >---multiple times--- > > > begin transaction --> | execute SQL --> cached result | --> throw business > > > exception --> call rollback > > >-- > > > | > > > > > > | > > > > > > | > > > but all SQL success, saga actuator don't run compensation <-- get result > > > from cache <-- saga actuator > > > > > > > > > All SQL is executed successfully, but the business program throws an > > > exception. So the actuator will judge > > > transaction is successful, and not to do compensation, which makes users > > > confused. > > > > > > > > > On this issue, I would like to ask for advice, should resolved in Saga > > > actuator or ShardingSphere? > > > A
Re: [Discuss] Should we support inner class param type defined in REST service class?
A new issue[1] is created. I'll analyse the root cause later. [1] https://issues.apache.org/jira/browse/SCB-1133 Yours sincerely Yao Haishi yhs0...@163.com On 1/25/2019 10:46,wjm wjm wrote: +1 but if too complex, can IMPL it in weak type engine. yhs0092 于2019年1月25日周五 上午9:55写道: Hello, I find out that currently it's not supported to define a inner class type in the REST service class as parameter. Should we allow such use case? For example, a REST service like below will cause javassist.NotFoundException: training.demo.provider.service.TestRestService.TestBodyParam @RestSchema(schemaId = "testSchema") @RequestMapping(path = "test") public class TestRestService { @PostMapping(path = "post") public String post(@RequestBody TestBodyParam body) { return null == body ? "null" : body.toString(); } public static class TestBodyParam { // fields omitted } } But inner class field in an independent param class is OK: @RestSchema(schemaId = "testSchema") @RequestMapping(path = "test") public class TestRestService { @PostMapping(path = "post") public String post(@RequestBody TestBodyParam body) { return null == body ? "null" : body.toString(); } } // define param type in an independent class public class TestBodyParam { private InnerBody innerBody; public static class InnerBody { // fields omitted } // fields omitted } As described above, in some case, the inner class param type works well. If you think this feature should be provided, I'll create a JIRA issue to track it. Yours sincerely Yao Haishi yhs0...@163.com
Re: [Discuss] Should we support inner class param type defined in REST service class?
+1 but if too complex, can IMPL it in weak type engine. yhs0092 于2019年1月25日周五 上午9:55写道: > Hello, I find out that currently it's not supported to define a inner > class type in the REST service class as parameter. Should we allow such use > case? > > > For example, a REST service like below will cause > javassist.NotFoundException: > training.demo.provider.service.TestRestService.TestBodyParam > @RestSchema(schemaId = "testSchema") > @RequestMapping(path = "test") > public class TestRestService { > @PostMapping(path = "post") > public String post(@RequestBody TestBodyParam body) { > return null == body ? "null" : body.toString(); > } > > public static class TestBodyParam { > // fields omitted > } > } > > > But inner class field in an independent param class is OK: > @RestSchema(schemaId = "testSchema") > @RequestMapping(path = "test") > public class TestRestService { > @PostMapping(path = "post") > public String post(@RequestBody TestBodyParam body) { > return null == body ? "null" : body.toString(); > } > } > // define param type in an independent class > public class TestBodyParam { > private InnerBody innerBody; > public static class InnerBody { > > // fields omitted > } > // fields omitted > } > > > As described above, in some case, the inner class param type works well. > If you think this feature should be provided, I'll create a JIRA issue to > track it. > > > Yours sincerely > > > Yao Haishi > yhs0...@163.com > >
Re: [Discuss]Business exception cannot be rollbacked inShardingSpherewith saga transaction.
When users execute SQL with saga transaction, ShardingSphere will execute the SQL immediately. After `commit` or `rollback` called, ShardingSphere starts to generate the saga execution graph. At this time, all SQL and their order in transaction has be known by ShardingSphere, so the saga execution graph can be generated. Specially, for `rollback`, ShardingSphere add a rollback request at the end of graph. -- Yi Yang (Sion) Apache ShardingSphere contributor -- Original -- From: "Willem Jiang"; Date: Fri, Jan 25, 2019 08:50 AM To: "dev"; Cc: "dev"; Subject: Re: [Discuss]Business exception cannot be rollbacked inShardingSpherewith saga transaction. Hi Sion, My question is how can we generate the saga execution graph with the rollback request at the first place? Willem Jiang Twitter: willemjiang Weibo: willem On Thu, Jan 24, 2019 at 6:04 PM tsubasaotl wrote: > > I think we think it too complicated. In fact, no matter what the exception > is, As long as the rollback is called, a rollback request is added at the end > of the call graph of saga. The actuator reads the rollback request and starts > to reverse the compensation. The saga definition should be > > > { > "policy": "BackwardRecovery", > "requests": [{ > "id": "xxx", > "datasource": "ds", > "type": "sql", > "transaction": { > ... > }, > "compensation": { > ... > }, > "parents": [], > }, > ... > ,{ > "id": "yyy", > "datasource": "ds", > "type": "sql", > "transaction": { > "sql": "ROLLBACK_EVENT", > ... > }, > "compensation": { > ... > }, > "parents": [the last ids of actual requests], > }] > } > > > > -- > Yi Yang (Sion) > Apache ShardingSphere contributor > > > > > > -- Original -- > From: "Zheng Feng"; > Date: Fri, Jan 18, 2019 10:26 AM > To: "dev"; > Cc: "dev"; > Subject: Re: [Discuss]Business exception cannot be rollbacked > inShardingSphere with saga transaction. > > > > I think it could be be resolved in ShardingSphere and catch the business > exception and mark the cached result "ROLLBACK_ONLY". > Anyway, this should be done before submitting to the saga actor. > > tsubasaotl ??2019??1??16?? 8:06?? > > > Hi, everyone. > > > > > > In ShardingSphere, SQL and their execution result will be cached in saga > > transaction manager. > > When users call `commit` or `rollback` method, the cached SQL will > > generate `SagaDefinition` > > and submit it to the saga actuator. > > > > > > Saga actuator do `Transaction` first and get execution result from cached. > > When the result is successful, the actuator will directly judge that the > > Transaction is successful > > and execute the next Transaction. > > When the result not found in cached or the result is failed, the actuator > > will do retry or compensation > > according to configuration. > > > > > > But there is a problem that saga actuator cannot rollback when business > > exception happened. > > The situation will happen in following workflow. > > > > > >---multiple times--- > > begin transaction --> | execute SQL --> cached result | --> throw business > > exception --> call rollback > >-- > > | > > > > | > > > > | > > but all SQL success, saga actuator don't run compensation <-- get result > > from cache <-- saga actuator > > > > > > All SQL is executed successfully, but the business program throws an > > exception. So the actuator will judge > > transaction is successful, and not to do compensation, which makes users > > confused. > > > > > > On this issue, I would like to ask for advice, should resolved in Saga > > actuator or ShardingSphere? > > And how to resolve better? > > > > > > -- > > Yi Yang (Sion) > > Apache ShardingSphere contributor
[Discuss] Should we support inner class param type defined in REST service class?
Hello, I find out that currently it's not supported to define a inner class type in the REST service class as parameter. Should we allow such use case? For example, a REST service like below will cause javassist.NotFoundException: training.demo.provider.service.TestRestService.TestBodyParam @RestSchema(schemaId = "testSchema") @RequestMapping(path = "test") public class TestRestService { @PostMapping(path = "post") public String post(@RequestBody TestBodyParam body) { return null == body ? "null" : body.toString(); } public static class TestBodyParam { // fields omitted } } But inner class field in an independent param class is OK: @RestSchema(schemaId = "testSchema") @RequestMapping(path = "test") public class TestRestService { @PostMapping(path = "post") public String post(@RequestBody TestBodyParam body) { return null == body ? "null" : body.toString(); } } // define param type in an independent class public class TestBodyParam { private InnerBody innerBody; public static class InnerBody { // fields omitted } // fields omitted } As described above, in some case, the inner class param type works well. If you think this feature should be provided, I'll create a JIRA issue to track it. Yours sincerely Yao Haishi yhs0...@163.com
Re: [discuss] change default settings of sync invocation executor
@yaohai...@huawei.com think about a business, normally 30 threads are enough, but when business is busy, maybe need 300 threads if use fixed thread pool, then after process boot, Whether the business is busy or not, must create 300 threads but if use thread pool with core/max/keepAlive configuration, thread pool will create thread when business is busy, and destroy thread when business is not so busy. @willem.jiang yes, the thread pool is for sync business logic, it's for worker thread but we change from old CPU count thread to tomcat default setting. Willem Jiang 于2019年1月24日周四 下午10:58写道: > If the thread pool with the cpu core is for the netty boss thread, I > think it should be fine. > But if the thread pool is for the worker thread, it could be a problem > if there are lots of requests need to be processed. > > Willem Jiang > > Twitter: willemjiang > Weibo: 姜宁willem > > On Thu, Jan 24, 2019 at 2:48 PM yhs0092 wrote: > > > > I agree that current default value of thread pool size is too small, but > I'm not sure about the disadvantages of the current fixed thread pool. > > Do you mean if multiple service instances is deployed on the same > machine, a fixed thread pool is not so flexible since the instances cannot > clean up some idle business thread? > > > > > > Yours sincerely > > > > > > Yao Haishi > > yhs0...@163.com > > > > > > On 1/24/2019 10:49,wjm wjm wrote: > > or default integrate only one ThreadPoolExecutor? > > because most customers TPS is not so high, no need to do this optimize > > > > wjm wjm 于2019年1月24日周四 上午10:35写道: > > > > currently we provide a default sync invocation executor: > > > > - default integrate two fixed thread pool > > - thread count for one pool is equals cpu count > > > > for most customers, thread count of one pool is too small, and fixed > > thread pool is not so good, so will change to: > > > > - default integrate two ThreadPoolExecutor > > - support to configure core/max thread count, keepAlive time and max > > queue size for one pool > > - default core thread: 25, same to tomcat > > - default max thread: 100, tomcat is 200, because we have 2 pool, so > > change to 100 > > - default keepAlive: 1 minute, same to tomcat > > - default max queue size: Integer.MAX_VALUE, same to tomcat > > > > >
Re: [Discuss]Business exception cannot be rollbacked inShardingSphere with saga transaction.
Hi Sion, My question is how can we generate the saga execution graph with the rollback request at the first place? Willem Jiang Twitter: willemjiang Weibo: 姜宁willem On Thu, Jan 24, 2019 at 6:04 PM tsubasaotl wrote: > > I think we think it too complicated. In fact, no matter what the exception > is, As long as the rollback is called, a rollback request is added at the end > of the call graph of saga. The actuator reads the rollback request and starts > to reverse the compensation. The saga definition should be > > > { > "policy": "BackwardRecovery", > "requests": [{ > "id": "xxx", > "datasource": "ds", > "type": "sql", > "transaction": { > ... > }, > "compensation": { > ... > }, > "parents": [], > }, > ... > ,{ > "id": "yyy", > "datasource": "ds", > "type": "sql", > "transaction": { > "sql": "ROLLBACK_EVENT", > ... > }, > "compensation": { > ... > }, > "parents": [the last ids of actual requests], > }] > } > > > > -- > Yi Yang (Sion) > Apache ShardingSphere contributor > > > > > > -- Original -- > From: "Zheng Feng"; > Date: Fri, Jan 18, 2019 10:26 AM > To: "dev"; > Cc: "dev"; > Subject: Re: [Discuss]Business exception cannot be rollbacked > inShardingSphere with saga transaction. > > > > I think it could be be resolved in ShardingSphere and catch the business > exception and mark the cached result "ROLLBACK_ONLY". > Anyway, this should be done before submitting to the saga actor. > > tsubasaotl 于2019年1月16日周三 下午8:06写道: > > > Hi, everyone. > > > > > > In ShardingSphere, SQL and their execution result will be cached in saga > > transaction manager. > > When users call `commit` or `rollback` method, the cached SQL will > > generate `SagaDefinition` > > and submit it to the saga actuator. > > > > > > Saga actuator do `Transaction` first and get execution result from cached. > > When the result is successful, the actuator will directly judge that the > > Transaction is successful > > and execute the next Transaction. > > When the result not found in cached or the result is failed, the actuator > > will do retry or compensation > > according to configuration. > > > > > > But there is a problem that saga actuator cannot rollback when business > > exception happened. > > The situation will happen in following workflow. > > > > > >---multiple times--- > > begin transaction --> | execute SQL --> cached result | --> throw business > > exception --> call rollback > >-- > > | > > > > | > > > > | > > but all SQL success, saga actuator don't run compensation <-- get result > > from cache <-- saga actuator > > > > > > All SQL is executed successfully, but the business program throws an > > exception. So the actuator will judge > > transaction is successful, and not to do compensation, which makes users > > confused. > > > > > > On this issue, I would like to ask for advice, should resolved in Saga > > actuator or ShardingSphere? > > And how to resolve better? > > > > > > -- > > Yi Yang (Sion) > > Apache ShardingSphere contributor
Re: [discuss] change default settings of sync invocation executor
If the thread pool with the cpu core is for the netty boss thread, I think it should be fine. But if the thread pool is for the worker thread, it could be a problem if there are lots of requests need to be processed. Willem Jiang Twitter: willemjiang Weibo: 姜宁willem On Thu, Jan 24, 2019 at 2:48 PM yhs0092 wrote: > > I agree that current default value of thread pool size is too small, but I'm > not sure about the disadvantages of the current fixed thread pool. > Do you mean if multiple service instances is deployed on the same machine, a > fixed thread pool is not so flexible since the instances cannot clean up some > idle business thread? > > > Yours sincerely > > > Yao Haishi > yhs0...@163.com > > > On 1/24/2019 10:49,wjm wjm wrote: > or default integrate only one ThreadPoolExecutor? > because most customers TPS is not so high, no need to do this optimize > > wjm wjm 于2019年1月24日周四 上午10:35写道: > > currently we provide a default sync invocation executor: > > - default integrate two fixed thread pool > - thread count for one pool is equals cpu count > > for most customers, thread count of one pool is too small, and fixed > thread pool is not so good, so will change to: > > - default integrate two ThreadPoolExecutor > - support to configure core/max thread count, keepAlive time and max > queue size for one pool > - default core thread: 25, same to tomcat > - default max thread: 100, tomcat is 200, because we have 2 pool, so > change to 100 > - default keepAlive: 1 minute, same to tomcat > - default max queue size: Integer.MAX_VALUE, same to tomcat > >
Re: [Discuss]Business exception cannot be rollbacked inShardingSphere with saga transaction.
I think we think it too complicated. In fact, no matter what the exception is, As long as the rollback is called, a rollback request is added at the end of the call graph of saga. The actuator reads the rollback request and starts to reverse the compensation. The saga definition should be { "policy": "BackwardRecovery", "requests": [{ "id": "xxx", "datasource": "ds", "type": "sql", "transaction": { ... }, "compensation": { ... }, "parents": [], }, ... ,{ "id": "yyy", "datasource": "ds", "type": "sql", "transaction": { "sql": "ROLLBACK_EVENT", ... }, "compensation": { ... }, "parents": [the last ids of actual requests], }] } -- Yi Yang (Sion) Apache ShardingSphere contributor -- Original -- From: "Zheng Feng"; Date: Fri, Jan 18, 2019 10:26 AM To: "dev"; Cc: "dev"; Subject: Re: [Discuss]Business exception cannot be rollbacked inShardingSphere with saga transaction. I think it could be be resolved in ShardingSphere and catch the business exception and mark the cached result "ROLLBACK_ONLY". Anyway, this should be done before submitting to the saga actor. tsubasaotl ??2019??1??16?? 8:06?? > Hi, everyone. > > > In ShardingSphere, SQL and their execution result will be cached in saga > transaction manager. > When users call `commit` or `rollback` method, the cached SQL will > generate `SagaDefinition` > and submit it to the saga actuator. > > > Saga actuator do `Transaction` first and get execution result from cached. > When the result is successful, the actuator will directly judge that the > Transaction is successful > and execute the next Transaction. > When the result not found in cached or the result is failed, the actuator > will do retry or compensation > according to configuration. > > > But there is a problem that saga actuator cannot rollback when business > exception happened. > The situation will happen in following workflow. > > >---multiple times--- > begin transaction --> | execute SQL --> cached result | --> throw business > exception --> call rollback >-- > | > > | > > | > but all SQL success, saga actuator don't run compensation <-- get result > from cache <-- saga actuator > > > All SQL is executed successfully, but the business program throws an > exception. So the actuator will judge > transaction is successful, and not to do compensation, which makes users > confused. > > > On this issue, I would like to ask for advice, should resolved in Saga > actuator or ShardingSphere? > And how to resolve better? > > > -- > Yi Yang (Sion) > Apache ShardingSphere contributor