Re: [Dev] WSO2 MB - To Be Dones I tracked so far
Hi HasithaH, On Config revamp : Noted. Will verify the referred name-spaces when modifying the configs. Currently I have added the new configuration access classes to andes, and working on injecting them to qpid configuration reference logic with minimal changes. Once its done, this can be committed. On Queue purge : Yes will plan a review within next couple of days and will double check the topic purge flow as well. Thanks On Sun, Nov 16, 2014 at 10:39 AM, Hasitha Hiranya hasit...@wso2.com wrote: Hi Sewwandi, Noticed that you have moved CQL related things to a package called cql. Please make sure you update andes-virtualhost.xml file with correct fully qualified class name. Update Asitha's store configuration doc as well. If you have already done, please ignore :) Configuration revamp by HasithaA should capture this as well. Thanks On Sun, Nov 16, 2014 at 10:36 AM, Hasitha Hiranya hasit...@wso2.com wrote: Hi Team, I Removed 1. nodeQueues 2. GlobalQueues 3. QueueAddress 4. TopicDeliveryWorker 5. Unnecessary column families created by CQL Message store. and committed. A few things to note: 1. MessageDeliveryThreadHandler has some unused methods now as TopicDeliveryWorker was removed. Need to review. (@HasithaA, any ideas?) 2. ClusterManager's globalQueueSyncId now renamed to nodeSyncID. Actually do we need it now? (@Sajini, WDYT?) Thanks On Sun, Nov 16, 2014 at 6:45 AM, Indika Sampath indi...@wso2.com wrote: On Sat, Nov 15, 2014 at 10:01 PM, Ramith Jayasinghe ram...@wso2.com wrote: On Sat, Nov 15, 2014 at 1:40 PM, Hasitha Hiranya hasit...@wso2.com wrote: Hi Team, Following are tasks TBD for M2 and M3 releases of MB. Thought of flushing my mind with the tasks that came up with last code review and some last for some time. 1. Add a warning if parts cache is growing. (Done) 2. Make message status block capital (Done) 3. Make message status commented (TBD) 4. Remove msgId2MsgData auto removal of IDs. And make it a concurrentHashMap. (Done) 5. Move Sent and Sent all status setting to correct places (). 6. Make send() of subscription impl logic free. If an exception came from there, catch it and do the work within QDW class. Need to define proper exceptions on why sending to subscriber failed (DLC sending if delivery rules failed, if exception came when sending try another subscriber if available, if not move them over in store with a new messageID, but keeping the old id somehow) (This is a task. Need to assign somebody). 7. When all nodes in the cluster is gone make the index again (Done by AsankaAb) 8. We decided to keep Acked by all and sent to all status, did not we? No need to write what Ramith suggested now. yes. 9. Add a delay when trying to delete slot to get over with Edge Case brought out by Indika (Is it solved now? Problem was elsewhere?) after the last perf tests we did, problem doesn't occur on H2, only with Cassandra ( CQL and Hector both). This is a real and Massive L1 This is fixed in cql. Problem with range query. Will do change in hector too. 10. Need to run all topic test cases and make them pass. 11. Need to introduce new ack handle mechanism (QOS 0,1,2) and AMQP ack means one of those. AMQP transactions, client acks also come into play here. We need to be careful. (This is a task. Should be assigned to somebody). 12. We still did not review purge and queue delete. Should be done soon. Topic deletion should be integrated with it. HasithaA can you schedule a review? +1 13. Hector code is committed now. We can choose hector or cql from config level. Are we maintaining both now? WDYT? My Opinion is we can keep both (atleast at this stage) This is config level option. Only need is enable necessary datasource and provider in master-datasources.xml and messageStore and andesContextStore in andes-virtualhosts.xml. Shall we come to conclusion after doing performance comparison ? 14. Need to fix packaging someday. I suggest one of us do this and commit (big bang). 15. Topic cluster wide tests needs to be written. There is a considerable number of scenarios. Shall we come up with scenarios (in a doc may be). the distribute. 16. Topic ack and requeing should be verified with the spec. Should we requeue topic messages if rejected by client, or if delivery failed? How this affect to MQTT impl? +1 lets check with the spec, and then chat 17. MQTT cluster tests needs to be written. Again lets see what are the scenarios and decide on a scope 18. Puppetizing MB deployment? Did we make any progress? We haven't done anything here. I checked with our cloud team there some puppet template written for 2.2.0 also. However, we are waiting on the initiative from automation team. 19. Remove zookeeper, node queue, qlobal queue, queueAddress, topic delivery worker concepts completely from the code. 20. We Must test and fix message counting. With concurrent
[Dev] WSO2 MB - To Be Dones I tracked so far
Hi Team, Following are tasks TBD for M2 and M3 releases of MB. Thought of flushing my mind with the tasks that came up with last code review and some last for some time. 1. Add a warning if parts cache is growing. (Done) 2. Make message status block capital (Done) 3. Make message status commented (TBD) 4. Remove msgId2MsgData auto removal of IDs. And make it a concurrentHashMap. (Done) 5. Move Sent and Sent all status setting to correct places (). 6. Make send() of subscription impl logic free. If an exception came from there, catch it and do the work within QDW class. Need to define proper exceptions on why sending to subscriber failed (DLC sending if delivery rules failed, if exception came when sending try another subscriber if available, if not move them over in store with a new messageID, but keeping the old id somehow) (This is a task. Need to assign somebody). 7. When all nodes in the cluster is gone make the index again (Done by AsankaAb) 8. We decided to keep Acked by all and sent to all status, did not we? No need to write what Ramith suggested now. 9. Add a delay when trying to delete slot to get over with Edge Case brought out by Indika (Is it solved now? Problem was elsewhere?) 10. Need to run all topic test cases and make them pass. 11. Need to introduce new ack handle mechanism (QOS 0,1,2) and AMQP ack means one of those. AMQP transactions, client acks also come into play here. We need to be careful. (This is a task. Should be assigned to somebody). 12. We still did not review purge and queue delete. Should be done soon. Topic deletion should be integrated with it. HasithaA can you schedule a review? 13. Hector code is committed now. We can choose hector or cql from config level. Are we maintaining both now? WDYT? 14. Need to fix packaging someday. 15. Topic cluster wide tests needs to be written. There is a considerable number of scenarios. 16. Topic ack and requeing should be verified with the spec. Should we requeue topic messages if rejected by client, or if delivery failed? How this affect to MQTT impl? 17. MQTT cluster tests needs to be written. 18. Puppetizing MB deployment? Did we make any progress? 19. Remove zookeeper, node queue, qlobal queue, queueAddress, topic delivery worker concepts completely from the code. 20. We Must test and fix message counting. With concurrent publishers and subscribers we still have message counting probs. 21. Apply config revamp. Suppose we can go to M2 when these are done. For M3 1. Fix all the UIs. This will take a considerable effort. 2. Merge remaining fixes from 2.2.0. Thanks -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* http://abeykoon.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] WSO2 MB - To Be Dones I tracked so far
On Sat, Nov 15, 2014 at 1:40 PM, Hasitha Hiranya hasit...@wso2.com wrote: Hi Team, Following are tasks TBD for M2 and M3 releases of MB. Thought of flushing my mind with the tasks that came up with last code review and some last for some time. 1. Add a warning if parts cache is growing. (Done) 2. Make message status block capital (Done) 3. Make message status commented (TBD) 4. Remove msgId2MsgData auto removal of IDs. And make it a concurrentHashMap. (Done) 5. Move Sent and Sent all status setting to correct places (). 6. Make send() of subscription impl logic free. If an exception came from there, catch it and do the work within QDW class. Need to define proper exceptions on why sending to subscriber failed (DLC sending if delivery rules failed, if exception came when sending try another subscriber if available, if not move them over in store with a new messageID, but keeping the old id somehow) (This is a task. Need to assign somebody). 7. When all nodes in the cluster is gone make the index again (Done by AsankaAb) 8. We decided to keep Acked by all and sent to all status, did not we? No need to write what Ramith suggested now. yes. 9. Add a delay when trying to delete slot to get over with Edge Case brought out by Indika (Is it solved now? Problem was elsewhere?) after the last perf tests we did, problem doesn't occur on H2, only with Cassandra ( CQL and Hector both). This is a real and Massive L1 10. Need to run all topic test cases and make them pass. 11. Need to introduce new ack handle mechanism (QOS 0,1,2) and AMQP ack means one of those. AMQP transactions, client acks also come into play here. We need to be careful. (This is a task. Should be assigned to somebody). 12. We still did not review purge and queue delete. Should be done soon. Topic deletion should be integrated with it. HasithaA can you schedule a review? +1 13. Hector code is committed now. We can choose hector or cql from config level. Are we maintaining both now? WDYT? My Opinion is we can keep both (atleast at this stage) 14. Need to fix packaging someday. I suggest one of us do this and commit (big bang). 15. Topic cluster wide tests needs to be written. There is a considerable number of scenarios. Shall we come up with scenarios (in a doc may be). the distribute. 16. Topic ack and requeing should be verified with the spec. Should we requeue topic messages if rejected by client, or if delivery failed? How this affect to MQTT impl? +1 lets check with the spec, and then chat 17. MQTT cluster tests needs to be written. Again lets see what are the scenarios and decide on a scope 18. Puppetizing MB deployment? Did we make any progress? We haven't done anything here. I checked with our cloud team there some puppet template written for 2.2.0 also. However, we are waiting on the initiative from automation team. 19. Remove zookeeper, node queue, qlobal queue, queueAddress, topic delivery worker concepts completely from the code. 20. We Must test and fix message counting. With concurrent publishers and subscribers we still have message counting probs. Lets open L2s for these to keep track. 21. Apply config revamp. Hasitha Amal can comment on this Suppose we can go to M2 when these are done. For M3 1. Fix all the UIs. This will take a considerable effort. 2. Merge remaining fixes from 2.2.0. And we need to keep on testing end to end fix issues ( M3 should have been all about bring stability, BUT given that there could be some other features/requirements we might have to deliver with v 3.0.0 - something we will decide next week) Thanks -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* http://abeykoon.blogspot.com -- Ramith Jayasinghe Technical Lead WSO2 Inc., http://wso2.com lean.enterprise.middleware E: ram...@wso2.com P: +94 777542851 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] WSO2 MB - To Be Dones I tracked so far
On Sat, Nov 15, 2014 at 10:01 PM, Ramith Jayasinghe ram...@wso2.com wrote: On Sat, Nov 15, 2014 at 1:40 PM, Hasitha Hiranya hasit...@wso2.com wrote: Hi Team, Following are tasks TBD for M2 and M3 releases of MB. Thought of flushing my mind with the tasks that came up with last code review and some last for some time. 1. Add a warning if parts cache is growing. (Done) 2. Make message status block capital (Done) 3. Make message status commented (TBD) 4. Remove msgId2MsgData auto removal of IDs. And make it a concurrentHashMap. (Done) 5. Move Sent and Sent all status setting to correct places (). 6. Make send() of subscription impl logic free. If an exception came from there, catch it and do the work within QDW class. Need to define proper exceptions on why sending to subscriber failed (DLC sending if delivery rules failed, if exception came when sending try another subscriber if available, if not move them over in store with a new messageID, but keeping the old id somehow) (This is a task. Need to assign somebody). 7. When all nodes in the cluster is gone make the index again (Done by AsankaAb) 8. We decided to keep Acked by all and sent to all status, did not we? No need to write what Ramith suggested now. yes. 9. Add a delay when trying to delete slot to get over with Edge Case brought out by Indika (Is it solved now? Problem was elsewhere?) after the last perf tests we did, problem doesn't occur on H2, only with Cassandra ( CQL and Hector both). This is a real and Massive L1 This is fixed in cql. Problem with range query. Will do change in hector too. 10. Need to run all topic test cases and make them pass. 11. Need to introduce new ack handle mechanism (QOS 0,1,2) and AMQP ack means one of those. AMQP transactions, client acks also come into play here. We need to be careful. (This is a task. Should be assigned to somebody). 12. We still did not review purge and queue delete. Should be done soon. Topic deletion should be integrated with it. HasithaA can you schedule a review? +1 13. Hector code is committed now. We can choose hector or cql from config level. Are we maintaining both now? WDYT? My Opinion is we can keep both (atleast at this stage) This is config level option. Only need is enable necessary datasource and provider in master-datasources.xml and messageStore and andesContextStore in andes-virtualhosts.xml. Shall we come to conclusion after doing performance comparison ? 14. Need to fix packaging someday. I suggest one of us do this and commit (big bang). 15. Topic cluster wide tests needs to be written. There is a considerable number of scenarios. Shall we come up with scenarios (in a doc may be). the distribute. 16. Topic ack and requeing should be verified with the spec. Should we requeue topic messages if rejected by client, or if delivery failed? How this affect to MQTT impl? +1 lets check with the spec, and then chat 17. MQTT cluster tests needs to be written. Again lets see what are the scenarios and decide on a scope 18. Puppetizing MB deployment? Did we make any progress? We haven't done anything here. I checked with our cloud team there some puppet template written for 2.2.0 also. However, we are waiting on the initiative from automation team. 19. Remove zookeeper, node queue, qlobal queue, queueAddress, topic delivery worker concepts completely from the code. 20. We Must test and fix message counting. With concurrent publishers and subscribers we still have message counting probs. Lets open L2s for these to keep track. 21. Apply config revamp. Hasitha Amal can comment on this Suppose we can go to M2 when these are done. For M3 1. Fix all the UIs. This will take a considerable effort. 2. Merge remaining fixes from 2.2.0. And we need to keep on testing end to end fix issues ( M3 should have been all about bring stability, BUT given that there could be some other features/requirements we might have to deliver with v 3.0.0 - something we will decide next week) Thanks -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* http://abeykoon.blogspot.com -- Ramith Jayasinghe Technical Lead WSO2 Inc., http://wso2.com lean.enterprise.middleware E: ram...@wso2.com P: +94 777542851 -- Indika Sampath Software Engineer WSO2 Inc. http://wso2.com Phone: +94 716 424 744 Blog: http://indikasampath.blogspot.com/ ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] WSO2 MB - To Be Dones I tracked so far
Hi Team, I Removed 1. nodeQueues 2. GlobalQueues 3. QueueAddress 4. TopicDeliveryWorker 5. Unnecessary column families created by CQL Message store. and committed. A few things to note: 1. MessageDeliveryThreadHandler has some unused methods now as TopicDeliveryWorker was removed. Need to review. (@HasithaA, any ideas?) 2. ClusterManager's globalQueueSyncId now renamed to nodeSyncID. Actually do we need it now? (@Sajini, WDYT?) Thanks On Sun, Nov 16, 2014 at 6:45 AM, Indika Sampath indi...@wso2.com wrote: On Sat, Nov 15, 2014 at 10:01 PM, Ramith Jayasinghe ram...@wso2.com wrote: On Sat, Nov 15, 2014 at 1:40 PM, Hasitha Hiranya hasit...@wso2.com wrote: Hi Team, Following are tasks TBD for M2 and M3 releases of MB. Thought of flushing my mind with the tasks that came up with last code review and some last for some time. 1. Add a warning if parts cache is growing. (Done) 2. Make message status block capital (Done) 3. Make message status commented (TBD) 4. Remove msgId2MsgData auto removal of IDs. And make it a concurrentHashMap. (Done) 5. Move Sent and Sent all status setting to correct places (). 6. Make send() of subscription impl logic free. If an exception came from there, catch it and do the work within QDW class. Need to define proper exceptions on why sending to subscriber failed (DLC sending if delivery rules failed, if exception came when sending try another subscriber if available, if not move them over in store with a new messageID, but keeping the old id somehow) (This is a task. Need to assign somebody). 7. When all nodes in the cluster is gone make the index again (Done by AsankaAb) 8. We decided to keep Acked by all and sent to all status, did not we? No need to write what Ramith suggested now. yes. 9. Add a delay when trying to delete slot to get over with Edge Case brought out by Indika (Is it solved now? Problem was elsewhere?) after the last perf tests we did, problem doesn't occur on H2, only with Cassandra ( CQL and Hector both). This is a real and Massive L1 This is fixed in cql. Problem with range query. Will do change in hector too. 10. Need to run all topic test cases and make them pass. 11. Need to introduce new ack handle mechanism (QOS 0,1,2) and AMQP ack means one of those. AMQP transactions, client acks also come into play here. We need to be careful. (This is a task. Should be assigned to somebody). 12. We still did not review purge and queue delete. Should be done soon. Topic deletion should be integrated with it. HasithaA can you schedule a review? +1 13. Hector code is committed now. We can choose hector or cql from config level. Are we maintaining both now? WDYT? My Opinion is we can keep both (atleast at this stage) This is config level option. Only need is enable necessary datasource and provider in master-datasources.xml and messageStore and andesContextStore in andes-virtualhosts.xml. Shall we come to conclusion after doing performance comparison ? 14. Need to fix packaging someday. I suggest one of us do this and commit (big bang). 15. Topic cluster wide tests needs to be written. There is a considerable number of scenarios. Shall we come up with scenarios (in a doc may be). the distribute. 16. Topic ack and requeing should be verified with the spec. Should we requeue topic messages if rejected by client, or if delivery failed? How this affect to MQTT impl? +1 lets check with the spec, and then chat 17. MQTT cluster tests needs to be written. Again lets see what are the scenarios and decide on a scope 18. Puppetizing MB deployment? Did we make any progress? We haven't done anything here. I checked with our cloud team there some puppet template written for 2.2.0 also. However, we are waiting on the initiative from automation team. 19. Remove zookeeper, node queue, qlobal queue, queueAddress, topic delivery worker concepts completely from the code. 20. We Must test and fix message counting. With concurrent publishers and subscribers we still have message counting probs. Lets open L2s for these to keep track. 21. Apply config revamp. Hasitha Amal can comment on this Suppose we can go to M2 when these are done. For M3 1. Fix all the UIs. This will take a considerable effort. 2. Merge remaining fixes from 2.2.0. And we need to keep on testing end to end fix issues ( M3 should have been all about bring stability, BUT given that there could be some other features/requirements we might have to deliver with v 3.0.0 - something we will decide next week) Thanks -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* http://abeykoon.blogspot.com -- Ramith Jayasinghe Technical Lead WSO2 Inc., http://wso2.com lean.enterprise.middleware E: ram...@wso2.com P: +94 777542851 -- Indika Sampath Software Engineer WSO2 Inc.
Re: [Dev] WSO2 MB - To Be Dones I tracked so far
Hi Sewwandi, Noticed that you have moved CQL related things to a package called cql. Please make sure you update andes-virtualhost.xml file with correct fully qualified class name. Update Asitha's store configuration doc as well. If you have already done, please ignore :) Configuration revamp by HasithaA should capture this as well. Thanks On Sun, Nov 16, 2014 at 10:36 AM, Hasitha Hiranya hasit...@wso2.com wrote: Hi Team, I Removed 1. nodeQueues 2. GlobalQueues 3. QueueAddress 4. TopicDeliveryWorker 5. Unnecessary column families created by CQL Message store. and committed. A few things to note: 1. MessageDeliveryThreadHandler has some unused methods now as TopicDeliveryWorker was removed. Need to review. (@HasithaA, any ideas?) 2. ClusterManager's globalQueueSyncId now renamed to nodeSyncID. Actually do we need it now? (@Sajini, WDYT?) Thanks On Sun, Nov 16, 2014 at 6:45 AM, Indika Sampath indi...@wso2.com wrote: On Sat, Nov 15, 2014 at 10:01 PM, Ramith Jayasinghe ram...@wso2.com wrote: On Sat, Nov 15, 2014 at 1:40 PM, Hasitha Hiranya hasit...@wso2.com wrote: Hi Team, Following are tasks TBD for M2 and M3 releases of MB. Thought of flushing my mind with the tasks that came up with last code review and some last for some time. 1. Add a warning if parts cache is growing. (Done) 2. Make message status block capital (Done) 3. Make message status commented (TBD) 4. Remove msgId2MsgData auto removal of IDs. And make it a concurrentHashMap. (Done) 5. Move Sent and Sent all status setting to correct places (). 6. Make send() of subscription impl logic free. If an exception came from there, catch it and do the work within QDW class. Need to define proper exceptions on why sending to subscriber failed (DLC sending if delivery rules failed, if exception came when sending try another subscriber if available, if not move them over in store with a new messageID, but keeping the old id somehow) (This is a task. Need to assign somebody). 7. When all nodes in the cluster is gone make the index again (Done by AsankaAb) 8. We decided to keep Acked by all and sent to all status, did not we? No need to write what Ramith suggested now. yes. 9. Add a delay when trying to delete slot to get over with Edge Case brought out by Indika (Is it solved now? Problem was elsewhere?) after the last perf tests we did, problem doesn't occur on H2, only with Cassandra ( CQL and Hector both). This is a real and Massive L1 This is fixed in cql. Problem with range query. Will do change in hector too. 10. Need to run all topic test cases and make them pass. 11. Need to introduce new ack handle mechanism (QOS 0,1,2) and AMQP ack means one of those. AMQP transactions, client acks also come into play here. We need to be careful. (This is a task. Should be assigned to somebody). 12. We still did not review purge and queue delete. Should be done soon. Topic deletion should be integrated with it. HasithaA can you schedule a review? +1 13. Hector code is committed now. We can choose hector or cql from config level. Are we maintaining both now? WDYT? My Opinion is we can keep both (atleast at this stage) This is config level option. Only need is enable necessary datasource and provider in master-datasources.xml and messageStore and andesContextStore in andes-virtualhosts.xml. Shall we come to conclusion after doing performance comparison ? 14. Need to fix packaging someday. I suggest one of us do this and commit (big bang). 15. Topic cluster wide tests needs to be written. There is a considerable number of scenarios. Shall we come up with scenarios (in a doc may be). the distribute. 16. Topic ack and requeing should be verified with the spec. Should we requeue topic messages if rejected by client, or if delivery failed? How this affect to MQTT impl? +1 lets check with the spec, and then chat 17. MQTT cluster tests needs to be written. Again lets see what are the scenarios and decide on a scope 18. Puppetizing MB deployment? Did we make any progress? We haven't done anything here. I checked with our cloud team there some puppet template written for 2.2.0 also. However, we are waiting on the initiative from automation team. 19. Remove zookeeper, node queue, qlobal queue, queueAddress, topic delivery worker concepts completely from the code. 20. We Must test and fix message counting. With concurrent publishers and subscribers we still have message counting probs. Lets open L2s for these to keep track. 21. Apply config revamp. Hasitha Amal can comment on this Suppose we can go to M2 when these are done. For M3 1. Fix all the UIs. This will take a considerable effort. 2. Merge remaining fixes from 2.2.0. And we need to keep on testing end to end fix issues ( M3 should have been all about bring stability, BUT given that there could be some other features/requirements we might have to deliver
