Re: [Dev] Fixing Message Counters and Queue Purging features for MB 2.1.1 after disruptor implementation
Hi Ishara, This is great !!! On Mon, Sep 9, 2013 at 12:30 AM, Ishara Premadasa ish...@wso2.com wrote: Hi Malinga, On Fri, Aug 30, 2013 at 2:42 PM, Malinga Purnasiri malin...@wso2.comwrote: hi Ishara, As i remember since per message count increment to DB kill the performance of MB considerably; we all thought of removing it for the moment. With the new disruptor changes we thought of updating queue count periodically (not in real-time). So i think its not good idea to depend on the queue count for the purging operation. instead .. What if we use [1] receive(timeout). We can set the timeout to (may be) 5 secs. Any ideas ? I used [1] as suggested with a timeout of 30 secs and got queue purging works fine. IIRC before introducing the disruptor we were unable to use receive(timeout) method from API due to the latency of retrieving messages from Queue, however the process has good speed now. With new fix when purge is clicked it will empty the Queue in blocking manner while a loading message is displayed and will return back after purging is done. Fixed and committed in r184738. Thanks! Ishara [1] http://docs.oracle.com/javaee/1.4/api/javax/jms/MessageConsumer.html#receive(long) On Wed, Aug 28, 2013 at 2:01 PM, Ishara Premadasa ish...@wso2.comwrote: Hi, Since we have removed the message counters after introducing the disruptor based message processing for MB 2.1.1 both the above features are currently not working and we are trying to fix this. As the previous MessageWriter, implementations from 2.1.0 are no longer used, now we don't keep message counts stored in Cassandra so the counters are always shown as zero. Due to this the current queue purge feature has too broken since it periodically checks the message counter and performs purge operation continuously till the total message count in the queue is zero. IMO we need to find an alternative way to get message counters work in the queue as the message writers are not used now. If not we may need to remove these two features from UI, since purging too depending on the counters. Thanks! Ishara -- Ishara Premasada Software Engineer, WSO2 Inc. http://wso2.com/ *Blog : http://isharapremadasa.blogspot.com/ Twitter : https://twitter.com/ishadil Mobile : +94 714445832* -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 -- Ishara Premasada Software Engineer, WSO2 Inc. http://wso2.com/ *Blog : http://isharapremadasa.blogspot.com/ Twitter : https://twitter.com/ishadil Mobile : +94 714445832* -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Fixing Message Counters and Queue Purging features for MB 2.1.1 after disruptor implementation
hi Ishara, As i remember since per message count increment to DB kill the performance of MB considerably; we all thought of removing it for the moment. With the new disruptor changes we thought of updating queue count periodically (not in real-time). So i think its not good idea to depend on the queue count for the purging operation. instead .. What if we use [1] receive(timeout). We can set the timeout to (may be) 5 secs. Any ideas ? [1] http://docs.oracle.com/javaee/1.4/api/javax/jms/MessageConsumer.html#receive(long) On Wed, Aug 28, 2013 at 2:01 PM, Ishara Premadasa ish...@wso2.com wrote: Hi, Since we have removed the message counters after introducing the disruptor based message processing for MB 2.1.1 both the above features are currently not working and we are trying to fix this. As the previous MessageWriter, implementations from 2.1.0 are no longer used, now we don't keep message counts stored in Cassandra so the counters are always shown as zero. Due to this the current queue purge feature has too broken since it periodically checks the message counter and performs purge operation continuously till the total message count in the queue is zero. IMO we need to find an alternative way to get message counters work in the queue as the message writers are not used now. If not we may need to remove these two features from UI, since purging too depending on the counters. Thanks! Ishara -- Ishara Premasada Software Engineer, WSO2 Inc. http://wso2.com/ *Blog : http://isharapremadasa.blogspot.com/ Twitter : https://twitter.com/ishadil Mobile : +94 714445832* -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] WSO2 Committers += Inosh Goonewardena
Congrats Inosh On Thu, Jun 27, 2013 at 10:27 AM, Kasun Weranga kas...@wso2.com wrote: Hi all, Its my pleasure to welcome Inosh Goonewardena as a WSO2 Committer. He has made great contributions to WSO2 BAM and other projects during the last few months, Thank you so much for all your hard work. Inosh, welcome aboard and keep up the good work. Thanks, KasunW ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Profiling of MB 2.1.0 Beta
Hi, After the MB 2.1.0 Beta; performance test released, we realized that we need to increase the numbers to further more. So after chatting with Srinath and team mates, I spent sometime to profile the MB with JProfiler. After looking at the *hotspots* and *VM Telemetry Views* found following issues in the MB. *** Message Expiration logic takes more time to execute and transfer the results. This is simply due to slice query results count set to max_int. * * *** Some executor pools had just 50 hard - coded limits. So when testing with parallel publishers and subscribers system fail to perform. Following is new performance numbers after the required changes Q Paradigm Message Size : 1K # of Producers : 20 # of Consumers : 250 # of Queues : 10 # of Messages : 50,000 Old Results : TPS is ~ 600 to 750 New Results : TPS is ~ 1300 to 1600 @Asanka Please we need to conduct comprehensive performance test by varying factors like message size, # of producers @Srinath As we discussed I will add this test suite to MB code base so when building the system we can ship this with the MB pack. So anytime we can run this without much hassle. Any thoughts on how we can increase this numbers, improve the test suite further more -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Profiling of MB 2.1.0 Beta
Will chat and write a test plan for this today. On Fri, Jun 21, 2013 at 8:35 AM, Srinath Perera srin...@wso2.com wrote: Hi Malinga, We need to also test with several MB2 nodes. 2, 4, and 8. Those nodes are big and we can run multiple MB in each node. Lets chat. --Srinath On Thu, Jun 20, 2013 at 9:28 PM, Malinga Purnasiri malin...@wso2.comwrote: Hi, After the MB 2.1.0 Beta; performance test released, we realized that we need to increase the numbers to further more. So after chatting with Srinath and team mates, I spent sometime to profile the MB with JProfiler. After looking at the *hotspots* and *VM Telemetry Views* found following issues in the MB. *** Message Expiration logic takes more time to execute and transfer the results. This is simply due to slice query results count set to max_int. * * *** Some executor pools had just 50 hard - coded limits. So when testing with parallel publishers and subscribers system fail to perform. Following is new performance numbers after the required changes Q Paradigm Message Size : 1K # of Producers : 20 # of Consumers : 250 # of Queues : 10 # of Messages : 50,000 Old Results : TPS is ~ 600 to 750 New Results : TPS is ~ 1300 to 1600 @Asanka Please we need to conduct comprehensive performance test by varying factors like message size, # of producers @Srinath As we discussed I will add this test suite to MB code base so when building the system we can ship this with the MB pack. So anytime we can run this without much hassle. Any thoughts on how we can increase this numbers, improve the test suite further more -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 -- Srinath Perera, Ph.D. Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] WSO2 Committers += Ishara Premadasa
Congratulations Ishara!! On Fri, Jun 21, 2013 at 11:00 AM, Bhathiya Jayasekara bhath...@wso2.comwrote: Congratulations ishara! On Fri, Jun 21, 2013 at 10:47 AM, Chanika Geeganage chan...@wso2.comwrote: Congratzzz Ishara On Fri, Jun 21, 2013 at 10:43 AM, Gihan Anuruddha gi...@wso2.com wrote: Congratulation Ishara !!! On Fri, Jun 21, 2013 at 10:21 AM, Shammi Jayasinghe sha...@wso2.comwrote: Hi, Its my pleasure to welcome Ishara Premadasa as WSO2 Committer. Ishara has been an excellent contributor for MB 2.1.0 and other related projects over the last few months and gave a great help on releasing Alpha, Beta and RC1 of MB 2.1.0. Thank you very much for all your hard work. Ishara, welcome aboard and keep up the good work. -- Best Regards,* Shammi Jayasinghe* Associate Tech Lead WSO2, Inc.; http://wso2.com, mobile: +94 71 4493085 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- --- W.G. Gihan Anuruddha Senior Software Engineer | WSO2, Inc. M: +94772272595 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Best Regards.. Chanika Geeganage Software Engineer WSO2, Inc.; http://wso2.com Mobile: +94773522586 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- *Bhathiya Jayasekara* *Software Engineer,* *WSO2 inc., http://wso2.com* * * *Phone: +94715478185* *LinkedIn: http://www.linkedin.com/in/bhathiyaj* *Twitter: https://twitter.com/bhathiyax* ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] WSO2 Message Broker 2.1.0 - RC1 Released!
WSO2 Message Broker 2.1.0 - RC1 Released! WSO2 MB team is pleased to announce the 2.1.0-RC1 release of the WSO2 Message Broker. WSO2 Message Broker 2.1.0, JMS message broker backed up with the wso2 distributed message brokering engine Andes is compliant with AMQP 0-91 version.The new RC1 version contains, • Hierarchical Topics Subscription feature • Queue Message browsing feature with added UI support • In memory message store feature with faster message publishing and consuming (queues/topics). • Message expiration feature • Support Number of Maximum Message Re-Delivery Tries Configuration • Support Message Re delivery Header Field (useful to JMS applications) • Providing test message sending support to a given queue from admin console • Providing queue purging ability from WSO2 MB admin console • Ability to cluster according to 5 deployment patterns. • Accurate count of messages residing in queues • Sorted queue list based on the descending order of message count • Working with port-offset in standalone mode • Ability to view inner details of the cluster using Management Console of any node in the cluster • Message delivery fine tuning capabilities and many bug fixes and improvements. • Ability to set Cassandra replication factor and read/write consistency levels • Many bug fixes and improvements You can download this distribution from http://dist.wso2.org/downloads/message-broker/2.1.0/rc1/wso2mb-2.1.0.zip 1. Extract the downloaded zip 2. Go to the bin directory in the extracted folder 3. Run the wso2server.sh or wso2server.bat as appropriate 4. Point your browser to the URL https://localhost:9443/carbon 5. Use admin, admin as the username and password to login as an admin and create a user account 6. Use the documentation to get to know on how to creating clients, using clients and deployment guide to do clustering. Fixed Issues Fixed issues are listed at the following url: https://wso2.org/jira/issues/?filter=11075 Known IssuesKnown issues are listed at the following url: https://wso2.org/jira/issues/?filter=11076 You can report issues at https://wso2.org/jira/browse/MB/fixforversion/10830 -- WSO2 Message Broker Team -- -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Encrypting the plain text passwords in config files
Hi, There is a requirement to encrypt some sensitive information used in the MB config file. for that I thought of using secure vault feature provided by carbon. For that task we might need to change the ciphertool.properties and ciphertext.properties in the [kernel/carbonhome/repository/conf/security]. Is that ok to go ahead with this change ? Please advice. On Mon, Jun 10, 2013 at 5:41 PM, Maninda Edirisooriya mani...@wso2.comwrote: Look at a sample at https://svn.wso2.org/repos/wso2/carbon/platform/branches/4.1.0/components/mediators/bam/org.wso2.carbon.mediator.bam.config/4.1.0/src/main/java/org/wso2/carbon/mediator/bam/config/CryptographyManager.java Thank you. * Maninda Edirisooriya* Software Engineer *WSO2, Inc. *lean.enterprise.middleware. *Blog* : http://maninda.blogspot.com/ *Phone* : +94 777603226 On Mon, Jun 10, 2013 at 5:05 PM, Malinga Purnasiri malin...@wso2.comwrote: Hi, We have a ticket listed in the jira for MB as [1]. Please post some ideas if anyone has achieved this before ? or got suggestions on this ? [1] https://wso2.org/jira/browse/MB-171 -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Encrypting the plain text passwords in config files
Hi Nirmal, Thanks for the info. but .. Instead of getting these files from kernel, you could add these files from the product level. Have a look at the bin.xml of a product distribution Is that the standard way of adding entries to cipertool.prop and cipertext.prop ? But in the trunk of the kernel; where those files located, has some commented-out lines (for product specific entries). and in the tutorial of the secure vault it says, add it to the files located in the kernel trunk and comment-out if its not used in the all products of the wso2. On Wed, Jun 12, 2013 at 10:51 PM, Nirmal Fernando nir...@wso2.com wrote: Hi Malinga, On Wed, Jun 12, 2013 at 10:33 PM, Malinga Purnasiri malin...@wso2.comwrote: Hi, There is a requirement to encrypt some sensitive information used in the MB config file. for that I thought of using secure vault feature provided by carbon. For that task we might need to change the ciphertool.properties and ciphertext.properties in the [kernel/carbonhome/repository/conf/security]. Is that ok to go ahead with this change ? Please advice. Why you need to do a change in the kernel level, if this is a MB specific change? Instead of getting these files from kernel, you could add these files from the product level. Have a look at the bin.xml of a product distribution. On Mon, Jun 10, 2013 at 5:41 PM, Maninda Edirisooriya mani...@wso2.comwrote: Look at a sample at https://svn.wso2.org/repos/wso2/carbon/platform/branches/4.1.0/components/mediators/bam/org.wso2.carbon.mediator.bam.config/4.1.0/src/main/java/org/wso2/carbon/mediator/bam/config/CryptographyManager.java Thank you. * Maninda Edirisooriya* Software Engineer *WSO2, Inc. *lean.enterprise.middleware. *Blog* : http://maninda.blogspot.com/ *Phone* : +94 777603226 On Mon, Jun 10, 2013 at 5:05 PM, Malinga Purnasiri malin...@wso2.comwrote: Hi, We have a ticket listed in the jira for MB as [1]. Please post some ideas if anyone has achieved this before ? or got suggestions on this ? [1] https://wso2.org/jira/browse/MB-171 -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Thanks regards, Nirmal Senior Software Engineer- Platform Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/ -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Encrypting the plain text passwords in config files
Thanks Prabath appreciated .. On Wed, Jun 12, 2013 at 11:12 PM, Prabath Siriwardena prab...@wso2.comwrote: Hi Malinga, Please talk to Asela - he will assist you on how to do this... Thanks regards, -Prabath On Wed, Jun 12, 2013 at 11:09 PM, Malinga Purnasiri malin...@wso2.comwrote: Hi Nirmal, Thanks for the info. but .. Instead of getting these files from kernel, you could add these files from the product level. Have a look at the bin.xml of a product distribution Is that the standard way of adding entries to cipertool.prop and cipertext.prop ? But in the trunk of the kernel; where those files located, has some commented-out lines (for product specific entries). and in the tutorial of the secure vault it says, add it to the files located in the kernel trunk and comment-out if its not used in the all products of the wso2. On Wed, Jun 12, 2013 at 10:51 PM, Nirmal Fernando nir...@wso2.comwrote: Hi Malinga, On Wed, Jun 12, 2013 at 10:33 PM, Malinga Purnasiri malin...@wso2.comwrote: Hi, There is a requirement to encrypt some sensitive information used in the MB config file. for that I thought of using secure vault feature provided by carbon. For that task we might need to change the ciphertool.properties and ciphertext.properties in the [kernel/carbonhome/repository/conf/security]. Is that ok to go ahead with this change ? Please advice. Why you need to do a change in the kernel level, if this is a MB specific change? Instead of getting these files from kernel, you could add these files from the product level. Have a look at the bin.xml of a product distribution. On Mon, Jun 10, 2013 at 5:41 PM, Maninda Edirisooriya mani...@wso2.com wrote: Look at a sample at https://svn.wso2.org/repos/wso2/carbon/platform/branches/4.1.0/components/mediators/bam/org.wso2.carbon.mediator.bam.config/4.1.0/src/main/java/org/wso2/carbon/mediator/bam/config/CryptographyManager.java Thank you. * Maninda Edirisooriya* Software Engineer *WSO2, Inc. *lean.enterprise.middleware. *Blog* : http://maninda.blogspot.com/ *Phone* : +94 777603226 On Mon, Jun 10, 2013 at 5:05 PM, Malinga Purnasiri malin...@wso2.comwrote: Hi, We have a ticket listed in the jira for MB as [1]. Please post some ideas if anyone has achieved this before ? or got suggestions on this ? [1] https://wso2.org/jira/browse/MB-171 -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Thanks regards, Nirmal Senior Software Engineer- Platform Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/ -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Thanks Regards, Prabath Mobile : +94 71 809 6732 http://blog.facilelogin.com http://RampartFAQ.com -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Encrypting the plain text passwords in config files
Hi Shelan, Noted ! thanks. On Mon, Jun 10, 2013 at 5:09 PM, Shelan Perera she...@wso2.com wrote: Hi, Have you looked into secure vault? You can use that to encrypt username and password of config files. thanks On Mon, Jun 10, 2013 at 5:05 PM, Malinga Purnasiri malin...@wso2.comwrote: Hi, We have a ticket listed in the jira for MB as [1]. Please post some ideas if anyone has achieved this before ? or got suggestions on this ? [1] https://wso2.org/jira/browse/MB-171 -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- *Shelan Perera* Senior Software Engineer ** Integration Technology Group *WSO2, Inc. : wso2.com* lean.enterprise.middleware. *Blog* : blog.shelan.org *Linked-i*n : http://www.linkedin.com/pub/shelan-perera/a/194/465 *Twitter* :https://twitter.com/#!/shelan *Mobile* : +94 772 604 402 -- Malinga Pathmal, Technical Lead, WSO2, Inc. : http://wso2.com/ Phone : (+94) 715335898 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Fwd: Less messages receive than send
Forwarding this to dev, since this may help others, whoever using in memory
mode of wso2 MB.
-- Forwarded message --
From: Malinga Purnasiri malin...@wso2.com
Date: Fri, Jun 7, 2013 at 12:25 PM
Subject: Re: Less messages received than sent
To: Srinath Perera srin...@wso2.com, Shammi Jayasinghe sha...@wso2.com,
Hasitha Hiranya hasit...@wso2.com, Ishara Premadasa ish...@wso2.com
had a chat with Srinath, regarding the message ignoring logic in in-mem
mode,
issue identified during the testing was : we just ignore the messages and
next time getting the messages from the queue, we just ignore the last read
pointer (*in persistence mode Srinath is doing it*) and we loose all the
pending messages due to that.
Now i have fixed it https://wso2.org/jira/browse/MB-293
On Wed, Jun 5, 2013 at 11:18 PM, Malinga Purnasiri malin...@wso2.comwrote:
Hi all,
Just look at the Following code block, (this from the
QueueDeliverWorker.java)
if (isInMemoryMode){
CassandraMessageStore messageStore =
ClusterResourceHolder.getInstance().getCassandraMessageStore();
ListQueueEntry messagesFromCassansdra =
messageStore.getNextQueueMessagesToDeliver(queue, messageCountToRead);
System.out.println(Size : +
messagesFromCassansdra.size());
System.out.println(messageCountToRead : +
messageCountToRead);
for(QueueEntry message: messagesFromCassansdra){
String queueName =
((AMQMessage)message.getMessage()).getMessageMetaData().getMessagePublishInfo().getRoutingKey().toString();
QueueDeliveryInfo queueDeliveryInfo =
getQueueDeliveryInfo(queueName);
if(!queueDeliveryInfo.messageIgnored){
if(queueDeliveryInfo.readButUndeliveredMessages.size() 5000){
System.out.println(limit exceeded :
+ messagesFromCassansdra.size());
queueDeliveryInfo.readButUndeliveredMessages.add(message);
System.out.println(Read but
undeliverd : + queueDeliveryInfo.readButUndeliveredMessages.size());
totalReadButUndeliveredMessages++;
System.out.println(totalReadButUndeliveredMessages AFTER : +
totalReadButUndeliveredMessages);
lastProcessedId =
message.getMessage().getMessageNumber();
*}else{*
* queueDeliveryInfo.messageIgnored =
true;*
*}*
}
}
* Just concentrate on the bold else statement, if the buffer is
growing more than 5000 elements, this code block just set ignore flag to
TRUE. but still it loose message from the from each loop
(for(QueueEntry message: messagesFromCassansdra){)
Am I correct ?
We need some sophisticate logic here, isnt it ? or am i missing some part
? Just confirm ... This is why we loosing few message when publish few
hundred messages.
I have solution here, will explain it later ...
--
Malinga Pathmal,
Technical Lead, WSO2, Inc. : http://wso2.com/
Phone : (+94) 715335898
--
Malinga Pathmal,
Technical Lead, WSO2, Inc. : http://wso2.com/
Phone : (+94) 715335898
--
Malinga Pathmal,
Technical Lead, WSO2, Inc. : http://wso2.com/
Phone : (+94) 715335898
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Fixing OOM issue in Message Broker
Hi,
When I'm working on the OOM issue in the MB, I have found that there is
some design imitations which will lead to it. Let me summaries my findings
in point form.
** Static Executor pool (org.wso2.andes.pool.AndesExecuter.java) issue,*
Unfortunately MB had static executor pool where we submit all the runnables
need to execute in parallel. Sooner when we send messages in a burst pool
will get exhausted (with runnables) and which will lead to increase the
internal queue backing the executor pool. Which will lead to OOM. This was
observed from the heap dump analysis with MAT.
*Solution* : We need to group the runnables based on the task nature
and send them to different executor pools.
** The way we inserting data to the Cassandra.*
Currently most of the time we are creating mutators and execute on the fly.
I have done some Benchmarks which simulate this situation. Following is
small code block for that ...
Sample ..
for (int i = 0; i 100; i++) {
MutatorString messageMutator = HFactory.createMutator(keyspaceOperator,
stringSerializer);
messageMutator.addInsertion(..);
messageMutator.execute();
}
When we looping and doing the execution on the fly, after few thousands
latency it takes very long time than we expected to execute the insertion.
According to the Cassandra documentations this will lead to another
side effect of sending so many messages over network, which will exhaust
network bandwidth too,
*Solution : Solution will be operate in batch mode. ex : BatchMutation*
** Message accumulation in LinkedBlockingQueue (observed from heap dump
with MAT)*
Inside the CassendraMessageStore we have used LinkedBlockingQueue to store
the messages temporary till we insert messages to Cassandra. But in there
we have huge bottleneck, producer to the queue inserting so fast but
consumer end its very slow. So this will lead to increase the blocking
queue and crate OOM.
*Solution : Add BatchMutation mode at the consumer end to make the consumer
fast. So Queue will have less messages in given time.*
** PublishMessageWriter run method is serial execution*
Inside the thread, we just take messages one by one and try to insert to
Cassandra, but when we have burst of messages this will create a
bottleneck. So we must introduce more parallelism to this.
-
Any ideas on this ?,
Note : I have done those above code changes and now MB can take messages in
a burst and work without OOM. But we need to design this and implement this
in a production quality mode.
--
Malinga Pathmal,
Technical Lead, WSO2, Inc. : http://wso2.com/
Phone : (+94) 715335898
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Issue in the org.wso2.carbon.event.core while getting subscription path
Hi, In the RegistryTopicManager.java class, inside getSubscriptionsPath(String) method there is an issue. Which will lead to erroneously deleting the Topics in the MB. In there it only checks EB_CONF_WS_SUBSCRIPTION_COLLECTION_NAME (ws.subscriptions). But if we have EB_CONF_JMS_SUBSCRIPTION_COLLECTION_NAME = jms.subscriptions, it wont check that condition. Shall i go and fix it or its as designed ? -Malinga ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Fresh 'svn co' compile issue
Hi all, I tried to mvn clean install on platform from outside the wso2 premises. but it failed unfortunately several times due to following error, http connection timeout to http://maven.wso2.org/nexus/content/groups/wso2-public/org/apache/sandesha2/sandesha2-persistence/1.6.1-wso2v1/sandesha2-persistence-1.6.1-wso2v1.pom o execute goal on project org.wso2.carbon.rm: Could not resolve dependencies for project org.wso2.carbon:org.wso2.carbon.rm:bundle:4.1.0: Failed to collect dependencies for [org.wso2.carbon:org.wso2.carbon.core:jar:4.1.0 (compile), org.apache.sandesha2:sandesha2:mar:1.6.1-wso2v1 (compile), org.apache.sandesha2:sandesha2-core:jar:1.6.1-wso2v1 (compile), org.apache.sandesha2:sandesha2-policy:jar:1.6.1-wso2v1 (compile), org.apache.sandesha2:sandesha2-rampart-integration:jar:1.6.1-wso2v1 (compile), org.apache.sandesha2:sandesha2-persistence:jar:1.6.1-wso2v1 (compile)]: Failed to read artifact descriptor for org.apache.sandesha2:sandesha2-persistence:jar:1.6.1-wso2v1: Could not transfer artifact org.apache.sandesha2:sandesha2-persistence:pom:1.6.1-wso2v1 from/to wso2-nexus (http://maven.wso2.org/nexus/content/groups/wso2-public/): Connection to http://maven.wso2.org refused: Connection timed out - [Help 1] but once I connect to wso2 VPN it was a success. After some research i have found myself that, http://maven.wso2.org could not be resolved outside the wso2 premises (may be due to proxy issue of wso2). and also i have found same repo also mirrored in http://dist.wso2.net/maven2 too which is accessible outside. --- If we can inform this to the carbon build team it would be better. affected module name is org.wso2.carbon.rm -Malinga ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
