[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_37198 ] Vadim Pesochinskiy commented on AMQ-855: I think you are right. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0.2, 4.0.1, 4.0 > Environment: any >Reporter: Vadim Pesochinskiy > Assigned To: Hiram Chirino >Priority: Critical > Fix For: 4.1 > > Attachments: ActiveMQMessageConsumer.java.patch3, > ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, > PrefetchSubscription.java.patch2, PrefetchSubscription.java.patch3, > PrefetchSubscription.patch, region.QueueSubscription.java.patch, > ZeroPrefetchConsumerTest.java.patch, ZeroPrefetchConsumerTest.java.patch3 > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36894 ] Vadim Pesochinskiy commented on AMQ-855: OK. I will take a look at it and the unit tests. Thanks. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy > Assigned To: Hiram Chirino >Priority: Critical > Fix For: 4.1 > > Attachments: ActiveMQMessageConsumer.patch, > ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36893 ] Hiram Chirino commented on AMQ-855: --- we need to also build some test cases for this. Any takers? > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy > Assigned To: Hiram Chirino >Priority: Critical > Fix For: 4.1 > > Attachments: ActiveMQMessageConsumer.patch, > ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36892 ] Hiram Chirino commented on AMQ-855: --- 'm going to start looking into it today. I may not use the DataResponse to return the message right away. I think I would still prefer to use the normal message dispatching flow and logic. What I'm thinking is that on timeout, or when receiveNoWait() is called, and no message has arrived to be dispatched, we dispatch a message to the consumer letting it know that no message was available. Then it can unblock. The nice thing about this is that you don't get timing issues between the client and server where the client times out but the broker actually dispatched the message. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy > Assigned To: Hiram Chirino >Priority: Critical > Fix For: 4.1 > > Attachments: ActiveMQMessageConsumer.patch, > ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Hi! Is there any chance someone can give me a rough estimate of when this issue can be looked at. I would really appreciate if someone reviewed the patch and told me if there are issues with it. Right now I have time to resolve them. Thanks. - Vadim. -- View this message in context: http://www.nabble.com/-jira--Created%3A-%28AMQ-855%29-Add-support-for-prefetchSize-%3D-0-tf2035055.html#a5952876 Sent from the ActiveMQ - Dev forum at Nabble.com.
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36782 ] Vadim Pesochinskiy commented on AMQ-855: James, I added implementation of pulling consumer whereby pullMessage() returns a response. Could you please browse through it and give me a feedback? Patches are for file versions PrefetechSubscription.java 1.15 and 1.22 ActiveMQMessageConsumer.java Current implementation does not work for me, because when recieve(long) call times out the prefetch window is increasing and we are back to where we started. Thanks a lot. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0.2, 4.0.1, 4.0 > Environment: any >Reporter: Vadim Pesochinskiy >Priority: Critical > Fix For: 4.1 > > Attachments: ActiveMQMessageConsumer.patch, PrefetchSubscription.patch > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36739 ] Maxim Fateev commented on AMQ-855: -- #1. I wasn't presize. Yes it holds references. But it doesn't really matter as these references are locked to the particular subsription and are not reassigned while this subscription is alive. #2. I think prefetch buffers are used to boost peformance. Pull model doesn't preclude their use. The question is if messages are assigned to subscriptions up front or only when space is available in prefetch buffers. I've already voted for it. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy >Priority: Critical > Fix For: 4.2 > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36738 ] Vadim Pesochinskiy commented on AMQ-855: #1. I thought that pending holds references, not messages. When message is actually dispatched it will become unavailable to other PrefetchSubscriptions. Maybe I am wrong. #2. I thought that push was employed primarely to boost performance. Whatever the reasons are, if you feel that this should be resolve could you please vote for it? Thanks a lot. Vadim. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy >Priority: Critical > Fix For: 4.2 > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36737 ] Maxim Fateev commented on AMQ-855: -- I think the failure scenario (slow consumer) is very common one and should be supported by ActiveMQ. The root cause of the problem is not the prefetch size. As James pointed out setting it to one is good enough. But the problem is that messages are assigned to consumers as soon as they are dispatched. If consuemer's prefetch buffer is full then they are kept in "pending" list (see PrefetchSubscription) implementation. This list is essentially unlimited. So messages assigned to slow consumer are never redispatched to overs unless slow consumer is disconnected or the whole broker is bounced. Standard way to solve this issue is to use "pull" model instead of "push". I.e. messages are not assigned to consumers up front but given to them when requested. I understand that "push" dispatch model of ActiveMQ is due to desire to support "selector expressions". But it doesn't mean that product should be unusable for customers that do not use selectors at all. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy >Priority: Critical > Fix For: 4.2 > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36736 ] Vadim Pesochinskiy commented on AMQ-855: > e.g. if we did implement prefetch of zero - which means don't deliver a > message to a consumer at all - > unless they perform a consumer.receive() - even then, the consumer could then > hang/deadlock and never > actually acknowledge or process the message. If consumer hangs and message is not processed is a different problem. Even if messages are lost in such case, nobody would blame AMQ. If you kill consumer and re-despatch, it would be brilliant. But if my consumers do not hang, crash or burn, can I just get my messages? James, could you please re-open this issue? I spend 1 month working on this project and now I will have to throw it all away and do explaining with my manager. I cannot apply my own fix, you will not fix it. What am I supposed to do now? > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy >Priority: Critical > Fix For: 4.2 > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36730 ] Vadim Pesochinskiy commented on AMQ-855: James & Hiram, I understand that you are trying to keep the scope of AMQ in check and I appreciate this as I believe we will benefit in quality. However, I just do not think I can proceed with using this product, because it does not satisfy my project's requirements. The requirements are not artificial and I am sure they cannot be resolved with either of the solutions that you proposed. I do not want to abuse your CPU with long explanation of why it is so. Due to whatever reasons I have multiple synchronous consumers (no listener, receive*(*) methods) and some of them are not used for extended periods of time. This results in the message broker not dispatching messages. Is this not a bug?!!! I do not know if setting prefetchSize=0 is the solution, but somehow AMQ needs to support the case when consumers are not getting any messages until they request them. - Vadim. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy >Priority: Critical > Fix For: 4.2 > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[
https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36718 ]
james strachan commented on AMQ-855:
BTW just to be absolutely clear - here's how I think we should fix Vadim's use
case.
If we had full support for priority based dispatching so that...
* all high priority messages are dispatched before *ANY* medium priority.
* all medium priority messages are dispatched before *ANY* low priority
* all low priority messages are then dispatched
Then the consumers all have a prefetch of 1 and each consumer subscribes to
high, medium and low priority messages.
The reason I don't see the prefetch of zero fixing the issue is as follows..
imagine the client did this...
message = highConsumer.receive(timeout);
if (message == null) {
// a high message could have just arrived now...
message = mediumConsumer.receive(timeout);
// a high message could have just arrived now...
}
if (message == null) {
// a high message could have just arrived now...
message = lowConsumer.receive(timeout);
// a high message could have just arrived now...
}
// a high message could have just arrived now...
there's still a chance the client would miss the high priority messages
arriving as they will spend a reasonable amount of time in wait loops waiting
to request-response to the broker.
The only way to truly guarrentee ordering is to get the broker to do it.
> Add support for prefetchSize = 0
>
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
>Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
>Reporter: Vadim Pesochinskiy
>Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10
> min, 1 min, 1 min. This sequence should finish in 10 minutes as one service
> will pick up the 10 minutes job, meanwhile the other one should manage the
> two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute
> jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes
> instead of 10.
> This is simplification of the real scenario where I have about 30 consumers
> submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors,
> which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after
> Job # 1500. Since senders are synchronously blocked this can result in
> time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system
> cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36714 ] Vadim Pesochinskiy commented on AMQ-855: I agree this functionality is essential for slow consumers. I do not see the hacking part in it. If prefetchSize is set to 0 client cannot expect messages dispatched to it, so it has to send an ack to get next msg. Of course push will work better when a lot of messages are sent. In this case AMQ-850 will fix it by timing out and disabling passive consumers. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy >Priority: Critical > Fix For: 4.2 > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36713 ] Andrew Lusk commented on AMQ-855: - Hiram, I don't really understand your comments here. There is a huge need for this. A push model just isn't appropriate for queues. Now, you could in theory hack together a "pull" model by setting prefetch size = 0, so that basically each ack is a request for the next message. This method is slower, but it does scale out as you add more consumers. This is a big problem for our internal customers. Systems like this need to be designed under the assumption that clients will not behave themselves. They will deadlock themselves, slow down arbitrarily, boxes will go up and down, etc. These things happen all the time in real life and shouldn't have adverse effects on other, well-behaved consumers. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy >Priority: Critical > Fix For: 4.2 > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36712 ] Vadim Pesochinskiy commented on AMQ-855: Well... I think there is a valid need ;-). I think that requesting not to create idle consumers is not valid. The solution that you proposed does not work for me. In my case JMS does not satisfy the complicated on-the-fly changing priorities that my system has. This is why I have to fiddle with the queues and multiple consumers, some of which maybe idle in certain cases. I do not see why this is not a valid requirement. I am willing to contribute time to solve this issue. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy >Priority: Critical > Fix For: 4.2 > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36711 ] Hiram Chirino commented on AMQ-855: --- I don't think this is a valid need. If each server/worker thread only creates 1 consumer with a prefetch of 1 then the jobs will be proccessed in 11 minutes. I think the problem is occuring becuse each sever/work thread is creating multiple consumers and each has a prefetch. Mutliple consumers can be avoided by using consumers that use destination wild cards. > Add support for prefetchSize = 0 > > > Key: AMQ-855 > URL: https://issues.apache.org/activemq/browse/AMQ-855 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 4.0, 4.0.1, 4.0.2 > Environment: any >Reporter: Vadim Pesochinskiy >Priority: Critical > Fix For: 4.2 > > > This feature would enable to support following test case: > 2 servers are processing 3 submitted jobs with following processing times 10 > min, 1 min, 1 min. This sequence should finish in 10 minutes as one service > will pick up the 10 minutes job, meanwhile the other one should manage the > two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute > jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes > instead of 10. > This is simplification of the real scenario where I have about 30 consumers > submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems: > Messages are sitting in prefetch buffer are not available to processors, > which results in a lot of idle time. > Order of processing is random. For some reason Job # 20 is processed after > Job # 1500. Since senders are synchronously blocked this can result in > time-outs. > Some requests are real-time, i.e. there is a user waiting, so the system > cannot wait, so AMQ-850 does not fix this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
