Re: In qpidd what's the maximum number of subscriptions to a queue node
On Sun, 2015-07-12 at 09:14 +0100, Fraser Adams wrote: > > As I say I think that ActiveMQ Apollo started out because of scaling > limitations of the original ActiveMQ and has evolved to a reactor > based > threading model built on hawt-dispatch (Java implementation of Grand > Central Dispatch https://en.wikipedia.org/wiki/Grand_Central_Dispatch > ) - > so a task parallel threading model rather than a shared state > threading > model. Given that the Proton reactor stuff seems to be, well, a > reaction > :-[ to this problem space it got me curious about the Qpid brokers. The proton reactive APIs are an attempt to find an API that works well for servers AND clients. Procedural APIs are good for clients but problematic on brokers which must "react" to unpredictable client activity. This aspect is not really related to performance. The proton engine is a single-threaded event processing state machine. To scale across cores you need to use epoll or similar to dispatch from a large number of connections to to a small thread pool that serializes work to each proton engine. That is exactly what qpidd and dispatch do. The proton C reactor also has events that can hook directly into epoll or select, e.g. accepting incoming connections. This is useful for writing portable (i.e. epoll/select agnostic) *singe threaded* servers, but it doesn't help with multi-core. I'm not sure yet if the C reactor can/should be extended to multi-core or if concurrency support belongs in the language binding layer. This is especially true of languages like Go with built-in concurrency support. See http://qpid.apache.org/proton/go/ for more - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: In qpidd what's the maximum number of subscriptions to a queue node
On 11 July 2015 at 17:52, Fraser Adams wrote: > Hey all, > Suppose I have a queue on qpidd, I know that I can have multiple consumer > clients subscribing to that queue node - I've used that several times to > provide a means of scaling out consumers, so if I have "n" consumers each > consumer receives roughly 1/n of the messages published onto the queue. > > I've never really pushed the limits of this and have tended to only have 10 > - 20 consumers, but it got me wondering: > a) What's the maximum theoretical limit for the number of consumers on such > a shared resource > b) what's the maximum practical limit - I'm not sure how the qpidd threading > model works (I *think* it's per connection), so I'm wondering at what point > we get into a position where the "stampeding herds" lock contention problem > kicks in. > > I know I should probably just stand up a test and give it a whirl, but > figured I'd ask first :-) > > I *think* that the answer to a) is that the theoretical limit is the maximum > number of link handles and is a 32 bit unsigned int so would be 4294967295, > though I suspect something else would get a bit sad before that limit is > reached. > Assuming you are talking about AMQP 1.0 link handles, I think those are assigned per-session, of which there can be 2^16 (0 is a valid value) per-connection, so the theoretical limit would actually be much higher. > > On point b. has anyone (most likely Gordon) explored the scaling limits of > qpidd? Obviously when Qpid started out servers tended to have something > between one and four cores, but now of course Moore's law tends to be > followed by increasing the number of cores, so I'm curious as to how qpidd > scales. I'm thinking of things like ActiveMQ Apollo where as I understand it > it uses hawt-dispatch (which I think is the Java equivalent of Apple's Grand > Central Dispatch pattern) and I know with Proton there has been a lot of > work migrating to a more reactive pattern, which makes me wonder if that's > an acknowledgement of any potential scaling limits in qpidd at present? If > there are known limits is there a roadmap to change the threading model. I'm > mostly just curious at the moment, but I guess it's a question that's going > to crop up more and more. > > Cheers, > Frase > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: In qpidd what's the maximum number of subscriptions to a queue node
On 07/11/2015 05:52 PM, Fraser Adams wrote: On point b. has anyone (most likely Gordon) explored the scaling limits of qpidd? Obviously when Qpid started out servers tended to have something between one and four cores, but now of course Moore's law tends to be followed by increasing the number of cores, so I'm curious as to how qpidd scales. There hasn't really been sufficient focus on different aspects of scalability in general. However in tests where a moderate number of connections (typically 10s, rather than 100s) transferred messages at their maximum rate, did not show significant benefit in increasing the number of worker-threads above 8 even if there were more cores available. These tests were focused on throughput and not on the number of connections, so I would always advise some benchmarking of scenarios similar to those anticipated in deployment. I'm thinking of things like ActiveMQ Apollo where as I understand it it uses hawt-dispatch (which I think is the Java equivalent of Apple's Grand Central Dispatch pattern) and I know with Proton there has been a lot of work migrating to a more reactive pattern, which makes me wonder if that's an acknowledgement of any potential scaling limits in qpidd at present? If there are known limits is there a roadmap to change the threading model. I think the limits are less in the threading model, which I think is a reasonable basis for scalable operation, and more in the locking and memory management. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: In qpidd what's the maximum number of subscriptions to a queue node
Hi Steve, Thanks for the response. Sorry yeah I remembered after I posted that there was a thread pool that was number of cores + 1, What I was meaning (in my own mind at least, if not what I wrote :-D ) was that the threading was *related* to connection (as opposed to per connection D'oh!!) that is to say if you have one connection with multiple sessions it will be bound to a single thread in the pool, so if you want concurrency on qpidd you would need to establish multiple connections (which isn't necessarily something you'd expect if you were used to JMS where the session is the unit of concurrency on the client side). On my thread saturation point I was really meaning that as the number of cores goes up the lock contention problem gets exponentially worse, so say 24 connections running flat out on say a four core box is one thing, but I'm wondering about the same 24 connections on a 24 core box and at what point will lock contention become the bottleneck? As I say I think that ActiveMQ Apollo started out because of scaling limitations of the original ActiveMQ and has evolved to a reactor based threading model built on hawt-dispatch (Java implementation of Grand Central Dispatch https://en.wikipedia.org/wiki/Grand_Central_Dispatch) - so a task parallel threading model rather than a shared state threading model. Given that the Proton reactor stuff seems to be, well, a reaction :-[ to this problem space it got me curious about the Qpid brokers. F. On 12/07/15 00:28, Steve Huston wrote: Hi Frase, You’ve done a good job at guessing the limits - I’ll let Gordon educate us on the rest. The threading model, though - definitely not per connection. There’s a pool of threads that can handle I/O - by default there are (number of cores) + 1, but you can change it by command line option. The I/O model is proactive (as opposed to reactive) and those threads handle what ever handles get I/O completions. Yes, at some point, it gets saturated but it’s driven by the amount of I/O going on concurrently, not by the sheer number of connections. So go for it - let us know how high you (well, the number of clients) can get :-) -Steve On Jul 11, 2015, at 12:52 PM, Fraser Adams wrote: Hey all, Suppose I have a queue on qpidd, I know that I can have multiple consumer clients subscribing to that queue node - I've used that several times to provide a means of scaling out consumers, so if I have "n" consumers each consumer receives roughly 1/n of the messages published onto the queue. I've never really pushed the limits of this and have tended to only have 10 - 20 consumers, but it got me wondering: a) What's the maximum theoretical limit for the number of consumers on such a shared resource b) what's the maximum practical limit - I'm not sure how the qpidd threading model works (I *think* it's per connection), so I'm wondering at what point we get into a position where the "stampeding herds" lock contention problem kicks in. I know I should probably just stand up a test and give it a whirl, but figured I'd ask first :-) I *think* that the answer to a) is that the theoretical limit is the maximum number of link handles and is a 32 bit unsigned int so would be 4294967295, though I suspect something else would get a bit sad before that limit is reached. On point b. has anyone (most likely Gordon) explored the scaling limits of qpidd? Obviously when Qpid started out servers tended to have something between one and four cores, but now of course Moore's law tends to be followed by increasing the number of cores, so I'm curious as to how qpidd scales. I'm thinking of things like ActiveMQ Apollo where as I understand it it uses hawt-dispatch (which I think is the Java equivalent of Apple's Grand Central Dispatch pattern) and I know with Proton there has been a lot of work migrating to a more reactive pattern, which makes me wonder if that's an acknowledgement of any potential scaling limits in qpidd at present? If there are known limits is there a roadmap to change the threading model. I'm mostly just curious at the moment, but I guess it's a question that's going to crop up more and more. Cheers, Frase - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: In qpidd what's the maximum number of subscriptions to a queue node
Hi Frase, You’ve done a good job at guessing the limits - I’ll let Gordon educate us on the rest. The threading model, though - definitely not per connection. There’s a pool of threads that can handle I/O - by default there are (number of cores) + 1, but you can change it by command line option. The I/O model is proactive (as opposed to reactive) and those threads handle what ever handles get I/O completions. Yes, at some point, it gets saturated but it’s driven by the amount of I/O going on concurrently, not by the sheer number of connections. So go for it - let us know how high you (well, the number of clients) can get :-) -Steve > On Jul 11, 2015, at 12:52 PM, Fraser Adams > wrote: > > Hey all, > Suppose I have a queue on qpidd, I know that I can have multiple consumer > clients subscribing to that queue node - I've used that several times to > provide a means of scaling out consumers, so if I have "n" consumers each > consumer receives roughly 1/n of the messages published onto the queue. > > I've never really pushed the limits of this and have tended to only have 10 - > 20 consumers, but it got me wondering: > a) What's the maximum theoretical limit for the number of consumers on such a > shared resource > b) what's the maximum practical limit - I'm not sure how the qpidd threading > model works (I *think* it's per connection), so I'm wondering at what point > we get into a position where the "stampeding herds" lock contention problem > kicks in. > > I know I should probably just stand up a test and give it a whirl, but > figured I'd ask first :-) > > I *think* that the answer to a) is that the theoretical limit is the maximum > number of link handles and is a 32 bit unsigned int so would be 4294967295, > though I suspect something else would get a bit sad before that limit is > reached. > > > On point b. has anyone (most likely Gordon) explored the scaling limits of > qpidd? Obviously when Qpid started out servers tended to have something > between one and four cores, but now of course Moore's law tends to be > followed by increasing the number of cores, so I'm curious as to how qpidd > scales. I'm thinking of things like ActiveMQ Apollo where as I understand it > it uses hawt-dispatch (which I think is the Java equivalent of Apple's Grand > Central Dispatch pattern) and I know with Proton there has been a lot of work > migrating to a more reactive pattern, which makes me wonder if that's an > acknowledgement of any potential scaling limits in qpidd at present? If there > are known limits is there a roadmap to change the threading model. I'm mostly > just curious at the moment, but I guess it's a question that's going to crop > up more and more. > > Cheers, > Frase > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org >
In qpidd what's the maximum number of subscriptions to a queue node
Hey all, Suppose I have a queue on qpidd, I know that I can have multiple consumer clients subscribing to that queue node - I've used that several times to provide a means of scaling out consumers, so if I have "n" consumers each consumer receives roughly 1/n of the messages published onto the queue. I've never really pushed the limits of this and have tended to only have 10 - 20 consumers, but it got me wondering: a) What's the maximum theoretical limit for the number of consumers on such a shared resource b) what's the maximum practical limit - I'm not sure how the qpidd threading model works (I *think* it's per connection), so I'm wondering at what point we get into a position where the "stampeding herds" lock contention problem kicks in. I know I should probably just stand up a test and give it a whirl, but figured I'd ask first :-) I *think* that the answer to a) is that the theoretical limit is the maximum number of link handles and is a 32 bit unsigned int so would be 4294967295, though I suspect something else would get a bit sad before that limit is reached. On point b. has anyone (most likely Gordon) explored the scaling limits of qpidd? Obviously when Qpid started out servers tended to have something between one and four cores, but now of course Moore's law tends to be followed by increasing the number of cores, so I'm curious as to how qpidd scales. I'm thinking of things like ActiveMQ Apollo where as I understand it it uses hawt-dispatch (which I think is the Java equivalent of Apple's Grand Central Dispatch pattern) and I know with Proton there has been a lot of work migrating to a more reactive pattern, which makes me wonder if that's an acknowledgement of any potential scaling limits in qpidd at present? If there are known limits is there a roadmap to change the threading model. I'm mostly just curious at the moment, but I guess it's a question that's going to crop up more and more. Cheers, Frase - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org