Re: Proton Performance Pictures (1 of 2)
On 04/09/14 01:34, Alan Conway wrote: On Thu, 2014-09-04 at 00:38 +0200, Leon Mlakar wrote: On 04/09/14 00:25, Alan Conway wrote: On Wed, 2014-09-03 at 12:16 -0400, Michael Goulish wrote: OK -- I just had a quick talk with Ted, and this makes sense to me now: count *receives* per second. I had it turned around and was worried about *sends* per second, and then got confused by issues of fanout. If you only count *receives* per second, and assume no discards, it seems to me that you can indeed make a fair speed comparison between sender --> receiver sender --> intermediary --> receiver and sender --> intermediary --> {receiver_1 ... receiver_n} and even sender --> {arbitrary network of intermediaries} --> {receiver_1 ... receiver_n} phew. So I will do it that way. That's right for throughput, but don't forget latency. A well behaved intermediary should have little effect on throughput but will inevitably add latency. Measuring latency between hosts is a pain. You can time-stamp messages at the origin host but clock differences can give you bogus numbers if you compare that to the time on a different host when the messages arrive. One trick is to have the messages arrive back at the same host where you time-stamped them (even if they pass thru other hosts in between) but that isn't always what you really want to measure. Maybe there's something to be done with NNTP, I've never dug into that. Have fun! To get a reasonably good estimate of the time difference between sender an receiver, one could exchange several timestamped messages, w/o intermediary, in both directions and get both sides to agree on the difference between them. Do that before the test, and then repeat the exchange at the end of the test to check for the drift. This of course assumes stable network latencies during these exchanges and is usable only in test environments. Exchanging several messages instead of just one should help eliminating sporadic instabilities. As I understand it that's pretty much what NTP does. http://en.wikipedia.org/wiki/Network_Time_Protocol says that NTP "can achieve better than one millisecond accuracy in local area networks under ideal conditions." That doesn't sound good enough to measure sub-millisecond latencies. I doubt that a home grown attempt at timing message exchanges will do better than NTP :( NTP may deserve further investigation however, Wikipedia probably makes some very broad assumptions about what your "ideal network conditions" are, its possible that it can be tuned better than that. I can easily get sub-millisecond round-trip latencies out of Qpid with a short message burst: qpidd --auth=no --tcp-nodelay qpid-cpp-benchmark --connection-options '{tcp-nodelay:true}' -q1 -m100 send-tp recv-tp l-min l-max l-avg total-tp 38816 30370 0.211.180.703943 Sadly if I tell qpidd to use AMQP 1.0 (and therefore proton), things degenerate very badly from a latency perspective. qpid-cpp-benchmark --connection-options '{tcp-nodelay:true,protocol:amqp1.0}' -q1 -m100 send-tp recv-tp l-min l-max l-avg total-tp 26086 19552 3.136.655.28913 However this may not be protons fault, the problem could be in qpidd's AMQP1.0 adapter layer. I'm glad to see that we're starting to measure these things for proton and dispatch, that will surely lead to improvement. Yes, you are correct, that's basically what NTP does ... and neither will work well with sub-millisecond ranges. I didn't realize that this is what you are after. There is a beast called http://en.wikipedia.org/wiki/Precision_Time_Protocol, though. A year ago we took a brief look into this but concluded that millisecond accuracy was good enough and that it was not worth the effort. And of course, it is also possible to attach a GPS receiver to both sending and receiving host. With decent drivers this should provide at least microsecond accuracy. Leon
Re: Proton Performance Pictures (1 of 2)
On 04/09/14 00:25, Alan Conway wrote: On Wed, 2014-09-03 at 12:16 -0400, Michael Goulish wrote: OK -- I just had a quick talk with Ted, and this makes sense to me now: count *receives* per second. I had it turned around and was worried about *sends* per second, and then got confused by issues of fanout. If you only count *receives* per second, and assume no discards, it seems to me that you can indeed make a fair speed comparison between sender --> receiver sender --> intermediary --> receiver and sender --> intermediary --> {receiver_1 ... receiver_n} and even sender --> {arbitrary network of intermediaries} --> {receiver_1 ... receiver_n} phew. So I will do it that way. That's right for throughput, but don't forget latency. A well behaved intermediary should have little effect on throughput but will inevitably add latency. Measuring latency between hosts is a pain. You can time-stamp messages at the origin host but clock differences can give you bogus numbers if you compare that to the time on a different host when the messages arrive. One trick is to have the messages arrive back at the same host where you time-stamped them (even if they pass thru other hosts in between) but that isn't always what you really want to measure. Maybe there's something to be done with NNTP, I've never dug into that. Have fun! To get a reasonably good estimate of the time difference between sender an receiver, one could exchange several timestamped messages, w/o intermediary, in both directions and get both sides to agree on the difference between them. Do that before the test, and then repeat the exchange at the end of the test to check for the drift. This of course assumes stable network latencies during these exchanges and is usable only in test environments. Exchanging several messages instead of just one should help eliminating sporadic instabilities. Leon
Re: Data interface in Proton-J
Fetched them, thank you. It does help indeed, At the first glance it looks like more than just 30% done. I even manged to integrate the stuff into my project and get the DataImpl instance through the factory with no problems. Will take a closer look in the next days, though. All this proton stuff is still pretty much new to me, so I might have some basic questions (like what exactly should narrow and widen do :-) later on. But let me first got through the sources and dig through whatever archives I manage to find. Cheers, Leon On 6/18/13 10:30 AM, Rob Godfrey wrote: OK - I've just committed [1] the initial pass at the Data implementation. Not yet implemented are the copy, append, narrow and widen operations... or much in the way of checking errors are handled the same way as the C, etc. If find issues and/or want to contribute patches you can attach comments/patch files to the JIRA [2]. Hope this helps, Rob [1] https://svn.apache.org/r1494067 [2] https://issues.apache.org/jira/browse/PROTON-339 On 14 June 2013 16:13, Rob Godfrey wrote: T [...]
Re: Data interface in Proton-J
Hi Rob, thanks for reply. It got me thinking ... since Data interface is not a history and will be around, to do things The Right Way I'll have to implement parts of it myself anyway. So I might as well take a shot in adding missing 70%, or at least some part of them, on top of your 30% :-). Is there any chance for you to you dig up what you've got so far and get it over to me somehow? Best regards, Leon Hi Leon, As you have noticed a pure Java implementation of the Data interface is not yet available (I think I have a 30% complete implementation sitting somewhere on an old laptop, but I never got round to finishing it). It will get implemented eventually, but it's not currently being worked on as far as I know. The Java implementation was built upon a codec which serializes objects in Java form into AMQP binary data. On the Java implementation of the Message class there is a method setBody(Section body). So, in Java, to send an "amqp value" message containing a map, you could do something like: import org.apache.qpid.proton.amqp.messaging.AmqpValue; import org.apache.qpid.proton.amqp.Symbol; ... Map map = new LinkedHashMap(); map.put("symbol", Symbol.valueOf("a symbol")); map.put("int", 21); ... AmqpValue body = new AmqpBody(map); message.setBody(body); Hope this helps, Rob On 14 June 2013 15:22, Leon Mlakar wrote: Hello everybody, I've been following this list for quite a while but so far had no reason to take part in discussions. However, these days I'm porting some C++ code that is using Proton-C to talk to peers to Java. The C++ code uses pn_data_put_* functions from codec module to compose the body of the message and pn_data_get_* counterparts to decompose it at the other end. In Java, the Proton-Api's Data interface (org.apache.qpid.proton.codec) seems to provide the same functionality. But as far as I could see no class is actually implementing the interface, and the DataFactoryImpl.createData(**long) is only throwing the exception which indicates that this operation is not supported. I've seen this in both 0.4 package and on the Git's trunk. Is this a part of functionality that is merely not implemented yet, or something that is not planned to get implemented at all? In either case, what would be the alternate methods to put together the message body in a way that could be understood by C++ code? And vice versa? The use of Java bindings to Proton-C is not the preferred way to go. Thanks and best regards, Leon
Data interface in Proton-J
Hello everybody, I've been following this list for quite a while but so far had no reason to take part in discussions. However, these days I'm porting some C++ code that is using Proton-C to talk to peers to Java. The C++ code uses pn_data_put_* functions from codec module to compose the body of the message and pn_data_get_* counterparts to decompose it at the other end. In Java, the Proton-Api's Data interface (org.apache.qpid.proton.codec) seems to provide the same functionality. But as far as I could see no class is actually implementing the interface, and the DataFactoryImpl.createData(long) is only throwing the exception which indicates that this operation is not supported. I've seen this in both 0.4 package and on the Git's trunk. Is this a part of functionality that is merely not implemented yet, or something that is not planned to get implemented at all? In either case, what would be the alternate methods to put together the message body in a way that could be understood by C++ code? And vice versa? The use of Java bindings to Proton-C is not the preferred way to go. Thanks and best regards, Leon