[jira] [Comment Edited] (QPID-4926) C++ CMake build system needs to include all of the tests from the autotools build
[ https://issues.apache.org/jira/browse/QPID-4926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13683855#comment-13683855 ] Andrew Stitcher edited comment on QPID-4926 at 6/14/13 10:28 PM: - I've added in the tests: * sasl_fed * sasl_fed_ex_dynamic * sasl_fed_ex_link * sasl_fed_ex_queue * sasl_fed_ex_route * ssl_test * msg_group_tests * cli_tests There may some more to add (especially in the msg_store area) but this covers all the regularly run tests I think. I'll leave this open for a few more days in case there are some more I notice. was (Author: astitcher): I've added in the tests: sasl_fed sasl_fed_ex_dynamic sasl_fed_ex_link sasl_fed_ex_queue sasl_fed_ex_route ssl_test msg_group_tests There may some more to add (especially in the msg_store area) but this covers all the regularly run tests I think. I'll leave this open for a few more days in case there are some more I notice. > C++ CMake build system needs to include all of the tests from the autotools > build > - > > Key: QPID-4926 > URL: https://issues.apache.org/jira/browse/QPID-4926 > Project: Qpid > Issue Type: Improvement > Components: Build Tools, C++ Broker, C++ Client >Reporter: Andrew Stitcher > > Currently the CMake build system does not include all the tests that the > autotools build system has. > In order to move fully over to cmake we need to be able run all the tests > under cmake. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-4926) C++ CMake build system needs to include all of the tests from the autotools build
[ https://issues.apache.org/jira/browse/QPID-4926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13683860#comment-13683860 ] ASF subversion and git services commented on QPID-4926: --- Commit 1493267 from [~astitcher] [ https://svn.apache.org/r1493267 ] QPID-4926: Add in overlooked cli_tests > C++ CMake build system needs to include all of the tests from the autotools > build > - > > Key: QPID-4926 > URL: https://issues.apache.org/jira/browse/QPID-4926 > Project: Qpid > Issue Type: Improvement > Components: Build Tools, C++ Broker, C++ Client >Reporter: Andrew Stitcher > > Currently the CMake build system does not include all the tests that the > autotools build system has. > In order to move fully over to cmake we need to be able run all the tests > under cmake. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-4926) C++ CMake build system needs to include all of the tests from the autotools build
[ https://issues.apache.org/jira/browse/QPID-4926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13683855#comment-13683855 ] Andrew Stitcher commented on QPID-4926: --- I've added in the tests: sasl_fed sasl_fed_ex_dynamic sasl_fed_ex_link sasl_fed_ex_queue sasl_fed_ex_route ssl_test msg_group_tests There may some more to add (especially in the msg_store area) but this covers all the regularly run tests I think. I'll leave this open for a few more days in case there are some more I notice. > C++ CMake build system needs to include all of the tests from the autotools > build > - > > Key: QPID-4926 > URL: https://issues.apache.org/jira/browse/QPID-4926 > Project: Qpid > Issue Type: Improvement > Components: Build Tools, C++ Broker, C++ Client >Reporter: Andrew Stitcher > > Currently the CMake build system does not include all the tests that the > autotools build system has. > In order to move fully over to cmake we need to be able run all the tests > under cmake. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-4926) C++ CMake build system needs to include all of the tests from the autotools build
[ https://issues.apache.org/jira/browse/QPID-4926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13683852#comment-13683852 ] ASF subversion and git services commented on QPID-4926: --- Commit 1493264 from [~astitcher] [ https://svn.apache.org/r1493264 ] QPID-4926: Add in tests missing from the cmake build > C++ CMake build system needs to include all of the tests from the autotools > build > - > > Key: QPID-4926 > URL: https://issues.apache.org/jira/browse/QPID-4926 > Project: Qpid > Issue Type: Improvement > Components: Build Tools, C++ Broker, C++ Client >Reporter: Andrew Stitcher > > Currently the CMake build system does not include all the tests that the > autotools build system has. > In order to move fully over to cmake we need to be able run all the tests > under cmake. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Created] (QPID-4928) Some of C++ CMake build tests need to start/stop a broker themselves
Andrew Stitcher created QPID-4928: - Summary: Some of C++ CMake build tests need to start/stop a broker themselves Key: QPID-4928 URL: https://issues.apache.org/jira/browse/QPID-4928 Project: Qpid Issue Type: Improvement Reporter: Andrew Stitcher CMake has a companion tool CTest which can be used to run tests individually. This means that each of the tests must be self contained. Currently a bunch of the tests rely on a previous test "start_broker" to start a required broker which is stopped by a subsequent test "stop_broker". The start and stop steps should be brought into the individual tests that need a broker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Created] (QPID-4927) C++ CMake tests need to use valgrind the same way that the autotools build does
Andrew Stitcher created QPID-4927: - Summary: C++ CMake tests need to use valgrind the same way that the autotools build does Key: QPID-4927 URL: https://issues.apache.org/jira/browse/QPID-4927 Project: Qpid Issue Type: Improvement Reporter: Andrew Stitcher To catch the same range of errors the cmake build tests need to use valgrind to run the C++ broker and C++ clients in the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Created] (QPID-4926) C++ CMake build system needs to include all of the tests from the autotools build
Andrew Stitcher created QPID-4926: - Summary: C++ CMake build system needs to include all of the tests from the autotools build Key: QPID-4926 URL: https://issues.apache.org/jira/browse/QPID-4926 Project: Qpid Issue Type: Improvement Components: Build Tools, C++ Broker, C++ Client Reporter: Andrew Stitcher Currently the CMake build system does not include all the tests that the autotools build system has. In order to move fully over to cmake we need to be able run all the tests under cmake. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-4925) [Java Broker] Timeout connections which do not complete connection handshake
[ https://issues.apache.org/jira/browse/QPID-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13683768#comment-13683768 ] ASF subversion and git services commented on QPID-4925: --- Commit 1493240 from [~godfrer] [ https://svn.apache.org/r1493240 ] QPID-4925 : [Java Broker] Timeout connections which do not complete connection handshake > [Java Broker] Timeout connections which do not complete connection handshake > > > Key: QPID-4925 > URL: https://issues.apache.org/jira/browse/QPID-4925 > Project: Qpid > Issue Type: Bug > Components: Java Broker >Reporter: Rob Godfrey >Assignee: Rob Godfrey > Fix For: 0.23 > > > Clients opening connections to the broker which do not complete the initial > handshake consume resources and are not visible through management tooling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Created] (QPID-4925) [Java Broker] Timeout connections which do not complete connection handshake
Rob Godfrey created QPID-4925: - Summary: [Java Broker] Timeout connections which do not complete connection handshake Key: QPID-4925 URL: https://issues.apache.org/jira/browse/QPID-4925 Project: Qpid Issue Type: Bug Components: Java Broker Reporter: Rob Godfrey Assignee: Rob Godfrey Fix For: 0.23 Clients opening connections to the broker which do not complete the initial handshake consume resources and are not visible through management tooling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Updated] (QPID-4348) HA Use independent sequence numbers for identifying messages.
[ https://issues.apache.org/jira/browse/QPID-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Conway updated QPID-4348: -- Description: Currently HA code uses queue sequence numbers to identify messasges. This assumes that message sequence is identical on primary and backup. Implementing new features (for example transactions) requires that we tolerate ordering differences between primary and backups. We introduces a new, queue-scoped HA sequence number managed by the HA plugin. This is used to identify messages independently of queue sequence numbers. Needs careful implementation to avoid bad performance effects. - Avoid adding headers, use the same mechansim as we have now for sending HA seq. nos. - Use an efficient index struct (e.g. dequeue) to map HA seq. to queue seq. The benefits: - required to support TX - required for re-use of stored messages for catchup - more robust - does not require primary and backups to have identical message ordering on their queues. was: Currently HA code uses queue sequence numbers to identify messasges. This assumes that message sequence is identical on primary and backup. Implementing new features (for example transactions) requires that we tolerate ordering differences between primary and backups. We introduces a new, queue-scoped HA sequence number managed by the HA plugin. This is used to identify messages independently of queue sequence numbers. Needs careful implementation to avoid bad performance effects. - Avoid adding headers, use the same mechansim as we have now for sending HA seq. nos. - Use an efficient index struct (e.g. dequeue) to map HA seq. to queue seq. The benefits: - required to support TX - required for re-use of stored messages for catchup - more robust - aovidb > HA Use independent sequence numbers for identifying messages. > - > > Key: QPID-4348 > URL: https://issues.apache.org/jira/browse/QPID-4348 > Project: Qpid > Issue Type: Improvement > Components: C++ Clustering >Affects Versions: 0.18 >Reporter: Alan Conway >Assignee: Alan Conway > > Currently HA code uses queue sequence numbers to identify messasges. > This assumes that message sequence is identical on primary and backup. > Implementing new features (for example transactions) requires that we tolerate > ordering differences between primary and backups. > We introduces a new, queue-scoped HA sequence number managed by the HA plugin. > This is used to identify messages independently of queue sequence numbers. > Needs careful implementation to avoid bad performance effects. > - Avoid adding headers, use the same mechansim as we have now for sending HA > seq. nos. > - Use an efficient index struct (e.g. dequeue) to map HA seq. to queue seq. > The benefits: > - required to support TX > - required for re-use of stored messages for catchup > - more robust - does not require primary and backups to have identical > message ordering on their queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Re: Qpid post-mortem and request for suggestions for (my) next release challenge (10M msgs/sec on Windows)
On existing broker failover - can you point me to where that behavior is documented? Because neither myself or anyone on the four teams I work with has come across the functionality you describe. I've never seen a client failover to another broker, only code to attempt to reconnect. Basic features we need: - externally adjustable retry / timeout on connections - to handle differences between LAN, WAN, and satellite internet. - updating broker list: How do you do this? Never seen it... - to prevent network splits, how are recovered brokers monitored? When a failed broker recovers, do clients switch back? How often / aggressively checked? - how is the application notified on broker failure, connection failover, recovery? Finally, we were ending up with LOTS of application complexity in SOA code when broker failure / recovery meant connection, sender and receiver objects had to be recreated. This was compounded by Connection being a different types of boost object than senders and receivers. We ended up building a bunch of code we were using everywhere to handle this, ended up collapsing it into a layer that just exposes send(queue name, message) and listen( queue name, callback) with all the functionality underneath, plus callback for status messages (broker failure, connection failover, recovery). On connection failure detection, I'll dig up my notes and send to the list, hopefully this has been cleaned up since 0.14. And anything you can think of for dynamically load balancing across brokers? Greatly appreciate the feedback and input... Kerry On Fri, Jun 14, 2013 at 4:09 AM, Gordon Sim wrote: > On 06/13/2013 02:37 PM, Kerry Bonin wrote: > >> I'm the system architect for a large program that has just shipped a major >> product, and Qpid is one of the foundations of our infrastructure. I >> thought I'd share a few things, and ask a few questions about my next >> step... >> > > Thanks for taking the time Kerry, it's always great to get feedback! > > >We shipped using Qpid 0.14, am updating now. Our product is (currently) >> 100% Windows, although most subsystems are implemented in cross-platform >> C++ or Python. We are essentially a video surveillance and access control >> system, so we have high volumes of events, low millions per day in bursts >> up into the low hundreds per second. In addition to event transport, our >> infrastructure is a ESB model SOA built over Qpid. We have high >> reliability >> requirements – no single points of failure, fast failover and recovery, >> active-active services, load balancing, and encryption throughout all. >> >> Our biggest challenges came from the large reliability gap between the >> Windows and *nix implementations. We've contributed all of our fixes back, >> but under high load we had issues that burnt a few man months of senior >> developer time. Its now running reasonably well for us. >> >> Broker failover was a challenge. We ended up building a wrapper over the >> Qpid client library to abstract all the connection, sender, receiver, ect. >> objects, so applications didn't have to deal with tearing down and >> building >> up these objects when a broker died. (Which happened often under high load >> in early testing, and still happens often in our worst case testbeds where >> we take them down or break connections to test reliability.) Since >> federation didn't work on Windows, system splits were unacceptable, so we >> also had to implement failover recovery. This also required distributed >> and maintaining an ordered list of brokers. Again, this was a pain, but is >> now working. It would be nice if the client handled these things for us. >> > > The client does have the ability to reconnect and re-establish the > sessions, senders and receiver automatically. A list of brokers can also be > provided and updated. There is additionally a utility that subscribes to > the 'failover exchange' to get updates. That latter mechanism could be > modified or copied for any sort of messaging based distribution of updates. > > Can you give a bit more detail on what is missing or doesn't work as > required? > > >Another serious challenge – can someone (if they haven't already) clean >> up >> the error propagation from the client to the application? Someone who was >> trying to recreate our broker failure detection asked me how we did it >> reliably – I had to give them a LIST of exceptions and error returns that >> we had to discover by trial and error to do this 100% of the time. This >> should be simple, and it isn't... >> > > Can you give a bit more detail on this? I know at one point there were > some cases where a ConnectionException would get thrown when failing first > to connect (rather than a TransportFailure as expected), but I believe that > was fixed already. > > >For our next release, we need to dramatically increase the volume of >> messages we handle. In leau of federation on Windows (when will that >> work?), I'm facing having
[jira] [Created] (QPID-4924) Python language examples based on the Python language bindings
Darryl L. Pierce created QPID-4924: -- Summary: Python language examples based on the Python language bindings Key: QPID-4924 URL: https://issues.apache.org/jira/browse/QPID-4924 Project: Qpid Issue Type: Bug Components: Python Client Reporter: Darryl L. Pierce Assignee: Darryl L. Pierce The current suite of Python language examples are based on the pure Python implementation and not on the Python language bindings generated by Swig. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Re: C++ code in coverity static analysis
Great. Thanks for keeping this up. If you manage the Coverity instance I'd like to support your effort by closing the loop on whittling down the Coverity issues. Is there a formal path for tracking what Coverity complains about: what's a false positive, what's a real problem, and when things get fixed? How do you want to work that? We've had some Coverity encounters before but never really followed up with a plan. http://apache-qpid-developers.2158895.n2.nabble.com/static-analysis-td5701419.html -Chuck - Original Message - > From: "Steve Huston" > To: dev@qpid.apache.org > Sent: Thursday, June 13, 2013 8:59:24 PM > Subject: C++ code in coverity static analysis > > Hi folks, > > A while back I registered Qpid-C++ for Coverity static analysis, but > didn't really keep it going. I am getting that going again and you may see > some email from coverity to that effect. I'll get it submitted regularly > and try to knock down the false positives that are obvious before asking > more people to look at it. > > FYI. > > -Steve > > > - > To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org > For additional commands, e-mail: dev-h...@qpid.apache.org > > - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Re: Qpid post-mortem and request for suggestions for (my) next release challenge (10M msgs/sec on Windows)
On 06/13/2013 02:37 PM, Kerry Bonin wrote: I'm the system architect for a large program that has just shipped a major product, and Qpid is one of the foundations of our infrastructure. I thought I'd share a few things, and ask a few questions about my next step... Thanks for taking the time Kerry, it's always great to get feedback! We shipped using Qpid 0.14, am updating now. Our product is (currently) 100% Windows, although most subsystems are implemented in cross-platform C++ or Python. We are essentially a video surveillance and access control system, so we have high volumes of events, low millions per day in bursts up into the low hundreds per second. In addition to event transport, our infrastructure is a ESB model SOA built over Qpid. We have high reliability requirements – no single points of failure, fast failover and recovery, active-active services, load balancing, and encryption throughout all. Our biggest challenges came from the large reliability gap between the Windows and *nix implementations. We've contributed all of our fixes back, but under high load we had issues that burnt a few man months of senior developer time. Its now running reasonably well for us. Broker failover was a challenge. We ended up building a wrapper over the Qpid client library to abstract all the connection, sender, receiver, ect. objects, so applications didn't have to deal with tearing down and building up these objects when a broker died. (Which happened often under high load in early testing, and still happens often in our worst case testbeds where we take them down or break connections to test reliability.) Since federation didn't work on Windows, system splits were unacceptable, so we also had to implement failover recovery. This also required distributed and maintaining an ordered list of brokers. Again, this was a pain, but is now working. It would be nice if the client handled these things for us. The client does have the ability to reconnect and re-establish the sessions, senders and receiver automatically. A list of brokers can also be provided and updated. There is additionally a utility that subscribes to the 'failover exchange' to get updates. That latter mechanism could be modified or copied for any sort of messaging based distribution of updates. Can you give a bit more detail on what is missing or doesn't work as required? Another serious challenge – can someone (if they haven't already) clean up the error propagation from the client to the application? Someone who was trying to recreate our broker failure detection asked me how we did it reliably – I had to give them a LIST of exceptions and error returns that we had to discover by trial and error to do this 100% of the time. This should be simple, and it isn't... Can you give a bit more detail on this? I know at one point there were some cases where a ConnectionException would get thrown when failing first to connect (rather than a TransportFailure as expected), but I believe that was fixed already. For our next release, we need to dramatically increase the volume of messages we handle. In leau of federation on Windows (when will that work?), I'm facing having to add code to manage pools of brokers, and dynamically load balance queues across brokers - once a broker is approaching 80% load, or dies, I need to move queues to other brokers, and coordinate these moves via my wrapper. I can do this, but I'd rather not. Does anyone have any other suggestions on how to handle 10M messages a second on (many) Windows boxes, spread across lots of queues at dynamic and unpredictably varying loads, that works with a fast failover and failover recovery mechanism? Kerry - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
