Re: [C++] Getting rid of file:line in QPID_MSG
FWIW, I've been bit in the past when I tried this, when a small subset of our messages were generic enough that we got multiple hits, and the developer performing the search guessed incorrectly which was the actual source location. On Wed, Jan 29, 2014 at 10:45 AM, Alan Conway acon...@redhat.com wrote: There's a convenience macro QPID_MSG in qpid/cpp/src/qpid/Msg.h that is used to format exception messages all over the place. The current implementation includes the filename:line source location where the macro is used. I propose to take the filename:line out. It is not necessary for developers (you can find the source of a message by grepping for the message text) and it adds a lot of useless non-information to messages that are read by users. It was my misguided self who put it in there long ago, if anyone thinks it is really useful please shout now and we'll argue about it, else I'll excise it. Silence = consent. Cheers, Alan. - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Documentation on flow control?
Hello! I'm working on our next product scale up, and qpidd remains a critical part of our infrastructure. One area where we are expecting challenges is around flow control, which is exacerbated by our product currently being Windows only (no federation or clustering.) For our 0.10 based release, flow control was ugly. Only QMF seems to let us keep track of when the system is in a state where producers are pushing faster than consumers are pulling, as client exceptions were rather unreliable signalling mechanisms, and didn't prevent us from crashing the broker, which we ended up having to deploy a watchdog for. I was wondering where I could find documentation on what is currently exposed for flow control, and any recommendations on best practices for using it? Also, I was wondering if there are any client feedback mechanisms other than QMF yet, or if I should start spending time integrating QMF based feedback into our clients? Any pointers are greatly appreciated! Kerry Bonin Lead Architect, UTC Fire Security Integration Platform Team
IRI encoding in broker URL strings?
We've run into an issue where a machine running a broker has a Unicode character in its name. We use SSL, and we're on Windows, so that means Microsoft Schannel SSL, which requires FQDNs when the machine is on a domain, which this is. I'm adding code today to escape the broker URL appropriately to see if that gets the string through the URL parser, does anyone know of any other issues I'm going to encounter trying to get this to work? We're currently on QPID 0.12, we'll be updating as soon as we get this release out the door, but I build from source w/ a few mods so making changes is fine for us. Any suggestions appreciated!
Re: Qpid post-mortem and request for suggestions for (my) next release challenge (10M msgs/sec on Windows)
On Mon, Jun 17, 2013 at 7:26 AM, Gordon Sim g...@redhat.com wrote: On 06/14/2013 03:58 PM, Kerry Bonin wrote: On existing broker failover - can you point me to where that behavior is documented? Because neither myself or anyone on the four teams I work with has come across the functionality you describe. I've never seen a client failover to another broker, only code to attempt to reconnect. It appears the reconnect_urls connection option is not in fact documented. Sorry about that. It takes a single url or a Variant::List of urls to try when reconnecting. Basic features we need: - externally adjustable retry / timeout on connections - to handle differences between LAN, WAN, and satellite internet. - updating broker list: How do you do this? Never seen it... There are two options. The first is that any url in the AMQP 0-10 format can itself contain multiple hosts, e.g. amqp:tcp:host1:port1,host2:**port2. The second is to use the reconnect_urls option as above. (When used in conjunction with the failover exchange there is a helper class that will receive updates and apply them: http://qpid.apache.org/books/**0.20/Programming-In-Apache-** Qpid/html/ch02s14.htmlhttp://qpid.apache.org/books/0.20/Programming-In-Apache-Qpid/html/ch02s14.html, something similar could be done for some other distribution mechanism). - to prevent network splits, how are recovered brokers monitored? When a failed broker recovers, do clients switch back? How often / aggressively checked? No, there is no switch back behaviour in the client. The new HA code allows a broker to be classed as in a backup or primary role and backups will reject or kick off any clients causing them to failover. Whatever cluster management solution was in use would then detect changes to primary and use QMF to tell each broker what their role was. I'd like to suggest that this is a serious deficiency. It would be nice if it was possible to have some HA features without having to deploy clustering. While the lack of clustering for Windows makes this an obvious problem for Windows users, I'd certainly argue that *nix users might also like to have failover and recovery without clustering. And without clustering, failover without recovery is kind of useless as a HA feature due to the split use case. (i.e. 2 clients talking through broker A, broker A fails and 2 clients failover to broker B. Broker A comes back online. Another client joins, connects to broker A. We now have a split, new client cannot see old clients.) For me, this means we have to leave our layer library in place. - how is the application notified on broker failure, connection failover, recovery? It isn't. Any threads using the connection will essentially block until either the connection was re-established or until the configured limit was reached and the client gives up trying. Now I write this I do recall a conversation on this topic with you some time back, with this being an issue for you. I'd like to suggest that this remains a serious deficiency. In most software, if a critical failure occurs down in middleware or its supporting infrastructure, it would be nice if the middleware library could report this to the application, so a system administrator could do something about it. While its certainly possible to rely on external monitoring systems to notify an admin, its also a good practice to have an application display some sort of error condition. A broker failure in an ESB SOA application is a critical failure, and the application needs to inform its user that it has lost connectivity to the system. For me, this also means we have to leave our layer library in place. Finally, we were ending up with LOTS of application complexity in SOA code when broker failure / recovery meant connection, sender and receiver objects had to be recreated. This was compounded by Connection being a different types of boost object than senders and receivers. That's strange. Those classes all use the Handle template so I can't see how they would be different in that regard. I don't suppose you recall the details? If I remember the root issue - if a broker fails, the Connection object dies, and so do all the associated Sender and Receiver objects. If I failover to another broker, I obtain a new Connection object, and must then obtain new Sender and Receiver objects from this new Connection object. This results in a decent bit of complexity propagating through our code, which was another reason why we created a wrapper over it - so we could call one send API (that stores a map of queue names to Sender objects) and use listener callbacks that register a list of callbacks associated with a queue name Receiver objects. On failover/recovery we recreate the Senders and Receivers once in one place, and update the maps. If the applications care about connection state, they ask for our status callbacks, as per comment
Re: Qpid post-mortem and request for suggestions for (my) next release challenge (10M msgs/sec on Windows)
On existing broker failover - can you point me to where that behavior is documented? Because neither myself or anyone on the four teams I work with has come across the functionality you describe. I've never seen a client failover to another broker, only code to attempt to reconnect. Basic features we need: - externally adjustable retry / timeout on connections - to handle differences between LAN, WAN, and satellite internet. - updating broker list: How do you do this? Never seen it... - to prevent network splits, how are recovered brokers monitored? When a failed broker recovers, do clients switch back? How often / aggressively checked? - how is the application notified on broker failure, connection failover, recovery? Finally, we were ending up with LOTS of application complexity in SOA code when broker failure / recovery meant connection, sender and receiver objects had to be recreated. This was compounded by Connection being a different types of boost object than senders and receivers. We ended up building a bunch of code we were using everywhere to handle this, ended up collapsing it into a layer that just exposes send(queue name, message) and listen( queue name, callback) with all the functionality underneath, plus callback for status messages (broker failure, connection failover, recovery). On connection failure detection, I'll dig up my notes and send to the list, hopefully this has been cleaned up since 0.14. And anything you can think of for dynamically load balancing across brokers? Greatly appreciate the feedback and input... Kerry On Fri, Jun 14, 2013 at 4:09 AM, Gordon Sim g...@redhat.com wrote: On 06/13/2013 02:37 PM, Kerry Bonin wrote: I'm the system architect for a large program that has just shipped a major product, and Qpid is one of the foundations of our infrastructure. I thought I'd share a few things, and ask a few questions about my next step... Thanks for taking the time Kerry, it's always great to get feedback! We shipped using Qpid 0.14, am updating now. Our product is (currently) 100% Windows, although most subsystems are implemented in cross-platform C++ or Python. We are essentially a video surveillance and access control system, so we have high volumes of events, low millions per day in bursts up into the low hundreds per second. In addition to event transport, our infrastructure is a ESB model SOA built over Qpid. We have high reliability requirements – no single points of failure, fast failover and recovery, active-active services, load balancing, and encryption throughout all. Our biggest challenges came from the large reliability gap between the Windows and *nix implementations. We've contributed all of our fixes back, but under high load we had issues that burnt a few man months of senior developer time. Its now running reasonably well for us. Broker failover was a challenge. We ended up building a wrapper over the Qpid client library to abstract all the connection, sender, receiver, ect. objects, so applications didn't have to deal with tearing down and building up these objects when a broker died. (Which happened often under high load in early testing, and still happens often in our worst case testbeds where we take them down or break connections to test reliability.) Since federation didn't work on Windows, system splits were unacceptable, so we also had to implement failover recovery. This also required distributed and maintaining an ordered list of brokers. Again, this was a pain, but is now working. It would be nice if the client handled these things for us. The client does have the ability to reconnect and re-establish the sessions, senders and receiver automatically. A list of brokers can also be provided and updated. There is additionally a utility that subscribes to the 'failover exchange' to get updates. That latter mechanism could be modified or copied for any sort of messaging based distribution of updates. Can you give a bit more detail on what is missing or doesn't work as required? Another serious challenge – can someone (if they haven't already) clean up the error propagation from the client to the application? Someone who was trying to recreate our broker failure detection asked me how we did it reliably – I had to give them a LIST of exceptions and error returns that we had to discover by trial and error to do this 100% of the time. This should be simple, and it isn't... Can you give a bit more detail on this? I know at one point there were some cases where a ConnectionException would get thrown when failing first to connect (rather than a TransportFailure as expected), but I believe that was fixed already. For our next release, we need to dramatically increase the volume of messages we handle. In leau of federation on Windows (when will that work?), I'm facing having to add code to manage pools of brokers, and dynamically load balance queues across brokers
[jira] [Created] (QPID-4345) Windows memory leak
Kerry Bonin created QPID-4345: - Summary: Windows memory leak Key: QPID-4345 URL: https://issues.apache.org/jira/browse/QPID-4345 Project: Qpid Issue Type: Bug Components: C++ Broker, C++ Client Affects Versions: 0.18, 0.12 Environment: Windows Reporter: Kerry Bonin Priority: Blocker Fix For: 0.19 In our large system performance test environments we see Broker memory growth until the process crashes. This appears to be reproducible simply by opening and closing connections as fast as possible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Updated] (QPID-4345) Windows memory leak
[ https://issues.apache.org/jira/browse/QPID-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kerry Bonin updated QPID-4345: -- Attachment: windows_close_leak_r1390053.patch Fix for Windows memory leak. One of our team leads, Ben Holm, developed this fix. It appears to resolve the leak in small scale testing, we are about to test it in our large scale environment. This appears to be an improperly handled corner case, where (at least on Windows) one buffer is leaked per connection close. This fix hands the buffer back to a managed pool. Windows memory leak --- Key: QPID-4345 URL: https://issues.apache.org/jira/browse/QPID-4345 Project: Qpid Issue Type: Bug Components: C++ Broker, C++ Client Affects Versions: 0.12, 0.18 Environment: Windows Reporter: Kerry Bonin Priority: Blocker Fix For: 0.19 Attachments: windows_close_leak_r1390053.patch In our large system performance test environments we see Broker memory growth until the process crashes. This appears to be reproducible simply by opening and closing connections as fast as possible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Updated] (QPID-4345) Windows memory leak
[ https://issues.apache.org/jira/browse/QPID-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kerry Bonin updated QPID-4345: -- Description: In our large system performance test environments we see Broker memory growth until the process crashes. This appears to be reproducible simply by opening and closing connections as fast as possible. Marked as 'Blocker' as it was blocking shipment of our Windows based product - the Broker would exhaust memory and crash itself every few days of operation. (was: In our large system performance test environments we see Broker memory growth until the process crashes. This appears to be reproducible simply by opening and closing connections as fast as possible.) Windows memory leak --- Key: QPID-4345 URL: https://issues.apache.org/jira/browse/QPID-4345 Project: Qpid Issue Type: Bug Components: C++ Broker, C++ Client Affects Versions: 0.12, 0.18 Environment: Windows Reporter: Kerry Bonin Priority: Blocker Fix For: 0.19 Attachments: windows_close_leak_r1390053.patch In our large system performance test environments we see Broker memory growth until the process crashes. This appears to be reproducible simply by opening and closing connections as fast as possible. Marked as 'Blocker' as it was blocking shipment of our Windows based product - the Broker would exhaust memory and crash itself every few days of operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Re: [jira] [Commented] (QPID-4257) Windows+SSL: Client hang on broker close, broker memory leak
No problem. I'm still investigating a remaining memory leak on the C++ broker on Windows, but that patch corrected most of it... On Tue, Aug 28, 2012 at 2:49 PM, Chuck Rolke (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/QPID-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13443451#comment-13443451] Chuck Rolke commented on QPID-4257: --- Kerry, Thanks for posting the patch. I will give it some attention. Chuck Windows+SSL: Client hang on broker close, broker memory leak Key: QPID-4257 URL: https://issues.apache.org/jira/browse/QPID-4257 Project: Qpid Issue Type: Bug Components: C++ Broker, C++ Client Affects Versions: 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19 Environment: Windows client on Windows 7, Windows broker on Server 2K8, using SSL. Reporter: Kerry Bonin Labels: patch Fix For: 0.19 Attachments: windows_ssl_hang_r1377724.patch Original Estimate: 168h Remaining Estimate: 168h Windows clients hang when broker dies via SSL connection - fetch never returns even w/ IMMEDIATE timeout. Windows broker memory grows linearly with SSL connection count, leak per connection, until broker exhausts memory and crashes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Created] (QPID-4257) Windows+SSL: Client hang on broker close, broker memory leak
Kerry Bonin created QPID-4257: - Summary: Windows+SSL: Client hang on broker close, broker memory leak Key: QPID-4257 URL: https://issues.apache.org/jira/browse/QPID-4257 Project: Qpid Issue Type: Bug Components: C++ Broker, C++ Client Affects Versions: 0.16, 0.15, 0.14, 0.13, 0.12, 0.17, 0.18, 0.19 Environment: Windows client on Windows 7, Windows broker on Server 2K8, using SSL. Reporter: Kerry Bonin Fix For: 0.19 Windows clients hang when broker dies via SSL connection - fetch never returns even w/ IMMEDIATE timeout. Windows broker memory grows linearly with SSL connection count, leak per connection, until broker exhausts memory and crashes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Updated] (QPID-4257) Windows+SSL: Client hang on broker close, broker memory leak
[ https://issues.apache.org/jira/browse/QPID-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kerry Bonin updated QPID-4257: -- Attachment: windows_ssl_hang_r1377724.patch Patch to correct. GetQueuedCompletionStatus was not being signaled properly when an SSL connection was closed remotely, so it never returned from a INFINITE timeout. Patched to clamp timeout to 1 second, existing logic handled return w/ numTransferred=0 case, and GetQueuedCompletionStatus did return false without signaled exit when SSL had been remotely closed. This corrected the hang. Added SslConnector::redirectDisconnect, as the eof() logic in qpid::client::TCPConnector wasn't properly signalling closure and releasing resources. Windows+SSL: Client hang on broker close, broker memory leak Key: QPID-4257 URL: https://issues.apache.org/jira/browse/QPID-4257 Project: Qpid Issue Type: Bug Components: C++ Broker, C++ Client Affects Versions: 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19 Environment: Windows client on Windows 7, Windows broker on Server 2K8, using SSL. Reporter: Kerry Bonin Labels: patch Fix For: 0.19 Attachments: windows_ssl_hang_r1377724.patch Original Estimate: 168h Remaining Estimate: 168h Windows clients hang when broker dies via SSL connection - fetch never returns even w/ IMMEDIATE timeout. Windows broker memory grows linearly with SSL connection count, leak per connection, until broker exhausts memory and crashes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Fixed critical bug (SSL hanging on Windows), question on patch submission...
I just tracked down an issue where on some of our Windows machines the QPIDD client would sometimes hang when the broker died while using an SSL connection. I fixed this in 0.12, but it appears to still exist in 0.16, and 0.18 RC is in process - I was wondering which branch I should submit a patch against?
Re: Client broker failover notification/callback?
Thank you for the replies! I'll probably end up creating a wrapper library to address this. Not very clean with the boost object wrapper behavior, but I'm not willing to write my own client at this time, so not much of an alternative. (I don't like the deeper code structure, its a pain to hack anything that needlessly complex.) Irregardless of how it is accomplished, more control is needed over failover to prevent network splits. The current model is essentially to accept the broker as a single point of failure, or deploy Linux clustering. I'd recommend an ordered broker list with monitoring and automatic fallback, subject to some flap mitigation rules. I also would expose more client state, I understand the desire to hide stuff so people don't use unsupported interfaces, but it is useful for serviceability and diagnostics to have a library expose basic health information. Out of honest curiosity, why don't you like callbacks? We've had to use threading with spin loops to get around the lack of callbacks for the messaging APIs, and we don't like having CPUs loads float high under low load even if the spin loop load drops gracefully under pressure, it feels inelegant and it raises power consumption. Personally, I like callbacks as long as threading models are clearly documented so its understood what can and can't be done safely, especially in conjunction with a decent sig/slot library to decouple threading models of subsystems. I've built servers that handle nearly 100k active sessions (video games) this way, FWIW... It would be nice if the Apache QPID team gave a little more support to Windows. No (official) Windows service, still no broker federation / clustering. Kerry Bonin On Tue, Jun 28, 2011 at 7:52 AM, Gordon Sim g...@redhat.com wrote: On 06/27/2011 08:19 PM, Kerry Bonin wrote: I was wondering if there was any existing way to know when a broker failover occurs (and which broker is active), other then hacking the client? In a clustered broker you can get notifications of changes to cluster membership. However other than that the c++ client library doesn't have any callbacks to notify of automatic reconnection attempts or to determine where a connection is connected to at any point in time (assuming it is connected). I don't like callbacks, at least at this level of the API, but I would agree that these would be nice concerns to address. Having some way to get some form of peer address would be nice, as would a way to be notified of disconnect and reconnect events. (Apologies for the delayed response on user thread btw) --**--**- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscribe@qpid.**apache.orgdev-subscr...@qpid.apache.org
Client broker failover notification/callback?
I was wondering if there was any existing way to know when a broker failover occurs (and which broker is active), other then hacking the client?
[jira] Commented: (QPID-2519) Allow Windows Broker to run as a Service
[ https://issues.apache.org/jira/browse/QPID-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12998370#comment-12998370 ] Kerry Bonin commented on QPID-2519: --- There is a good deal of duplicated and strangely flowing code there - as I said I basically just took the MSDN sample service code and converted it into a class. For me there was never a justification to clean it up further, as it was such special case code. Allow Windows Broker to run as a Service Key: QPID-2519 URL: https://issues.apache.org/jira/browse/QPID-2519 Project: Qpid Issue Type: New Feature Components: C++ Broker Environment: Windows Reporter: Kerry Bonin Assignee: Steve Huston Fix For: Future Attachments: broker_as_service_Qpid_Logging.patch, broker_as_service_r921371.patch The Windows version of the C++ Broker should have the ability to run as a Windows Service, to correspond with the ability of the *nix version to run as a daemon, as this is a common requirement for critical software infrastructure elements of enterprise class software on the Windows platform. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Commented: (QPID-2519) Allow Windows Broker to run as a Service
[ https://issues.apache.org/jira/browse/QPID-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12996780#comment-12996780 ] Kerry Bonin commented on QPID-2519: --- I should have time this week to (finally) help finish this... Allow Windows Broker to run as a Service Key: QPID-2519 URL: https://issues.apache.org/jira/browse/QPID-2519 Project: Qpid Issue Type: New Feature Components: C++ Broker Environment: Windows Reporter: Kerry Bonin Assignee: Steve Huston Fix For: Future Attachments: broker_as_service_Qpid_Logging.patch, broker_as_service_r921371.patch The Windows version of the C++ Broker should have the ability to run as a Windows Service, to correspond with the ability of the *nix version to run as a daemon, as this is a common requirement for critical software infrastructure elements of enterprise class software on the Windows platform. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Are there any complete examples of durable / persistent transport on Windows?
Appreciate any help...
Re: Are there any complete examples of durable / persistent transport on Windows?
I'm interested in the classic durable subscription case - two clients and a broker, client 2 has a durable subscription to a queue, client 1 sends something to that queue while client 2 is disconnected. When client 2 reconnects it would like to receive the messages it missed. Ideally this could include a power cycle of the broker, hence my interest in the database persistence module. I've enabled the mssql persistence module, but the next steps are unclear - I've found conflicting information on how to setup the subscription, and am hoping to avoid stepping into the broker to figure out what I'm missing or doing wrong. We're using the C++ broker and have been building from recent sources. I'm also curious if there is support / plans for durability on the client side - having the client queue sent messages if the broker is temporarily offline. Thanks! On Tue, Feb 8, 2011 at 11:39 AM, Andrew Stitcher astitc...@redhat.comwrote: On Tue, 2011-02-08 at 11:29 -0600, Kerry Bonin wrote: Appreciate any help... Could you be a little clearer with what you want? In the current qpid implementation (in C++ broker at least) durability of messages isn't related to the transport (which is a networking concept), but to a persistence plugin. I don't know if this is also true for the Java broker (if you wanted that) but I'd think it's pretty likely to be true there too. Andrew - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: [jira] Updated: (QPID-2519) Allow Windows Broker to run as a Service
Thanks for working on this cleanup, I've been a bit swamped and haven't had time to return to change the style or fix the logging. For what its worth, we've been using this in a major product release in system test for a few months, zero issues on the service to date... On Mon, Sep 27, 2010 at 11:40 AM, Daniel Sack (JIRA) qpid-...@incubator.apache.org wrote: [ https://issues.apache.org/jira/browse/QPID-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Sack updated QPID-2519: -- Attachment: broker_as_service_Qpid_Logging.patch Hi, Scince we need this patch for our production system, I tried to implement parts of the comments 1. Please reformat according to Qpid coding style (http://qpid.apache.org/qpid-c-documentation.html) - no tabs, consistent brace style - didn't have the time for it 2. There's a global WinService in QpiddBroker.cpp - what's that for? - Responsible for all the daemon stuff which is needed for the windows platform. 3. Lots of printfs and no Qpid logging statements, and no exception throwing on error. Please review these and probably should be throwing on errors; outputs should probably be using logging to be redirected as the user specified. - triedto fix this Allow Windows Broker to run as a Service Key: QPID-2519 URL: https://issues.apache.org/jira/browse/QPID-2519 Project: Qpid Issue Type: New Feature Components: C++ Broker Reporter: Kerry Bonin Attachments: broker_as_service_Qpid_Logging.patch, broker_as_service_r921371.patch The Windows version of the C++ Broker should have the ability to run as a Windows Service, to correspond with the ability of the *nix version to run as a daemon, as this is a common requirement for critical software infrastructure elements of enterprise class software on the Windows platform. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: [jira] Updated: (QPID-2519) Allow Windows Broker to run as a Service
If you don't specify an account to run under, then it defaults (via
SCM) to LocalService. If I try and specify LocalService then it I
get that error, which led me to suspect it had to do with the service
name string.
On Mon, Sep 27, 2010 at 1:06 PM, Daniel Sack daniel.s...@techtalk.at wrote:
I tried it out to run it as LocalService but if I try to start the broker
with service.mmc I 'll get the message:
Windows could not start the qpidd service on Local Computer.
Error 1053: The service did not respond to the start or control request in a
timely fashion.
There is also not qpid broker process running.
Lg,
Daniel
-Original Message-
From: Kerry Bonin [mailto:kerrybo...@gmail.com]
Sent: Montag, 27. September 2010 19:18
To: Daniel Sack
Subject: Re: [jira] Updated: (QPID-2519) Allow Windows Broker to run as a
Service
By default I believe it runs as LocalService. I added support for the SCM
API to allow specification of an alternative user, but when I tested it I
couldn't get it to work as NetworkService. I suspect that was only because I
wasn't specifying the user in the string format that SCM expected (something
like {hostname}/{account name}), but I got distracted after playing for a few
mins and haven't returned yet.
On Mon, Sep 27, 2010 at 11:55 AM, Daniel Sack daniel.s...@techtalk.at wrote:
Hi,
Just a small question:
Currently it's only possible for me to run the qpid-broker service under
some user credentials which are not built in service users like:
LocalSystem or NetworkService
If we want to start the broker as LocalSystem it won't start with error
1503.
Can you answer me why?
We have tried to use xyntservice
(http://www.codeproject.com/KB/system/xyntservice.aspx) which was not really
working with qpid 0.6 cpp broker.
It' always crashed somewhere in kernel.dll with some corrupted heap, ...
But we were able to run the broker as Local System account, maybe is the
problem why the broker crashes, I don't know.
Br,
Daniel Sack
-Original Message-
From: Kerry Bonin [mailto:kerrybo...@gmail.com]
Sent: Montag, 27. September 2010 18:43
To: dev@qpid.apache.org
Subject: Re: [jira] Updated: (QPID-2519) Allow Windows Broker to run
as a Service
Thanks for working on this cleanup, I've been a bit swamped and haven't had
time to return to change the style or fix the logging.
For what its worth, we've been using this in a major product release in
system test for a few months, zero issues on the service to date...
On Mon, Sep 27, 2010 at 11:40 AM, Daniel Sack (JIRA)
qpid-...@incubator.apache.org wrote:
[
https://issues.apache.org/jira/browse/QPID-2519?page=com.atlassian.ji
r a.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Sack updated QPID-2519:
--
Attachment: broker_as_service_Qpid_Logging.patch
Hi,
Scince we need this patch for our production system, I tried to
implement parts of the comments 1. Please reformat according to Qpid
coding style (http://qpid.apache.org/qpid-c-documentation.html) - no
tabs, consistent brace style
- didn't have the time for it
2. There's a global WinService in QpiddBroker.cpp - what's that for?
- Responsible for all the daemon stuff which is needed for the windows
platform.
3. Lots of printfs and no Qpid logging statements, and no exception
throwing on error. Please review these and probably should be throwing on
errors; outputs should probably be using logging to be redirected as the
user specified.
- triedto fix this
Allow Windows Broker to run as a Service
Key: QPID-2519
URL: https://issues.apache.org/jira/browse/QPID-2519
Project: Qpid
Issue Type: New Feature
Components: C++ Broker
Reporter: Kerry Bonin
Attachments: broker_as_service_Qpid_Logging.patch,
broker_as_service_r921371.patch
The Windows version of the C++ Broker should have the ability to run as a
Windows Service, to correspond with the ability of the *nix version to run
as a daemon, as this is a common requirement for critical software
infrastructure elements of enterprise class software on the Windows
platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscr...@qpid.apache.org
-
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscr...@qpid.apache.org
-
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact
Re: Implementing clustering support for Qpid Java broker
On the C++ Broker, just remember that the technology choice made for clustering (OpenAIS/Corosync) is a Linux only solution, and that codebase (~100k SLOC) was not written for portability. Our project needs a Windows option, and this is already hurting us. I (and my team) would be willing to spend some cycles supporting any reasonable effort to address this, even if its not C++... On Sun, Sep 19, 2010 at 4:18 AM, Lahiru Gunathilake glah...@gmail.com wrote: On Sun, Sep 19, 2010 at 11:37 AM, Etienne Antoniutti Di Muro etienne.antoniu...@adaptivebytes.com wrote: Andrew, I'll have a look at services provided by Terracotta, however at a first glance, I hit the search button on their site, with 'virtual synchrony' - no result! While if you google 'spread.org: virtual synchrony' -it's there. I'll take some time to investigate deeper into Terracotta services. However it all depends on what clustering is intended for. Moreover, in order to keep the project compact, does it make sense building a java API for the C++ broker clustering solution ? I think there's C++ broker clustering already done ! I would like to use another Apache project like Tribes. Lahiru Lahiru Regards Etienne On Thu, Sep 16, 2010 at 8:57 AM, Andrew Kennedy andrewinternatio...@gmail.com wrote: On 15 Sep 2010, at 22:56, Etienne Antoniutti Di Muro wrote: However I do have some proposals, you can have a look here http://portal.acm.org/citation.cfm?id=1101143 As a starter, I'd need someone to help me understand Qpid architecture, at least an overview. For those of us that aren't paying for an ACM subscription, Etienne's thesis is also available here: http://middleware05.objectweb.org/WSProceedings/DoctoralSymposium05/a3-diMuro.pdf I think that functional homogeneity could probably be provided by Terracotta or similar, but I am unsure of the licensing restrictions. Andrew. -- -- andrew d kennedy ? do not fold, bend, spindle, or mutilate ; - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: qmfconsole question - where is queue type ?
Sorry, been working with the QPID messaging APIs so long I'm
forgetting the underlying AMQP syntax...
We have two communication models - one uses request/response (direct)
which we create with an address string of name; {create:always,
node:{type:queue}}, the other is a pub/sub which we create with an
address string of name; {create:always, node:{type:topic}}.
We're trying to create a diagnostic tool to make sure this is setup
correctly, and thought we'd try and use the QMF API to enumerate
queues and exchanges and their bindings. I can't find the binding
info, and I'm not sure how to map back to what was provided to the
Address class...
On Mon, Jul 19, 2010 at 4:40 AM, Gordon Sim g...@redhat.com wrote:
On 07/14/2010 09:34 PM, Kerry Bonin wrote:
I'm playing with the qmfconsole APIs and examples with the goal of
bringing up a monitoring and diagnostic tool for our qpid based
system. I'm having a difficult time determining the queue node type
{queue|topic}. Any suggestions?
The queue|topic distinction is at this point a concept of the messaging API
only. A 'queue' in the API maps on to an AMQP 0-10 queue and a 'topic' maps
on to an AMQP 0-10 exchange.
On related note, I didn't see how to
map between queues and exchanges...
Not quite sure what you mean here... are you trying to get the bindings? Or
something else?
-
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscr...@qpid.apache.org
-
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscr...@qpid.apache.org
qmfconsole question - where is queue type ?
I'm playing with the qmfconsole APIs and examples with the goal of
bringing up a monitoring and diagnostic tool for our qpid based
system. I'm having a difficult time determining the queue node type
{queue|topic}. Any suggestions? On related note, I didn't see how to
map between queues and exchanges...
Thanks!
-
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Closed: (QPID-2676) Python egg doesn't work on Windows platform...
[ https://issues.apache.org/jira/browse/QPID-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kerry Bonin closed QPID-2676. - Resolution: Invalid After spending some time to come up to speed on Python to attempt to validate what my Python team was telling me, I have come to the conclusion that my Python team was wrong. Our Python lead had a failure on his box, but it was due to a corrupted Python installation, and I had not been informed that no other members had ever attempted to validate. My sincere apologies for posting this issue, and I will personally validate any information from my Python team before posting it here again. Python egg doesn't work on Windows platform... -- Key: QPID-2676 URL: https://issues.apache.org/jira/browse/QPID-2676 Project: Qpid Issue Type: Bug Components: Python Client Affects Versions: 0.7 Environment: Python egg on Windows Reporter: Kerry Bonin Assignee: Rafael H. Schloming Attachments: setup.py, stack_trace.txt Patch to be attached to this JIRA to create a Python egg works on Mac Ubuntu platforms, but on Windows it fails with an XML parsing error -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Commented: (QPID-2676) Python egg doesn't work on Windows platform...
[ https://issues.apache.org/jira/browse/QPID-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882314#action_12882314 ] Kerry Bonin commented on QPID-2676: --- Sorry for the delay - I was just told the error occurs on Windows 7 64-bit, which apparently worked fine using an earlier qpid dev build. The error is the one I posted, something about an unknown url for the amqp-0-10.dtd. Python egg doesn't work on Windows platform... -- Key: QPID-2676 URL: https://issues.apache.org/jira/browse/QPID-2676 Project: Qpid Issue Type: Bug Components: Python Client Affects Versions: 0.7 Environment: Python egg on Windows Reporter: Kerry Bonin Assignee: Rafael H. Schloming Attachments: setup.py, stack_trace.txt Patch to be attached to this JIRA to create a Python egg works on Mac Ubuntu platforms, but on Windows it fails with an XML parsing error -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Issue Comment Edited: (QPID-2676) Python egg doesn't work on Windows platform...
[ https://issues.apache.org/jira/browse/QPID-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882314#action_12882314 ] Kerry Bonin edited comment on QPID-2676 at 6/24/10 4:33 PM: Sorry for the delay - I was just told the error occurs on Windows 7 64-bit, which apparently worked fine using an earlier qpid dev build. The error is the one I posted, something about an unknown url for the amqp-0-10.dtd. I don't know enough Python to go much further myself. The Python team here won't be able to look at this in the very near future, so I wanted to pass this on. As to the question you raised - I'm now not sure if this in fact an egg issue, which is what I was originally told. was (Author: kbonin): Sorry for the delay - I was just told the error occurs on Windows 7 64-bit, which apparently worked fine using an earlier qpid dev build. The error is the one I posted, something about an unknown url for the amqp-0-10.dtd. Python egg doesn't work on Windows platform... -- Key: QPID-2676 URL: https://issues.apache.org/jira/browse/QPID-2676 Project: Qpid Issue Type: Bug Components: Python Client Affects Versions: 0.7 Environment: Python egg on Windows Reporter: Kerry Bonin Assignee: Rafael H. Schloming Attachments: setup.py, stack_trace.txt Patch to be attached to this JIRA to create a Python egg works on Mac Ubuntu platforms, but on Windows it fails with an XML parsing error -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Created: (QPID-2676) Python egg doesn't work on Windows platform...
Python egg doesn't work on Windows platform... -- Key: QPID-2676 URL: https://issues.apache.org/jira/browse/QPID-2676 Project: Qpid Issue Type: Bug Components: Python Client Affects Versions: 0.7 Environment: Python egg on Windows Reporter: Kerry Bonin Patch to be attached to this JIRA to create a Python egg works on Mac Ubuntu platforms, but on Windows it fails with an XML parsing error -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Updated: (QPID-2676) Python egg doesn't work on Windows platform...
[ https://issues.apache.org/jira/browse/QPID-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kerry Bonin updated QPID-2676: -- Attachment: setup.py Attached updated setup.py local changes: - edit the setup.py to make it build an egg - make it so that the setup.py will work whether setuptools is installed or not (ie distutils only will work, but if setuptools is installed it will use that instead). The egg created with this fails on Windows. Python egg doesn't work on Windows platform... -- Key: QPID-2676 URL: https://issues.apache.org/jira/browse/QPID-2676 Project: Qpid Issue Type: Bug Components: Python Client Affects Versions: 0.7 Environment: Python egg on Windows Reporter: Kerry Bonin Attachments: setup.py Patch to be attached to this JIRA to create a Python egg works on Mac Ubuntu platforms, but on Windows it fails with an XML parsing error -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Python egg support ?
Just had another talk with the Python lead. He is generating eggs now, and they work fine on Mac and Ubuntu, but are failing on Windows somewhere in the xml spec parsing. According to him the 'egg' is a common enough delivery medium for Python that he's surprised this wasn't caught in an existing unit test. I've created QPID-2676 and attached our updated setup.py that works with and without setuptools and was used to generate the egg which failed on Windows. On Thu, Jun 17, 2010 at 5:57 AM, Rafael Schloming rafa...@redhat.com wrote: Kerry Bonin wrote: Follow up comment from another Python developer here, abbreviated (I'm not the best filter here since I'm more of a C++/Java/Cg/Lua developer, have only briefly touched Python.) - setup.py is based on disttools, which predates eggs - simple fix would be to change setup.py to use setuptools instead. - recommended fix is to change setup.py to use 'distribute' - Reference link: http://www.mcguru.net/download/presentations/michipug-2006-02-02-Intro-to-setuptools.pdf Another Python developer here just told me he's made the setuptools change to setup.py in our local fork, I'll generate a patch and attach to a JIRA as soon as he checks his changes in. I'm happy to add optional support for setuptools if there is a clean way to do that. I would like to keep the scripts working with distutils as well though since that is available by default on any python install without having to get additional packages. Give me a ping when you post your JIRA and I'll have a look. --Rafael - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Updated: (QPID-2676) Python egg doesn't work on Windows platform...
[ https://issues.apache.org/jira/browse/QPID-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kerry Bonin updated QPID-2676: -- Attachment: stack_trace.txt Stack trace for Windows problem Python egg doesn't work on Windows platform... -- Key: QPID-2676 URL: https://issues.apache.org/jira/browse/QPID-2676 Project: Qpid Issue Type: Bug Components: Python Client Affects Versions: 0.7 Environment: Python egg on Windows Reporter: Kerry Bonin Attachments: setup.py, stack_trace.txt Patch to be attached to this JIRA to create a Python egg works on Mac Ubuntu platforms, but on Windows it fails with an XML parsing error -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Python egg support ?
One of the teams using QPID here uses Python, and is looking at updating to a more recent version of the code to keep in sync with those of us working in C++ on the Windows platforms. I was informed this morning that some changes have been made that make it difficult to generate a python egg using the current code. Can someone from the Python side of the house point me to any information on how they can use the latest code in a Python application currently based on EasyInstall and eggs ? Thanks... - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Python egg support ?
I just talked to him - his install aborts with an error, but he was using an older working copy, we'll update (with our patches to c++ side) and retry, thanks... On Wed, Jun 16, 2010 at 12:44 PM, Rafael Schloming rafa...@redhat.com wrote: Kerry Bonin wrote: One of the teams using QPID here uses Python, and is looking at updating to a more recent version of the code to keep in sync with those of us working in C++ on the Windows platforms. I was informed this morning that some changes have been made that make it difficult to generate a python egg using the current code. Can someone from the Python side of the house point me to any information on how they can use the latest code in a Python application currently based on EasyInstall and eggs ? All the python code uses distutils which I believe will generate eggs by default. I don't use eggs myself, so I haven't tried firsthand. Do you have anymore details on what the issue might be? --Rafael - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Python egg support ?
OK, he tried again with latest. According to him, the bdist egg command was removed, and what is generated by default (the ...egg-info) is script data to allow a local install, but no egg is actually generated. We need to know how to generate an actual egg so we can redistribute that, not the entire qpid build tree. Does that make sense ? On Wed, Jun 16, 2010 at 2:11 PM, Kerry Bonin kerrybo...@gmail.com wrote: I just talked to him - his install aborts with an error, but he was using an older working copy, we'll update (with our patches to c++ side) and retry, thanks... On Wed, Jun 16, 2010 at 12:44 PM, Rafael Schloming rafa...@redhat.com wrote: Kerry Bonin wrote: One of the teams using QPID here uses Python, and is looking at updating to a more recent version of the code to keep in sync with those of us working in C++ on the Windows platforms. I was informed this morning that some changes have been made that make it difficult to generate a python egg using the current code. Can someone from the Python side of the house point me to any information on how they can use the latest code in a Python application currently based on EasyInstall and eggs ? All the python code uses distutils which I believe will generate eggs by default. I don't use eggs myself, so I haven't tried firsthand. Do you have anymore details on what the issue might be? --Rafael - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Python egg support ?
Follow up comment from another Python developer here, abbreviated (I'm not the best filter here since I'm more of a C++/Java/Cg/Lua developer, have only briefly touched Python.) - setup.py is based on disttools, which predates eggs - simple fix would be to change setup.py to use setuptools instead. - recommended fix is to change setup.py to use 'distribute' - Reference link: http://www.mcguru.net/download/presentations/michipug-2006-02-02-Intro-to-setuptools.pdf Another Python developer here just told me he's made the setuptools change to setup.py in our local fork, I'll generate a patch and attach to a JIRA as soon as he checks his changes in. On Wed, Jun 16, 2010 at 3:42 PM, Kerry Bonin kerrybo...@gmail.com wrote: OK, he tried again with latest. According to him, the bdist egg command was removed, and what is generated by default (the ...egg-info) is script data to allow a local install, but no egg is actually generated. We need to know how to generate an actual egg so we can redistribute that, not the entire qpid build tree. Does that make sense ? On Wed, Jun 16, 2010 at 2:11 PM, Kerry Bonin kerrybo...@gmail.com wrote: I just talked to him - his install aborts with an error, but he was using an older working copy, we'll update (with our patches to c++ side) and retry, thanks... On Wed, Jun 16, 2010 at 12:44 PM, Rafael Schloming rafa...@redhat.com wrote: Kerry Bonin wrote: One of the teams using QPID here uses Python, and is looking at updating to a more recent version of the code to keep in sync with those of us working in C++ on the Windows platforms. I was informed this morning that some changes have been made that make it difficult to generate a python egg using the current code. Can someone from the Python side of the house point me to any information on how they can use the latest code in a Python application currently based on EasyInstall and eggs ? All the python code uses distutils which I believe will generate eggs by default. I don't use eggs myself, so I haven't tried firsthand. Do you have anymore details on what the issue might be? --Rafael - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Patches for committing ?
https://issues.apache.org/jira/browse/QPID-2519 Patch to allow broker to run as a Windows service, its been in system test here for over a month, no issues found, and we're getting ready to ship a major product that relies on this feature. On Wed, Jun 16, 2010 at 4:05 PM, Marnie McCormack marnie.mccorm...@googlemail.com wrote: All, Looking over the JIRAs this week, I have the impression that may be quite a few overlooked patches out there. If you're a contributer and you have a patch that no-one has picked up - please can you shout now about it ? Thanks, Marnie - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Best practice for sending UTF-16 wstring and Binary blobs via messaging API for C++ Python ?
Is the recommended practice for sending a UTF-16 string to convert to UTF-8 and send as std::string? Ideally, is there an example available to send receive a wstring via the messaging API? I understand the amqp 0-10 codec supports this type, but its not clear how to send something not directly supported by the Variant class. On a related note, what is the recommended practice for sending a binary blob using the messaging API? It wasn't clear from QPID-2452 if it is safe to stuff a binary array into a string type. Is there any example available to send receive a binary blob via the messaging API? And my apologies if this has appeared in recent example code, I'm using a slightly dated trunk... Thanks! - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
SSL connection string for messaging::Connection URL in C++ Python on Windows?
I've been working on enabling SSL for our Windows Python based application. I've finally got silent root and server certificate generators installers working to bring up the brokers, now trying to figure out how to specify an SSL connection using the messaging APIs. Any pointers? Thanks... - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Suggestions for Broker fault tolerance on Windows platform?
I've been walking down a rat hole and thought I'd take a moment to ask the list how I should proceed - I could use some advice from someone with deeper QPID experience... My requirement is relatively simple - I've got a number of clients using QPID , all currently Windows platforms, with clients in C++, Python, and Java. We need the system to handle broker process / host failure, preferably in a manner transparent to the client applications (although we can insert code between the client applications and the QPID client library.) We can run multiple brokers, and had planned on deploying a broker on each server box, and wanted at least some sort of failover solution. Our problem - how to proceed on Windows? QPID Clustering is built over Corosync, which I believe limits it to a subset of *nix platforms. Federation with failover would work, but Federation doesn't work on Windows. (I'm still looking at QPID-2199...) Failover code in progress appears tied to clustering, which is out for Windows. Should I just roll my own standalone failover, forking if necessary to expose lower features? (I don't want to do this!) Can I leverage the Failover code in the trunk and manage amq.failover without the Corosync dependent code? Are there any simpler approaches I'm missing? Comments greatly appreciated... - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Suggestions for Broker fault tolerance on Windows platform?
I like the idea of generalizing the failover scheme. Any suggestions on where to start? An externally configurable list of brokers published to amq.failover would seem an appropriate first step, and then work on mechanisms to update those lists. I'm looking over failover examples now... On Wed, May 5, 2010 at 1:39 PM, Alan Conway acon...@redhat.com wrote: On 05/05/2010 02:04 PM, Kerry Bonin wrote: I've been walking down a rat hole and thought I'd take a moment to ask the list how I should proceed - I could use some advice from someone with deeper QPID experience... My requirement is relatively simple - I've got a number of clients using QPID , all currently Windows platforms, with clients in C++, Python, and Java. We need the system to handle broker process / host failure, preferably in a manner transparent to the client applications (although we can insert code between the client applications and the QPID client library.) We can run multiple brokers, and had planned on deploying a broker on each server box, and wanted at least some sort of failover solution. Our problem - how to proceed on Windows? QPID Clustering is built over Corosync, which I believe limits it to a subset of *nix platforms. Federation with failover would work, but Federation doesn't work on Windows. (I'm still looking at QPID-2199...) Failover code in progress appears tied to clustering, which is out for Windows. Should I just roll my own standalone failover, forking if necessary to expose lower features? (I don't want to do this!) Can I leverage the Failover code in the trunk and manage amq.failover without the Corosync dependent code? Are there any simpler approaches I'm missing? Comments greatly appreciated... The client side of failover doesn't involve corosync at all. Clients subscribe to the amq.failover exchange to get updates on the list of brokers they can use for failover, and if they detect a connection failure they iterate over this list till they get a connection. All this is standard AMQP. The cluster plugin on the broker generates those updates based on corosync membership changes. If you had an alternate plugin generating membership updates then it could drive the existing client failover with no changes on the client. Of course this assumes some non-corosync source for the membership info but you'd need that anyway in a roll-your-own scheme. I'd be happy to help out if you're interested in implementing something like this. Generalizing the failover-list notification scheme is preferable to forking and I think will be useful in other contexts as well. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Suggestions for Broker fault tolerance on Windows platform?
Hello Gordon! I've seen reconnection working, I haven't tried URL lists, and I hadn't seen mention of resending unconfirmed messages. I'll look at both of those now, and thanks for the pointer! We've moved to the messaging API already, so this may make a great short term solution for us... On Wed, May 5, 2010 at 2:31 PM, Gordon Sim g...@redhat.com wrote: On 05/05/2010 08:22 PM, Kerry Bonin wrote: I like the idea of generalizing the failover scheme. Any suggestions on where to start? An externally configurable list of brokers published to amq.failover would seem an appropriate first step, and then work on mechanisms to update those lists. I'm looking over failover examples now... The messaging api in c++ and python both support reconnection and can take a list of urls to try. They will replay any unconfirmed messages on reconnect as well. So if your overall solution doesn't require actual message replication, this may work for you. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Commented: (QPID-2199) Federation connections initiated from windows brokers stuck in connecting state
[ https://issues.apache.org/jira/browse/QPID-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12863497#action_12863497 ] Kerry Bonin commented on QPID-2199: --- I've modified the AsynchConnector class in the Windows AsynchIO.cpp code to perform asynchronous connects and the links are still reported stuck in connecting state. I'm now digging into how link state is managed after the connection succeeds. Federation connections initiated from windows brokers stuck in connecting state - Key: QPID-2199 URL: https://issues.apache.org/jira/browse/QPID-2199 Project: Qpid Issue Type: Bug Components: C++ Broker Affects Versions: 0.5 Environment: Windows XP, qpid 0.5 from .msi installer, hosts scengsrv and jlaughlin, qpidbroker.exe running under cmd.exe, other tools running under cygwin Ubuntu Hardy, qpid 0.5 compiled from source with unused return value patches, host santa-anna Reporter: Jeff Laughlin Attachments: linux.cap, linux.out, windows.cap, windows.out My Windows broker can't establish federation link to other brokers running on windows or linux; links remain in connecting state forever. Packet sniffer reveals strange and inconsistent things. Linux broker can successfully establish links to windows brokers, however. Python client tools seems to work fine with both windows and linux brokers, from both cygwin python and linux python. Here's a terminal session transcript that illustrates this behavior. I'm starting with two fresh instances of qpid on two different windows hosts, jlaughlin and scengsrv. Both have been configured with a durable alarms exchange and durable alarmd queue that have been bound together. First I try to create the link in push mode, so qpid-route is commanding jlaughlin (the localhost) to establish a link to scengsrv jlaugh...@jlaughlin ~/Downloads/qpid-0.5/python/commands $ python2.5 qpid-route queue add scengsrv jlaughlin alarms alarmd --ack 1 --src-local After a moment I check my links jlaugh...@jlaughlin ~/Downloads/qpid-0.5/python/commands $ python2.5 qpid-route link list HostPortTransport Durable State Last Error = scengsrv5672tcp N Connecting Hm it's still connecting. Wait a few moments. jlaugh...@jlaughlin ~/Downloads/qpid-0.5/python/commands $ python2.5 qpid-route link list HostPortTransport Durable State Last Error = scengsrv5672tcp N Connecting Still connecting. Darn. Lets try going the other way, commanding scengsrv to link to jlaughlin. jlaugh...@jlaughlin ~/Downloads/qpid-0.5/python/commands $ python2.5 qpid-route queue add scengsrv jlaughlin alarms alarmd --ack 1 jlaugh...@jlaughlin ~/Downloads/qpid-0.5/python/commands $ python2.5 qpid-route link list scengsrv HostPortTransport Durable State Last Error = jlaughlin 5672tcp N Connecting Still no good. Now lets try commanding jlaughlin to connect to my linux box, santa-anna jlaugh...@jlaughlin ~/Downloads/qpid-0.5/python/commands $ python2.5 qpid-route queue add santa-anna jlaughlin alarms alarmd --ack 1 --src-local jlaugh...@jlaughlin ~/Downloads/qpid-0.5/python/commands $ python2.5 qpid-route link list HostPortTransport Durable State Last Error = santa-anna 5672tcp N Connecting jlaugh...@jlaughlin ~/Downloads/qpid-0.5/python/commands $ python2.5 qpid-route link list HostPortTransport Durable State Last Error = santa-anna 5672tcp N Connecting Negative, ghost rider, the pattern is full. Enough of this, lets command the linux host, santa-anna, to connect to my jlaughlin windows host: jlaugh...@jlaughlin ~/Downloads/qpid-0.5/python/commands $ python2.5 qpid-route queue add santa-anna jlaughlin alarms alarmd --ack 1 jlaugh...@jlaughlin ~/Downloads/qpid-0.5/python/commands $ python2.5 qpid-route link list santa-anna HostPortTransport Durable State Last Error = jlaughlin 5672tcp N Operational It works! Yay Linux! Still why is the windows
Re: Help requested w/ federation - what am I doing wrong?
OK, thanks for the pointer to that issue, I remember reading it a
while ago, didn't realize it was still open (I love JIRA, but issue
search isn't very effective, at least for me - I often don't find
highly relevant open issues...) Can you think of any other open
issues that might bite me in this area? In the mean time, I guess I'm
blocked by 2199, if that is the case I'll go fix it...
Kerry
On Wed, Apr 21, 2010 at 4:33 PM, Steve Huston shus...@riverace.com wrote:
Hi Kerry,
--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications
http://www.riverace.com
-Original Message-
From: Kerry Bonin [mailto:kerrybo...@gmail.com]
Sent: Wednesday, April 21, 2010 4:05 PM
To: qpid-dev
Subject: Help requested w/ federation - what am I doing wrong?
This is on XP SP2, using recent build (r921371 + my QPID-2519
patch applied but not being used.)
I think I'm doing this correctly, but I never see anything
from the receiver on the second broker. As I understand it,
shouldn't clients be able to subscribe to topics and see
published messages no matter which broker they are connected to?
You're not doing anything wrong that I can see, but Windows has a known
problem in this area: https://issues.apache.org/jira/browse/QPID-2199
Nobody has had time to address it yet, so I don't know if this is a big
or small problem. I suspect it's not terribly big.
Let me know if you need some help navigating this.
Best regards,
-Steve
My end goal here is to add some fault tolerance to a Windows
AMQP QPID system, where clients can switch to another broker
if their current broker dies. Since someone chose a Linux
only solution for clustering, I think my simplest option is
to leverage Federation, with something like
ResilientConnection to manage a list of brokers, and fail
client connections over to other brokers on connection
failure. This is from my federation test case, and I'm
probably setting something up incorrectly...
Greatly appreciate the help!
Kerry Bonin
I bring up two brokers:
start 5680 /Dd:\dev\qpid-r921371\cpp\build\src\debug
qpidd.exe --data-dir=.\qpidd.data.5680 --auth=no --port=5680
--load-module=qmfconsoled.dll
start 5681 /Dd:\dev\qpid-r921371\cpp\build\src\debug
qpidd.exe --data-dir=.\qpidd.data.5681 --auth=no --port=5681
--load-module=qmfconsoled.dll
Create exchanges
python D:\dev\qpid-r921371\tools\src\py\qpid-config -a
localhost:5680 add exchange topic fed.topic
python D:\dev\qpid-r921371\tools\src\py\qpid-config -a
localhost:5681 add exchange topic fed.topic
Create routes
python D:\dev\qpid-r921371\tools\src\py\qpid-route dynamic
add localhost:5680 localhost:5681 fed.topic
python D:\dev\qpid-r921371\tools\src\py\qpid-route dynamic
add localhost:5681 localhost:5680 fed.topic
This appears to work correctly :
D:\dev\qpid\binpython d:\dev\qpid-r921371\tools\src\py\qpid-route
route map localhost:5680
Finding Linked Brokers:
localhost:5680... Ok
localhost:5681... Ok
Dynamic Routes:
Exchange fed.topic:
localhost:5681 = localhost:5680
Static Routes:
none found
Now trimmed from my C++ testbed...
// Setup URLs and Addresses
std::string urlA = amqp:tcp:127.0.0.1:5680;
std::string urlB = amqp:tcp:127.0.0.1:5681;
std::string queue = fed.topic;
Address addressTx( queue );
Address addressRx( queue );
int64_t timeout = 1000;
// Setup transmitter on 5680
Connection connectionTxA;
connectionTxA.open( urlA );
Session sessionTxA = connectionTxA.newSession();
Sender senderTxA = sessionTxA.createSender( addressTx );
// Setup listeners on 5680 and 5681
Connection connectionRxA;
connectionRxA.open( urlA );
Session sessionRxA = connectionRxA.newSession();
Receiver receiverRxA = sessionRxA.createReceiver( addressRx );
Connection connectionRxB;
connectionRxB.open( urlB );
Session sessionRxB = connectionRxB.newSession();
Receiver receiverRxB = sessionRxB.createReceiver( addressRx );
// Transmit to 5680
Message messageOut;
MapContent contentOut( messageOut );
contentOut[id] = 1234;
contentOut[name] = Request;
contentOut.encode();
senderTxA.send( messageOut );
// Local listener sees the message
Message messageRxA;
if( receiverRxA.fetch( messageRxA, qpid::sys::Duration(
timeout ) ) ) {
MapView contentRxA( messageRxA );
std::cout local received: contentRxA std::endl;
sessionRxA.acknowledge();
}
else
std::cout local timeout std::endl;
// Remote never does...
Message messageRxB;
if( receiverRxB.fetch( messageRxB, qpid::sys::Duration(
timeout ) ) ) {
MapView contentRxB( messageRxB );
std::cout remote received: contentRxB std::endl;
sessionRxB.acknowledge();
}
else
std::cout remote timeout std::endl;
-
Apache Qpid - AMQP Messaging Implementation
Project: http
Re: Contributer Patches - Committers Requested
I'd also suggest the patch I made w/ QPID-1904 - The C++ AbsTime Duration values are not set to a consistent epoch, which causes problems when you try using QMF timestamps across Linux and Windows, and even on Windows when you try and talk between Python and C++ clients. I submitted a patch which forces the epoch to that specified in AMQP, instead of inheriting it from the platform, which seems quite wrong for a network protocol across platforms with differing native epochs. The epoch is calculated once, and the conversion was already relative to a base, so there should be no performance hit. On Thu, Apr 22, 2010 at 8:27 AM, Emmanuel Bourg ebo...@apache.org wrote: Rajith Attapattu a écrit : I have taken care of QPID-2508 and QPID-2522 from Emmanuel. I left the MINA patch to Martin as he is more familiar with that side. Thanks all for taking care of the patches. There is also an involved patch waiting for a thorough review in QPID-2433. https://issues.apache.org/jira/browse/QPID-2433 Emmanuel Bourg - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Issue Comment Edited: (QPID-2519) Allow Windows Broker to run as a Service
[ https://issues.apache.org/jira/browse/QPID-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12858943#action_12858943 ] Kerry Bonin edited comment on QPID-2519 at 4/20/10 1:38 PM: This patch implements most of my worklist for the feature. - No changes to cross-platform qpidd.cpp - Preserve single executable. - Preserve service encapsulation in separate class (WinService) - Encapsulate basic SCM features via command line - install, start, stop, uninstall - Allow basic SCM options at install - account, dependent service list - Runs on XP through Win7, Server 2003 2008 (passed a few weeks system test) Features I still want and plan to add, but can't get to right now, hope to get back to soon... - Rename service - Set service description text - Change default account name - Change startup error level - Clean up event log use for error reporting - add use message files for proper formatting. Please let me know if anyone has any issues or concerns with these changes, and I'll do my best to fix them ASAP. We've already built this into a major product release, its worked well in test so far... Kerry Bonin Systems Architect United Technologies Fire Security, formerly GE Security was (Author: kbonin): This patch implements most of my worklist for the feature. - No changes to cross-platform qpidd.cpp - Preserve single executable. - Preserve service encapsulation in separate class (WinService) - Encapsulate basic SCM features via command line - install, start, stop, uninstall - Allow basic SCM options at install - account, dependent service list - Runs on XP through Win7, Server 2003 2008 (passed a few weeks system test) Features I still want and plan to add, but can't get to right now, hope to get back to soon... - Rename service - Set service description text - Change default account name - Change startup error level - Clean up event log use for error reporting - add use message files for proper formatting. Please let me know if anyone has any issues or concerns with these changes, and I'll do my best to fix them ASAP. We've already built this into a major product release (GE Security / United Technologies Fire Security), its worked well in test so far... Kerry Bonin Systems Architect United Technologies Fire Security, formerly GE Security Allow Windows Broker to run as a Service Key: QPID-2519 URL: https://issues.apache.org/jira/browse/QPID-2519 Project: Qpid Issue Type: New Feature Components: C++ Broker Reporter: Kerry Bonin Attachments: broker_as_service_r921371.patch The Windows version of the C++ Broker should have the ability to run as a Windows Service, to correspond with the ability of the *nix version to run as a daemon, as this is a common requirement for critical software infrastructure elements of enterprise class software on the Windows platform. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Issue Comment Edited: (QPID-2519) Allow Windows Broker to run as a Service
[ https://issues.apache.org/jira/browse/QPID-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12858943#action_12858943 ] Kerry Bonin edited comment on QPID-2519 at 4/20/10 1:38 PM: This patch implements most of my worklist for the feature. - No changes to cross-platform qpidd.cpp - Preserve single executable. - Preserve service encapsulation in separate class (WinService) - Encapsulate basic SCM features via command line - install, start, stop, uninstall - Allow basic SCM options at install - account, dependent service list - Runs on XP through Win7, Server 2003 2008 (passed a few weeks system test) Features I still want and plan to add, but can't get to right now, hope to get back to soon... - Rename service - Set service description text - Change default account name - Change startup error level - Clean up event log use for error reporting - add use message files for proper formatting. Please let me know if anyone has any issues or concerns with these changes, and I'll do my best to fix them ASAP. We've already built this into a major product release, its worked well in test so far... Kerry Bonin was (Author: kbonin): This patch implements most of my worklist for the feature. - No changes to cross-platform qpidd.cpp - Preserve single executable. - Preserve service encapsulation in separate class (WinService) - Encapsulate basic SCM features via command line - install, start, stop, uninstall - Allow basic SCM options at install - account, dependent service list - Runs on XP through Win7, Server 2003 2008 (passed a few weeks system test) Features I still want and plan to add, but can't get to right now, hope to get back to soon... - Rename service - Set service description text - Change default account name - Change startup error level - Clean up event log use for error reporting - add use message files for proper formatting. Please let me know if anyone has any issues or concerns with these changes, and I'll do my best to fix them ASAP. We've already built this into a major product release, its worked well in test so far... Kerry Bonin Systems Architect United Technologies Fire Security, formerly GE Security Allow Windows Broker to run as a Service Key: QPID-2519 URL: https://issues.apache.org/jira/browse/QPID-2519 Project: Qpid Issue Type: New Feature Components: C++ Broker Reporter: Kerry Bonin Attachments: broker_as_service_r921371.patch The Windows version of the C++ Broker should have the ability to run as a Windows Service, to correspond with the ability of the *nix version to run as a daemon, as this is a common requirement for critical software infrastructure elements of enterprise class software on the Windows platform. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
How do we run python tools after recent changes?
I used to be able to get qpid-route, ect. to run by updating my PYTHONPATH, but now I get ImportError: No module named qmf.console even with that set correctly. Any updated methods? Thanks...
Re: Access to qpidd original arg list from {platform specific}QpiddBroker.cpp ?
FWIW, I've been doing some reading, and talking to friends who are still
active on the exploit side of the security world, and they've convinced me
the default user should be NetworkService. Short version is a) it is the
official Microsoft recommendation, and b) all the attack plans post process
injection / remote execution begin with privilege escalation, so its best if
your process starts with the lowest possible local privileges. In the real
world once process injection / remote execution is achieved there are so
many unpatched privilege escalation vectors on Windows that a network
service has already lost the machine, so you might as well make the next
step as hard as possible.
Account, password and depends now work (and are delivered to our test team),
looking at QPID-1423 today...
On Thu, Apr 1, 2010 at 4:27 PM, Steve Huston shus...@riverace.com wrote:
This sounds great, Kerry. Looking forward to seeing it.
-Steve
-Original Message-
From: Kerry Bonin [mailto:kerrybo...@gmail.com]
Sent: Thursday, April 01, 2010 1:30 PM
To: dev@qpid.apache.org
Subject: Re: Access to qpidd original arg list from {platform
specific}QpiddBroker.cpp ?
LocalService still has access to the network, and is
generally preferable over NetworkService - NetworkService
tries to use the domain account privileges it is running
under, whereas LocalService is treated as an anonymous
network agent. A hacked service running under NetworkService
will therefore have more privileges in the local network than
one running under LocalService, so unless you need to
interact with the domain, LocalService is preferred for
network services.
On the service abstraction issue, I've now been able to
contain all the Windows service code to the Windows specific
QpiddBroker.cpp, and reverted all my changes to qpidd.cpp.
(Although I am adding a WinService .cpp/.h with my generic
service helper class.) So the platform abstraction is as
clean as possible on the Windows side. The only other
interaction between the service code and the broker is during
shutdown, in which case I preserve the Broker * and call the
shutdown() API.
I've added --start-type={auto|demand|disabled}, and now
working on --account, --password, and --depends (register
list of dependent services), all as per SCM conventions.
Also looking at QPID-1423...
On Thu, Apr 1, 2010 at 12:14 PM, Andrew Stitcher
astitc...@redhat.comwrote:
On Thu, 2010-04-01 at 11:46 -0500, Kerry Bonin wrote:
There is a UAC issue on installing a service, in that if
you try and
use
the
--install command while logged in as a user that does not have the
privilege
to install services, it will (properly) fail. So that is
working as
it should - in our testing here we use that command during our
installation process, which is running with elevated
privileges, so
it works at that time, as it should.
As for normal use, I have the broker by default install
itself under
the LocalSystem account, which IIRC is the recommended
account for
services,
and
this account has reduced privileges on later Windows. If
you NEED
to run
it
with more or less privileges, you can create or use an existing
account appropriately, and just tell the broker to use
that account
- you only
need
to provide the credentials once, and Windows SCM manages
the token
generation and caching as per normal SCM rules, and when you try
installing
it you would obviously need sufficient rights to use that account
AND install a service.
Doesn't it have to be the 'NETWORK SERVICE' account for it to have
access to the network? which you'd think it would need! In
more recent
versions of windows the service accounts are split.
I'm pretty sure this all meets guidelines, as I'm never violating
any security rules that I know of... (and I'm a hardcore
security /
crypto geek, 25+ years...)
If you would like me to disable the self-installation features
before
patch,
I certainly can...
No need, I was working from a faulty memory, and I can't
think of any
good reason to exclude this functionality. I suspect the
much bigger
issue is going to be how the Unix daemonisation and the Windows
service code are both abstracted so that the code becomes easier to
maintain.
Andrew
-
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscr...@qpid.apache.org
-
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Access to qpidd original arg list from {platform specific}QpiddBroker.cpp ?
One question - should I create a new jira, or use an existing, to submit the
patch ?
On Fri, Apr 2, 2010 at 11:34 AM, Steve Huston shus...@riverace.com wrote:
Thanks for checking into best practices here, Kerry!
-Steve
-Original Message-
From: Kerry Bonin [mailto:kerrybo...@gmail.com]
Sent: Friday, April 02, 2010 11:33 AM
To: dev@qpid.apache.org
Subject: Re: Access to qpidd original arg list from {platform
specific}QpiddBroker.cpp ?
FWIW, I've been doing some reading, and talking to friends
who are still active on the exploit side of the security
world, and they've convinced me the default user should be
NetworkService. Short version is a) it is the official
Microsoft recommendation, and b) all the attack plans post
process injection / remote execution begin with privilege
escalation, so its best if your process starts with the
lowest possible local privileges. In the real world once
process injection / remote execution is achieved there are so
many unpatched privilege escalation vectors on Windows that a
network service has already lost the machine, so you might as
well make the next step as hard as possible.
Account, password and depends now work (and are delivered to
our test team), looking at QPID-1423 today...
On Thu, Apr 1, 2010 at 4:27 PM, Steve Huston
shus...@riverace.com wrote:
This sounds great, Kerry. Looking forward to seeing it.
-Steve
-Original Message-
From: Kerry Bonin [mailto:kerrybo...@gmail.com]
Sent: Thursday, April 01, 2010 1:30 PM
To: dev@qpid.apache.org
Subject: Re: Access to qpidd original arg list from {platform
specific}QpiddBroker.cpp ?
LocalService still has access to the network, and is generally
preferable over NetworkService - NetworkService tries to use the
domain account privileges it is running under, whereas
LocalService
is treated as an anonymous network agent. A hacked
service running
under NetworkService will therefore have more privileges in the
local network than one running under LocalService, so unless you
need to interact with the domain, LocalService is preferred for
network services.
On the service abstraction issue, I've now been able to
contain all
the Windows service code to the Windows specific QpiddBroker.cpp,
and reverted all my changes to qpidd.cpp. (Although I am adding a
WinService .cpp/.h with my generic service helper class.) So the
platform abstraction is as clean as possible on the
Windows side.
The only other interaction between the service code and
the broker
is during shutdown, in which case I preserve the Broker *
and call
the
shutdown() API.
I've added --start-type={auto|demand|disabled}, and now
working on
--account, --password, and --depends (register list of dependent
services), all as per SCM conventions. Also looking at
QPID-1423...
On Thu, Apr 1, 2010 at 12:14 PM, Andrew Stitcher
astitc...@redhat.comwrote:
On Thu, 2010-04-01 at 11:46 -0500, Kerry Bonin wrote:
There is a UAC issue on installing a service, in that if
you try and
use
the
--install command while logged in as a user that does
not have
the
privilege
to install services, it will (properly) fail. So that is
working as
it should - in our testing here we use that command
during our
installation process, which is running with elevated
privileges, so
it works at that time, as it should.
As for normal use, I have the broker by default install
itself under
the LocalSystem account, which IIRC is the recommended
account for
services,
and
this account has reduced privileges on later Windows. If
you NEED
to run
it
with more or less privileges, you can create or use
an existing
account appropriately, and just tell the broker to use
that account
- you only
need
to provide the credentials once, and Windows SCM manages
the token
generation and caching as per normal SCM rules, and
when you try
installing
it you would obviously need sufficient rights to use that
account AND install a service.
Doesn't it have to be the 'NETWORK SERVICE' account for
it to have
access to the network? which you'd think it would need! In
more recent
versions of windows the service accounts are split.
I'm pretty sure this all meets guidelines, as I'm never
violating any security rules that I know of... (and I'm a
hardcore
security /
crypto geek, 25+ years...)
If you would like me to disable the self-installation
features
before
patch,
I certainly can...
No need, I was working from a faulty memory, and I can't
think of any
good reason to exclude
Re: Access to qpidd original arg list from {platform specific}QpiddBroker.cpp ?
There is a UAC issue on installing a service, in that if you try and use the --install command while logged in as a user that does not have the privilege to install services, it will (properly) fail. So that is working as it should - in our testing here we use that command during our installation process, which is running with elevated privileges, so it works at that time, as it should. As for normal use, I have the broker by default install itself under the LocalSystem account, which IIRC is the recommended account for services, and this account has reduced privileges on later Windows. If you NEED to run it with more or less privileges, you can create or use an existing account appropriately, and just tell the broker to use that account - you only need to provide the credentials once, and Windows SCM manages the token generation and caching as per normal SCM rules, and when you try installing it you would obviously need sufficient rights to use that account AND install a service. I'm pretty sure this all meets guidelines, as I'm never violating any security rules that I know of... (and I'm a hardcore security / crypto geek, 25+ years...) If you would like me to disable the self-installation features before patch, I certainly can... Kerry On Thu, Apr 1, 2010 at 9:33 AM, Andrew Stitcher astitc...@redhat.comwrote: On Tue, 2010-03-30 at 11:21 -0500, Kerry Bonin wrote: Hello, and thanks for the comments! First, thank you Andrew for GetCommandLine() - 20 something years on Windows, and I don't remember seeing that one before, certainly made this simpler. On the subject of the command line in general for a service - I agree that under most normal use a config file should be used, I just wanted to make sure the command line was usable... On the security of self-installing services - if the service is doing much more than installing itself, especially if it contains baked in credentials, ect., that would be a very bad thing. What I've done is essentially the equivalent of sc create|start|stop|delete wrap as a convenience function, nothing more, and the calls execute with the same privilege level a user has available to them at the command line. ISTR that the issue is related to UAC, but I admit I can't quite see what the issue could be. One security related issue I'd suggest is that if running qpid as a service then we should run as an unprivileged user though with network access. qpidd only shuffles bits around a network so doesn't seem to need elevated privileges. I'm not sure how this fits exactly, but I assume that you'd need to create a new user account on installation and run the service using it. This would obviously require admin privileges. Andrew - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Access to qpidd original arg list from {platform specific}QpiddBroker.cpp ?
LocalService still has access to the network, and is generally preferable
over NetworkService - NetworkService tries to use the domain account
privileges it is running under, whereas LocalService is treated as an
anonymous network agent. A hacked service running under NetworkService will
therefore have more privileges in the local network than one running under
LocalService, so unless you need to interact with the domain, LocalService
is preferred for network services.
On the service abstraction issue, I've now been able to contain all the
Windows service code to the Windows specific QpiddBroker.cpp, and reverted
all my changes to qpidd.cpp. (Although I am adding a WinService .cpp/.h
with my generic service helper class.) So the platform abstraction is as
clean as possible on the Windows side. The only other interaction between
the service code and the broker is during shutdown, in which case I preserve
the Broker * and call the shutdown() API.
I've added --start-type={auto|demand|disabled}, and now working on
--account, --password, and --depends (register list of dependent services),
all as per SCM conventions. Also looking at QPID-1423...
On Thu, Apr 1, 2010 at 12:14 PM, Andrew Stitcher astitc...@redhat.comwrote:
On Thu, 2010-04-01 at 11:46 -0500, Kerry Bonin wrote:
There is a UAC issue on installing a service, in that if you try and use
the
--install command while logged in as a user that does not have the
privilege
to install services, it will (properly) fail. So that is working as it
should - in our testing here we use that command during our installation
process, which is running with elevated privileges, so it works at that
time, as it should.
As for normal use, I have the broker by default install itself under the
LocalSystem account, which IIRC is the recommended account for services,
and
this account has reduced privileges on later Windows. If you NEED to run
it
with more or less privileges, you can create or use an existing account
appropriately, and just tell the broker to use that account - you only
need
to provide the credentials once, and Windows SCM manages the token
generation and caching as per normal SCM rules, and when you try
installing
it you would obviously need sufficient rights to use that account AND
install a service.
Doesn't it have to be the 'NETWORK SERVICE' account for it to have
access to the network? which you'd think it would need! In more recent
versions of windows the service accounts are split.
I'm pretty sure this all meets guidelines, as I'm never violating any
security rules that I know of... (and I'm a hardcore security / crypto
geek, 25+ years...)
If you would like me to disable the self-installation features before
patch,
I certainly can...
No need, I was working from a faulty memory, and I can't think of any
good reason to exclude this functionality. I suspect the much bigger
issue is going to be how the Unix daemonisation and the Windows service
code are both abstracted so that the code becomes easier to maintain.
Andrew
-
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Access to qpidd original arg list from {platform specific}QpiddBroker.cpp ?
Hello, and thanks for the comments! First, thank you Andrew for GetCommandLine() - 20 something years on Windows, and I don't remember seeing that one before, certainly made this simpler. On the subject of the command line in general for a service - I agree that under most normal use a config file should be used, I just wanted to make sure the command line was usable... On the security of self-installing services - if the service is doing much more than installing itself, especially if it contains baked in credentials, ect., that would be a very bad thing. What I've done is essentially the equivalent of sc create|start|stop|delete wrap as a convenience function, nothing more, and the calls execute with the same privilege level a user has available to them at the command line. On the account / password arguments for the service - these are NOT retained by the service in any way - they are merely passed to the SCM as part of service creation in the exact same way the sc command lets you specify the obj and password arguments - they are then used by the SCM to create a security token which is placed in the registry along with the service data, and used when the service is started. On the event log, I hadn't seen that option yet, I'll look for it and not add another one... Kerry On Mon, Mar 29, 2010 at 4:15 PM, Steve Huston shus...@riverace.com wrote: Looks nice, Kerry! I don't have many answers for you at this point, but one note... Use of event log There is already a logging option to put stuff in the windows event log. As a side note, if anyone has any other suggestions related to this or other service features, please let me know and I'll see if I can add it. The installer build is going to be checked in sometime soon. This builds a WiX-based installer; it may be nice to add the ability to install the broker as a service during the install. -Steve - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Access to qpidd original arg list from {platform specific}QpiddBroker.cpp ?
Hello Steve! I'll take a look at fixing QPID-1423 next, hadn't seen it before... Kerry On Tue, Mar 30, 2010 at 11:52 AM, Steve Huston shus...@riverace.com wrote: Hi Kerry, Hello, and thanks for the comments! You're welcome - thanks for working on a very useful addition! On the event log, I hadn't seen that option yet, I'll look for it and not add another one... Great - also please note QPID-1423 if you wish to make some improvement there: https://issues.apache.org/jira/browse/QPID-1423 Thanks, -Steve On Mon, Mar 29, 2010 at 4:15 PM, Steve Huston shus...@riverace.com wrote: Looks nice, Kerry! I don't have many answers for you at this point, but one note... Use of event log There is already a logging option to put stuff in the windows event log. As a side note, if anyone has any other suggestions related to this or other service features, please let me know and I'll see if I can add it. The installer build is going to be checked in sometime soon. This builds a WiX-based installer; it may be nice to add the ability to install the broker as a service during the install. -Steve - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
C++ Messaging API questions...
I'm looking at migrating to messaging, and have a few questions. r/e Message : - If headers has an accessor, why doesn't it work? It only returns an empty map. - Why isn't timestamp exposed? (Trust TZ issues aside, it is common to want this data, why require recreation?) r/e Receiver : - Why did we lose the callback pattern? I now have to (at the least) spin a thread across all my receivers to collect asynchronously. FWIW, I'd be very happy to help fix these, as I need them myself...
Re: C++ Messaging API questions...
r/e headers - I do see the same thing you listed (when I use quotation
marks, I'm on Win32). Your comment implies the basis for my question -
digging into the source I now see properties are internally split into
MessageProperties and DeliveryProperties, and I was expecting to see those
fields when I called getHeaders().
r/e timestamps
- One common use case we have is to have clients synchronize distributed
data sets, which we do by providing a snapshot via request - response as
well as an update broadcast via publish - subscribe. We use timestamps to
determine when an update event was sent relative to the snapshot. We were
including our own timestamp field in the message body for this, and thought
we could use message timestamps.
- Another common use case is to broadcast events. We care when these were
sent, and historically have included a UTC event time in the message that is
used (in part) to identify the event. Again, we could include our own
timestamp in the message body for this, but we thought we could use the
message timestamp for this - so long as it is the time of creation or first
transmission. We haven't seen documentation on the expected behavior of
message timestamps across federated and/or clustered systems, or through
durable queues.
r/e the asynchronous APIs, this is more of a nice to have, but we have C++
and Python developers who liked not having to create service threads to read
asynchronously for the previous API. Event loops are common, but not always
used...
On Tue, Mar 16, 2010 at 12:20 PM, Gordon Sim g...@redhat.com wrote:
On 03/16/2010 03:58 PM, Kerry Bonin wrote:
I'm looking at migrating to messaging, and have a few questions.
r/e Message :
- If headers has an accessor, why doesn't it work? It only returns an
empty
map.
It works for me (are you using trunk?):
$ ./examples/messaging/spout --address 'my-queue; {create: always}'
--property my-property=my-value
$ ./examples/messaging/drain --address 'my-queue; {create: always}'
Message(properties={my-property:my-value,
spout-id:eab53080-8f16-48db-b4cf-5cf9bb200ec0:0}, content='')
The above uses Message::getHeaders(), what do you see when you run those
examples?
(Note I'd actually like to change that to getProperties() for consistency
with the python client).
- Why isn't timestamp exposed? (Trust TZ issues aside, it is common to
want this data, why require recreation?)
How would you expect the timestamp to be used?
r/e Receiver :
- Why did we lose the callback pattern? I now have to (at the least) spin
a
thread across all my receivers to collect asynchronously.
The idea is that some sort of generic dispatcher would be layered on top of
the current api. The plan is to extend or generalise the
Session::nextReceiver() pattern so that Sessions and indeed Connections
could be integrated into a common event loop.
FWIW, I'd be very happy to help fix these, as I need them myself...
Excellent, and thanks for the feedback, the API could do with some more
eyes on it.
-
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: C++ Messaging API questions...
On Tue, Mar 16, 2010 at 1:29 PM, Gordon Sim g...@redhat.com wrote:
On 03/16/2010 06:05 PM, Kerry Bonin wrote:
r/e headers - I do see the same thing you listed (when I use quotation
marks, I'm on Win32). Your comment implies the basis for my question -
digging into the source I now see properties are internally split into
MessageProperties and DeliveryProperties, and I was expecting to see those
fields when I called getHeaders().
The messaging API is intended to abstract out protocol specifics. So there
are explicit setters/getters for attributes of a message (e.g. userid or
ttl) and these will be mapped on to the underlying protocol used (in the
case of AMQP 0-10, onto fields in Message- or Delivery- Properties). Then
there are generic properties available to applications for their own
purposes. These are name value pairs. In AMQP 0-10 they map onto
MessageProperites::applicationHeaders.
r/e timestamps
- One common use case we have is to have clients synchronize distributed
data sets, which we do by providing a snapshot via request - response as
well as an update broadcast via publish - subscribe. We use timestamps to
determine when an update event was sent relative to the snapshot. We were
including our own timestamp field in the message body for this, and
thought
we could use message timestamps.
- Another common use case is to broadcast events. We care when these were
sent, and historically have included a UTC event time in the message that
is
used (in part) to identify the event. Again, we could include our own
timestamp in the message body for this, but we thought we could use the
message timestamp for this - so long as it is the time of creation or
first
transmission. We haven't seen documentation on the expected behavior of
message timestamps across federated and/or clustered systems, or through
durable queues.
So the timestamp is set by the sender and refers to the creation of the
message or to the time of the event the message represents? (i.e. the
detailed semantics are application specific?)
Yes, that's how we've used message timestamps historically - our send API
would set the timestamp to the current time if it had not been previously
set during message construction, and this value would be propagated to all
recipients. If this concept differs from the meaning of timestamp in AMQP,
we can certainly return to an application field for timestamps. In either
case, it would be useful to clarify formally (somewhere) what timestamps are
used for in AMQP, and if they have any value to an application use it would
seem reasonable to expose them.
r/e the asynchronous APIs, this is more of a nice to have, but we have C++
and Python developers who liked not having to create service threads to
read
asynchronously for the previous API. Event loops are common, but not
always
used...
I agree its nice to have something there out of the box. However even in
the old 0-10 specific API the only really usable solution was
SubscriptionManager::run(). I think you could replace the functionality of
that method with:
typedef std::mapstd::string, MessageListener* Callbacks;
Callbacks callbacks;
//populate per subscription listeners
Listener* default = //default callback
while (true) {
Receiver r = session.nextReceiver();
Callbacks::const_iterator i = callbacks.find(r.getName());
if (i == callbacks.end()) {
default-received(r.fetch());
} else {
i-second-received(r.fetch());
}
}
You can also customise that loop more, e.g. by timing out if nothing is
received after a certain time, or having different strategies for
dispatching the messages to listeners/callbacks.
The above of course still requires a thread per session, but that is no
worse than the old API. The plan is to add the ability to tie in waiting on
one or more connection to the same pattern.
Re: [jira] Commented: (QPID-2395) fd leak in Connection
FWIW...
I've always required my teams to follow a practice where any class which
takes ownership of a resource is required to guarantee it is released
properly, and to treat leaking the resource as something worth of throwing
an assert or exception unless the object was explicitly ordered to leak the
resource.
In this case, I'd recommend adding an argument to open to enable/disable
automatic close on delete, with the default behavior being to close on
delete. That way if there is a corner case requiring such behavior, it is
possible to achieve with the public API, yet the default behavior will be to
handle the resource safely.
In this specific use case I've seen too many leaks occurring during
exception stack unwinding that were difficult to resolve cleanly...
On Fri, Feb 12, 2010 at 6:32 AM, Steve Huston (JIRA)
qpid-...@incubator.apache.org wrote:
[
https://issues.apache.org/jira/browse/QPID-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12832982#action_12832982]
Steve Huston commented on QPID-2395:
Stefan, the constructor doesn't initialize the fd; the open() call does, so
we're not really talking about RAII here. Thus, requiring a corresponding
close() is legitimate.
Also, if the destructor were to close the fd, we'd introduce subtle
problems where objects may be copied and one of the copies goes out of
scope, closing the fd out from under the others.
fd leak in Connection
-
Key: QPID-2395
URL: https://issues.apache.org/jira/browse/QPID-2395
Project: Qpid
Issue Type: Bug
Components: C++ Client
Affects Versions: 0.6
Environment: Linux dev-gmx001 2.6.31.8-GMX-CORE2_64 #1 SMP Tue
Dec 15 10:21:34 CET 2009 x86_64 Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
GenuineIntel GNU/Linux
Debian GNU/Linux 3.1
gcc version 4.3.3 (Debian 4.3.3-13.1)
Reporter: Daniel Etzold
Assignee: Steve Huston
Fix For: 0.7
Hi,
with the following code we are able to reproduce a fd leak:
#include qpid/client/Connection.h
int main(int argc, char** argv)
{
while (1) {
qpid::client::Connection connection;
connection.open(localhost, 5672);
//connection.close();
}
}
When closing the connection explicitely as shown in the comment the fd
leak no longer occurs. It seems that the destructor of Connection lacks
closing the connection.
Regards,
Daniel
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Created: (QPID-2402) qpid::messaging.Message::setTTL() unit ambiguity can cause unexpectected behavior
qpid::messaging.Message::setTTL() unit ambiguity can cause unexpectected behavior - Key: QPID-2402 URL: https://issues.apache.org/jira/browse/QPID-2402 Project: Qpid Issue Type: Bug Components: C++ Broker, C++ Client Reporter: Kerry Bonin qpid::messaging.Message::setTTL( boost::uint64_t ttl ) - does not declare its units. The currently exposed time types are AbsTime and Duration, which seems to imply they should be used for encapsulating time. When a Duration object is passed to setTTL, the inline operator int64_t() returns the member variable nanoseconds. When this value is propagated to the broker, it is interpreted around /cpp/src/qpid/broker/Message # 353 to be in millisecond units. I would recommend: - document Message.setTTL units clearly - pick ms or ns - if ms, then overload setTTL to take a Duration object so we can use without error by converting in setTTL - if ns, then somewhere downstream (OutgoingMessage#351 ?) the ns needs to get converted to ms -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Is anyone working on C++ broker TTL ?
Created QPID-2402 to capture the API ambiguity and behavior, called a bug as I can use the API in the way it appears to be appropriate, and see A problem which impairs or prevents the functions of the product. On Thu, Feb 11, 2010 at 10:26 AM, Steve Huston shus...@riverace.com wrote: I'm not Gordon or Rafi, but I thought we were trying to keep boost out of the API as well. I do agree that qpid::sys::Duration should not be exposed; something like qpid::messaging::Duration (which may just be a typedef for uint64_t) would be better. -Steve -Original Message- From: Andrew Stitcher [mailto:astitc...@redhat.com] Sent: Thursday, February 11, 2010 9:44 AM To: dev@qpid.apache.org Cc: Gordon Sim; Rafael Schloming Subject: Re: Is anyone working on C++ broker TTL ? On Wed, 2010-02-10 at 16:51 -0600, Kerry Bonin wrote: OK... (for the C++ code) qpid::messaging.Message::setTTL( boost::uint64_t ttl ); I was using this with a qpid::sys::Duration() object, which stores time in a boost::posix_time::ptime object. This is documented to store in units of nanoseconds. The Duration object (as several others) is meant to be entirely internal to the qpid code and not to leak to the client API. At least we should have explicitly decided it was part of the API if we want it there. It may look convenient, but it's really not been designed to be useful for any more than the internals of qpid. According to /cpp/src/qpid/broker/Message # 351-2, TTL is supposed to be set in milliseconds. When I manually send ms, it works fine. I would recommend: - document Message.setTTL units clearly - pick ms or ns - if ms, then overload setTTL to take a Duration object so we can use without error by converting in setTTL If we decided to keep the qpid time objects in the client API then we need to recast all times in the API to take (only) the qpid Duration and AbsTime types. - if ns, then somewhere downstream (OutgoingMessage#351 ?) the ns needs to get converted to ms This does raise the interesting issue of specifying time in the client API and doing it consistently - Gordon/Rafi do you have any thoughts WRT the new API? Andrew - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Is anyone working on C++ broker TTL ?
Created QPID-2402 to capture the API ambiguity and behavior, called a bug as I can use the API in the way it appears to be appropriate, and see A problem which impairs or prevents the functions of the product. On Thu, Feb 11, 2010 at 10:26 AM, Steve Huston shus...@riverace.com wrote: I'm not Gordon or Rafi, but I thought we were trying to keep boost out of the API as well. I do agree that qpid::sys::Duration should not be exposed; something like qpid::messaging::Duration (which may just be a typedef for uint64_t) would be better. -Steve -Original Message- From: Andrew Stitcher [mailto:astitc...@redhat.com] Sent: Thursday, February 11, 2010 9:44 AM To: dev@qpid.apache.org Cc: Gordon Sim; Rafael Schloming Subject: Re: Is anyone working on C++ broker TTL ? On Wed, 2010-02-10 at 16:51 -0600, Kerry Bonin wrote: OK... (for the C++ code) qpid::messaging.Message::setTTL( boost::uint64_t ttl ); I was using this with a qpid::sys::Duration() object, which stores time in a boost::posix_time::ptime object. This is documented to store in units of nanoseconds. The Duration object (as several others) is meant to be entirely internal to the qpid code and not to leak to the client API. At least we should have explicitly decided it was part of the API if we want it there. It may look convenient, but it's really not been designed to be useful for any more than the internals of qpid. According to /cpp/src/qpid/broker/Message # 351-2, TTL is supposed to be set in milliseconds. When I manually send ms, it works fine. I would recommend: - document Message.setTTL units clearly - pick ms or ns - if ms, then overload setTTL to take a Duration object so we can use without error by converting in setTTL If we decided to keep the qpid time objects in the client API then we need to recast all times in the API to take (only) the qpid Duration and AbsTime types. - if ns, then somewhere downstream (OutgoingMessage#351 ?) the ns needs to get converted to ms This does raise the interesting issue of specifying time in the client API and doing it consistently - Gordon/Rafi do you have any thoughts WRT the new API? Andrew - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Is anyone working on C++ broker TTL ?
(sorry for the double post there, had a gmail UI problem...) On Thu, Feb 11, 2010 at 11:11 AM, Andrew Stitcher astitc...@redhat.comwrote: On Thu, 2010-02-11 at 11:26 -0500, Steve Huston wrote: I'm not Gordon or Rafi, but I thought we were trying to keep boost out of the API as well. Absolutely, sorry not to mention it! Ideally the client API should export no other classes/interfaces at all. Essentially the reasoning here is: 1. Accidentally exporting other interfaces from your header files can lead to symbol conflict and confusion. 2. Leaking eg. boost leaks an API which has no stability guarantee. 3. Requiring the use of boost limits the client programmer. 4. The more we include in the API that isn't truly necessary the harder it gets to maintain the API. 5. Equally if we leak internal classes into the API then we have to be careful to change their interfaces as someone external may be relying on them. etc. etc. Andrew - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Is anyone working on C++ broker TTL ?
I can understand if messages live some reasonable interval past their TTL expiration, especially if it is a broker configurable argument. In our case, some developers wanted to set the TTL explicitly to 30 seconds, then saw their messages persist overnight, which meant something wasn't working as expected, hence my JIRA... On the subject of default settings (and those broker config settings, thanks for them, btw, hadn't seen that option yet) is we are experiencing quite a challenge with the 'state' of the documentation. Other areas of concern include the JSON dictionary like arguments on queue creation. I've seen docs appearing in the 'book' folder, need to look there more, but as a developer I wish there was a consistent place I could find accurate docs, and point my team to. If this was a wiki I'd be happy to contribute myself, as I discover things in the code that I think should be in the docs... On Thu, Feb 11, 2010 at 11:59 AM, Alan Conway acon...@redhat.com wrote: On 02/10/2010 02:54 PM, Kerry Bonin wrote: I just validated a report from a few of my developers that TTL doesn't seem to be working with the 0.6 broker, is anyone working on this ? If not I can have a look. If it does work I'd like to compare notes to see what we're doing wrong... Thx! (may be QPID-1418, QPID-1575 ?) This has been seen by my Python developer using 0.5 api w/ 0.6 broker, and I just saw in 0.6 trunk C++ using Messaging. In the C++ I'm creating w/ type:topic, calling qpid::messaging::Message.setTTL( TIME_SEC * 2 ), and never see the events go away. Not sure if this is a broker issue or a messaging issue... Messages are not deleted immediately when their TTL expires. There are two ways TTL messages can get dropped on the broker. If a consumer is looking for the next message on a queue, any stale messages at the front of the queue are dropped. There is also a periodic cleanup operation that removes stale messages from the queue but the default interval is 10 minutes. You can set a smaller interval in seconds with the --queue-purge-interval option or config file setting. If your main concern with TTL is ensuring stale messages don't reach consumers then the default settings are fine. If your main concern is limiting the size of queues in memory you probably want a smaller interval. The periodic cleanup operation currently involves a linear sweep of all queues with TTL messages so there may be some impact on performance during the sweep if you have a lot of deep queues.
Is anyone working on C++ broker TTL ?
I just validated a report from a few of my developers that TTL doesn't seem to be working with the 0.6 broker, is anyone working on this ? If not I can have a look. If it does work I'd like to compare notes to see what we're doing wrong... Thx! (may be QPID-1418, QPID-1575 ?) This has been seen by my Python developer using 0.5 api w/ 0.6 broker, and I just saw in 0.6 trunk C++ using Messaging. In the C++ I'm creating w/ type:topic, calling qpid::messaging::Message.setTTL( TIME_SEC * 2 ), and never see the events go away. Not sure if this is a broker issue or a messaging issue...
Re: Is anyone working on C++ broker TTL ?
OK... (for the C++ code) qpid::messaging.Message::setTTL( boost::uint64_t ttl ); I was using this with a qpid::sys::Duration() object, which stores time in a boost::posix_time::ptime object. This is documented to store in units of nanoseconds. According to /cpp/src/qpid/broker/Message # 351-2, TTL is supposed to be set in milliseconds. When I manually send ms, it works fine. I would recommend: - document Message.setTTL units clearly - pick ms or ns - if ms, then overload setTTL to take a Duration object so we can use without error by converting in setTTL - if ns, then somewhere downstream (OutgoingMessage#351 ?) the ns needs to get converted to ms Kerry On Wed, Feb 10, 2010 at 2:08 PM, Kerry Bonin kerrybo...@gmail.com wrote: In my test case (trunk C++ via messaging) I fire one message w/ 2 sec TTL each time I run, then see how many messages I can read, and they never go away until I kill the broker. I'm not acknowledging anything, as I should have multiple listeners for these events. On Wed, Feb 10, 2010 at 2:01 PM, Carl Trieloff cctriel...@redhat.comwrote: On 02/10/2010 02:54 PM, Kerry Bonin wrote: I just validated a report from a few of my developers that TTL doesn't seem to be working with the 0.6 broker, is anyone working on this ? If not I can have a look. If it does work I'd like to compare notes to see what we're doing wrong... Thx! (may be QPID-1418, QPID-1575 ?) This has been seen by my Python developer using 0.5 api w/ 0.6 broker, and I just saw in 0.6 trunk C++ using Messaging. In the C++ I'm creating w/ type:topic, calling qpid::messaging::Message.setTTL( TIME_SEC * 2 ), and never see the events go away. Not sure if this is a broker issue or a messaging issue... They should be reaped at mgnt interval if no client connected, or when someone trys to consume messages. Are they not being reaped on an idle queue or you getting expired messages at the consumer? Carl. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Management problem with rc5? Re: Vote for 0.6 Release
It didn't work for me until I updated my Python to 2.6. I created QPID-2356 to recommend reporting an unsupported Python version... On Tue, Feb 9, 2010 at 7:41 AM, Gordon Sim g...@redhat.com wrote: On 02/08/2010 10:11 PM, Alan Conway wrote: I built installed the cpp stuff, ran some C++ clients, kicked the tires on the cluster, looks good Then I installed the python tools and... nothing works. qpid-config, qpid-stat, they just sit there till they time out. qpid-tool connects but never gets out of the waiting for periodic update state. With trace logging on I can see the clients do connect and send their initial commands, but then they get stuck waiting I think for management events that don't seem to arrive, although the periodic management log messages are there alright. Can someone try this out and confirm that I'm doing something monumentally stupid and it all really works just fine? It works for me. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Status of Windows SSL / ACL ?
Hello! I had a few questions regarding the state of broker security for Windows platforms. On the SSL front, I see QPID-1403 has been resolved, and I can now build w/ BUILD_SSL enabled. What I haven't been able to get working yet is to get my broker to actually load a certificate - ::CertFindCertificateInStore fails. On this issue, do we have a known good set of instructions to generate a working system? The windows error (Cannot find object or property) is so vague I'm not sure why it isn't loading my cert. I'm generating a self-signed cert w/ openssl, loading as trusted root cert, generating a client cert w/ the CN of my node, converting to pkcs12 along w/ password free private key and importing to the personal store w/ friendly name of node, not sure what else I need to do... On the ACL front, QPID-1842 is still keeping Windows link from working. Any idea when this will be fixed? On a related note, I have time to help fix these, as I need secure broker (with clustering) on the Windows platform working in the next few months for my project here. Appreciate any pointers, and please let me know if there is anything I can do to help... Kerry Bonin Sr. Systems Architect GE Security
[jira] Created: (QPID-2356) Have tools validate python version against required features...
Have tools validate python version against required features... --- Key: QPID-2356 URL: https://issues.apache.org/jira/browse/QPID-2356 Project: Qpid Issue Type: New Feature Components: python tools Affects Versions: 0.6 Reporter: Kerry Bonin Priority: Minor Current python tools do not appear to validate the installed python runtime against what is required. For example, I had Python 2.5 installed and qpid 0.5 tools worked fine. When I updated to the 0.6 rc trunk, at some point I started getting a cryptic error message: Failed: TypeError: - __init__() got an unexpected keyword argument 'host'. My local python expert informed me that some module being loaded likely required a newer python version than I had, and upgrading from 2.5 to 2.6 resolved the error. It would be nice if the tools could report that the local python runtime version was not supported. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
python management tools no longer working (for me) on WinXP
I've searched the bug list and google, can't find this issue. These used to work fine for me w/ 0.5, I'm currently running trunk. Platform is WinXP(x32), Python 2.5.4. Running from command box, PYTHONPATH set, executing from python / commands folder. All the qpid python tools now report: (even something simple like python qpid-config) Failed: TypeError: - __init__() got an unexpected keyword argument 'host' I get this only when the broker is running (connection refused otherwise). I'm setting up another dev box to see if its isolated to something local...
[jira] Created: (QPID-2344) qpid/messaging/uuid.h r899657 breaks Windows build...
qpid/messaging/uuid.h r899657 breaks Windows build... - Key: QPID-2344 URL: https://issues.apache.org/jira/browse/QPID-2344 Project: Qpid Issue Type: Bug Components: C++ Client Affects Versions: 0.6 Environment: Visual Studio 2008 Reporter: Kerry Bonin Priority: Blocker The friend declarations break the VS2008 build - the compiler doesn't seem to automatically apply the Microsoft declspec from the operators with the friend function keyword. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Updated: (QPID-2344) qpid/messaging/uuid.h r899657 breaks Windows build...
[ https://issues.apache.org/jira/browse/QPID-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kerry Bonin updated QPID-2344: -- Attachment: uuid.patch Patch to fix QPID-2344 Adds QPID_CLIENT_EXTERN to friend function operators to fix VS2008 build. qpid/messaging/uuid.h r899657 breaks Windows build... - Key: QPID-2344 URL: https://issues.apache.org/jira/browse/QPID-2344 Project: Qpid Issue Type: Bug Components: C++ Client Affects Versions: 0.6 Environment: Visual Studio 2008 Reporter: Kerry Bonin Priority: Blocker Attachments: uuid.patch Original Estimate: 1h Remaining Estimate: 1h The friend declarations break the VS2008 build - the compiler doesn't seem to automatically apply the Microsoft declspec from the operators with the friend function keyword. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Commented: (QPID-2344) qpid/messaging/uuid.h r899657 breaks Windows build...
[ https://issues.apache.org/jira/browse/QPID-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12802380#action_12802380 ] Kerry Bonin commented on QPID-2344: --- FWIW - The code in the current trunk, at least as of r 900866, doesn't contain the fix. qpid/messaging/uuid.h r899657 breaks Windows build... - Key: QPID-2344 URL: https://issues.apache.org/jira/browse/QPID-2344 Project: Qpid Issue Type: Bug Components: C++ Client Affects Versions: 0.6 Environment: Visual Studio 2008 Reporter: Kerry Bonin Assignee: Steve Huston Priority: Blocker Fix For: 0.7 Attachments: uuid.patch Original Estimate: 1h Remaining Estimate: 1h The friend declarations break the VS2008 build - the compiler doesn't seem to automatically apply the Microsoft declspec from the operators with the friend function keyword. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Errors building trunk?
Thought I'd mention this - I get some errors building a clean trunk on Ubuntu and Windows. On Windows, I'm running VC9 SP1 on XP (fully patched), CMake 2.8.0, boost 1.3.9 against r896143, clean checkout. cpp\src\qmf\engine\ResilientConnection.cpp gives me an error on line 416, 'write not a member of global namespace'. This causes qmfengine build to fail. On Ubuntu, (my first build attempt there), Ubuntu x64, gcc 4.4.1, boost 1.40.0.cmake2, against r895713, clean checkout. cpp/src/qpid/broker/SaslAuthenticator.cpp line 26 gives me error 'type qualifiers ignored on function return type'.
Questions - bug and process...
Hello!
We're playing with the qpid messaging code in the trunk and running into a
few problems.
First, a process question - what is the recommended process for submitting
info like this? Search for existing bug, if not found open one in ASF
JIRA? Related question - is the recommended process captured anywhere?
http://qpid.apache.org/getting-involved.html seemed like the logical place,
but no obvious answer after a brief search...
To the issues:
1) The ReplyTo message property isn't being propagated from sender to
receiver. This is easily notable by trying the examples\messaging
client/server, which fails with a 'Error: no reply address specified
for...'. I noticed this in my own attempt at a request - response pattern,
in which the Address returned from the received message is empty.
2) The nomenclature for specifying the auto-delete node property as part of
an address string isn't working as shown in
qpid\cpp\examples\messaging\client.cpp#49 : Address foo( #name;
{create:always, type:queue, node-properties:{x-ampq0-10-auto-delete:true}}
); The queue is created, but not with an auto-delete attribute set.
FWIW, I'm still learning the underlying qpid code or I'd be posting fixes,
it'll take me a few more days to grok data flow through this code, and I'm
still learning my way around boost. I thought I'd raise these to see if I'm
missing something obvious, and to see if I should be opening bugs for these
sorts of issues...
Thanks!
Kerry
Building C++\Windows trunk?
I'm having some issues getting the trunk to build - most notably namespace collisions between the global namespace types in qpid/sys/windows.IntegerTypes.h and those defined in boost cstdint - first hit is in DispatchHandle.cpp which uses these typenames, and I didn't see any way the compiler could pick one. I don't see others complaining about a broken build, so I'm wondering if I'm missing something? I'm using CMake 2.8.0, Boost 1.35, and MSVC 2008. Appreciate any help on this - I'm recommending we redirect a very large project here at work (GE) over AMQP/QPID (C++, .Net and Python), and I'll be taking point on the integration, so I'd like to be able to start building from the trunk. On a related note, I should be able to start contributing work on the Win side of things, as I need a few features from the *nix broker on the Win32/64 platform... Thanks! Kerry Bonin (posting via gmail instead of kerry.bo...@ge.com as GE mail nukes external mailing list subs...)
