Re: One subscription, multiple consumers.
It's... - ActiveMQ 5.14- Python 3 (running in Docker)- Latest version of Qpid Proton (installed via pip3) J. Jeremy Gooch http://goochgooch.co.uk From: Robbie Gemmell To: "users@qpid.apache.org" Sent: Monday, 10 April 2017, 15:55 Subject: Re: One subscription, multiple consumers. Answers here are likely to be quite dependent on which server, and which version of it, that you are actually using. On 10 April 2017 at 10:06, Jeremy Gooch wrote: > Hello, > Using Qpid Proton (mainly Python), I know you can have multiple consumers > against one queue and they operate in a race condition. > But can the same happen with a subscription? In other words, how do I > configure multiple consumers against one subscription so that messages are > distributed amongst them? This has come up as a requirement to support an > auto-scaling client, who wants to make multiple connections to a durable > subscription but not receive duplicate messages. > Thanks, > J. > > Jeremy Gooch http://goochgooch.co.uk - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
qpid-jms-client writing passwords to the log
When connecting to ActiveMQ via JMS I am using failover and also specifingy the SSL parameters in the connection URI - as specified @ https://qpid.apache.org/releases/qpid-jms-0.21.0/docs/index.html Questions: * Is it possible to disable the logging of the JMS connection URI? * If not, is it possible to mask out the URI's password parameters before logging? * Failing that, is it possible to disable ALL logging performed by the components which identify themselves as 'FailoverProvider' and 'AmqpProvider' - if so, how? The logging level of my client application (NiFi) seems to have absolutely zero effect on this. I am using NiFi to connect to ActiveMQ, using qpid-jms-client 0.11.1 (with javax.jms-api 2.0.1) and to do this I have developed a NiFi processor which uses org.apache.qpid.jms.jndi.JmsInitialContextFactory and javax.jms.ConnectionFactory (specifying the environment via a Hashtable) in accordance with the guidance at https://qpid.apache.org/releases/qpid-jms-0.21.0/docs/index.html Note: I cannot use a later version of qpid-jms-client (because later versions require Java 8, my project is Java 7). Observations: In the NiFi log, regardless of ActiveMQ's logging level or even if it is running, I see lots of log entries, emitted from something called 'AmqpProvider' - they appear there, regardless of NiFi's logging level. When the NiFi user stops/starts the processor, the NiFi processor opens/closes the JMS connection, and this results in: * an *INFO*-level log entry from 'AmqpProvider' reporting the "o.a.qpid.jms.sasl.SaslMechanismFinder Best match for SASL auth was SASL-PLAIN" * an *INFO*-level log entry from 'FailoverProvider' reporting the connection either "connected to remote Broker" or "Connection attempt [...] failed". * a *DEBUG*-level log entry from 'AmqpProvider' reporting that connection has been opened/closed. * a *DEBUG*-level log entry from 'FailoverProvider' reporting the connection is in-progress / being created (Executing Failover task: create) / being destroyed (Executing Failover task: destroyed) There are also periodic entries from 'AmqpProvider' at *DEBUG *level, stating "proton.trace IN: CH[0] : Empty Frame". In ALL of ^these log entries, the JMS connection URI is displayed, in full, showing all the URI parameters, including those for transport.keyStorePassword and transport.trustStorePassword, in plaintext. I don't want that to happen. If the passwords can be starred out, that would be good. Ideally, I'd also like to be able to switch the logging off as its pretty chatty, and I really don't want DEBUG level logging happening unless my client application's logging level is set to DEBUG mode. NiFi doesn't seem to use log4j - I can't see how its logging config file can be modified to set the logging level for 'FailoverProvider' and 'AmqpProvider'. Please help! -- View this message in context: http://qpid.2158936.n2.nabble.com/qpid-jms-client-writing-passwords-to-the-log-tp7662189.html Sent from the Apache Qpid users mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-jms-client writing passwords to the log
As described at http://qpid.apache.org/releases/qpid-jms-0.21.0/docs/index.html#logging, the client uses SLF4J and requires the application provide a binding for it and matching logging implementation then configure it according to its needs. It logs nothing by default unless loaded alongside suitable logging binding and implementation which is configured to output its particular loggers at some level. In this case, something else in your application is providing an SLF4J binding and logging implementation, which is being configured (or not configured) in such a way as to cause output of the loggers used by the client to at least debug level. This may be Log4J, or something else entirely, you will need to investigate your application to find out what and then change that or configure it appropriately. The proton trace entries you mentioned aren't of a format I recogonise, and the client wont emit proton tracing unless explicitly configured, so I'm a bit confused by that point. We will take a look in terms of improving the rest when loggers are enabled. Robbie On 11 April 2017 at 14:12, Vince Cole wrote: > When connecting to ActiveMQ via JMS I am using failover and also specifingy > the SSL parameters in the connection URI - as specified @ > https://qpid.apache.org/releases/qpid-jms-0.21.0/docs/index.html > > Questions: > * Is it possible to disable the logging of the JMS connection URI? > * If not, is it possible to mask out the URI's password parameters before > logging? > * Failing that, is it possible to disable ALL logging performed by the > components which identify themselves as 'FailoverProvider' and > 'AmqpProvider' - if so, how? The logging level of my client application > (NiFi) seems to have absolutely zero effect on this. > > I am using NiFi to connect to ActiveMQ, using qpid-jms-client 0.11.1 (with > javax.jms-api 2.0.1) and to do this I have developed a NiFi processor which > uses org.apache.qpid.jms.jndi.JmsInitialContextFactory and > javax.jms.ConnectionFactory (specifying the environment via a Hashtable) in > accordance with the guidance at > https://qpid.apache.org/releases/qpid-jms-0.21.0/docs/index.html > > Note: I cannot use a later version of qpid-jms-client (because later > versions require Java 8, my project is Java 7). > > Observations: > > In the NiFi log, regardless of ActiveMQ's logging level or even if it is > running, I see lots of log entries, emitted from something called > 'AmqpProvider' - they appear there, regardless of NiFi's logging level. > > When the NiFi user stops/starts the processor, the NiFi processor > opens/closes the JMS connection, and this results in: > * an *INFO*-level log entry from 'AmqpProvider' reporting the > "o.a.qpid.jms.sasl.SaslMechanismFinder Best match for SASL auth was > SASL-PLAIN" > * an *INFO*-level log entry from 'FailoverProvider' reporting the connection > either "connected to remote Broker" or "Connection attempt [...] failed". > * a *DEBUG*-level log entry from 'AmqpProvider' reporting that connection > has been opened/closed. > * a *DEBUG*-level log entry from 'FailoverProvider' reporting the connection > is in-progress / being created (Executing Failover task: create) / being > destroyed (Executing Failover task: destroyed) > > There are also periodic entries from 'AmqpProvider' at *DEBUG *level, > stating "proton.trace IN: CH[0] : Empty Frame". > > In ALL of ^these log entries, the JMS connection URI is displayed, in full, > showing all the URI parameters, including those for > transport.keyStorePassword and transport.trustStorePassword, in plaintext. > > I don't want that to happen. If the passwords can be starred out, that would > be good. > > Ideally, I'd also like to be able to switch the logging off as its pretty > chatty, and I really don't want DEBUG level logging happening unless my > client application's logging level is set to DEBUG mode. > > NiFi doesn't seem to use log4j - I can't see how its logging config file can > be modified to set the logging level for 'FailoverProvider' and > 'AmqpProvider'. > > Please help! > > > > -- > View this message in context: > http://qpid.2158936.n2.nabble.com/qpid-jms-client-writing-passwords-to-the-log-tp7662189.html > Sent from the Apache Qpid users mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: One subscription, multiple consumers.
With ActiveMQ 5 I think your only option would be Virtual Topics, http://activemq.apache.org/virtual-destinations.html On 11 April 2017 at 13:17, Jeremy Gooch wrote: > It's... > - ActiveMQ 5.14- Python 3 (running in Docker)- Latest version of Qpid Proton > (installed via pip3) > J. > Jeremy Gooch http://goochgooch.co.uk > > From: Robbie Gemmell > To: "users@qpid.apache.org" > Sent: Monday, 10 April 2017, 15:55 > Subject: Re: One subscription, multiple consumers. > > Answers here are likely to be quite dependent on which server, and > which version of it, that you are actually using. > > On 10 April 2017 at 10:06, Jeremy Gooch wrote: >> Hello, >> Using Qpid Proton (mainly Python), I know you can have multiple consumers >> against one queue and they operate in a race condition. >> But can the same happen with a subscription? In other words, how do I >> configure multiple consumers against one subscription so that messages are >> distributed amongst them? This has come up as a requirement to support an >> auto-scaling client, who wants to make multiple connections to a durable >> subscription but not receive duplicate messages. >> Thanks, >> J. >> >> Jeremy Gooch http://goochgooch.co.uk > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > > > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-jms-client writing passwords to the log
OK thanks, will ask the NiFi guys -- View this message in context: http://qpid.2158936.n2.nabble.com/qpid-jms-client-writing-passwords-to-the-log-tp7662189p7662193.html Sent from the Apache Qpid users mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [VOTE] Release Qpid Dispatch Router 0.8.0 (RC2)
-1 I hate to do this to my own vote, but Chuck Rolke has found an issue that is a blocker: https://issues.apache.org/jira/browse/DISPATCH-742 Please stand by for RC3 with a fix for this issue. -Ted On 04/07/2017 10:34 AM, Ted Ross wrote: Please cast your vote on this thread for releasing RC2 as the official 0.8.0. Thanks, -Ted - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org -- Ted Ross Senior Principal Software Engineer Red Hat 314 Littleton Road Westford, MA 01886 tr...@redhat.com T: +1-978-392-3950 M: +1-978-399-4122 - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-jms-client writing passwords to the log
I'm afraid I don't even know really know what NiFi is, but essentially it should be pretty obvious from your classpath what is happening, and it may or may not involve NiFi directly. In the past, SLF4J would complain if there was more than one logging binding available, so its possible there is only one, particularly if you dont know you are supplying it. You'll then need to configure it appropriately to your (/its) needs, or perhaps even change to an entirely different logging implemetnation of your choice. Log4J historically did default to debug level if not configured, so if that is being supplied elsewhere then it might be your issue, you've introduced another element and might need to supply appropriate configuration for it (/update the existing configuration if there is some). On 11 April 2017 at 14:49, Vince Cole wrote: > OK thanks, will ask the NiFi guys > > > > -- > View this message in context: > http://qpid.2158936.n2.nabble.com/qpid-jms-client-writing-passwords-to-the-log-tp7662189p7662193.html > Sent from the Apache Qpid users mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [VOTE] Release Qpid Dispatch Router 0.8.0 (RC2)
Can you send a "[CANCELLED] [VOTE] Release Qpid Dispatch Router 0.8.0 (RC2) " mail, just for the record. Robbie On 11 April 2017 at 14:51, Ted Ross wrote: > -1 > > I hate to do this to my own vote, but Chuck Rolke has found an issue that is > a blocker: > > https://issues.apache.org/jira/browse/DISPATCH-742 > > Please stand by for RC3 with a fix for this issue. > > -Ted > > > On 04/07/2017 10:34 AM, Ted Ross wrote: >> >> Please cast your vote on this thread for releasing RC2 as the official >> 0.8.0. >> >> Thanks, >> >> -Ted >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >> For additional commands, e-mail: users-h...@qpid.apache.org >> > > -- > Ted Ross > Senior Principal Software Engineer > Red Hat > 314 Littleton Road > Westford, MA 01886 > tr...@redhat.com T: +1-978-392-3950 M: +1-978-399-4122 > > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
[CANCELLED] [VOTE] Release Qpid Dispatch Router 0.8.0 (RC2)
This vote thread is cancelled pending a new release candidate. -Ted On 04/11/2017 09:51 AM, Ted Ross wrote: -1 I hate to do this to my own vote, but Chuck Rolke has found an issue that is a blocker: https://issues.apache.org/jira/browse/DISPATCH-742 Please stand by for RC3 with a fix for this issue. -Ted On 04/07/2017 10:34 AM, Ted Ross wrote: Please cast your vote on this thread for releasing RC2 as the official 0.8.0. Thanks, -Ted - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org -- Ted Ross Senior Principal Software Engineer Red Hat 314 Littleton Road Westford, MA 01886 tr...@redhat.com T: +1-978-392-3950 M: +1-978-399-4122 - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: One subscription, multiple consumers.
If I understand this correctly, the limitation with regard to only a single consumer on one topic is in order to maintain compliance with the JMS specification. ActiveMQ's Virtual Destinations give a workaround for this where a producer will publish to a topic as normal whilst allowing a consumer to connect as if to a queue, thereby enabling the consumer to operate multiple connections to increase its message consumption rate. The AMQP model has the concept of sessions within a connection. Is it possible therefore to have multiple sessions within a connection against a single subscription without using Virtual Destinations? We could then use a client with some sort of concurrency feature to operate multiple sessions and increase message consumption. J. Jeremy Gooch http://goochgooch.co.uk From: Robbie Gemmell To: "users@qpid.apache.org" Sent: Tuesday, 11 April 2017, 14:49 Subject: Re: One subscription, multiple consumers. With ActiveMQ 5 I think your only option would be Virtual Topics, http://activemq.apache.org/virtual-destinations.html On 11 April 2017 at 13:17, Jeremy Gooch wrote: > It's... > - ActiveMQ 5.14- Python 3 (running in Docker)- Latest version of Qpid Proton > (installed via pip3) > J. > Jeremy Gooch http://goochgooch.co.uk > > From: Robbie Gemmell > To: "users@qpid.apache.org" > Sent: Monday, 10 April 2017, 15:55 > Subject: Re: One subscription, multiple consumers. > > Answers here are likely to be quite dependent on which server, and > which version of it, that you are actually using. > > On 10 April 2017 at 10:06, Jeremy Gooch wrote: >> Hello, >> Using Qpid Proton (mainly Python), I know you can have multiple consumers >> against one queue and they operate in a race condition. >> But can the same happen with a subscription? In other words, how do I >> configure multiple consumers against one subscription so that messages are >> distributed amongst them? This has come up as a requirement to support an >> auto-scaling client, who wants to make multiple connections to a durable >> subscription but not receive duplicate messages. >> Thanks, >> J. >> >> Jeremy Gooch http://goochgooch.co.uk > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > > > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: One subscription, multiple consumers.
For the most part topic subscriptions only ever had a single subscriber asociated with them, until at some point someone thought it would be useful to support more. Some systems can do that, others cant, others still have specific mechanisms to achieve the same effect. This is all completely independent of the concepts of sessions and connections. You could have a single session with lots of subscribers on it, or lots of sessions with a single subscriber, but that doesnt particularly influence whether they can share a single subscription or not, the messaging system itself does. Various messaging systems now support doing that in their own way, hence my original question what you were using. In ActiveMQ 5.x its abilities are closely related to JMS 1.1, and I beleive its only method of achieving the shared subscribers effect is Virtual Destinations, effectively turning each subscriber for a given subscription into a specific queue consumer as you note. We recently mapped out a mechanism to support shared subscriptions over AMQP 1.0 such that the JMS 2.0 shared subscriptions functionlity could be supported for the Qpid JMS client. I'm only aware of 2 brokers that already implement support for the precise mechanism it uses, ActiveMQ Artemis (I think 2.x is needed) and the upcoming Qpid Broker-J 7.0.0. The Qpid C++ broker also supports a variant of shared susbcriptions over AMQP 1.0, but does not currently support the precise mechanism required by the JMS 2.0 client. You could similarly use those mechanisms from other AMQP 1.0 clients, or alternatively there may be other broker-specific mechanisms for achieving the same effect, but as far as ActiveMQ 5.x is concerned, I beleive your only option is Virtual Topics. Robbie On 11 April 2017 at 16:57, Jeremy Gooch wrote: > If I understand this correctly, the limitation with regard to only a single > consumer on one topic is in order to maintain compliance with the JMS > specification. ActiveMQ's Virtual Destinations give a workaround for this > where a producer will publish to a topic as normal whilst allowing a consumer > to connect as if to a queue, thereby enabling the consumer to operate > multiple connections to increase its message consumption rate. > > The AMQP model has the concept of sessions within a connection. Is it > possible therefore to have multiple sessions within a connection against a > single subscription without using Virtual Destinations? We could then use a > client with some sort of concurrency feature to operate multiple sessions and > increase message consumption. > J. > > Jeremy Gooch http://goochgooch.co.uk > > From: Robbie Gemmell > To: "users@qpid.apache.org" > Sent: Tuesday, 11 April 2017, 14:49 > Subject: Re: One subscription, multiple consumers. > > With ActiveMQ 5 I think your only option would be Virtual Topics, > http://activemq.apache.org/virtual-destinations.html > > On 11 April 2017 at 13:17, Jeremy Gooch wrote: >> It's... >> - ActiveMQ 5.14- Python 3 (running in Docker)- Latest version of Qpid Proton >> (installed via pip3) >> J. >> Jeremy Gooch http://goochgooch.co.uk >> >> From: Robbie Gemmell >> To: "users@qpid.apache.org" >> Sent: Monday, 10 April 2017, 15:55 >> Subject: Re: One subscription, multiple consumers. >> >> Answers here are likely to be quite dependent on which server, and >> which version of it, that you are actually using. >> >> On 10 April 2017 at 10:06, Jeremy Gooch wrote: >>> Hello, >>> Using Qpid Proton (mainly Python), I know you can have multiple consumers >>> against one queue and they operate in a race condition. >>> But can the same happen with a subscription? In other words, how do I >>> configure multiple consumers against one subscription so that messages are >>> distributed amongst them? This has come up as a requirement to support an >>> auto-scaling client, who wants to make multiple connections to a durable >>> subscription but not receive duplicate messages. >>> Thanks, >>> J. >>> >>> Jeremy Gooch http://goochgooch.co.uk >> >> - >> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >> For additional commands, e-mail: users-h...@qpid.apache.org >> >> >> > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > > > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: One subscription, multiple consumers.
On 04/11/2017 12:42 PM, Robbie Gemmell wrote: For the most part topic subscriptions only ever had a single subscriber asociated with them, until at some point someone thought it would be useful to support more. Some systems can do that, others cant, others still have specific mechanisms to achieve the same effect. This is all completely independent of the concepts of sessions and connections. You could have a single session with lots of subscribers on it, or lots of sessions with a single subscriber, but that doesnt particularly influence whether they can share a single subscription or not, the messaging system itself does. Various messaging systems now support doing that in their own way, hence my original question what you were using. In ActiveMQ 5.x its abilities are closely related to JMS 1.1, and I beleive its only method of achieving the shared subscribers effect is Virtual Destinations, effectively turning each subscriber for a given subscription into a specific queue consumer as you note. We recently mapped out a mechanism to support shared subscriptions over AMQP 1.0 such that the JMS 2.0 shared subscriptions functionlity could be supported for the Qpid JMS client. I'm only aware of 2 brokers that already implement support for the precise mechanism it uses, ActiveMQ Artemis (I think 2.x is needed) and the upcoming Qpid Broker-J 7.0.0. The Qpid C++ broker also supports a variant of shared susbcriptions over AMQP 1.0, but does not currently support the precise mechanism required by the JMS 2.0 client. You could similarly use those mechanisms from other AMQP 1.0 clients, or alternatively there may be other broker-specific mechanisms for achieving the same effect, but as far as ActiveMQ 5.x is concerned, I beleive your only option is Virtual Topics. That is correct, you'd need to use Virtual Topic subscriptions for this scenario. Robbie On 11 April 2017 at 16:57, Jeremy Gooch wrote: If I understand this correctly, the limitation with regard to only a single consumer on one topic is in order to maintain compliance with the JMS specification. ActiveMQ's Virtual Destinations give a workaround for this where a producer will publish to a topic as normal whilst allowing a consumer to connect as if to a queue, thereby enabling the consumer to operate multiple connections to increase its message consumption rate. The AMQP model has the concept of sessions within a connection. Is it possible therefore to have multiple sessions within a connection against a single subscription without using Virtual Destinations? We could then use a client with some sort of concurrency feature to operate multiple sessions and increase message consumption. J. Jeremy Gooch http://goochgooch.co.uk From: Robbie Gemmell To: "users@qpid.apache.org" Sent: Tuesday, 11 April 2017, 14:49 Subject: Re: One subscription, multiple consumers. With ActiveMQ 5 I think your only option would be Virtual Topics, http://activemq.apache.org/virtual-destinations.html On 11 April 2017 at 13:17, Jeremy Gooch wrote: It's... - ActiveMQ 5.14- Python 3 (running in Docker)- Latest version of Qpid Proton (installed via pip3) J. Jeremy Gooch http://goochgooch.co.uk From: Robbie Gemmell To: "users@qpid.apache.org" Sent: Monday, 10 April 2017, 15:55 Subject: Re: One subscription, multiple consumers. Answers here are likely to be quite dependent on which server, and which version of it, that you are actually using. On 10 April 2017 at 10:06, Jeremy Gooch wrote: Hello, Using Qpid Proton (mainly Python), I know you can have multiple consumers against one queue and they operate in a race condition. But can the same happen with a subscription? In other words, how do I configure multiple consumers against one subscription so that messages are distributed amongst them? This has come up as a requirement to support an auto-scaling client, who wants to make multiple connections to a durable subscription but not receive duplicate messages. Thanks, J. Jeremy Gooch http://goochgooch.co.uk - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org -- Tim Bish twitter: @tabish121 blog: http://timbish.blogspot.com/ - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
