Re: hundreds of threads trying to connect to vm "localhost"?

be specifying "?brokerName=" where is the actual broker name? Jim On Mon, Jan 16, 2017 at 11:23 AM James A. Robinson <jim.robin...@gmail.com> wrote: > Several times now we've witnessed a cluster spinning off hundreds of > threads and spewing warnings like: >

hundreds of threads trying to connect to vm "localhost"?

Several times now we've witnessed a cluster spinning off hundreds of threads and spewing warnings like: 2017-01-16 11:10:35,376 [p209148636-3304] WARN BrokerRegistry - Broker localhost not started so using amq-prod-1 instead A thread dump shows hundreds of threads along these lines: "ActiveMQ

Re: A rookie need help

I don't know where you got that link from, it looks like an example from documentation, where you're expected to fill in some details (the path to the version you want). Look at the links for Windows or Linux on this page: http://activemq.apache.org/activemq-591-release.html Those link to a

Re: stomp EOF Exceptions

A packet capture on the machines indicates a perfectly valid sequence of exchanges between the JMS publisher and the consumer with the broker when the test is succeeding. I'll run another test when it is in a failure mode to see what is changing. Jim

Re: stomp EOF Exceptions

#issuecomment-157861049 On Thu, Apr 21, 2016 at 9:18 AM James A. Robinson <jim.robin...@gmail.com> wrote: > > Interesting: > > https://github.com/chirino/stomp-benchmark/issues/11 > > > On Wed, Apr 20, 2016 at 10:54 AM James A. Robinson <jim.robin...@gmail.com> >

Re: stomp EOF Exceptions

Interesting: https://github.com/chirino/stomp-benchmark/issues/11 On Wed, Apr 20, 2016 at 10:54 AM James A. Robinson <jim.robin...@gmail.com> wrote: > I can't immediately tell if this is a problem with the stomp client or > with activemq, but I suspect the latter given what

stomp EOF Exceptions

I can't immediately tell if this is a problem with the stomp client or with activemq, but I suspect the latter given what I was seeing. A monitor I put together to test that messages could be passed on the transports configured for the broker started failing this morning, throwing EOF Exceptions

confusion re DataDirectory MBean Attribute

Folks, In my /etc/activemq/activemq.xml I've got this defined: and that's where the data files are written: $ ls /var/log/activemq/activemq-data/amq-prod-1 db-342.log db-343.log db-344.log db-345.log db-380.log db-381.log db.data db.redo lock scheduler So why does the attribute

Re: Stranded - ActiveMQ constantly Crashing

Hi, Are there any queues where there is no movement vs. ones where there is a steady shift in enqueued/dequeued counts? Or any topic where there are one or more consumers and any pending messages to be dequeued? I wouldn't typically expect a memory limit on a queue to cause a consumer to not be

Re: Stranded - ActiveMQ constantly Crashing

​Hi, ​ Looking for a queue or topic that had a large number of unconsumed messages ​ ​ would probably be a good star ​​ t. ​ If your ​server has the webconsole activated you can look at the queues and topics, ​ ​ for example if your server had a webconsole on port 8161:

Re: Storage Leak - How to analyse

You might see if you can get debugging output similar to what KahaDB offers: http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html I had a similar situation and the KahaDB debugging showed that the problem was due to un-acked persistent messages sprinkled throughout the

Testing a network of brokers

Hi, I've written a client to exercise the network of brokers in my setup, the intent is to use this to help monitor that all members of the network can pull messages from each other. Currently it's set up to generate N tests, one for each possible combination of a broker producer and broker

Re: Monitoring ActiveMQ

Through the JMX service there are an extensive number of mbeans available to determine the state of the broker and its queues and topics: http://activemq.apache.org/jmx.html It possible to access the JMX MBeans over HTTP, and of course you can use tools that make direct use of the JMX service.

Re: ActiveMQ Object Message to json transformation not working

JAR causes this (though I haven't made any > effort to trace the code); am I missing something? > > Tim > > On Thu, Mar 31, 2016 at 1:08 AM, James A. Robinson <j...@highwire.org> > wrote: > > > I think what it boils down to is figuring out where you need to put

Re: Activemq HA without shared Database or Shared file system

Some people are using it. I wanted to, but decided I wasn't comfortable relying on it. On Thu, Mar 31, 2016 at 19:15 Natarajan, Rajeswari < rajeswari.natara...@sap.com> wrote: > Is replicated level DB store production ready now. > > > -Original Message- > Fr

Re: Activemq HA without shared Database or Shared file system

I'm not aware of any other choice. I initially tried to use the replicated leveldb system but ran into too many stability issues. On Thu, Mar 31, 2016 at 3:17 PM Natarajan, Rajeswari < rajeswari.natara...@sap.com> wrote: > Hi, > > Would like to know if ActiveMQ supports HA with message

Re: ActiveMQ Object Message to json transformation not working

I think what it boils down to is figuring out where you need to put the jar to make it available to the class loader in the java instance that is running your broker. So, for example, on my Linux setup I have a directory /usr/share/activemq/lib that contains the jars needed to run ActiveMQ and

Re: NFS v4 locks "given up" w/o any logging?

was seeing. Jim On Wed, Mar 23, 2016 at 7:21 AM James A. Robinson <j...@highwire.org> wrote: > Thank you for the information. Would you be able to tell me if your > NetApp settings for the lock lease are the standard values? I'm told that > it's 30 seconds, I was thinkin

Re: NFS v4 locks "given up" w/o any logging?

Thank you for the information. Would you be able to tell me if your NetApp settings for the lock lease are the standard values? I'm told that it's 30 seconds, I was thinking about whether or not I needed to have that raised much higher to prevent the particular issue I'm seeing (the tcp layer

Re: NFS v4 locks "given up" w/o any logging?

To answer my own question, no (at least not NFS 4): https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-nfs.html On Mon, Mar 21, 2016 at 3:46 PM James A. Robinson <jim.robin...@gmail.com> wrote: > I suppose I should pose the

Re: NFS v4 locks "given up" w/o any logging?

I suppose I should pose the question to folks that use NFS, do you use TCP or UDP for your NFS stack? On Mon, Mar 21, 2016 at 3:29 PM James A. Robinson <jim.robin...@gmail.com> wrote: > Running a packet trace while the problem was occurring let me see that the > problem is that t

Re: NFS v4 locks "given up" w/o any logging?

Running a packet trace while the problem was occurring let me see that the problem is that the NFS layer isn't even being involved for 3 minutes due to the TCP layer retrying for 3 minute before notifying the caller of the error. The client sends SYN packets, doesn't get an ACK, and finally times

Re: NFS v4 locks "given up" w/o any logging?

, or I could raise the server side lease time to match the 180 seconds the client will try for. On Fri, Mar 18, 2016 at 8:26 AM, James A. Robinson <j...@highwire.org> wrote: > Yes, the combination of settings in place right now could add up to 3 > minutes: > > On the client s

NFS v4 locks "given up" w/o any logging?

Is it common that an activemq broker might give up its NFS v4 lock w/o logging any sort of message? I've got two brokers that logged this: broker-a which held the log: 2016-03-17 15:01:51,113 [yMonitor Worker] WARN Transport - Transport Connection to: tcp://104.232.16.4:62269 failed:

Re: NFS v4 locks "given up" w/o any logging?

Yes, the combination of settings in place right now could add up to 3 minutes: On the client side: nfsvers=4,proto=tcp, hard,timeo=600,retrans=2,ac,acregmin=3,acregmax=60,acdirmin=30,fg,retry=120, sharecache,lookupcache=all,cto So right now it's got a 60 second timeo value, and it will retry up

Re: NFS v4 locks "given up" w/o any logging?

ped them detect lost locks. Anyone who has an HA configuration using NFS that they know works for failover care to share exactly what mount settings they are using? On Fri, Mar 18, 2016 at 8:51 AM, James A. Robinson <jim.robin...@gmail.com> wrote: > Yes indeed there was a problem w/

Re: ActiveMQ-CPP Exception connecting to broker

Based on your numbers it's probably not ephemeral port exhaustion. The default TIME_WAIT is 4 minutes on Solaris, and to use up 32,767 sockets within 4 minutes, you'd need you to be sending an average of 136 messages per second. Jim

Re: ActiveMQ-CPP Exception connecting to broker

; wrote: > On Thu, 10 Mar 2016 16:13:32 +, "James A. Robinson" > <j...@highwire.org> wrote: > > It happens infrequently and unpredictibly. For example sometimes it > fails after 25 times and sometimes it works for several 100 before > going wrong. >

Re: ActiveMQ-CPP Exception connecting to broker

You don't say how frequently it performs this action? If it is frequent, and I were debugging it, the first thing I would check for would be that you haven't run out of socket filehandles. On a unix system running something like "netstat -nt" can tell you what state the socket filehandles are

Re: question for users of NFS master/slave setups

Thank you for the input. The way I originally set up the Replicated LevelDB servers was to create 2 sets of 3 hosts: 1a 1b 1c 2a 2b 2c So within the "1" and "2" clusters each had one master and two slaves, and then I networked the "1" and "2" clusters together, My intent was to create highly

Re: question for users of NFS master/slave setups

On Tue, Mar 1, 2016 at 7:41 AM, Tim Bain wrote: > Another possibility: the paths that each broker uses to reach the lock file > don't resolve to the same file in NFS. > In my case they resolve to the same server IP and export path.​

Re: question for users of NFS master/slave setups

On Tue, Mar 1, 2016 at 7:02 AM, artnaseef wrote: > So something is very wrong then. NFS should *not* allow two NFS clients to > obtain the same lock. > > Three possible explanations come to mind: > > * The lock file is getting incorrectly removed (I've never seen ActiveMQ >

Re: question for users of NFS master/slave setups

On Mon, Feb 29, 2016 at 7:08 PM, artnaseef wrote: > Something sounds very wrong there. The NFS lock file should prevent more > than one broker writing to the store at a time. > > Is all of /var/log/activemq/activemq-data/ shared across all of the > brokers? > ​Hi,

Re: question for users of NFS master/slave setups

​This is interesting. When i use a purposefully slow set of 3 brokers sharing an NFS 4 mount, I found that it's very very easy to get them into a bad state. As simple a procedure as starting them in sequence and then restarting them in the same sequence nets me errors like: 2016-02-29

Re: question for users of NFS master/slave setups

Hi, Thank you for the reply. Do you happen to know what the mount settings actually are in your setup? Yes, we are using nfs4 for this. Jim On Thu, Feb 25, 2016 at 20:21 artnaseef wrote: > I've used it successfully more than once without any specific tuning to > NFS. >

question for users of NFS master/slave setups

Hi folks, Would any of you folks here with an ActiveMQ NFS master/slave setup that you think has been very reliable be willing to share your NFS client/server settings? Right now I'm playing around wth testing some of the suggestions from http://www.billharlan.com/papers/NFS_for_clusters.html

Question re Jetty and ActiveMQ 5.13.0 async support

I'm working on upgrading from activemq 5.11.0 to 5.13.0. When I make an HTTP GET request against the older version of the service, when the queue doesn't have any pending messages, the request hangs until one is ready. On the newer version it returns a fatal status 500 error and throws an

Re: Question re Jetty and ActiveMQ 5.13.0 async support

Never mind, it appears the problem was due to how the install was run, there was another section that was overriding the web.xml using an older version, not the one distributed with 5.13.0. On Wed, Feb 10, 2016 at 12:47 PM James A. Robinson <jim.robin...@gmail.com> wrote: > I'

KahaDB vs LevelDB

Is KahaDB considered the more robust backing store of the two options? We just ran into a variation of https://issues.apache.org/jira/browse/AMQ-5459 and I couldn't see any way to recover it. Jim

Re: ActiveMQ deployment

So when I was building my system I had wanted to use M/S, but the documentation had indicated the old M/S was deprecated in favor of the newer replicated LevelDB store. There are some stability issues with replicated LevelDB (w/ the code handling the zookeeper connection). Do you use an older

Re: AMQ pauses sending to consumers

Have you configured your machine to allow for JMX connections? You might see if you can connect via something like jconsole and see what the state of memory and cpu is, the active threads, etc. On Thu, Aug 20, 2015 at 8:49 AM Richard Sinek rsi...@intouchgps.com wrote: The producer continues to

Re: zookeeper reconnects

the recommendation we come to. But I don't want to say on the official documentation that it has to be = if = would work fine. Tim On Fri, Jul 24, 2015 at 10:39 AM, James A. Robinson j...@highwire.org wrote: So about 40 hours after fixing the settings so that both activemq and zookeeper

Re: zookeeper reconnects

connections. If I'm right, then this indicates the instability I saw can be masked over by making sure the two agree on the session timeout, but that the fundamental fragility of the activemq zookeeper client code is still a potential risk. Jim On Wed, Jul 22, 2015 at 6:27 PM James A. Robinson j

Re: Monitoring?

Hi, You're talking about http://rhq-project.github.io/rhq/? On Fri, Jul 24, 2015 at 11:42 AM barry.barn...@wellsfargo.com wrote: RHQ is a great tool to use to monitor brokers and queues/topics.

Re: Monitoring?

that much stuff or not retrieve values super often. As with everything related to performance, YMMV so plan to measure how much throughput you can actually do, but don't expect to get all stats for all destinations and clients every 100ms. On Jul 23, 2015 1:52 PM, James A. Robinson j...@highwire.org

Re: Monitoring?

at 11:39 AM, James A. Robinson j...@highwire.org wrote: What tools do you folks find to be effective for monitoring the health of your brokers? I see several nagios plugins do things like check queue depth. I also note that most of the plugins I see have no concept of monitoring

Re: ActiveMQ replicated leveldb with 2 brokers(replica = 1)

My understanding has been that the election algorithm requires [(n/2)+1] (where n is the configured size of the cluster) nodes to reach consensus on who the master is. So [(2/2)+1 = 2] nodes, meaning you have no way to lose a node and still run. Unless I'm mistaken, that's typically why you see

Monitoring?

What tools do you folks find to be effective for monitoring the health of your brokers? I see several nagios plugins do things like check queue depth. I also note that most of the plugins I see have no concept of monitoring a failover capable cluster. Jim

zookeeper reconnects

Anyone using replicated leveldb and seeing a somewhat regular set of errors about lost connections to zookeeper? 2015-07-20 09:26:33,568 [hWire.org:2181)] INFO ClientCnxn - Opening socket connection to server zk2.mydomain.org/xxx.xxx.xxx.xxx:2181. Will not attempt to authenticate using SASL

Re: zookeeper reconnects

Hrm... I'm wondering if this is due to the zookeeper server having a default session timeout of 40 seconds vs. a lower one I set for the activemq node...

Re: ActiveMQ master-slave topology issue[BUG]

-write that you asked about, so I think it's worth doing. Tim On Fri, Jun 19, 2015 at 10:19 AM, James A. Robinson j...@highwire.org wrote: On Mon, Jun 15, 2015 at 7:08 AM Tim Bain tb...@alumni.duke.edu wrote: It seems pretty clear that the assumption that acquiring a single file lock

Re: ActiveMQ master-slave topology issue[BUG]

On Mon, Jun 15, 2015 at 7:08 AM Tim Bain tb...@alumni.duke.edu wrote: It seems pretty clear that the assumption that acquiring a single file lock without doing any further checks will provide thread-safety in all cases is not an accurate one. As I see it, here are the failures of the

Re: ActiveMQ master-slave topology issue[BUG]

All the issues you've seen with NFS matches up well with everything I've seen dealing with NFS storage over the past 18 years. I had always thought maybe we're just doing it wrong, but I think everything you outlined looks familiar to me. When we decided to try using ActiveMQ a few months ago

Re: AMQ-5082 (replicated LevelDB)

I've had a dev cluster running for a little while now and twice I've seen interruptions where the cluster didn't recover, didn't select a new master. I had hoped AMQ-5082 fixed that issue but it looks like there might be additional problems. How many of you folks are running replicated leveldb,

Re: jetty9

There should be a few lines below the one you pasted that indicate the actual missing symbol, e.g., \nsymbol: class WebSocket\nlocation: package org.eclipse.jetty.websocket. Are there those details in your build error? Jim On Wed, May 13, 2015 at 5:29 AM lukas83 tre...@arch.ethz.ch wrote:

Re: jetty9

Looking at https://issues.apache.org/jira/browse/AMQ-5517 I would guess there are some additional dependencies missing from the pom.xml, but I'm afraid I don't know what those are. On Wed, May 13, 2015 at 5:43 AM James A. Robinson j...@highwire.org wrote: There should be a few lines below

Re: Network of Brokers recommendations

all of the other clusters so you'd use a hub-and-spoke topology so you only have to specify the hub node in your config. On Fri, Mar 13, 2015 at 12:21 AM, James A. Robinson jim.robin...@gmail.com wrote: Hi folks, I joined this list because we're starting to look into setting up

Re: Creating a Network of Replicated LevelDB broker clusters

On Tue, Mar 31, 2015 at 12:08 PM, wonderkind kevin...@am.sony.com wrote: Is your broker running pretty clean when you send messages through the fabric of network of replicated master/slaves? Yes, though I'm only running the examples/openwire/swissarmy producer/consumer example using the

Re: Creating a Network of Replicated LevelDB broker clusters

On Tue, Mar 31, 2015 at 1:32 PM, wonderkind kevin...@am.sony.com wrote: What version of ActiveMQ are you running with? I am working with 5.11.1.. I'm running 5.11.1 plus the patch in ticket AMQ-5082. Jim

Re: Creating a Network of Replicated LevelDB broker clusters

On Mon, Mar 30, 2015 at 2:09 PM, wonderkind kevin...@am.sony.com wrote: How big is your network? Do you see any significant degradation with a network of replicated levelDB brokers? I haven't started stress testing the configuration yet, the current plan is to have network two clusters of

Re: Creating a Network of Replicated LevelDB broker clusters

On Mon, Mar 30, 2015 at 4:57 PM, wonderkind kevin...@am.sony.com wrote: Do you have a sample activemq.xml configuration file you could share for your 3 x 3 cluster? I am trying to do the same, with everything running on my Mac initially. On the amq-prod-1 cluster my network connector url is

Re: Creating a Network of Replicated LevelDB broker clusters

On Fri, Mar 27, 2015 at 11:48 AM, wonderkind kevin...@am.sony.com wrote: Has anyone ever created a network of brokers, with each broker been a Master/Slave replicated LevelDB store? My understanding has been that, for replicated LevelDB, you need a set of three brokers, one master two slave,

Re: Multiple Producers not scaling well

What OS are you running this under? Are you seeing a lot of threads in your JVM? Are you seeing a lot of GC activity? On Mon, Mar 23, 2015 at 9:43 AM, phill.tomlinson philltomlin...@fico.com wrote: Thanks Tim. I tried without message persistence and although I did get higher throughput it

Re: Highly available and scalable criss crossed brokers architectural advice

On Wed, Mar 18, 2015 at 1:10 AM, Fred Moore fred.moor...@gmail.com wrote: Questions: Q1\ Is this approach a viable one? Q2\ Are there better ways of obtaining a similar QoS (better in the sense of ...more simple and/or more robust and/or more standard) Q3\ Are there any specific

Re: ActiveMQ 5.11 Dynamic Camel Route - Message Namespaces

I suspect it's because the question looks related more to Apache Camel than to ActiveMQ and people here might not be using the former. Are you using the same Camel library on both revisions of the ActiveMQ broker? if you set xpath logNamespaces=true

Re: ActiveMQ 5.11 Dynamic Camel Route - Message Namespaces

On Wed, Mar 18, 2015 at 8:17 AM, mtod m...@thetods.net wrote: I tried the XPath attributes you recommended but no luck I get the same error. Maybe I should have used a different subject line :) So for the 1st recommendation I had: xpath logNamespaces=true

Re: ActiveMQ 5.11 Dynamic Camel Route - Message Namespaces

On Wed, Mar 18, 2015 at 10:49 AM, mtod m...@thetods.net wrote: INFO | Created default XPathFactory com.sun.org.apache.xpath.internal.jaxp.XPathFactoryImpl@343985bd INFO | Creating new XPath expression in pool. Namespaces on XPath expression: [me: {system -

Network of Brokers recommendations

Hi folks, I joined this list because we're starting to look into setting up ActiveMQ for our developers. My preference at this time is to set up a highly reliable system that doesn't have single points of failure. Due to the concern about single points of failure I didn't want to introduce a

Re: Data Migration from KahaDB to LevelDB?

On Thu, Mar 12, 2015 at 9:26 PM, Tim Bain tb...@alumni.duke.edu wrote: This doesn't really work for topics; you need to get the new messages to collect on b2 while the old consumer drains b1, and only then have the old consumer disconnect and reconnect on b2; you can't have the consumer

Re: Data Migration from KahaDB to LevelDB?

On Thu, Mar 12, 2015 at 8:58 AM, Tim Bain tb...@alumni.duke.edu wrote: And there would need to be a change to allow a non-durable subscriber to disconnect from the old broker without unsubscribing the consumer and to resume that subscription once the connection is made to the new broker (and

Re: Replicated LevelDB status (production worthy?)

/jimrobinson/activemq/commit/d272a116ff5c0916a6044d657f99df48f264bd2a On Tue, Mar 10, 2015 at 4:57 PM, James A. Robinson j...@highwire.org wrote: Working my way through the code and the debug log from the test, I see that the ZooKeeper group is getting emptied out after session expiration occurs

Re: Replicated LevelDB status (production worthy?)

:10:53,490 | DEBUG | ZooKeeper state change dispatcher thread | ZooKeeper group for 01 changed: Map() On Sat, Mar 7, 2015 at 8:52 PM, James A. Robinson j...@highwire.org wrote: I think I've got the use a case represented for https://issues.apache.org/jira/browse/AMQ-5082 I could use

Re: Data Migration from KahaDB to LevelDB?

On Fri, Mar 6, 2015 at 6:07 AM, underflow underf...@async.eu wrote: - Another idea was to create a network of brokers with the original instance (w/ kahadb persistence) and the new instance (w/ leveldb persistence) + resending all content, if required... I'll be curious to see what advice you

Re: Replicated LevelDB status (production worthy?)

On Wed, Mar 4, 2015 at 12:29 PM, James A. Robinson j...@highwire.org wrote: Thanks. I'm pretty sure AMQ-5082 is what I'm seeing on 5.11.1. I'll see if I can get the cycles to set up a unit test to replicate the issue. I think I've got the use a case represented for https://issues.apache.org

Re: help with timeout

On Fri, Mar 6, 2015 at 8:37 AM, arun196 arun.bi...@gmail.com wrote: I have a consumer that processes orders from messages received from activemq. I would like to set a timeout in case the order processing takes over 5 minutes. I'm sorry, it's not clear to me from your description what it is

Re: java.lang.IllegalArgumentException: Invalid connect parameters: {connectionTimeout=1000}

On Thu, Mar 5, 2015 at 2:28 PM, arun196 arun.bi...@gmail.com wrote: brokerURL=failover:(tcp://localhost:61616?keepAlive=true,tcp://localhost:61617?keepAlive=true,tcp://localhost:61618?keepAlive=true)?connectionTimeout=1000amp;jms.redeliveryPolicy.maximumRedeliveries=4 I don't see

Re: Replicated LevelDB status (production worthy?)

can search the archives on Nabble), and I don't recall any discussion of anyone finding a solution. But JIRA is the authoritative place to find out whether anyone has reported and/or fixed this issue (or any other). On Mar 3, 2015 8:23 AM, James A. Robinson jim.robin...@gmail.com wrote: Hi

Replicated LevelDB status (production worthy?)

Hi folks, While testing out ActiveMQ I've been building clusters VirtualBox. I've been spinning up two 3-node Replicated LevelDB stores on my laptop. I've noticed that the clusters can sometimes get into a state where none of the nodes is the master. It appears to me as though it's an issue

Re: Message Source time vs Receive time

I suspect one issue with figuring out the time a message was received by a subscriber is that an efficient client will typically buffer up messages. The page http://activemq.apache.org/activemq-message-properties.html says that JMSTimestamp is the time the message was sent, and I believe that