We need to run the tests again but you are right. Looking at the Dispatch Router config, the links are in and out. thx.
On Fri, Jul 8, 2016 at 5:47 PM, Ted Ross <tr...@redhat.com> wrote: > > > On 07/08/2016 11:42 AM, Olivier Mallassi wrote: > >> Hi Rob >> >> We have the following topology: >> >> * Publisher -> Dispatcher -> Broker1 or Broker 2 (Java) -> Dispatcher >> (same >> instance) -> Consumer >> * each Broker is configured (before the run) >> - with an exchange perf.topic (amqp topic) >> - with a queue (perf.queue) that has a binding (binding key + header >> based >> filtering) to the perf.topic exchange >> >> * the Dispatcher is configured with autoLink >> * The publisher publish in perf.topic (kind of distributed topic) >> * The consumer consume from perf.queue (distributed queue) >> * Dispatch version is 0.6.0 and Broker is in version 6.0.0. >> > > I think your autoLink _in_ from perf.topic is spurious. I suspect that > there are no consumers on the router consuming from "perf.topic" and > therefore the listener attached to the temporary queue is never issued > credit and it accumulates all perf.topic messages. > > > >> We have the expected behavior : published messages are load-balanced on >> brokers and consumed as expected. >> >> Yet, and we observed the OOM while running the test for a longer time, >> when >> the dispatcher connect, it creates a queue with a UUID (more or less) >> name. >> that queue is bind to perf.topic with # as a binding key (so every >> messages >> sent by the publisher fill up this queue thus the OOM because it is in >> memory). >> >> It looks to be "expected". >> Is there a way to avoid this? are we missing something? >> >> Many thx. >> >> oliv/ >> >> >> >> >> >> >> On Fri, Jul 8, 2016 at 5:16 PM, Rob Godfrey <rob.j.godf...@gmail.com> >> wrote: >> >> Hi Adel, >>> >>> I'm a bit confused... you say >>> >>> As no consumer is connected to this queue, all messages are kept >>>> >>> in-memory which causes the OutOfMemory exception in the broker. >>> >>> but then also say >>> >>> Consumer connected to that queue (Is it the dispatcher?): >>>> Name: qdlink.OyY41QAJRZ4JGGg >>>> Mode: MOVE >>>> >>> >>> Does the queue have a consumer or not? >>> >>> As to holding on to the messages... the heap dump shows that the message >>> is >>> in the queue (which we know) and has not been evicted by flow to disk >>> (we'd >>> need to work out why)... >>> >>> On average what size are the messages you are sending? Do you see >>> messages >>> flow through this topic at all? >>> >>> As to the "weird" queue name - what is happening is that we are creating >>> an >>> internal temporary queue when you subscribe to an exchange object in the >>> broker... for that we use a UUID as the queue name. >>> >>> -- Rob >>> >>> On 8 July 2016 at 15:48, Adel Boutros <adelbout...@live.com> wrote: >>> >>> I forgot to add that when I execute a qdmanage command to delete the >>>> "connector" to the broker, the weird queue disappears. >>>> >>>> Adel >>>> >>>> Date: Fri, 8 Jul 2016 07:45:49 -0700 >>>>> From: adelbout...@live.com >>>>> To: users@qpid.apache.org >>>>> Subject: Re: OutOfMemory exception with a Qpid Java Broker linked to >>>>> >>>> qpid dispatcher >>>> >>>>> >>>>> Fixing bad formatting on Nabbledue to Hotmail: >>>>> >>>>> Hello, >>>>> >>>>> We are doing some performance tests with a Qpid Java Broker connected >>>>> >>>> to >>> >>>> a >>>> >>>>> Qpid dispatcher. >>>>> We have noticed that after some time, the broker dies with an >>>>> >>>> OutOfMemory >>> >>>> exception and java heap is dumped. >>>>> After analyzing the heap dump and checking the broker, we have noticed >>>>> >>>> 2 >>> >>>> weird behaviors: >>>>> >>>>> 1) A queue is created on the broker with a weird Name bound on our >>>>> >>>> topic >>> >>>> (perf.topic) with a binding key "#". It will receive all messages. >>>>> This queue seems to be created only when the dispatcher is present and >>>>> >>>> is >>> >>>> configured. >>>>> As no consumer is connected to this queue, all messages are kept >>>>> >>>> in-memory >>>> >>>>> which causes the OutOfMemory exception in the broker. >>>>> >>>>> Weird Queue definition >>>>> Name: 18bb86dd-2953-4c6f-8eb2-56e5128963f3 >>>>> Type: standard >>>>> State: ACTIVE >>>>> Durable: false >>>>> Lifespan: DELETE_ON_NO_OUTBOUND_LINKS >>>>> Persist Messages: NEVER >>>>> Inbound: 0 msg/s (0.00 B/s) >>>>> Outbound: 0 msg/s (0.00 B/s) >>>>> Size: 2374 msgs (0.00 B) >>>>> Pre-fetched: 0 msgs (0.00 B) >>>>> Oldest Message Age: 356.064 secs >>>>> Enforced Max. Ttl(ms): 0 >>>>> Enforced Min. Ttl(ms): 0 >>>>> Exclusive: LINK >>>>> >>>>> Consumer connected to that queue (Is it the dispatcher?): >>>>> Name: qdlink.OyY41QAJRZ4JGGg >>>>> Mode: MOVE >>>>> >>>>> >>>>> 2) All messages even those consumed successfully by the consumers are >>>>> >>>> still >>>> >>>>> present on the broker >>>>> >>>>> From the heap dump, we can see the Berkley DB is keeping a reference to >>>>> >>>> all >>>> >>>>> messages. Is this also coming from the above weird queue? >>>>> >>>>> >>>>> PS: If we only use the dispatcher instead, we have none of the weird >>>>> behaviors >>>>> >>>>> Extract from the heap dump (Object holding >>>>> reference to one of the message header. "Validated" is one the message >>>>> header fields we set and which is already received by a consumer) >>>>> >>>>> char[9] @ 0xf59ff3d8 VALIDATED >>>>> '- value java.lang.String @ 0xf59ff3c0 VALIDATED >>>>> '- value java.util.HashMap$Entry @ 0xf59ff310 >>>>> '- [3] java.util.HashMap$Entry[8] @ 0xf59ff2c0 >>>>> '- table java.util.HashMap @ 0xf59ff188 >>>>> '- _appProperties >>>>> org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0 @ 0xf59fef88 >>>>> '- _metaData >>>>> >>>>> >>>> >>> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$MessageDataSoftRef >>> >>>> @ 0xf59fef70 >>>>> '- _messageDataRef >>>>> >>>>> >>>> >>> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage >>> >>>> @ 0xf59fef20 >>>>> '- _handle >>>>> >>>> org.apache.qpid.server.protocol.v1_0.Message_1_0 @ >>>> >>>>> 0xf59feef0 >>>>> '- _message >>>>> org.apache.qpid.server.message.AbstractServerMessageImpl$Reference @ >>>>> 0xf59621c8 >>>>> '- _message >>>>> org.apache.qpid.server.queue.StandardQueueEntry @ 0xf5962188 >>>>> '- _next >>>>> >>>> org.apache.qpid.server.queue.StandardQueueEntry >>>> >>>>> @ 0xf5962130 >>>>> - this$0 >>>>> >>>>> >>>> org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0$MessageHeader_1_0 >>> @ >>> >>>> 0xf59ff1f0 >>>>> >>>>> Broker Config >>>>> 1 Virtual Host Node and 1 Virtual Host configured with Berkley DB types >>>>> >>>> (BDB >>>> >>>>> with the default configuration). >>>>> 1 Topic (perf.topic) >>>>> bound to a queue (perfQueue) with a binding key and a jms-selector >>>>> filter >>>>> bound to an "alternate exchange" of a type fanout which is also >>>>> >>>> bound >>>> >>>>> to a queue but with no binding key (empty string) >>>>> >>>>> Dispatcher Config >>>>> >>>>> qdmanage -b amqp://localhost:10454 create --type=address >>>>> >>>> prefix=perfQueue >>> >>>> waypoint=true name=perf.queue.addr >>>>> qdmanage -b amqp://localhost:10454 create --type=address >>>>> >>>> prefix=perf.topic >>>> >>>>> waypoint=true name=perf.topic.addr >>>>> qdmanage -b amqp://localhost:10454 create --type=connector >>>>> role=route-container addr=localhost port=10455 >>>>> name=localhost.broker.10455.connector >>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink >>>>> >>>> addr=perfQueue >>> >>>> dir=out connection=localhost.broker.10455.connector >>>>> name=localhost.broker.10455.perfQueue.out >>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink >>>>> >>>> addr=perfQueue >>> >>>> dir=in connection=localhost.broker.10455.connector >>>>> name=localhost.broker.10455.perfQueue.in >>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink >>>>> >>>> addr=perf.topic >>> >>>> dir=out connection=localhost.broker.10455.connector >>>>> name=localhost.broker.10455.perf.topic.out >>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink >>>>> >>>> addr=perf.topic >>> >>>> dir=in connection=localhost.broker.10455.connector >>>>> name=localhost.broker.10455.perf.topic.in >>>>> >>>>> Clients config >>>>> 3 JMS Consumers connected each to perfQueue on the dispatcher >>>>> 2 JMS producers connected each to perf.topic on the dispatcher >>>>> >>>>> With the above config, we send a number of messages of which only 1/3 >>>>> >>>> will >>>> >>>>> be routed to the "alternate exchange" and never consumed. >>>>> >>>>> >>>>> Versions >>>>> Qpid Java Broker: 6.0.0 >>>>> Qpid Dispatch: 0.6.0 >>>>> JMS: 0.9.0 >>>>> >>>>> Regards, >>>>> Adel >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> >>>> >>>> >>> http://qpid.2158936.n2.nabble.com/OutOfMemory-exception-with-a-Qpid-Java-Broker-linked-to-qpid-dispatcher-tp7647065p7647066.html >>> >>>> Sent from the Apache Qpid users mailing list archive at Nabble.com. >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >>>>> For additional commands, e-mail: users-h...@qpid.apache.org >>>>> >>>>> >>>> >>>> >>> >> > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
