Hi Gordon,
----- Original Message -----
> From: "Gordon Sim" <g...@redhat.com>
> To: users@qpid.apache.org
> Sent: Thursday, December 15, 2011 3:33:22 PM
> Subject: Re: Weird behavior of flow_to_disk policy
>
> On 12/13/2011 03:33 PM, Pavel Moravec wrote:
> > Hi all,
> > I noticed a weird behavior of flow-to-disk policy. When the
> > relevant queue is utilized over its max-queue-count and when I am
> > sending and receiving messages there, my sender gets "Enqueue
> > capacity threshold".
> >
> > One particular scenario:
> > qpid-config add queue ToFillQueue --durable --max-queue-count=1000
> > --limit-policy flow-to-disk --file-count 2 --file-size 2
> > ./spout -c 1000 ToFillQueue
> > ./spout -c 1000 ToFillQueue
> > (due to flow control, we have to break the program)
> > ./drain -c 1000 ToFillQueue
>
> At this point the journal still has 1000 messages in it. The queue
> also
> has 1000 messages, so any further messages will be written to journal
> as
> part of the flow-to-disk behaviour...
I was under the impression that flow_to_disk is applied when max-queue-count
messages are in memory, not in the whole queue.. Then the behavior makes sense.
So, this means that once FTD is applied, messages (msg bodies) will be stored
to disk and not to memory until the queue depth drops below max-queue-count.
And there will be messages on disk while memory not "fully utilized"(*) until
there will be no message in the journal.
(*)memory fully utilized = memory keeps max-queue-count messages.
That is unexpected but in my opinion acceptable behavior - the second option in
having max-queue-count in memory every time would gain some performance but
more complex to implement.
>
> > ./spout -c 1000 ToFillQueue
> > 2011-12-13 16:15:50 warning Broker closed connection: 501, Enqueue
> > capacity threshold exceeded on queue "ToFillQueue".
> > (JournalImpl.cpp:587)
> > 2011-12-13 16:15:50 error session-busy: Session detached by peer
> > session-busy: Session detached by peer
>
> I think this is simply a journal capacity problem. It cannot
> accommodate
> 2000 messages. Journal size is 2*64k*2=256k, which assuming the
> smallest
> enqueue record of 128 bytes would allow for only 2000 in theory and
> in
> practice you get less (I believe due to other control records plus
> file
> boundaries etc).
Yes, I chosen the journal size just for this example. The journal itself stores
1536 of the smallest messages.
>
> > qpid-tool shows:
> >
> > arguments {u'qpid.file_size': 2,
> > u'qpid.file_count': 2, u'qpid.max_count': 1000,
> > u'qpid.policy_type': u'flow_to_disk'}
> > altExchange absent
> > msgTotalEnqueues 2536
> > msgTotalDequeues 1000
> > msgTxnEnqueues 0
> > msgTxnDequeues 0
> > msgPersistEnqueues 0
> > msgPersistDequeues 0
> > msgDepth 1536
> >
> > Interestingly, when I pump messages to the queue (without any
> > draining) to fill the journals, the exception is raised when
> > exactly the same number of msgTotalEnqueues appears.
> >
> > Playing with more scenarios, sometimes the msgTotalEnqueues is
> > higher, sometimes even lower than the total capacity of the
> > queue(+its journals).
> >
> > Almost the same result I achieve when sending durable messages
> > (with the main difference that all messages go to journal direclt,
> > not storing any in memory).
> >
> > Anyway, it seems for me that consuming a message from flow-to-disk
> > queue (when the policy is already applied) is wrongly implemented.
> >
> > I noticed this on qpid 0.10 and 0.12 while 0.8 seems ok.
> >
> > Kind regards,
> > Pavel
> >
> >
> >
> > ---------------------------------------------------------------------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact: mailto:users-subscr...@qpid.apache.org
> >
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscr...@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscr...@qpid.apache.org
