[
https://issues.apache.org/jira/browse/QPID-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Moravec updated QPID-5880:
--------------------------------
Attachment: QPID-5880.patch
Simple idea:
- in Queue::enqueue, set PersistableMessage::persistencyID "manually" to some
unique number that is identical to all message instances that has common
SharedState - e.g. to the pointer to SharedState
- during journal recovery, if we recover a message with already seen
persistencyID, use the previous seen instead with its SharedState and
PersistableMessage bits
Known limitation:
- message annotations added to some queue (e.g. due to queue sequencing
enabled) will be either over-written or shared to all other queues during
recovery
The patch contains a new QueueSettings option to enable (bu default disabled)
this feature on per queue basis. This somehow limits the limitation above.
Isn't storing pointer to SharedState to the disk (via persistencyID) some sort
of security breach? (I dont think so, but worth to ask)
Can't manual setup of persistencyID break something in store? (AFAIK no as
uniqueness of the ID is assured: 1) a new / different message with the same
persistencyID can appear only after the previous instance is gone from memory,
and 2) only queues with the option enabled are checked for message coupling)
Will it work well in cluster? No, it won't. As when primary broker gets 1
message to an exchange that distributes it to 100 queues, th broker updates
backup brokers via 100 individual "enqueue 1 message to queue q[1-100]" events.
So backup brokers consume more memory than primary - the same amount like
primary does not share SharedState at all.
Not sure if the patch is reasonable due to the cluster/HA point.
> [C++ broker] Make memory usage consistent after broker restart
> --------------------------------------------------------------
>
> Key: QPID-5880
> URL: https://issues.apache.org/jira/browse/QPID-5880
> Project: Qpid
> Issue Type: Improvement
> Components: C++ Broker
> Affects Versions: 0.28
> Reporter: Pavel Moravec
> Assignee: Pavel Moravec
> Priority: Minor
> Attachments: QPID-5880.patch
>
>
> In scenario "send messages via (fanout) exchange into multiple (tens,
> hundreds) queues", qpid broker keeps SharedState just once per the received
> message, to save RAM requirements.
> Assuming the messages are durable and stored to the disk, broker restart and
> journal recovery creates one SharedState for each message copy in each queue.
> This has negative impact to memory consumed by the broker, and it could
> potentially prevent broker successfull startup due to RAM exhausted - while
> the broker was running fine prior the restart attempt.
> It is required to have some "unifying" mechanism in journal recovery that
> creates just one SharedState per identical copies of a message spread over
> more queues.
> Ideally, this could be resolved by durable topics but that would require much
> bigger change.
> My proposal (to be discussed in reviewboard) is rather a fast fix supplying
> the original demand.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
