Java Broker Design - Operational Logging has been edited by Martin Ritchie (Jun 25, 2009).

(View changes)

Content:

Operational Logging

The current logging configuration in the Java broker is focused for developers. Logging is performed on a class basis and as a result it is not easy to enable logging to get an operational view of the broker. Some work has been done to create configuration files that set the levels on various classes to provide the operational view of the broker (see Debug using Log4j). While this provides some good detail it does not provide the full picture.

The page will document the logging information that would be useful to provide, a suggested approach that should be followed and guidelines to developers for adding operational logging messages.

Overview

Currently logging is performed added in an ad-hoc manner by the devloper, ususlly to assist in the developemnt of the code base.

Log messages should be aimed at helping to provide users and/or support staff with information on the health of the broker; and - in the case where there has been some issue - help them diagnose the cause of that issue.

As such these messages should be readable without knowledge of the Qpid code base, they should not be so frequenet as to impact the performance of the broker but should be frequent enough such that diagnosis of issues is possible.

Log messages should occur whenever a significant event occurs, for instance the creation or destruction of a connection to the broker. The log message should contain enough information to be able to correlate the message with a business process event. In the case of a connection open the remote address, the login name, the virtual host, and the application id should be included in the log message.

Log messages at INFO level and above are expected to be turned on in a production environment.

Logging Content

There are two types of logging that are valuable to the operation of a Qpid broker:

  1. State updates
  2. Statisticsl updates.

Status Updates

The following model objects should log updates on state changes, creation / descruction events:

  • Broker
  • VirtualHost
  • MessageStore
  • Connection
  • Queue
  • Exchange
  • Binding

Whilst this will usually mean a single log instruction for an event such as creation/destruction the models may record state that can then be reported with the periodic statistics update. The following is an example what details may be logged:

Broker
Total Incoming message (count / bytes)
Total Outgoing message (count / bytes)
Peak Incoming message (count / bytes)
Peak Outgoing message (count / bytes)
Heap current usage
Heap peak usage

VirtualHost
Number of active connections
Total number of connections made
Total Incoming message (count / bytes)
Total Outgoing message (count / bytes)
Peak Incoming message (count / bytes)
Peak Outgoing message (count / bytes)

MessageStore
Disk space used
Memory used
Entities Stored (Queue, Exchange, Binding, Message)
Total Incoming message (count / bytes)
Total Outgoing message (count / bytes)
Peak Incoming message (count / bytes)
Peak Outgoing message (count / bytes)

Connection
Number of sessions
Number of consumers
Total Incoming message (count / bytes)
Total Outgoing message (count / bytes)
Peak Incoming message (count / bytes)
Peak Outgoing message (count / bytes)

Queue
Subcription count
Binding Count
Current Queue size (count / bytes)
Messages sent (count / bytes)

Exchange
Bound queues
Total Messages sent (count / bytes)

Subscription
Total Outgoing messages (count / bytes)
Peak Outgoing message (count / bytes)
Unacknowledged size (count / bytes)

Statistics updates

In addition to status events being logged, we should periodically log statistics. Each model object may have a set of statistics that they wish to report but it is expected that the statistics will be based on the period between logs. There may also be desire to log at more than one interval. i.e once per minute, hour, day.

Broker
Incoming message rate (count / bytes)
Outgoing message rate (count / bytes)
Min / Max / Average message size
Last Activity / Idle Notification

VirtualHost
Incoming message rate (count / bytes)
Outgoing message rate (count / bytes)
Min / Max / Average message size
Last Activity / Idle Notification

MessasgeStore
Incoming message rate (count / bytes)
Outgoing message rate (count / bytes)
Min / Max / Average message size
Last Activity / Idle Notification

Connection
Incoming message rate (count / bytes)
Outgoing message rate (count / bytes)
Min / Max / Average message size
Last Activity / Idle Notification

Queue
Incoming message rate (count / bytes)
Outgoing message rate (count / bytes)
Min / Max / Average message size
Last Activity / Idle Notification

Exchange
Message rate (count / bytes)
Min / Max / Average message size
Last Activity / Idle Notification

Subscription
Outgoing message rate (count / bytes)
Min / Max / Average message size
Last Activity / Idle Notification

As will be mentioned in the guidelines section, any collection and reporting must be mindful of the performance overhead in doing so.

Logging format

To reduce the amount of lookups required, by the user, each log statement should include all the information required to understand the context of the message. e.g. A subscription log statement should include details about the associated connection, vhost, channel and queue.

In order to keep the amount of data logged to the essential the message should be short and unambiguous - but easily recognisable. So the following formats are recommend for each model:

Conection
"con-<ip>-<username>-<UID>" : the UID will allow the disabmbigiuation if the same client has more than one connection.
VirtualHost
"vh-<name>"
Channel
"c-<name>"
queue
"q-<name>"
subscription
"s-[<selector>][-<properties>]-<UID>" : again a UID is required here to disabmbiguate multiple subscriptions.

Logging Hierarchy

When looking at augmenting the logging of the broker it makes sence to take a step back and provide an operational based logging hierarchy qpid. in addition to the developer focused org.apache.qpid..

The hierarchy of loggers should be structured such that it is easy to enable start or stop monitoring at runtime. The suggested hierarchy is as follows:

qpid ─┬─ Broker
      │
      ├─ Connection ─── <id> ─── Subscription ─── <id>
      │
      └─ VirtualHost ─── <name> ─┬─ Binding 
                                 │
                                 ├─ Exchange ─── <name> 
                                 │
                                 ├─ Queue ─── <name>
                                 │
                                 └─ MessageStore 

This would allow the quick enabling or disabling of the various logging events.

Guidelines for logging changes

To date there has been no discussion around what, who or even when to log and each developer has been left provide logging that they see fit. As mentioned earlier Log messages should occur whenever a significant event occurs, The log message should contain enough information to be able to correlate the message with a business process event. In the case of a connection open the remote address, the login name, the virtual host, and the application id should be included in the log message.

The log statements should be short and the reader should not need to refer to previous log statements in order to fully understand the situation. i.e. A new consumer log statement should include the connnection and virtualhost details rather than just the connection details and so requiring the user to read back to find the details around the connection creation.

All log statements that perform any computation before the log call must be wrapped with the is<LEVEL>Enabled calls to remove the computation should the log statement not be required.

When adding a new log statement a comment should be placed before hand highlighting if this is on the critical message routing path to allow reviewers to better gauge the potential impact. In addition the performance suit should be run with and without the log statement to get an actual measure of the impact.

--------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:commits-subscr...@qpid.apache.org

Reply via email to