|
Page Edited :
qpid :
Network IO Interface
Network IO Interface has been edited by Aidan Skinner (Jul 06, 2009). Content:Objective and Scope.OverviewThe broker is prone to heap exhaustion, leading to OutOfMemory errors. The servers main memory usage is storing messages on queues, Producer flow control is needed to manage that area and is out of scope for this work. The other major source of memory consumption are the network buffers used to hold unprocessed AMQP frames once they have been read from the socket. These are currently able to grow uninhibited, and there is no means available to control them. Further, the transport layer is poorly implemented and difficult to work in. Improving encapsulation is an explicit goal of this work. Problem StatementWhen the broker is unable to process frames as quickly as they are being sent these buffers begin to fill up and the broker has no way to limit those. For the broker to effectively manage its memory usage, it needs to be able to at least place an upper bound on the size of it's network buffers. It also has no way to know how large those buffers are. 1 High message throughput on one publishing connectionThis is where the publishing client is sending a consistently sustained high rate of messages to the broker, and is more likely to happen where some of the messages are persistent Data from the client gets out of the client side buffers and into the broker side buffers. The broker is processing messages onto the queues as fast as it can, but gets backed up and the broker side buffers grow until eventually the broker OOM's, the heap filled by the MINA buffers along with the data in queues. 2 Medium-High throughput on several publishing connectionsThis is where the are multiple connections sending data to the broker, at varying rates from medium to high (as measured in Qpid terms as greater than the ave/max throughputs measured in test). The broker threads are being managed by the JVM in terms of processor time i.e. yielding to each other in an unpredictable way. As for case 1 above, with the caveat that it's far harder to predict how long it'd take to happen and how the TCP socket level behaviour will impact the client. It's also a more likely real world scenario i.e. probably more than one connection for MDS publication (for example) would result in a set of growing buffers resulting in OOM. 3. Environmental issue prevents broker from processing buffer dataThis is where the broker, for example, runs out of SAN or something else (disk for logging etc) and so cannot process the messages out of the buffers and onto the queues. CPU ? As for case 1, assuming that the broker just needs to be up to have data being buffered which previous tests certainly indicate i.e. a publisher can happily pump data into a disabled broker for some time before the connection gets killed. 4. Slow client causes broker network buffers to growThis is where the client is not reading data as fast as the broker is sending it. The data will be buffered on the broker, increasing memory usage. This is particularly problematic when sending messages to the client since the payload will then be in both the message store and the network buffers at the same time, doubling usage per message. Current Architecture
Inside Qpid, data is read from a socket and placed in a buffer. A separate thread then takes this buffer and attempts to parse it as an AMQP command. this AMQP command is then put on a second buffer. Finally a third thread reads the command and processes it. Currently the two buffers between these three threads are unbounded. This means that data is read from the network as fast as possible with no regard as to whether the broker has the capacity to process it. Queues are themselves a kind of buffer between client applications. From a queue the message can be assigned to be send to a client. At this point a delivery command is placed in another buffer awaiting sending on the network. When received by the client a similar process to receiving on the broker occurs The whole process looks something like this Client App sends message -> (MINA Buffer) Of all the buffers above, only the TCP buffers are bounded (the Delivery Queue Buffer in the client is potentially bounded by prefetch, although prefetch is not set on bytes but on messages which may be of arbitrary size), every other buffer is a potential source of out of memory exceptions. From the above we can see that there are many potential sources of OutOfMemoryExceptions. We need to consider where we may get unbounded growth, what scenarios will cause that, and what other ways we have to mitigate those risks. In general we get growth of the IO (MINA) buffers when sender and receiver are operating at mismatched rates (i.e. the Client and Broker). We will get unbounded growth of the queue if the sending client is producing at a faster rate than the receiving client can process. Exclusions: / Assumptions
High-Level Technical ArchitectureFunctional Requirements
Non Functional Requirements
Architecture DesignCommon should have an interface which all transport plugins can implement and which the server and client can use. The interface would include a means to set the standard socket options and to limit it's total memory usage. TCP itself has a flow control mechanism which kicks in when the receiver of data cannot read from the socket as fast as data is being sent. TCP sockets use send and receive buffers to attempt to smooth the flow from application to network and maximize network performance. By limitting the rate at which the application reads from the network to the rate at which is can process the data the sender of the data is throttled to that production rate. Overview of DesignCommon will hold a transport layer interface which the existing MINA transport will be ported too. We will also port the 0-10 client o.a.q.transport.network.io package to that interface. This interface should be quite simple. Methods to send, receive, flush, close, open, listen and a method to set TCP options are likely to be sufficient. These would operate on a QpidByteBuffer, essentially MinaByteBuffer to avoid having to fix our use of expanding buffers at the same time. The server and client both use common for their network layer, and will need to be updated to use the new interface. They will need to pass through the configured socket options. When processing the incoming data, one frame at a time will be processed and that frames processing will be completed before the next one is read. There will be no other data structures used to hold unprocessed frames. The server will need to be substantively modified to push the MINA specific parts into the appropriate plugin. Implementing fixed size IO buffers would require replacing MINA with an alternative implmentation of the transport layer. the first step in any such process would be to clearly encapsulate the concept of a transport layer using the existing MINA code as the initial implementation. The next step would be to adapt the already used 0-10 transport to sit clearly behind the same interface. Finally a network transport more atuned to the needs of a broker (supporting large numbers of incoming connections) could be developed from the base of the existing 0-10 transport. The changes would impact the client and broker where they interface with the transport layer. In the first phase the client and broker would be altered to use this implementation independent interface. Once this work had been completed the client and broker could be tested and released using the MINA implementation proving that no adverse impact from the encapsulation of the transport layer had occurred. Once the first phase of the work has been completed, alternative transport implementations could be developed. this would require no code changes to the client or broker, but would require system testing to prove that behaviour was correct when using the alternative transport implementation. Implementing a bounded buffer transport without implementing AMQP flow control incurs no extra cost (although it obviously only addresses the issues solved by buffer bounding). Bounding the buffers attempts to address the issue of regulating incoming data flow to the rate at which it can be processed by the receiver. This will generally occur when the sender is capable of sending bursts of data at a high rate. This is most evident with persistent messages where the rate at which messages can be persisted to disk is much lower than the rate at which they can be sent over the network. Fundamentally if it is not able to process messages at the rate at which they are being sent, Qpid should not accept them, pretending to do so is giving the application a false impression about what Qpid is doing, and is potentially only deferring an issue to a later point when there will be a great deal of message loss. Further more the behaviour is non-JMS compliant (JMS expect publishing to be a synchronous activity). Breakdown of component parts.
TestingTesting under load and handling error conditions (unexpected disconnection etc) will need to be carried out. New load tests which accurately simulate application workloads need to be developed so that we can provide accurate configuration guidance. This needs to be carried out on Windows, Linux and Solaris as they have significantly different OS in all permutations of client/server. New unit tests will need to be written to cover the transport plugins and the new interfaces. Existing test coverage in this area is minimal. ImpactThere is a potential effect upon performance, we will need to measure this once it has been implemented to quantify what effect, if any, it has had. Compatibility / Migration Implications
Risks
|
Unsubscribe or edit your notifications preferences
