Performance issues with large object handling in DDMWriter
----------------------------------------------------------
Key: DERBY-867
URL: http://issues.apache.org/jira/browse/DERBY-867
Project: Derby
Type: Improvement
Components: Network Server, Performance
Versions: 10.1.1.0, 10.2.0.0
Reporter: Bryan Pendleton
Priority: Minor
There are some performance issues related to the way that DDMWriter handles
extremely large network messages, such as those that arise in DERBY-125 and
DERBY-492. The performance problems involve both memory usage and CPU cycles.
Among the problems that could be investigated and improved:
1) The DDMWriter internal byte array grows, but never shrinks (except when the
DDMWriter as a whole is collected). This means that there may be some memory
wastage when the buffer grows to a large value, then is later used for only
small buffers.
2) When the buffer grows, it tends to double in size. When you start gettting
to moderately large buffer sizes, this may mean that the buffer over-expands
during a growth event, and grows to a larger size than is needed, again leading
to memory wastage.
3) I'm not sure what would happen if memory were unavailable when the buffer
wanted to grow; I think it would be a very hard error. There may be ways to
handle this better, such as: waiting a while to see if memory becomes
available; writing previously chained messages out of the buffer to free up
space, etc. At the very least, it would be nice if an out-of-memory problem
were handled cleanly, for example by terminating this connection but leaving
the rest of the Network Server online and available for other users.
4) Perhaps we could eliminate the expansion all together by instead of
expanding have a pool of buffers available to all the DDMWriters. If a
DDMWriter needs to write more, it picks up a new buffer and returns it to the
pool when no longer needed.
5) Large DDM objects which require Layer B segmentation are written as a single
object, then later the segmentation is added after the payload data is
complete. This processing is handled by endDdm() and finalizeDssLength(). This
code currently copies the bytes around in the buffer two "extra" times: once to
insert the extended length bytes at the front of the large DDM object, and a
second time to insert the DSS Continuation Headers every 32K bytes. Given the
API between DDMWriter and its callers, it may be impossible to totally remove
this memory copying, but it does seem like we could reduce from 2 full copies
to 1, by combining the insertion of the extended length bytes with the
continuation header segmentation. It might also be possible to detect a large
object once it spills out from one DSS block into the second block, and hence
avoid copying for subsequent blocks, even if those first two blocks do have to
be copied once.
There may be other opportunities in this part of the code; this issue is just a
collection of the ideas that came up while the developers were discussing bugs
DERBY-125, DERBY-170, DERBY-491, and DERBY-492. A review of the processing that
occurs in those test cases should be performed as part of working on this bug.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
