vamossagar12 commented on a change in pull request #10798:
URL: https://github.com/apache/kafka/pull/10798#discussion_r703333661
##########
File path:
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java
##########
@@ -505,6 +506,14 @@ private void closeOpenIterators() {
}
}
+ private ByteBuffer createDirectByteBufferAndPut(byte[] bytes) {
+ ByteBuffer directBuffer = ByteBuffer.allocateDirect(bytes.length);
Review comment:
> @vamossagar12 great to see you're about to add direct buffer support,
I was actually also looking into this. Just a few comments: as pointed out
already, one important aspect to keep in mind with direct buffers is that
allocation and disposal is more expensive than with regular heap buffers, so
it's important to re-use them, it's common practice to keep a pool of direct
buffers around and re-use them on a per-call basis (most I/O libraries to this,
e.g., netty, Crail, etc.). You can keep queues of free and used buffers and
move them around during state store operations.
Thanks @patrickstuedi , that's a great suggestion. I think it makes sense
given the expense in allocation/disposal. Couple of questions there:
1) I think the factor to consider there would be how to manage the pool of
used/free buffers? As already pointed by @cadonna above, the only case where
we can expect concurrent access to state stores would be for IQ and otherwise
it's single threaded. For the single threaded case, it's going to be straight
forward but for IQ, what happens if a request comes in and none of the direct
byte buffers are `free` ? Do we re-allocate on the fly?
2) This might be a small issue but calling it out. We need to allocate some
capacity to the direct byte buffer. Currently, I am iterating over the list of
keys and values and assigning the max length- which again mayn't be the most
efficient way. If I were to create the pool, then I would need to know the
capacity to allocate upfront. Can this be exposed as a config for the users in
that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
