[
https://issues.apache.org/jira/browse/GEODE-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17435605#comment-17435605
]
Darrel Schneider commented on GEODE-9752:
-----------------------------------------
It looks to me like Anthony's ChunkedWriteHandler would solve Dan's big problem
of serializing the whole response into one ByteBuf.
But it does require that the whole result is known to the ChunkedInput we
implement. And because of concurrency issues the ChunkedInput we implement will
need to have a stable view of all the values being returned. So in Dan's
example of zrange it will need to create a new ArrayList and we keep it alive
until netty has written all the chunks back to the client. So the ArrayList can
end up getting promoted and cause a more expensive gc in the server's future.
I agree that it could be worse (we could also have to copy all the elements in
the ArrayList) but it also seems like it could be better if as we iterate over
data structure we start writing the result. In some cases this might not be
efficient because we need to finish the iteration before we know the size of
the result and we need to write the size first. But in other cases it seems
best to "cut out the middle man" and write the result data directly to a netty
ByteBuf.
> Limit Memory Consumption for Read Operation
> -------------------------------------------
>
> Key: GEODE-9752
> URL: https://issues.apache.org/jira/browse/GEODE-9752
> Project: Geode
> Issue Type: Improvement
> Components: redis
> Affects Versions: 1.15.0
> Reporter: Wayne
> Priority: Major
>
> The "read" commands can be made more memory friendly by streaming back their
> result to netty a "batch" at a time. They can get the netty ByteBuf and write
> the result directly to it. Once the buffer contains a certain number of bytes
> (say 4k) it do a write and flush. Once that completes it can then use the
> same buffer to send the next 4k bytes to the client. Writing the response
> directly to the netty ByteBuf will also produce less garbage. The only
> downside to it is that the writing will be done while holding the stripe
> lock. This probably will not be any slower unless the buffer fills up while
> we still hold the lock. The last buffer (the one that is not full) can be
> done after the lock is released just as we currently do by returning a
> RedisResponse outside the lock and then asking it to write itself to netty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
