Hi,weihao One concern is the lifecycle of the pending serialized TsBlock in `QueryExecution`.
What happens if the client disconnects or stops fetching halfway through a fragmented result? Is the retained buffer released on query cancellation, timeout, and session close? Also, how are repeated or stale offsets handled during retries to avoid mixing fragments from different result blocks? Hongzhi Gao On 2026-11-12 09:02 UTC Weihao Li wrote: > Hi all, > > > I'd like to raise a design for discussion before submitting the > implementation. > > > Problem: A single row containing an OBJECT/BLOB field can reach hundreds of > MB. The DataNode writes the whole serialized TsBlock into one > TSFetchResultsResp.queryResult (or TSExecuteStatementResp.queryResult), and > the existing limits (fetchSize in rows, MAX_RETURN_SIZE = 8MB) cannot cap a > single oversized row. So: > > > - The response frame is as large as the oveversized row; > - TElasticFramedTransport grows its read/write buffers to the frame size and > keeps them for the lifetime of the connection — one such response leaves > roughly the row size resident on each of the server connection thread and the > client, per active connection, which easily cause OOM. > > > Proposal: We recently solved the same problem for DataNode-to-DataNode > exchange (MPPDataExchange large TsBlock fragmentation). This proposal applies > the same idea to the client-server path, simplified to a pure pull model: > > > - Add optional offset/totalLength fields to > TSFetchResultsReq/TSFetchResultsResp (and the two fields to > TSExecuteStatementResp). A client that sets offset (0 on the first request) > declares it can assemble fragments; old clients and servers keep the existing > behavior. > - The server cuts a serialized TsBlock into fragments that fit a bounded > payload (reuses the existing conf > `mpp_data_exchange_max_payload_size_in_bytes`); the unsent tail is kept on > QueryExecution and continued by the next request. Fragments are zero-copy > slice() views; a TsBlock is still serialized only once. > - IoTDBRpcDataSet (shared by Session and JDBC) reassembles fragments before > TsBlockSerde.deserialize, so SessionDataSet, RowRecord, Field and JDBC users > are unaffected. > - Small results take exactly the same RPCs as today; only an oversized block > costs one additional round trip, because the response that completes the > fragment does not piggyback new rows (this keeps the per-response row count > within the existing fetchSize bound). > > > Compatibility. Protocol changes are purely additive optional fields, so > rolling upgrades and other language clients are unaffected. The execute path > negotiates via an enable_result_chunking key in the openSession configuration > map. > > > Thanks, > Weihao Li
