Hi, hongzhi
Thanks for the review. Here are answers about your points. 1. Concern about lifecycle of the pending TsBlock The `releaseResource` method will be reached when query canceled, timeout, or session closed and we will call `clear` method to release pending TsBlock there. So there is no problem about memory safety. 2. Concern about repeated or stale offsets Fragments from different blocks can never be mixed, because both sides validate strictly and we will fail query when validation failed: - On the server, the pending fragment holds exactly one block plus its nextOffset. A request is accepted only if req.offset == pending.nextOffset when a fragment is pending, or req.offset == 0 when nothing is pending. Any other value (stale offset, offset ahead of the server, or a non-zero offset with no pending fragment) is rejected with an error status and the dataset fails — we never append bytes to the wrong block. - On the client, IoTDBRpcDataSet validates every fragment: the returned offset must equal the accumulated buffer length, and when the block completes the assembled size must equal the totalLength carried by the first fragment. A fragment belonging to a different block would fail these checks immediately. Thanks, Weihao Li At 2026-11-12 18:03:30, "Hongzhi Gao via dev" <[email protected]> wrote: >Hi,weihao > >One concern is the lifecycle of the pending serialized TsBlock in >`QueryExecution`. > >What happens if the client disconnects or stops fetching halfway through a >fragmented result? Is the retained buffer released on query cancellation, >timeout, and session close? > >Also, how are repeated or stale offsets handled during retries to avoid mixing >fragments from different result blocks? > >Hongzhi Gao > >On 2026-11-12 09:02 UTC Weihao Li wrote: >> Hi all, >> >> >> I'd like to raise a design for discussion before submitting the >> implementation. >> >> >> Problem: A single row containing an OBJECT/BLOB field can reach hundreds of >> MB. The DataNode writes the whole serialized TsBlock into one >> TSFetchResultsResp.queryResult (or TSExecuteStatementResp.queryResult), and >> the existing limits (fetchSize in rows, MAX_RETURN_SIZE = 8MB) cannot cap a >> single oversized row. So: >> >> >> - The response frame is as large as the oveversized row; >> - TElasticFramedTransport grows its read/write buffers to the frame size and >> keeps them for the lifetime of the connection — one such response leaves >> roughly the row size resident on each of the server connection thread and >> the client, per active connection, which easily cause OOM. >> >> >> Proposal: We recently solved the same problem for DataNode-to-DataNode >> exchange (MPPDataExchange large TsBlock fragmentation). This proposal >> applies the same idea to the client-server path, simplified to a pure pull >> model: >> >> >> - Add optional offset/totalLength fields to >> TSFetchResultsReq/TSFetchResultsResp (and the two fields to >> TSExecuteStatementResp). A client that sets offset (0 on the first request) >> declares it can assemble fragments; old clients and servers keep the >> existing behavior. >> - The server cuts a serialized TsBlock into fragments that fit a bounded >> payload (reuses the existing conf >> `mpp_data_exchange_max_payload_size_in_bytes`); the unsent tail is kept on >> QueryExecution and continued by the next request. Fragments are zero-copy >> slice() views; a TsBlock is still serialized only once. >> - IoTDBRpcDataSet (shared by Session and JDBC) reassembles fragments before >> TsBlockSerde.deserialize, so SessionDataSet, RowRecord, Field and JDBC users >> are unaffected. >> - Small results take exactly the same RPCs as today; only an oversized block >> costs one additional round trip, because the response that completes the >> fragment does not piggyback new rows (this keeps the per-response row count >> within the existing fetchSize bound). >> >> >> Compatibility. Protocol changes are purely additive optional fields, so >> rolling upgrades and other language clients are unaffected. The execute path >> negotiates via an enable_result_chunking key in the openSession >> configuration map. >> >> >> Thanks, >> Weihao Li
