github-actions[bot] commented on code in PR #66810:
URL: https://github.com/apache/doris/pull/66810#discussion_r3803535519
##########
be/src/exec/sink/writer/paimon/jni_paimon_write_backend.cpp:
##########
@@ -428,22 +438,45 @@ Status
JniPaimonWriter::_write_projected_block(RuntimeState* state, Block& block
block.get_by_position(i).name = _sink.column_names[i];
}
- // Pipeline: Doris Block → Arrow Schema → Arrow RecordBatch → IPC Stream →
JNI direct buffer
- //
- // Step 1: Build Arrow schema from the projected Block.
+ // Build the Arrow schema once, then convert and send bounded row ranges.
This avoids holding
+ // Arrow arrays and an IPC buffer for the entire Doris block at the same
time.
// Paimon write timestamps are transported as civil-time fields. The Java
writer uses the
// pinned Paimon target type to preserve NTZ values or convert LTZ values
with the session zone.
// Variant V2 is transported losslessly as its value/metadata pair,
including nested Variant.
std::shared_ptr<arrow::Schema> arrow_schema;
RETURN_IF_ERROR(get_paimon_arrow_schema_from_block(block, &arrow_schema));
- // Step 2: Convert Doris Block columns to an Arrow RecordBatch.
+ const size_t block_rows = block.rows();
+ const size_t block_bytes = block.bytes();
+ const size_t average_row_bytes =
+ std::max<size_t>(1, block_bytes / block_rows + (block_bytes %
block_rows != 0));
+ const size_t rows_per_batch = BlockBudget(state->batch_size(),
_arrow_batch_size_bytes)
Review Comment:
[P1] Split skewed ranges by their actual size
`rows_per_batch` is derived once from the whole block's average, so it does
not bound the bytes in any selected range. For example, with the default 8 MiB
target, 100 rows totaling about 36 MiB (one 12 MiB row and about 24 MiB spread
across the rest) yield roughly 22 rows per range; the range containing the
large row can exceed 16 MiB even though that row would fit when sent alone.
`estimated_ipc_bytes` only preallocates the C++ stream and cannot constrain
Java decoding, so the new 16 MiB `RootAllocator` rejects an otherwise writable
block. Please choose `end_row` from the selected rows' actual or serialized
bytes before invoking Java, and add a skewed-row regression test.
##########
fe/be-java-extensions/paimon-connector/src/main/java/org/apache/doris/paimon/PaimonJniWriter.java:
##########
@@ -158,10 +155,18 @@ public void open(String serializedTable, Map<String,
String> hadoopConfig,
throw new IllegalArgumentException(
"PaimonJniWriter requires a native memory manager");
}
+ if (arrowMemoryLimitBytes <= 0) {
+ throw new IllegalArgumentException(
+ "PaimonJniWriter requires a positive Arrow memory
limit");
+ }
+ if (allocator != null) {
+ throw new IllegalStateException("PaimonJniWriter is already
open");
+ }
this.preExecutionAuthenticator =
PreExecutionAuthenticatorCache.getAuthenticator(hadoopConfig);
this.arrowConverter = new
PaimonArrowConverter(ZoneId.of(timeZone));
preExecutionAuthenticator.execute(() -> {
try {
+ this.allocator = new RootAllocator(arrowMemoryLimitBytes);
Review Comment:
[P2] Preserve Java Arrow limit failures across JNI
This finite allocator makes Arrow decode OOM an expected hard-limit outcome,
but `write()` catches it as `Throwable` and wraps it in a generic
`RuntimeException`. The C++ caller then uses `GetJniExceptionMsg()`, which
always returns `JNI_ERROR`, so hitting this configured limit is not reported as
`QUERY_MEMORY_EXCEEDED` and bypasses memory-specific handling and diagnostics.
This is a separate boundary from the existing C++ Arrow-allocation thread.
Please preserve an explicit Java Arrow OOM category through the wrapper and
translate it to the Doris memory-limit status at JNI return.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
