comphead commented on code in PR #7692:
URL: https://github.com/apache/datafusion-comet/pull/7692#discussion_r4637037096
##########
native/core/src/local.rs:
##########
@@ -122,6 +133,12 @@ pub extern "system" fn
Java_org_apache_comet_local_NativeLocal_createQuery(
);
JVMClasses::init(env);
let serialized_plan = env.convert_byte_array(serialized_query)?;
+ let mut scan_partitions_vec =
Vec::with_capacity(scan_partitions.len(env)?);
+ for i in 0..scan_partitions.len(env)? {
+ let bytes = scan_partitions.get_element(env, i)?;
+ let bytes = unsafe { JByteArray::from_raw(&*env, bytes.into_raw())
};
+ scan_partitions_vec.push(env.convert_byte_array(bytes)?);
+ }
Review Comment:
Each `get_element` result stays a live local reference until `createQuery`
returns, so a scan of 1024 partitions holds 1024 of them in one JNI frame. The
jni 0.22 docs for `delete_local_ref` recommend freeing references in loops like
this, and `credential_bridge.rs` does it for the same reason
(`env.delete_local_ref(element)`, with the comment about the local frame).
Converting with `env.convert_byte_array(&bytes)` and then calling
`env.delete_local_ref(bytes)`, or using `.auto()`, avoids that. Reading
`scan_partitions.len(env)?` once also saves the second JNI call.
##########
native/core/src/local.rs:
##########
@@ -305,6 +403,12 @@ pub extern "system" fn
Java_org_apache_comet_local_NativeLocal_nextBatch(
) -> jlong {
try_unwrap_or_throw(&e, |env| {
let result = next(env, id, arrays, schemas);
+ if result.is_err() {
+ let partition = QUERIES.lock().get(&id).map(|entry|
entry.query.partition());
+ if let Some(partition) = partition {
+ FAILED_PARTITIONS.lock().insert(id, partition);
+ }
+ }
if result.is_err() || matches!(result, Ok(-1)) {
close(id);
Review Comment:
`LocalQueryIterator.executionFailure` reads the failed partition and then
closes the query, and the task completion listener closes it on every other
path. If `nextBatch` closed only at end of stream and left a failed query in
`QUERIES` until `close`, `failedPartition` could read `entry.query.partition()`
from the live entry. `FAILED_PARTITIONS`, its removal in `close` and the extra
count in `activeQueries` would then not be needed. `PartitionOrder::fail`
already drops the running partitions on an error, so I expect the entry to hold
no running partition while it waits for `close`.
##########
native/local/src/lib.rs:
##########
@@ -344,6 +362,11 @@ impl Stream for PartitionOrder {
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) ->
Poll<Option<Self::Item>> {
loop {
self.start_partitions();
+ if self.running.is_empty() {
+ return Poll::Ready(None);
+ }
+ let turn = self.next - self.running.len();
+ self.turn.store(turn, Ordering::Release);
let Some(partition) = self.running.front_mut() else {
return Poll::Ready(None);
};
Review Comment:
With this check, the `else` of the `let Some(partition) =
self.running.front_mut() else` two lines below can no longer run, so one of the
two exits can go. `self.next - self.running.len()` cannot underflow on an empty
queue, and `turn` is read only right after an error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]