sunchao commented on code in PR #6048:
URL: https://github.com/apache/datafusion-comet/pull/6048#discussion_r4053871744
##########
spark/src/main/scala/org/apache/comet/Tracing.scala:
##########
@@ -23,6 +23,16 @@ object Tracing {
private val nativeLib = new Native
+ /**
+ * Emits the Arrow memory counters for the JVM side. `jvm_arrow_imported` is
the part of
+ * `jvm_arrow_allocated` that came from native over the C Data Interface, so
the difference is
+ * the Arrow memory the JVM allocated itself. See
[[CometArrowImportAllocator]].
+ */
+ def logArrowMemory(): Unit = {
+ nativeLib.logMemoryUsage("jvm_arrow_allocated",
CometArrowAllocator.getAllocatedMemory)
+ nativeLib.logMemoryUsage("jvm_arrow_imported",
CometArrowImportAllocator.getAllocatedMemory)
Review Comment:
### Correctness
[P2] Track foreign buffer charges separately from the whole import allocator
This child also pays for memory allocated by the JVM. Arrow 18.3's
`ArrayImporter` allocates its owning `ArrowArray` using the supplied allocator,
and `BitVectorHelper.loadValidityBuffer` allocates a bitmap there when a native
all-valid vector has no validity buffer. A component probe using this head's
`ArrowImporter` with 4,096 integers reports root = child = 17,024 bytes,
comprising 16,384 foreign payload bytes and 640 JVM-created bytes. The
documented subtraction reports zero and loses those 640 bytes.
The error can include an entire UDF output: the interface permits allocating
with `inputs.head.getAllocator`, and `Data.exportVector(root, ...)` does not
transfer that allocation to the root. A second probe leaves 32,768 bytes of
JVM-created output in this child after input cleanup, so all of it is
subtracted. Both probes release cleanly.
Please measure foreign buffer charges separately, or expose this as
import-allocator accounting without claiming the difference measures
JVM-created memory. Add assertions for an absent validity buffer and an output
allocated from the input allocator. The current positive-child and
parent-rollup assertions pass in both cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
