liujiayi771 commented on PR #5717:
URL:
https://github.com/apache/incubator-gluten/pull/5717#issuecomment-2111724082
I have added some codes in the `release` method of `ArrowNativeMemoryPool`
to check the peak memory.
```java
@Override
public void release() throws Exception {
System.out.println("peak=" + listener.peak() +", current=" +
listener.current());
if (arrowPool.getBytesAllocated() != 0) {
LOGGER.warn(
String.format(
"Arrow pool still reserved non-zero bytes, "
+ "which may cause memory leak, size: %s. ",
Utils.bytesToString(arrowPool.getBytesAllocated())));
}
arrowPool.close();
}
```
I also added some logs in arrow codes to check the peak memory.
```c++
Result<RecordBatchGenerator> CsvFileFormat::ScanBatchesAsync(
const std::shared_ptr<ScanOptions>& scan_options,
const std::shared_ptr<FileFragment>& file) const {
auto this_ = checked_pointer_cast<const CsvFileFormat>(shared_from_this());
auto source = file->source();
auto reader_fut =
OpenReaderAsync(source, *this, scan_options,
::arrow::internal::GetCpuThreadPool());
auto generator = GeneratorFromReader(std::move(reader_fut),
scan_options->batch_size);
WRAP_ASYNC_GENERATOR_WITH_CHILD_SPAN(
generator, "arrow::dataset::CsvFileFormat::ScanBatchesAsync::Next");
std::cout << "memory=" << default_memory_pool()->bytes_allocated() << ",
max=" << default_memory_pool()->max_memory() << std::endl;
return generator;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
