andygrove commented on PR #5543: URL: https://github.com/apache/datafusion-comet/pull/5543#issuecomment-5800557350
@mbutrovich, on your two top-level points. I've split committed benchmark results out of #5487's documentation item into a checkbox of their own, so the documentation box can be ticked on merge without them. The ~800 bytes per partition I couldn't reproduce when writing the issue up, so I haven't filed one. Caching relations of 1 to 64 partitions at 100 and 10,000 rows each, through both write paths and from a native Parquet scan, `CometArrowAllocator` is back to where it started while the relation is cached, after a read, and after uncaching, and every allocator the cache path uses is a child of it. Rather than file something I can't substantiate, I've taken the note out of the description. Verified on the default profile: `CometInMemoryCacheSuite` 53/53, and it also passes on Spark 3.5 and 4.0. It compiles against 3.4, 3.5, 4.0 and 4.1, and scalafix, scalastyle and spotless are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
