viirya commented on code in PR #7258:
URL: https://github.com/apache/datafusion-comet/pull/7258#discussion_r4405242750
##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -258,11 +258,21 @@ object Utils extends CometTypeShim with Logging {
val dictionaryProvider: CDataDictionaryProvider = new
CDataDictionaryProvider
val codec = CompressionCodec.createCodec(SparkEnv.get.conf)
Review Comment:
Since this is already being touched: `CompressionCodec.createCodec` runs
once per batch inside `batches.map`, which means a reflective construction
(and, for LZ4, a fresh `LZ4Factory` lookup) for every tiny batch. All batches
in a partition are serialized on one thread, so creating the codec once before
the `map` should be enough and leaves the output unchanged. I haven't measured
how much of the remaining OFF time this accounts for, so treat it as optional;
it would lower the per-batch cost for every broadcast input, not just the
batched path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
