yihua commented on code in PR #19737:
URL: https://github.com/apache/hudi/pull/19737#discussion_r3859877899
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/blob/BatchedBlobReader.scala:
##########
@@ -691,17 +715,16 @@ object BatchedBlobReader {
// Apply mapPartitions
val result = df.mapPartitions { partition =>
- // Create storage and reader for this partition
- val storage = HoodieStorageUtils.getStorage(broadcastConf.value)
- val reader = new BatchedBlobReader(storage, maxGapBytes, lookaheadSize)
+ // Create reader for this partition
+ val reader = new BatchedBlobReader(
+ HoodieStorageUtils.getStorage(_, broadcastConf.value), maxGapBytes,
lookaheadSize)
Review Comment:
If we want to add a cache, it should not be added at this reader layer; it
should be added in the `HoodieStorage` layer. For now, the cache is overkill,
and per-blob instantiation overhead is extremely low, so keeping the current
implementation for simplicity. If this becomes a bottleneck, we can optimize
this in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
