wombatu-kun commented on code in PR #18375:
URL: https://github.com/apache/hudi/pull/18375#discussion_r3035477979
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/lance/SparkLanceReaderBase.scala:
##########
@@ -76,8 +76,21 @@ class SparkLanceReaderBase(enableVectorizedReader: Boolean)
extends SparkColumna
val filePath = file.filePath.toString
if (requiredSchema.isEmpty && partitionSchema.isEmpty) {
- // No columns requested - return empty iterator
- Iterator.empty
+ // No column data needed (e.g. count() on a non-partitioned table). Open
the Lance file to
+ // get the actual row count and return that many empty rows, so Spark
computes the correct count.
+ val countAllocator = HoodieArrowAllocator.newChildAllocator(
+ getClass.getSimpleName + "-count-" + file.filePath,
HoodieSparkLanceReader.LANCE_DATA_ALLOCATOR_SIZE)
+ try {
+ val lanceReader = LanceFileReader.open(file.filePath.toString,
countAllocator)
+ try {
+ val rowCount = lanceReader.numRows()
+ Iterator.fill(rowCount.toInt)(InternalRow.empty)
+ } finally {
Review Comment:
Good point. Switched to `Math.toIntExact()` which will throw
`ArithmeticException` on overflow instead of silently wrapping. `Iterator.fill`
in Scala is already lazy (backed by a counter), so no materialization concern —
but the int limit on `Iterator.fill` size is a real constraint. In practice
Lance files won't approach 2B rows, but better to fail clearly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
