fabientra opened a new issue, #13570:
URL: https://github.com/apache/gluten/issues/13570
### Backend
VL (Velox)
### Bug description
With the native Iceberg scan (`IcebergScanTransformer`), a simple count over
one partition of an Iceberg table returns fewer rows than vanilla Spark in most
runs. The number of missing rows changes from run to run (0 to 543 here). No
task fails, no task is retried, and the logs have no error or warning.
In the same engine, with the same query:
- `spark.gluten.sql.columnar.batchscan=false` (vanilla Iceberg reader, rest
of the plan in Gluten) is exact every time.
- The same 15 data files, read as plain Parquet with the native
`FileSourceScanExecTransformer` (``parquet.`<dir>/{file1,...,file15}` ``), are
exact every time. The result is identical to vanilla Spark on the same path, 5
runs out of 5.
So the rows seem to be lost in the Iceberg split path (Gluten's Iceberg
split info or Velox's Iceberg split reader). Parquet decoding, S3 reads,
filters and aggregation return correct results on the plain Parquet path.
We found it in production-like tests: an Iceberg overwrite of about 8.2
billion rows committed 49 to 398 fewer rows than vanilla Spark in 4 runs out of
5, with all tasks successful.
**Query** (table and value names changed):
```sql
SELECT count(*) AS n, count(DISTINCT id) AS ids, bit_xor(xxhash64(id)) AS
id_hash
FROM db.events
WHERE name IN (<27 event names>)
AND server_created_at >= CAST('2026-11-04' AS DATE) AND server_created_at
< CAST('2026-11-05' AS DATE)
AND app_id = '<one app>' AND platform = 'android'
```
Expected (vanilla Spark, and Gluten with `batchscan=false`): `n =
39,125,071`.
**Table**
- Iceberg 1.10.0, Glue catalog, `S3FileIO`, data on S3. Partition spec:
`day(server_created_at)`, `bucket(app_id)`.
- No delete files in the partition (position or equality).
- The Velox Iceberg doc marks partitioned tables as "Fallback mostly". Here
the scan of a table partitioned by `day` and `bucket` is offloaded
(`IcebergScanTransformer`), and it returns wrong results without a fallback or
an error.
- The matching rows are in 15 data files of one partition (the partition has
83 data files, 292 M rows, 19.4 GB).
- In our first analysis, all lost rows came from one file: 3,668,268 rows,
520 MB, 5 row groups, `split_offsets` = `[4, 132840796, 259353067, 386848326,
513486252]`. With the default 128 MB split size, Iceberg plans this file as
several file scan tasks. But a split size of 1 GB, so that no file is cut into
several tasks, does not stop the loss (see below).
**Results** (same query, executors with 3 cores, 5g heap, 10g off-heap)
| Setup | Runs | Missing rows per run |
|---|---|---|
| 1.8.0-SNAPSHOT nightly (2026-10-27), native Iceberg scan | 11 | −76, −211,
−18, −433, −32, −38, −1, 0, −80, −64, −38 |
| same, `prefetchRowGroups=0` | 5 | −66, −64, 0, −543, −85 |
| same, `maxCoalescedDistance=0B` | 5 | 0, −122, −9, −57, −320 |
| same, `loadQuantum=8MB` | 5 | −62, −270, −203, −112, 0 |
| same, `fileHandleCacheEnabled=false` | 5 | −61, −152, −402, −9, −245 |
| same, `IOThreads=0` and `SplitPreloadPerDriver=0` (the 8.2 B row job) | 3
| −398, −146, −49 |
| same, read option `split-size` = 1 GB (no file is cut: 4 tasks instead of
39) | 5 | −420, −416, −63, −423, −142 (control in the same application, default
split size: −104, 0, −324, −70, −195) |
| 1.7.0-SNAPSHOT nightly (2026-06-18), Graviton3 nodes only | 5 | 0, −212,
−323, −38, −18 |
| 1.8.0-SNAPSHOT, `batchscan=false` | 3 | 0, 0, 0 |
| 1.8.0-SNAPSHOT, the same 15 files as plain Parquet, native scan | 5 | 0,
0, 0, 0, 0 (identical to vanilla Spark) |
(The Velox settings are static configs, so each of them ran in its own Spark
application.)
**Other observations**
- The lost rows come in fixed groups: two runs on two different applications
lost the same 9 rows (same count, same `id_hash`). A 1.7.0-SNAPSHOT run and a
1.8.0-SNAPSHOT run lost the same 38 rows. Only which groups go missing changes
between runs.
- The lost rows are close together: most of them are in a narrow range of
`user_id` values.
- The issue is in 1.7.0-SNAPSHOT (June 2026) too, so it is not a recent
regression. We did not find a related fix on `main` up to the 2026-11-06
nightly.
- On the plain Parquet path, the count differs from the Iceberg count by
7,083 rows also with Gluten off. We did not investigate this difference: the
comparison above is always Gluten against vanilla Spark on the same path.
- Possibly related: #12545 (silent data loss when preload threads fail). But
`IOThreads=0` does not fix our case.
### Gluten version
main branch
(Nightly jars from nightlies.apache.org/gluten/nightly-release-jdk17:
`gluten-velox-bundle-spark3.5_2.12-linux_aarch64-1.8.0-SNAPSHOT.jar` built
2026-10-27T02:24:53Z, sha256
`f7b1b71a849a56662284e5f5670baaa75c0a1c4a64dbac20415e6e6908b1e166`, and
`...-1.7.0-SNAPSHOT.jar` built 2026-06-18T01:03:52Z, sha256
`05795d60495b901b3888561240440d528ffecee8dc57af4439fac8cd5b22008a`.)
### Spark version
Spark-3.5.x
### Spark configurations
```
spark.plugins=org.apache.gluten.GlutenPlugin
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=10g
spark.executor.memory=5g
spark.executor.memoryOverhead=6g
spark.executor.cores=3
spark.driver.defaultJavaOptions=-Dio.netty.tryReflectionSetAccessible=true
spark.executor.defaultJavaOptions=-XX:+UseTransparentHugePages
-Dio.netty.tryReflectionSetAccessible=true -XX:TrimNativeHeapInterval=5000
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.<cat>=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<cat>.type=glue
spark.sql.catalog.<cat>.io-impl=org.apache.iceberg.aws.s3.S3FileIO
spark.sql.shuffle.partitions=1000
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.minExecutors=80
spark.dynamicAllocation.maxExecutors=200
spark.decommission.enabled=true
spark.storage.decommission.enabled=true
```
### System information
- Apache Spark 3.5.6 on Kubernetes (EKS), Iceberg 1.10.0
(`iceberg-spark-runtime-3.5_2.12`), Java 17 (Temurin 17.0.15), Ubuntu 22.04
base image.
- linux/aarch64: AWS Graviton executors (r6gd, r7gd, r8gd). The
1.7.0-SNAPSHOT runs used r7gd only, because that arm64 jar needs SVE.
- All runs are on linux/aarch64, with the official nightly jars (the
releases have no aarch64 jar). We could not test x86. The plain Parquet read on
the same aarch64 jar is exact.
### Relevant logs
No error or warning. All tasks succeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]