comphead opened a new issue, #25433:
URL: https://github.com/apache/datafusion/issues/25433
### Describe the bug
For one Parquet file, `SELECT *` returns 6 rows and `SELECT count(*)`
returns 3. The scan and the
count disagree with each other in the same session.
The file has a row group that declares `num_rows = 3`, while each of its two
column chunks declares
`num_values = 6`. `count(*)` is answered from the row group metadata and the
scan is driven by the
column chunks, and the two are never reconciled. Wrapping the scan in a
subquery does not change
the count.
Whatever the right row count for such a file is, DataFusion should not
report two different answers
for it.
### To Reproduce
DataFusion 55.0.0 (`datafusion-cli 55.0.0`).
The file is attached below. It is 853 bytes, holds a single column `s` of
type
`struct<dup: bigint, dup: bigint>`, and was written by Spark 4.1.3 with:
```scala
spark.range(3)
.selectExpr("named_struct('dup', id, 'dup', id + 100) as s")
.repartition(1)
.write.mode("overwrite").parquet("/tmp/dup-struct")
```
Its footer schema and column chunk metadata:
```
message spark_schema {
required group s {
required int64 dup;
required int64 dup;
}
}
path_in_schema num_values row_group_num_rows
s, dup 6 3
s, dup 6 3
```
Then, in `datafusion-cli`:
```sql
DESCRIBE SELECT * FROM '/tmp/dup-struct.parquet';
+-------------+------------------------------------------------------+-------------+
| column_name | data_type |
is_nullable |
+-------------+------------------------------------------------------+-------------+
| s | Struct("dup": non-null Int64, "dup": non-null Int64) | NO
|
+-------------+------------------------------------------------------+-------------+
SELECT s['dup'] AS dup FROM '/tmp/dup-struct.parquet';
+-----+
| dup |
+-----+
| 0 |
| 100 |
| 1 |
| 101 |
| 2 |
| 102 |
+-----+
6 row(s) fetched.
SELECT count(*) AS n FROM '/tmp/dup-struct.parquet';
+---+
| n |
+---+
| 3 |
+---+
SELECT count(*) AS n FROM (SELECT * FROM '/tmp/dup-struct.parquet');
+---+
| n |
+---+
| 3 |
+---+
```
### Expected behavior
`count(*)` and the scan agree on the number of rows in the file.
### Additional context
For reference, two other readers return 3 rows for the same file and the
same projection:
| Reader | rows | `s.dup` values |
| --- | --- | --- |
| DataFusion 55.0.0 | 6 | `0, 100, 1, 101, 2, 102` |
| Spark 4.1.3 | 3 | `0, 100, 1` |
| DuckDB 1.1.2 | 3 | `0, 100, 1` |
Both of those clamp to the row group's `num_rows`, which is also the number
`count(*)` reports here.
A file written by DataFusion itself for the same logical data is consistent
(`num_values = 3` on
each chunk) and reads back as 3 rows, so this only shows up for a file whose
chunk `num_values`
exceeds the row group's `num_rows`:
```sql
COPY (SELECT named_struct('dup', v, 'dup', v + 100) AS s FROM (VALUES
(0),(1),(2)) t(v))
TO '/tmp/df-written.parquet' STORED AS PARQUET;
SELECT s['dup'] FROM '/tmp/df-written.parquet'; -- 3 rows: 0, 1, 2
SELECT count(*) FROM '/tmp/df-written.parquet'; -- 3
```
<details>
<summary>The attached Parquet file, base64 (853 bytes, md5
27aa66dd2b78cdb47168ecf004c8326b)</summary>
Decode with:
```sh
base64 -d > /tmp/dup-struct.parquet <<'EOF'
UEFSMRUAFWAVRBWEprnjBRwVDBUAFQgVCAAAMAAADQEAZA0IAAENCABlDQg8AgAAAAAAAABmAAAAAAAAABUAFWAVRBWEprnjBRwV
DBUAFQgVCAAAMAAADQEAZA0IAAENCABlDQg8AgAAAAAAAABmAAAAAAAAABkRAhkYCAAAAAAAAAAAGRgIZgAAAAAAAAAVAhkWAAAZ
EQIZGAgAAAAAAAAAABkYCGYAAAAAAAAAFQIZFgAAGRwWCBVyFgAAABkcFnoVchYAAAAVAhlMSAxzcGFya19zY2hlbWEVAgA1ABgB
cxUEABUEJQAYA2R1cAAVBCUAGANkdXAAFgYZHBksJgAcFQQZJQgAGSgBcwNkdXAVAhYMFo4BFnImCDwYCGYAAAAAAAAAGAgAAAAA
AAAAABYAKAhmAAAAAAAAABgIAAAAAAAAAAAAGRwVABUAFQIAPAAAFugCFRQW7AEVPgAmABwVBBkFGSgBcwNkdXAVAhYMFo4BFnIm
ejwYCGYAAAAAAAAAGAgAAAAAAAAAABYAKAhmAAAAAAAAABgIAAAAAAAAAAAAGQw8AAAW/AIVFBaqAhU+ABacAhYGJggW5AEUAAAZ
LBgYb3JnLmFwYWNoZS5zcGFyay52ZXJzaW9uGAU0LjEuMwAYKW9yZy5hcGFjaGUuc3Bhcmsuc3FsLnBhcnF1ZXQucm93Lm1ldGFk
YXRhGOQBeyJ0eXBlIjoic3RydWN0IiwiZmllbGRzIjpbeyJuYW1lIjoicyIsInR5cGUiOnsidHlwZSI6InN0cnVjdCIsImZpZWxk
cyI6W3sibmFtZSI6ImR1cCIsInR5cGUiOiJsb25nIiwibnVsbGFibGUiOmZhbHNlLCJtZXRhZGF0YSI6e319LHsibmFtZSI6ImR1
cCIsInR5cGUiOiJsb25nIiwibnVsbGFibGUiOmZhbHNlLCJtZXRhZGF0YSI6e319XX0sIm51bGxhYmxlIjpmYWxzZSwibWV0YWRh
dGEiOnt9fV19ABhKcGFycXVldC1tciB2ZXJzaW9uIDEuMTYuMCAoYnVpbGQgNDAyYzM4MTBjMzcyZDI5NjAzZTE4MTc3MWFjZWJm
ZWNjNzFiZWY2MSkZLBwAABwAAACFAgAAUEFSMQ==
EOF
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
