mattfaltyn opened a new issue, #1557:
URL: https://github.com/apache/iceberg-go/issues/1557
### Apache Iceberg version
`main` at `9097100f4aa3bae419644e3b1f25d3fe29ce8fb7`. The affected code is
also present in v0.6.0.
### Please describe the bug
A filtered Arrow scan can silently drop rows written before a column was
added, even when that column's `InitialDefault` satisfies the filter.
For example, start with a v3 table containing two rows, add optional
`new_col int` with an initial default of `42`, and scan the old file:
- An unfiltered projection returns both rows with `new_col = 42`, as
expected.
- A scan filtered by `new_col == 42` returns no rows and no error.
- `PlanFiles` retains the old data file, so the loss occurs during read-time
filtering rather than file planning.
Expected: both old rows are returned because their projected value is `42`.
Actual: the result is empty.
The following package-local regression test reproduces the problem using the
existing `buildV3TableWithRows` helper:
```go
func collectInitialDefaultScan(
t *testing.T,
scan *Scan,
tasks []FileScanTask,
) (ids []int64, defaults []int32) {
t.Helper()
_, records, err := scan.ReadTasks(t.Context(), tasks)
require.NoError(t, err)
for rec, recErr := range records {
require.NoError(t, recErr)
require.NotNil(t, rec)
if rec.NumRows() > 0 {
idCol := rec.Column(0).(*array.Int64)
defaultCol := rec.Column(1).(*array.Int32)
for i := range int(rec.NumRows()) {
ids = append(ids, idCol.Value(i))
defaults = append(defaults, defaultCol.Value(i))
}
}
rec.Release()
}
return ids, defaults
}
func TestInitialDefaultFilterDropsOldFileRepro(t *testing.T) {
tbl := buildV3TableWithRows(t,
`[{"id":1,"data":"a"},{"id":2,"data":"b"}]`)
txn := tbl.NewTransaction()
require.NoError(t, txn.UpdateSchema(true, false).
AddColumn(
[]string{"new_col"},
iceberg.PrimitiveTypes.Int32,
"",
false,
iceberg.Int32Literal(42),
).
Commit())
var err error
tbl, err = txn.Commit(t.Context())
require.NoError(t, err)
control := tbl.Scan(WithSelectedFields("id", "new_col"))
controlTasks, err := control.PlanFiles(t.Context())
require.NoError(t, err)
require.Len(t, controlTasks, 1)
controlIDs, controlDefaults := collectInitialDefaultScan(t, control,
controlTasks)
require.Equal(t, []int64{1, 2}, controlIDs)
require.Equal(t, []int32{42, 42}, controlDefaults)
matching := tbl.Scan(
WithSelectedFields("id", "new_col"),
WithRowFilter(iceberg.EqualTo(iceberg.Reference("new_col"),
int32(42))),
)
matchingTasks, err := matching.PlanFiles(t.Context())
require.NoError(t, err)
require.Len(t, matchingTasks, 1, "planning retains the old file")
matchingIDs, matchingDefaults := collectInitialDefaultScan(t, matching,
matchingTasks)
t.Logf("control IDs=%v defaults=%v", controlIDs, controlDefaults)
t.Logf("matching-filter IDs=%v defaults=%v", matchingIDs,
matchingDefaults)
require.Equal(t, controlIDs, matchingIDs,
"every old row projects new_col=42 and must satisfy new_col ==
42")
require.Equal(t, controlDefaults, matchingDefaults)
}
```
Required imports:
```go
import (
"testing"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/iceberg-go"
"github.com/stretchr/testify/require"
)
```
Run:
```text
go test ./table -run '^TestInitialDefaultFilterDropsOldFileRepro$' -count=2
-v
```
Observed on both runs:
```text
control IDs=[1 2] defaults=[42 42]
matching-filter IDs=[] defaults=[]
```
### Root cause
[`TranslateColumnNames`](https://github.com/apache/iceberg-go/blob/9097100f4aa3bae419644e3b1f25d3fe29ce8fb7/visitors.go#L499-L508)
treats a field missing from the file schema as null: it folds `IsNull` to true
and every other predicate to `AlwaysFalse`, without consulting the evolved
field's `InitialDefault`.
[`getRecordFilter`](https://github.com/apache/iceberg-go/blob/9097100f4aa3bae419644e3b1f25d3fe29ce8fb7/table/arrow_scanner.go#L728-L740)
turns that `AlwaysFalse` into `dropFile = true`, and this happens [before
projection](https://github.com/apache/iceberg-go/blob/9097100f4aa3bae419644e3b1f25d3fe29ce8fb7/table/arrow_scanner.go#L1113-L1139).
Projection later [materializes the correct
`InitialDefault`](https://github.com/apache/iceberg-go/blob/9097100f4aa3bae419644e3b1f25d3fe29ce8fb7/table/arrow_utils.go#L1090-L1096),
but the file has already been discarded.
### Suggested fix
When translating a bound predicate for a field absent from the file schema,
evaluate the predicate against the field's `InitialDefault` and fold it to the
corresponding constant result. Preserve the existing null behavior only when no
non-null initial default applies.
A regression matrix should cover matching and non-matching predicates, null
predicates, and both old and newly written files.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
