markhoerth opened a new issue, #14461:
URL: https://github.com/apache/gravitino/issues/14461
### Version
main branch
### Describe what's wrong
`POST /v1/{prefix}/namespaces/{ns}/tables/{table}/plan`:
1. An incremental scan includes the start snapshot's files. The REST spec
defines `start-snapshot-id` as exclusive, and Iceberg's REST client sends it
from the exclusive `appendsBetween`/`appendsAfter`. A start equal to the end
returns that snapshot's files instead of nothing.
(`CatalogWrapperForREST.createFilePlanScanTasks` uses `fromSnapshotInclusive`.)
2. A filter on an unknown column, with the wrong case while `case-sensitive`
is true, or with a value of the wrong type returns 500 `RuntimeException` with
the table's whole schema struct in the message. The `ValidationException`
raised while files are planned is wrapped into a `RuntimeException`. An unknown
`stats-fields` name returns a NullPointerException text.
3. The spec's constant filters `{"type": "true"}` and `{"type": "false"}`
are refused with 400 "Cannot parse missing field: term"; Iceberg's parser
accepts only JSON booleans.
4. On a v3 table, deletion vectors stored in one Puffin file all resolve to
the last one in the plan response, so a client applies the wrong deletion
vector: deleted rows come back and live rows disappear. Iceberg's
`TableScanResponseParser.serializeScanTasks` indexes delete files by location
only.
### Error message and/or stacktrace
```
plan {"filter":{"type":"eq","term":"nope","value":1}} 500
RuntimeException "Scan planning failed ... Cannot find field 'nope' in struct:
struct<...>"
plan {"filter":{"type":"true"}} 400 "Cannot parse
missing field: term"
```
### How to reproduce
1. Append three snapshots S1, S2, S3 to a table and plan with
`start-snapshot-id` S1 and `end-snapshot-id` S3: S1's files are returned.
2. Plan with a filter on a column the table does not have: 500.
3. Plan with `{"filter":{"type":"true"}}`: 400.
4. On a v3 table, delete rows from several data files in one Spark DELETE,
then plan: tasks reference the wrong deletion vector.
### Additional context
Item 4 is a defect in Iceberg's response serializer; the server should
refuse such a plan until Iceberg's format can reference deletion vectors that
share a file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
