chenwyi2 opened a new issue, #17427:
URL: https://github.com/apache/iceberg/issues/17427
### Apache Iceberg version
1.10.1
### Query engine
Starrocks
### Please describe the bug 🐞
### Query engine
REST Catalog server-side scan planning.
### Please describe the bug
The reference implementation of REST server-side scan planning keeps all
planned
`FileScanTask` objects in the singleton `InMemoryPlanningState`, even after
clients
have successfully fetched every plan task.
As a result, repeated successful scans cause retained heap usage to grow
approximately linearly with the number of planned files.
The retained state is stored in the following maps:
Map<String, List<FileScanTask>> planTaskToFileScanTasks;
Map<String, String> planTaskToNext;
Map<String, PlanStatus> asyncPlanningStates;
The normal successful fetch path only reads these maps:
List<FileScanTask> fileScanTasks =
IN_MEMORY_PLANNING_STATE.fileScanTasksForPlanTask(planTask);
return FetchScanTasksResponse.builder()
.withFileScanTasks(fileScanTasks)
.withPlanTasks(IN_MEMORY_PLANNING_STATE.nextPlanTask(planTask))
.withSpecsById(table.specs())
.build();
Neither fileScanTasksForPlanTask nor nextPlanTask removes the fetched
entries.
The initial task returned by synchronous planTableScan or asynchronous
fetchPlanningResult is also retained.
Currently, the state is released only when cancelPlanning calls
InMemoryPlanningState.cancelPlan.
However, according to the REST Catalog OpenAPI specification, cancellation
is not
required after scan tasks have been fetched for every plan task. This
implies that a
successful fetch lifecycle must eventually release fetched task state without
requiring an explicit cancellation request.
Specification:
https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml
Relevant implementation:
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/InMemoryPlanningState.java
Impact
In a long-running REST catalog deployment based on this implementation, we
observed:
Heap usage continuously increasing after successful scans
Tens of millions of retained FileScanTask objects
More than 20 GB of heap retained by planning state
Full GC unable to reclaim the objects
No active scans corresponding to most retained tasks
No cancellation requests because all plan tasks had already been fetched
successfully
In one observation window, approximately 25 million FileScanTask objects were
retained after about 22 hours of operation.
This may eventually cause excessive GC pressure or an OutOfMemoryError.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
