j1wonpark opened a new issue, #4113:
URL: https://github.com/apache/amoro/issues/4113
### What happened?
When a table's `self-optimizing.group` is changed while an optimizing
process is running, the process is forcefully closed via
`completeProcess(false)`. However, `completeProcess()` unconditionally updates
the `lastOptimizedSnapshotId` and `lastOptimizedChangeSnapshotId` snapshot
markers regardless of the `success` parameter.
This causes `SchedulingPolicy.isTablePending()` to return `false` (because
`lastOptimizedSnapshotId == currentSnapshotId`), so the scheduler permanently
skips the table. The table remains stuck in PENDING status until new data
arrives and changes the current snapshot.
**Expected behavior:** When a process is closed without success
(`completeProcess(false)`), the snapshot markers should not be updated since no
actual optimization was performed. The table should be eligible for re-planning
in the new optimizer group.
**Actual behavior:** The snapshot markers are updated even on failure,
causing the scheduler to believe the table has already been optimized at the
current snapshot, permanently skipping it.
### Affects Versions
master
### What table formats are you seeing the problem on?
Iceberg, Mixed-Iceberg, Mixed-Hive
### What engines are you seeing the problem on?
AMS
### How to reproduce
1. Create a table and insert data so that optimization is triggered
2. Wait for the table to enter an optimizing state (MINOR_OPTIMIZING,
MAJOR_OPTIMIZING, etc.) with a running `optimizingProcess`
3. Change the optimizer group: `ALTER TABLE ... SET TBLPROPERTIES
('self-optimizing.group' = '<another_group>')`
4. Ensure the new group has active optimizers
5. Observe that the table transitions to PENDING but never progresses to
PLANNING
6. The table stays stuck in PENDING indefinitely until new data arrives
This also affects other `completeProcess(false)` paths such as task retry
exhaustion and process failures.
### Relevant log output
```shell
```
### Anything else
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
