bobbai00 opened a new pull request, #5856: URL: https://github.com/apache/texera/pull/5856
### What changes were proposed in this PR? Removes the two `Thread.sleep(500)` waits in `SyncExecutionResource.executeWorkflowSync` and reads results as soon as the execution reaches `COMPLETED`. The wait was redundant: the engine commits each operator's result storage synchronously — `OutputManager.closeOutputStorageWriterIfNeeded` joins the writer thread (forcing the Iceberg commit) before the worker reports `COMPLETED`, and a failed commit surfaces as `FAILED`/`KILLED`, never `COMPLETED`. Adds `ResultPersistedOnCompletionSpec` to lock that invariant. ### Any related issues, documentation, discussions? Closes #5855 ### How was this PR tested? `WorkflowExecutionService/testOnly ...e2e.ResultPersistedOnCompletionSpec` — 2 cases (a single source, and a multi-region scan→keyword→count DAG) that read result storage the instant the run reports `COMPLETED` (no wait) and assert the committed `getCount` equals the rows actually readable. The existing `OutputPortStorageWriterThreadSpec` covers the commit/failure-propagation path. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 (1M context) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
