kokila-19 opened a new pull request, #5566:
URL: https://github.com/apache/hive/pull/5566
…query execution.
When concurrent write fails in iceberg, the query is retried. On retry, the
previous execution's commit info in resourceMap is
available which leads to FileNotFoundException.
### What changes were proposed in this pull request?
Reset the resourceMap in Querystate in releaseContext()
### Why are the changes needed?
When concurrent write fails in iceberg, the query might fail with 'Found
conflicting files'. This typically occurs when two or more operations conflict
over writing to the same table, leading to a situation where the write cannot
be completed successfully due to file-level conflicts.
When this error occurs, the query is retried based on the retry strategies
defined.
During retry, in prepareContext(),
```
if (!driverContext.isRetrial()) {
closeInProcess(false);
}
```
closeInProcess is called which in return invokes releaseContext() which
removes the queryState from SessionState.
However, the driverContext still holds the same queryState, and this
queryState is added back to SessionState.
```
if (SessionState.get() != null) {
QueryState queryState = getQueryState();
SessionState.get().addQueryState(queryState.getQueryId(), queryState);
}
```
When the query is retried, depending on the tez session pool's availability
, it can be assigned a different Tez session than the one used during the
previous execution. If the application ID is different than the previous
execution, then job ID will also change. If we don't clear the resourceMap , it
will contain both old + new job ID. This will lead to FileNotFoundException
when the files of old job id is accessed.
### Does this PR introduce _any_ user-facing change?
No
### Is the change a dependency upgrade?
No
### How was this patch tested?
Tested Manually in a cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
