krutoileshii commented on issue #3462:
URL: https://github.com/apache/polaris/issues/3462#issuecomment-3762590928
Could this be coming from here? If FileIO fails's wouldn't this still hold
connection for manifest?
```
private Stream<TaskEntity> getManifestTaskStream(
TaskEntity cleanupTask,
TableMetadata tableMetadata,
FileIO fileIO,
IcebergTableLikeEntity tableEntity,
PolarisMetaStoreManager metaStoreManager,
PolarisCallContext polarisCallContext) {
// read the manifest list for each snapshot. dedupe the manifest files
and schedule a
// cleanupTask
// for each manifest file and its data files to be deleted
return tableMetadata.snapshots().stream()
.flatMap(sn -> sn.allManifests(fileIO).stream())
// distinct by manifest path, since multiple snapshots will contain
the same
// manifest
.collect(Collectors.toMap(ManifestFile::path, Function.identity(),
(mf1, mf2) -> mf1))
.values()
.stream()
.filter(mf -> TaskUtils.exists(mf.path(), fileIO))
.map(
mf -> {
// append a random uuid to the task name to avoid any
potential conflict
// when
// storing the task entity. It's better to have duplicate
tasks than to risk
// not storing the rest of the task entities. If a duplicate
deletion task
// is
// queued, it will check for the manifest file's existence and
simply exit
// if
// the task has already been handled.
String taskName = cleanupTask.getName() + "_" + mf.path() +
"_" + UUID.randomUUID();
LOGGER
.atDebug()
.addKeyValue("taskName", taskName)
.addKeyValue("tableIdentifier",
tableEntity.getTableIdentifier())
.addKeyValue("metadataLocation",
tableEntity.getMetadataLocation())
.addKeyValue("manifestFile", mf.path())
.log("Queueing task to delete manifest file");
return new TaskEntity.Builder()
.setName(taskName)
.setId(metaStoreManager.generateNewEntityId(polarisCallContext).getId())
.setCreateTimestamp(clock.millis())
.withTaskType(AsyncTaskType.MANIFEST_FILE_CLEANUP)
.withData(
ManifestFileCleanupTaskHandler.ManifestCleanupTask.buildFrom(
tableEntity.getTableIdentifier(), mf))
.setId(metaStoreManager.generateNewEntityId(polarisCallContext).getId())
// copy the internal properties, which will have storage
info
.setInternalProperties(cleanupTask.getInternalPropertiesAsMap())
.build();
});
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
