kotwal-itpro commented on PR #8819: URL: https://github.com/apache/hop/pull/8819#issuecomment-6088621284
Thanks @mattcasters, and thanks for approving #8795. Fixed in 23af45a90f: **Commit vs. pipeline status.** You're right that `setRunning(true)` came too late. The commit listener is now inserted at index 0 of the live `getExecutionFinishedListeners()` list (it's registered from `processRow`, after `startThreads` put the built-in listener first), so the commit runs while `getStatus()` is still `Running`, before the built-in listener flips the status and frees VFS. The `setRunning` workaround is gone; `setErrors` stays. `statusIsNotFinishedBeforeTheSnapshotExists` polls `getStatus()` from another thread while 3 copies write 3,000 rows, and checks the snapshot exists the first time it sees `Finished`. On the previous commit it saw `Finished` with no snapshot on every run. **Overwrite of an empty table.** `validateNoConflictingData()` is now always set for overwrites (and `validateNoConflictingDeletes()` for partition replaces); `validateFromSnapshot` only when there was a starting snapshot, as you described. `overwriteOfAnEmptyTableDoesntRemoveAConcurrentFirstAppend` creates a table without snapshots, has another writer append its first snapshot during an Overwrite run, and checks the run fails and that snapshot is kept. It failed before the change. **Hard link and copy path.** On the `Paths.get` part: in commons-vfs2 2.10, which Hop uses, `FileObject.getURI()` returns a `java.net.URI`, so this is `Paths.get(URI)`, and the link does get created. `localPublishUsesAHardLink` checks that the published `v<N>.metadata.json` has `unix:nlink == 2`. Your other points were right though, and are fixed: - Only missing hard-link support falls through now (`UnsupportedOperationException`, or a `FileSystemException` with no existing target). `NoSuchFileException` and other errors fail the commit. - A local file system without hard links no longer uses `getOutputStream`: it publishes with `Files.newOutputStream(target, CREATE_NEW)`, an exclusive create, so it's atomic too, and a partial file from a failed copy is deleted. - On non-local file systems the version file is still written after an `exists` check (single-writer, as documented), and a partial file from a failed write is deleted; if that delete fails, it's a `CommitStateUnknownException`. **Legacy gzip name.** `v<N>.metadata.json.gz` is probed in the forward walk and matched by `metadataVersion`; `versionHintIsAStartingPoint` now includes it. Tests: lakehouse 91, Spark 179, RAT clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
