kotwal-itpro commented on PR #8819: URL: https://github.com/apache/hop/pull/8819#issuecomment-6087616401
Thanks again @mattcasters, all three were real. Fixed in db38765528 (rebased onto main): **`moveTo` on non-local file systems, and the local fallback.** Nothing in a commit ever deletes or replaces an existing version file now. The hard link is the only atomic route, on local file systems that support it. Everywhere else, including HDFS, object stores and local file systems without hard links, `publish` checks that `v<N>.metadata.json` doesn't exist and then writes the version file directly from the temp file, instead of `moveTo` (which deletes an existing destination) or `Files.move` (which ends in a replacing `rename`). A failed write there is reported as `CommitStateUnknownException`, since the file may already be visible. The class Javadoc and the docs PR now say path tables are single-writer on every non-local file system, HDFS included. I didn't add HDFS non-overwriting rename or S3 conditional puts; I think those belong in a follow-up with real HDFS and S3 tests. `nonLocalPublishNeverReplacesAVersion` checks on Hop's `ram://` that an existing version is left al one and the commit fails, and `pathTableOnANonLocalFileSystem` runs a create and an append there. **Commit after the pipeline is marked finished.** As you suggested: for the duration of the commit the pipeline reports itself as running again (`setRunning(true)` on entry, `setRunning(false)` in a `finally`), so a status poll sees `Running` rather than `Finished` until the snapshot attempt has ended, and a failed commit is counted with `setErrors` on the transform before the listener returns, so it's part of the result before `pipelineCompleted()` releases the waiters. `failedCommitIsAnErrorWhenThePipelineFinishes` makes an Overwrite fail validation (another writer appends while the run is writing) and checks `getErrors() > 0` once `waitUntilFinished()` returns. It reported 0 errors before this change. **Coalesce and extra options.** When either is set, the local writer now logs that it only applies to the Spark engine and is ignored. There are no table properties on the transform today, so `Map.of()` doesn't drop anything the user set; if we add them, the local writer should pass them through. Tests: lakehouse 88, Spark 179, RAT clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
