CarlosJuncher03 commented on issue #7908: URL: https://github.com/apache/hop/issues/7908#issuecomment-5281364368
In my case, the dependency is related to the result of the preparation rather than simply whether a specific workflow has executed. I have a sequential data-loading process that combines data extracted from an API with reference data extracted from a database. The API provides the entities that will be transformed and loaded into the destination, while the database extraction provides reference data used for key lookups, domain validation, and business-rule validation. During the workflow, these datasets are extracted and stored in S3. A later pipeline uses DuckDB/SQL to compare and validate this data and decide whether each record should be inserted, updated, or deleted. The important part is that this pipeline cannot safely run in isolation. For example, there is an initial pipeline that generates the reference/base dataset used by the subsequent validation pipelines. If someone executes the DuckDB validation pipeline directly before that base has been generated — or using a base produced by a previous execution — the pipeline may produce incorrect insert/update/delete decisions. So I agree that checking only whether a workflow or pipeline "has executed" could become complicated because execution state and time windows would also need to be considered. Expressing this as required preparation results may actually be a better abstraction. In this scenario, the requirement could be something like: * the required reference datasets must exist; * they must belong to the same execution/load cycle; * they must be complete and valid; * optionally, some execution identifier or metadata could prove that the preparation phase completed successfully. Then the pipeline would not necessarily depend on a specific workflow filename. Instead, it would declare the conditions or artifacts that must be satisfied before execution is allowed. The workflow would still be responsible for orchestration, but the pipeline could protect itself against being executed when its required preparation state is not available. I think this would address the original problem while avoiding turning Hop into a dependency scheduler. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
