bamaer opened a new pull request, #9457:
URL: https://github.com/apache/hop/pull/9457

   Items 1 and 2 of #9323: Hop reads dataset metadata from a data catalog, and 
a workflow can check datasets before it goes on.
   
   - A `data-catalog` metadata type with a pluggable catalog type, like 
execution information locations. DataHub is the first type, in the DataHub 
plugin: URL, token, environment, timeouts, trust store, and a Test button in 
the editor.
   - A new action, **Check data catalog**: for a list of tables, files or 
catalog ids it checks that the catalog has them, that they aren't deprecated, 
that their data changed within a maximum age, and that their latest assertion 
runs passed. A failed check is a false result with 0 errors; a configuration 
problem is an error. What happens when the catalog can't answer is an option.
   - Tables and files resolve to the same dataset ids the lineage sinks write, 
through the engine's lineage identity code.
   
   The API (`org.apache.hop.datacatalog`) and the action are marked 
experimental.
   
   Points worth checking:
   - The DataHub queries use fields DataHub has had since 1.0 (checked against 
the 1.0, 1.7 and 1.8 schemas).
   - Freshness comes from DataHub operations. The DataHub lineage sink doesn't 
send those yet, so a table that only Hop writes has no last change; follow-up 
for #9324.
   - The Data Catalog type is in the default client; DataHub comes with the 
full client or the marketplace.
   
   Tests: unit tests in engine, the DataHub plugin and the action. IT 
`datacatalog` runs the action against a DataHub GraphQL stub (8 cases, plus a 
check of what was asked). Also tried against a local DataHub 1.7.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to