bamaer opened a new pull request, #9545:
URL: https://github.com/apache/hop/pull/9545

   Items 1, 2 and 4–9 of #9324 (item 3 was #9479), one commit per item, 
followed by fixes from review.
   
   - **Lineage sinks as metadata (1).** A `lineage-sink` metadata type, 
OpenLineage or DataHub, selected in a pipeline or workflow run configuration. A 
child run also reports to its parent's sink. The DataHub sink takes its URL, 
token and environment from a DataHub data catalog. The `HOP_LINEAGE_*` 
variables work as before.
   - **Sink health (2).** Tools > Lineage sinks in Hop GUI, and a section on 
the Hop Server status page: events sent, refused, waiting and spooled, and the 
last error.
   - **Transformation logic (4).** The SQL of Table input and Execute SQL goes 
to the OpenLineage `sql` facet and to DataHub's transformation logic. Formula 
and Calculator expressions become task properties `logic.<field>`, because 
DataHub only takes SQL there.
   - **Platform instance per connection (5).** The DataHub data catalog maps 
relational connections to platform instances. Their tables are then named 
`<instance>.<db>.<schema>.<table>`, as DataHub's SQL sources name them.
   - **Documentation (6).** Publishing can include what `hop doc` writes, as 
DataHub documents (1.7 and later). They are related to the pipeline and its 
tasks, under a document for the project. Parameters and notes are only included 
when asked for.
   - **Publishing without running (7).** `hop datahub-publish` and a new 
**Publish to DataHub** action publish pipelines and workflows: their tasks, the 
tables and files they declare, and file layouts as schemas. Names resolve as a 
run of each file resolves them, and no database is queried.
   - **Unit tests as assertions (8).** Each golden data set becomes an 
assertion run (passed, failed or error) on the datasets the tested transform 
writes, and an OpenLineage `hopUnitTest` run facet. The test's own run, of a 
modified copy of the pipeline, doesn't go to sinks selected in a run 
configuration.
   - **AI Assistant (9).** A sharing option that adds what a data catalog says 
about the pipeline's tables and files: descriptions, owners, tags, glossary 
terms and deprecation. The catalog is read in the background.
   
   Tested with unit tests in the engine and the affected plugins, and with a 
new IT project, `integration-tests/lineage-sinks`, which runs a DataHub 
stand-in in the container. It covers sink routing and inheritance, publishing, 
documents, unit test assertions, transformation logic and platform instances. 
The `openlineage` IT passes too, and I checked against a live DataHub 1.7.
   
   Not in this PR:
   - Child workflows of a unit test run still report their lineage.
   - Results of the Execute tests transform don't reach run-configuration sinks.
   
   Fixes #9324
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to