bamaer opened a new pull request, #9341:
URL: https://github.com/apache/hop/pull/9341
Builds on #9339 and #9340; the last commit is the new one.
A lineage sink that writes DataHub aspects to GMS directly, not through
DataHub's
OpenLineage endpoint (which replaces the producer, ingests asynchronously
and makes a
flow per transform).
- Pipelines and workflows are DataFlows, transforms and actions DataJobs in
hop order,
each run a DataProcessInstance with its status and parent run.
- Dataset URNs match DataHub's own sources (files, S3/GCS/ABS, databases),
with schemas,
column lineage and copy edges.
- Transforms and actions removed from a pipeline or workflow are
soft-deleted in DataHub.
- Uses the OpenLineage sink's spool and retries; the token can be encrypted.
- Configured with `HOP_LINEAGE_DATAHUB_*` variables; manual page under
Technology > DataHub.
Tested live against DataHub 1.0, 1.7 and 1.8 with token auth: nested
workflows, a
Pipeline Executor with 500 rows, a DataHub outage during a run, renamed
transforms.
43 unit tests.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [x] Run `mvn clean install apache-rat:check` to make sure basic checks
pass. A more thorough check will be performed on your pull request
automatically.
- [x] If you have a group of commits related to the same change, please
squash your commits into one and force push your branch using `git rebase -i`.
- [x] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable.
To make clear that you license your contribution under the [Apache License
Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
you have to acknowledge this by using the following check-box.
- [x] I hereby declare this contribution to be licensed under the [Apache
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
- [ ] In any other case, please file an [Apache Individual Contributor
License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]