bamaer opened a new pull request, #9341:
URL: https://github.com/apache/hop/pull/9341

   Builds on #9339 and #9340; the last commit is the new one.
   
   A lineage sink that writes DataHub aspects to GMS directly, not through 
DataHub's
   OpenLineage endpoint (which replaces the producer, ingests asynchronously 
and makes a
   flow per transform).
   
   - Pipelines and workflows are DataFlows, transforms and actions DataJobs in 
hop order,
     each run a DataProcessInstance with its status and parent run.
   - Dataset URNs match DataHub's own sources (files, S3/GCS/ABS, databases), 
with schemas,
     column lineage and copy edges.
   - Transforms and actions removed from a pipeline or workflow are 
soft-deleted in DataHub.
   - Uses the OpenLineage sink's spool and retries; the token can be encrypted.
   - Configured with `HOP_LINEAGE_DATAHUB_*` variables; manual page under 
Technology > DataHub.
   
   Tested live against DataHub 1.0, 1.7 and 1.8 with token auth: nested 
workflows, a
   Pipeline Executor with 500 rows, a DataHub outage during a run, renamed 
transforms.
   43 unit tests.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   - [x] Run `mvn clean install apache-rat:check` to make sure basic checks 
pass. A more thorough check will be performed on your pull request 
automatically.
   - [x] If you have a group of commits related to the same change, please 
squash your commits into one and force push your branch using `git rebase -i`.
   - [x] Mention the appropriate issue in your description (for example: 
`addresses #123`), if applicable.
   
   To make clear that you license your contribution under the [Apache License 
Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
   you have to acknowledge this by using the following check-box.
   
   - [x] I hereby declare this contribution to be licensed under the [Apache 
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
   - [ ] In any other case, please file an [Apache Individual Contributor 
License Agreement](https://www.apache.org/licenses/icla.pdf).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to