bamaer opened a new issue, #9323:
URL: https://github.com/apache/hop/issues/9323
### What would you like to happen?
### What would you like to happen?
#9034 sends lineage from Hop to DataHub. This issue is the other direction:
let pipelines
and workflows use what DataHub knows about a dataset. DataHub stays the
owner of that
metadata; Hop only reads it.
1. **DataHub connection metadata type.** A `datahub-connection` type (URL,
token,
environment, platform instance, TLS truststore) with a dialog and a Test
button. The
features below use it. The lineage sink can move to it later; its
variables keep working.
2. **Workflow action: Check DataHub dataset.** Check one or more datasets
before
continuing: latest assertion results, freshness against a maximum age,
and deprecation.
It succeeds or fails like the other check actions. The assertions
themselves stay in DataHub.
3. **Mask fields by DataHub tag or glossary term.** An option in Mask fields
to mask every
field whose column carries a given tag or term (for example `PII`), with
the masking
method chosen per tag. Off by default and selected explicitly. The fields
are resolved
when the transform starts and logged, so each run shows what was masked.
4. **Get fields from DataHub.** A button in transform dialogs with a field
table, starting
with the file inputs: pick a dataset and fill names, types, lengths and
formats from its
schema. Design time only: the pipeline keeps its own field definitions,
so a run never
changes because DataHub changed.
Out of scope: changing metadata in DataHub from Hop, and changing a
pipeline's fields
at run time.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms, Actions, Metadata
### Issue Priority
Priority: 2
### Issue Component
Component: Metadata, Component: Transforms, Component: Actions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
