[
https://issues.apache.org/jira/browse/NIFI-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218124#comment-16218124
]
Aidan Cope commented on NIFI-3709:
----------------------------------
I have been working on some enhancements to the work that was being done by
Koji here:
https://github.com/ijokarumawak/nifi/tree/nifi-3709-2/nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task
The work in progress can be seen here:
https://github.com/ikethecoder/nifi/tree/nifi-3709-2/nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task
Would like to get feedback on the proposed changes I am doing. It is for a
client, and their requirements require quite a detailed tracking of provenance
data from NiFi to Atlas. To this end, I added a parameter to the reporter task
to indicate the granularity of the Atlas processes - with the two options being
"By Flow Path" and "By Flow File". By Flow Path is the existing approach by
Koji. The "By Flow File" uses the File UUID as the qualifier name for the
process, rather than the component Id. The client wants to be able to track
files through the complete NiFi Flow over time, which can not be easily done
using the way it was implemented. I think both use strategies are valid,
depending on the use case and requirement - which is why I made it as a
parameter on the reporting task.
But would really appreciate feedback on whether this is a viable approach and
if it can be considered for a pull request?
> Export NiFi flow dataset lineage to Apache Atlas
> ------------------------------------------------
>
> Key: NIFI-3709
> URL: https://issues.apache.org/jira/browse/NIFI-3709
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
>
> While Apache NiFi has provenance and event level lineage support within its
> data flow, Apache Atlas also does manage lineage between dataset and process
> those interacting with such data.
> It would be beneficial for users who use both NiFi and Atlas and if they can
> see end-to-end data lineage on Atlas lineage graph, as some type of dataset
> are processed by both NiFi and technologies around Atlas such as Storm,
> Falcon or Sqoop. For example, Kafka topics and Hive tables.
> In order to make this integration happen, I propose a NiFi reporting task
> that analyzes NiFi flow then creates DataSet and Process entities in Atlas.
> The challenge is how to design NiFi flow dataset level lineage within Atlas
> lineage graph.
> If we just add a single NiFi process and connect every DataSet from/to it, it
> would be too ambiguous since it won't be clear which part of a NiFi flow
> actually interact with certain dataset.
> But if we put every NiFi processor as independent process in Atlas, it would
> be too granular, too. Also, we already have detailed event level lineage in
> NiFi, we wouldn't need the same level in Atlas.
> If we can group certain processors in a NiFI flow as a process in Atlas, it
> would be a nice granularity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
