Hello, I'm researching tools to store and visualize data lineage to help debug issues in my ETL pipeline, for instance, why a particular piece of data got filtered out prior to loading into the database or why it didn't get merged into an output CSV file. Does Atlas have any limitations in visualizing or searching lineage where a data rows has either been split or merged from/to a large batch file containing like 500,000 rows and I want to find only the rows that got filtered in a certain direction?
Also would it be possible to join event streams from NiFi provenance with higher level business process events that I send to Atlas through Kafka to get a full picture of what happened and also to take advantage of the split, route and merge events that are already there? Thanks, Eric
