joewitt commented on PR #9171: URL: https://github.com/apache/nifi/pull/9171#issuecomment-2282894218
Understood on Slack. It is well understood about that limitation. Please put the necessary context in the PR or JIRA so reviewers can participate and/or know their comments are being factored in. You can copy/paste from slack. I better understand the intent based on your recent comment. Effectively while the key information is provided in our current provenance event/flowfile model it puts too much strain on the destination database to determine what the lineage graph looked like in terms of ordering. This is because we provide the necessary relationships and timestamps but it would require the downstream database to reconstruct them once all events are completed as timestamp is currently a key indicator of order of operations for a given set of events associated with a single flow file. I can see how this could be beneficial to resolve. What other solutions were considered to address this? The proposed change creates a link by the long identifier of all possible 'previous' events for a given provenance event and adds an alternative linkage over the current model which is purely based on the relationship of a flow file, the type of event, and any parent flowfiles from which it stems. The core issue is ordering within the lifespan of a single flowfile. That seems solvable with a single 'previous event identifier'. In the case the previous event identifier is not established this is the start of this flowfile's life. That could mean it was a created flowfile or it was the result of various different sources being combined into a new child object. We already know and capture the parent flowfile identifier so this chain can be followed by the downstream database. I left a couple comments as it relates to changes in the abstractions as well (Abstract repository, UpdateableEvent...). It isn't clear to me why such changes are desirable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
