[
https://issues.apache.org/jira/browse/NIP-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne reassigned NIP-40:
-----------------------------
Assignee: Mark Payne
> Content and FlowFile Repository API Cleanup
> -------------------------------------------
>
> Key: NIP-40
> URL: https://issues.apache.org/jira/browse/NIP-40
> Project: NiFi Improvement Proposal
> Issue Type: Improvement
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> h2. Motivation
> Several small, related gaps in the Content Repository and FlowFile Repository
> API (nifi-framework-api) make it harder than necessary to implement t he
> repository interfaces and to reason about repository behavior. This NIP
> groups a few backward-compatible cleanups to that API.
> # *No component or Connector context on writes.* When a ProcessSession
> creates content or commits FlowFile changes, it calls
> {{ContentRepository.create(...) }}and
> {{{}FlowFileRepository.updateRepository(...){}}}. Neither call tells the
> repository which component the session is acting on behalf of, nor which
> Connector (managed flow) that component belongs to, so the repository cannot
> attribute an operation to a component or organize data by Connector. There
> have been several instances when I wanted to know which component a Content
> Claim was for; e.g., if we know that it's for a Processor that statistically
> will create large FlowFiles we may not want to append its content to an
> existing Resource Claim but instead create a new Resource Claim. This would
> help avoid scenarios where Content Repository holds onto a large file just
> because one small FlowFile shares the same Resource Claim.
> # *The session depends on framework-internal stream classes.*
> {{StandardProcessSession}} reads content through
> {{{}ContentClaimInputStream{}}}, and the write cache reuses claims through
> {{{}ContentClaimOutputStream.newContentClaim(){}}}. These abstractions are
> good and make sense, but both classes live in framework internals
> (nifi-framework-components), not in the more public
> {{{}nifi-framework-api{}}}, so a repository that wants to provide its own
> read stream or claim-reuse behavior has no public abstraction to implement
> and must bind to framework internals.
> # *Content Repository should own the maximum appendable claim size.* The
> claim rollover size is read from the
> {{nifi.content.claim.max.appendable.size}} property and threaded to the write
> cache, while a repository that has its own notion of where a storage unit
> ends has no way to declare it. The two values must be kept consistent by
> hand. While participation from Process Session is necessary for the max
> appendable size to work properly, it should be the responsibility of the
> pluggable {{ContentRepository}} to dictate what max size should be used.
> h2. Proposed Solution
> Three backward-compatible additions to the repository SPI:
> * *Component/Connector context.* Pass a small, immutable context object on
> each create/update describing the component and Connector the session is
> acting for. It is populated by the framework and passed straight through;
> stock repositories ignore it.
> * *Public stream abstractions.* Expose the content-claim input-stream
> abstraction and the output-stream claim-reuse contract in nifi-framework-api,
> and have {{StandardProcessSession}} and the write cache depend on those
> public types rather than the framework-internal classes.
> * *Repository-owned appendable size.* Let {{ContentRepository}} report its
> maximum appendable claim size, and have the framework source the rollover
> size from the repository. The existing property remains the default the stock
> repository reports. All additions are default methods or newly exposed types;
> no existing type or method is removed, changed, or deprecated. The stock
> FileSystemRepository and WriteAheadFlowFileRepository behave exactly as they
> do today.
> h2. Framework API Changes (nifi-framework-api)
> *Component/Connector context*
> * New {{LossTolerance}} enum ({{{}LOSS_TOLERANT{}}},
> {{{}LOSS_INTOLERANT{}}}). While we could keep with the existing {{boolean
> lossTolerant}} the general pattern that NiFi has tended toward is to drop
> booleans in interfaces/APIs in favor of more descriptive enums.
> * New {{ContentClaimCreationContext}} (component id, Connector id, loss
> tolerance) and {{FlowFileUpdateContext}} (component id, Connector id).
> * New {{default}} {{ContentRepository.create(ContentClaimCreationContext)}}
> and {{{}FlowFileRepository.updateRepository(Collection<RepositoryRecord>,
> FlowFileUpdateContext){}}}, each delegating to the existing method.{*}{{*}}
>
> *Public stream abstractions*
> * Relocate {{ContentClaimInputStream}} and {{ContentClaimOutputStream}} from
> {{nifi-framework-components}} to {{nifi-framework-api}}
> *Repository-owned appendable size*
> * New {{default}} {{ContentRepository.getMaxAppendableClaimBytes()}}
> returning the configured default (so stock behavior is unchanged); the
> framework sources the rollover size from the repository. The value must be a
> cheap in-memory read,
> since it is consulted on the write hot path. The existing
> {{{}create(boolean){}}}, {{{}updateRepository(Collection){}}}, and the
> {{nifi.content.claim.max.appendable.size}} property are all retained and not
> deprecated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
