Mark Payne created NIP-40:
-----------------------------
Summary: Content and FlowFile Repository API Cleanup
Key: NIP-40
URL: https://issues.apache.org/jira/browse/NIP-40
Project: NiFi Improvement Proposal
Issue Type: Improvement
Reporter: Mark Payne
h2. Motivation
Several small, related gaps in the Content Repository and FlowFile Repository
API (nifi-framework-api) make it harder than necessary to implement t he
repository interfaces and to reason about repository behavior. This NIP
groups a few backward-compatible cleanups to that API.
# *No component or Connector context on writes.* When a ProcessSession creates
content or commits FlowFile changes, it calls {{ContentRepository.create(...)}}
and {{{}FlowFileRepository.updateRepository(...){}}}. Neither call tells the
repository which component the session is acting on behalf of, nor which
Connector (managed flow) that component belongs to, so the repository cannot
attribute an operation to a component or organize data by Connector. There have
been several instances when I wanted to know which component a Content Claim
was for; e.g., if we know that it's for a Processor that statistically will
create large FlowFiles we may not want to append its content to an existing
Resource Claim but instead create a new Resource Claim. This would help avoid
scenarios where Content Repository holds onto a large file just because one
small FlowFile shares the same Resource Claim.
# *The session depends on framework-internal stream classes.*
{{StandardProcessSession}} reads content through
{{{}ContentClaimInputStream{}}}, and the write cache reuses claims through
{{{}ContentClaimOutputStream.newContentClaim(){}}}.
These abstractions are good and make sense, but both classes live in framework
internals (nifi-framework-components), not in the
more public {{{}nifi-framework-api{}}}, so a repository that wants to provide
its own read stream or claim-reuse behavior has no public abstraction to
implement and must bind to framework internals.
# *Content Repository should own the maximum appendable claim size.* The claim
rollover size is read from the {{nifi.content.claim.max.appendable.size}}
property and threaded to the write cache, while a repository that has its own
notion of where a storage unit ends has no way to declare it. The two values
must be kept
consistent by hand. While participation from Process Session is necessary for
the max appendable size to work properly, it should be the responsibility of
the pluggable {{ContentRepository}} to dictate what max size should be used.
h2. Proposed Solution
Three backward-compatible additions to the repository SPI:
* *Component/Connector context.* Pass a small, immutable context object on each
create/update describing the component and Connector the session is acting for.
It is populated by the framework and passed straight through; stock repositories
ignore it.
* *Public stream abstractions.* Expose the content-claim input-stream
abstraction and the output-stream claim-reuse contract in nifi-framework-api,
and have {{StandardProcessSession}} and the write cache depend on those public
types rather than the framework-internal classes.
* *Repository-owned appendable size.* Let {{ContentRepository}} report its
maximum appendable claim size, and have the framework source the rollover size
from the repository. The existing property remains the default the stock
repository reports.
All additions are default methods or newly exposed types; no existing type or
method is removed, changed, or deprecated. The stock FileSystemRepository and
WriteAheadFlowFileRepository behave exactly as they do today.
h2. Framework API Changes (nifi-framework-api)
*Component/Connector context*
* New {{LossTolerance}} enum ({{{}LOSS_TOLERANT{}}}, {{{}LOSS_INTOLERANT{}}}).
While we could keep with the existing {{boolean lossTolerant}} the general
pattern that NiFi has tended toward is to drop booleans in interfaces/APIs in
favor of more descriptive enums.
* New {{ContentClaimCreationContext}} (component id, Connector id, loss
tolerance) and {{FlowFileUpdateContext}} (component id, Connector id).
* New {{default}} {{ContentRepository.create(ContentClaimCreationContext)}} and
{{{}FlowFileRepository.updateRepository(Collection<RepositoryRecord>,
FlowFileUpdateContext){}}},
each delegating to the existing method.{*}{*}
*Public stream abstractions*
* Relocate {{ContentClaimInputStream}} and {{ContentClaimOutputStream}} from
{{nifi-framework-components}} to {{nifi-framework-api}}
*Repository-owned appendable size*
* New {{default}} {{ContentRepository.getMaxAppendableClaimBytes()}} returning
the configured default (so stock behavior is unchanged); the framework sources
the rollover size from the repository. The value must be a cheap in-memory read,
since it is consulted on the write hot path.
The existing {{{}create(boolean){}}}, {{{}updateRepository(Collection){}}}, and
the
{{nifi.content.claim.max.appendable.size}} property are all retained and not
deprecated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
