[
https://issues.apache.org/jira/browse/NIP-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIP-40:
--------------------------
Description:
h2. Motivation
Several small, related gaps in the Content Repository and FlowFile Repository
API (nifi-framework-api) make it harder than necessary to implement t he
repository interfaces and to reason about repository behavior. This NIP
groups a few backward-compatible cleanups to that API.
# *No component or Connector context on writes.* When a ProcessSession creates
content or commits FlowFile changes, it calls {{{}ContentRepository.create(...)
and FlowFileRepository.updateRepository(...){}}}. Neither call tells the
repository which component the session is acting on behalf of, nor which
Connector (managed flow) that component belongs to, so the repository cannot
attribute an operation to a component or organize data by Connector. There have
been several instances when I wanted to know which component a Content Claim
was for; e.g., if we know that it's for a Processor that statistically will
create large FlowFiles we may not want to append its content to an existing
Resource Claim but instead create a new Resource Claim. This would help avoid
scenarios where Content Repository holds onto a large file just because one
small FlowFile shares the same Resource Claim.
# *The session depends on framework-internal stream classes.*
{{StandardProcessSession}} reads content through
{{{}ContentClaimInputStream{}}}, and the write cache reuses claims through
{{{}ContentClaimOutputStream.newContentClaim(){}}}. These abstractions are good
and make sense, but both classes live in framework internals
(nifi-framework-components), not in the more public {{{}nifi-framework-api{}}},
so a repository that wants to provide its own read stream or claim-reuse
behavior has no public abstraction to implement and must bind to framework
internals.
# *Content Repository should own the maximum appendable claim size.* The claim
rollover size is read from the {{nifi.content.claim.max.appendable.size}}
property and threaded to the write cache, while a repository that has its own
notion of where a storage unit ends has no way to declare it. The two values
must be kept consistent by hand. While participation from Process Session is
necessary for the max appendable size to work properly, it should be the
responsibility of the pluggable {{ContentRepository}} to dictate what max size
should be used.
h2. Proposed Solution
Three backward-compatible additions to the repository SPI:
* *Component/Connector context.* Pass a small, immutable context object on
each create/update describing the component and Connector the session is acting
for. It is populated by the framework and passed straight through; stock
repositories ignore it.
* *Public stream abstractions.* Expose the content-claim input-stream
abstraction and the output-stream claim-reuse contract in nifi-framework-api,
and have {{StandardProcessSession}} and the write cache depend on those public
types rather than the framework-internal classes.
* *Repository-owned appendable size.* Let {{ContentRepository}} report its
maximum appendable claim size, and have the framework source the rollover size
from the repository. The existing property remains the default the stock
repository reports. All additions are default methods or newly exposed types;
no existing type or method is removed, changed, or deprecated. The stock
FileSystemRepository and WriteAheadFlowFileRepository behave exactly as they do
today.
h2. Framework API Changes (nifi-framework-api)
*Component/Connector context*
* New {{LossTolerance}} enum ({{{}LOSS_TOLERANT{}}}, {{{}LOSS_INTOLERANT{}}}).
While we could keep with the existing {{boolean lossTolerant}} the general
pattern that NiFi has tended toward is to drop booleans in interfaces/APIs in
favor of more descriptive enums.
* New {{ContentClaimCreationContext}} (component id, Connector id, loss
tolerance) and {{FlowFileUpdateContext}} (component id, Connector id).
* New {{default}} {{ContentRepository.create(ContentClaimCreationContext)}}
and {{{}FlowFileRepository.updateRepository(Collection<RepositoryRecord>,
FlowFileUpdateContext){}}}, each delegating to the existing method.{*}{{*}}
*Public stream abstractions*
* Relocate {{ContentClaimInputStream}} and {{ContentClaimOutputStream}} from
{{nifi-framework-components}} to {{nifi-framework-api}}
*Repository-owned appendable size*
* New {{default}} {{ContentRepository.getMaxAppendableClaimBytes()}} returning
the configured default (so stock behavior is unchanged); the framework sources
the rollover size from the repository. The value must be a cheap in-memory read,
since it is consulted on the write hot path. The existing
{{{}create(boolean){}}}, {{{}updateRepository(Collection){}}}, and the
{{nifi.content.claim.max.appendable.size}} property are all retained and not
deprecated.
was:
h2. Motivation
Several small, related gaps in the Content Repository and FlowFile Repository
API (nifi-framework-api) make it harder than necessary to implement t he
repository interfaces and to reason about repository behavior. This NIP
groups a few backward-compatible cleanups to that API.
# *No component or Connector context on writes.* When a ProcessSession creates
content or commits FlowFile changes, it calls {{ContentRepository.create(...)
}}and {{{}FlowFileRepository.updateRepository(...){}}}. Neither call tells the
repository which component the session is acting on behalf of, nor which
Connector (managed flow) that component belongs to, so the repository cannot
attribute an operation to a component or organize data by Connector. There have
been several instances when I wanted to know which component a Content Claim
was for; e.g., if we know that it's for a Processor that statistically will
create large FlowFiles we may not want to append its content to an existing
Resource Claim but instead create a new Resource Claim. This would help avoid
scenarios where Content Repository holds onto a large file just because one
small FlowFile shares the same Resource Claim.
# *The session depends on framework-internal stream classes.*
{{StandardProcessSession}} reads content through
{{{}ContentClaimInputStream{}}}, and the write cache reuses claims through
{{{}ContentClaimOutputStream.newContentClaim(){}}}. These abstractions are good
and make sense, but both classes live in framework internals
(nifi-framework-components), not in the more public {{{}nifi-framework-api{}}},
so a repository that wants to provide its own read stream or claim-reuse
behavior has no public abstraction to implement and must bind to framework
internals.
# *Content Repository should own the maximum appendable claim size.* The claim
rollover size is read from the {{nifi.content.claim.max.appendable.size}}
property and threaded to the write cache, while a repository that has its own
notion of where a storage unit ends has no way to declare it. The two values
must be kept consistent by hand. While participation from Process Session is
necessary for the max appendable size to work properly, it should be the
responsibility of the pluggable {{ContentRepository}} to dictate what max size
should be used.
h2. Proposed Solution
Three backward-compatible additions to the repository SPI:
* *Component/Connector context.* Pass a small, immutable context object on
each create/update describing the component and Connector the session is acting
for. It is populated by the framework and passed straight through; stock
repositories ignore it.
* *Public stream abstractions.* Expose the content-claim input-stream
abstraction and the output-stream claim-reuse contract in nifi-framework-api,
and have {{StandardProcessSession}} and the write cache depend on those public
types rather than the framework-internal classes.
* *Repository-owned appendable size.* Let {{ContentRepository}} report its
maximum appendable claim size, and have the framework source the rollover size
from the repository. The existing property remains the default the stock
repository reports. All additions are default methods or newly exposed types;
no existing type or method is removed, changed, or deprecated. The stock
FileSystemRepository and WriteAheadFlowFileRepository behave exactly as they do
today.
h2. Framework API Changes (nifi-framework-api)
*Component/Connector context*
* New {{LossTolerance}} enum ({{{}LOSS_TOLERANT{}}}, {{{}LOSS_INTOLERANT{}}}).
While we could keep with the existing {{boolean lossTolerant}} the general
pattern that NiFi has tended toward is to drop booleans in interfaces/APIs in
favor of more descriptive enums.
* New {{ContentClaimCreationContext}} (component id, Connector id, loss
tolerance) and {{FlowFileUpdateContext}} (component id, Connector id).
* New {{default}} {{ContentRepository.create(ContentClaimCreationContext)}}
and {{{}FlowFileRepository.updateRepository(Collection<RepositoryRecord>,
FlowFileUpdateContext){}}}, each delegating to the existing method.{*}{{*}}
*Public stream abstractions*
* Relocate {{ContentClaimInputStream}} and {{ContentClaimOutputStream}} from
{{nifi-framework-components}} to {{nifi-framework-api}}
*Repository-owned appendable size*
* New {{default}} {{ContentRepository.getMaxAppendableClaimBytes()}} returning
the configured default (so stock behavior is unchanged); the framework sources
the rollover size from the repository. The value must be a cheap in-memory read,
since it is consulted on the write hot path. The existing
{{{}create(boolean){}}}, {{{}updateRepository(Collection){}}}, and the
{{nifi.content.claim.max.appendable.size}} property are all retained and not
deprecated.
> Content and FlowFile Repository API Cleanup
> -------------------------------------------
>
> Key: NIP-40
> URL: https://issues.apache.org/jira/browse/NIP-40
> Project: NiFi Improvement Proposal
> Issue Type: Improvement
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> h2. Motivation
> Several small, related gaps in the Content Repository and FlowFile Repository
> API (nifi-framework-api) make it harder than necessary to implement t he
> repository interfaces and to reason about repository behavior. This NIP
> groups a few backward-compatible cleanups to that API.
> # *No component or Connector context on writes.* When a ProcessSession
> creates content or commits FlowFile changes, it calls
> {{{}ContentRepository.create(...) and
> FlowFileRepository.updateRepository(...){}}}. Neither call tells the
> repository which component the session is acting on behalf of, nor which
> Connector (managed flow) that component belongs to, so the repository cannot
> attribute an operation to a component or organize data by Connector. There
> have been several instances when I wanted to know which component a Content
> Claim was for; e.g., if we know that it's for a Processor that statistically
> will create large FlowFiles we may not want to append its content to an
> existing Resource Claim but instead create a new Resource Claim. This would
> help avoid scenarios where Content Repository holds onto a large file just
> because one small FlowFile shares the same Resource Claim.
> # *The session depends on framework-internal stream classes.*
> {{StandardProcessSession}} reads content through
> {{{}ContentClaimInputStream{}}}, and the write cache reuses claims through
> {{{}ContentClaimOutputStream.newContentClaim(){}}}. These abstractions are
> good and make sense, but both classes live in framework internals
> (nifi-framework-components), not in the more public
> {{{}nifi-framework-api{}}}, so a repository that wants to provide its own
> read stream or claim-reuse behavior has no public abstraction to implement
> and must bind to framework internals.
> # *Content Repository should own the maximum appendable claim size.* The
> claim rollover size is read from the
> {{nifi.content.claim.max.appendable.size}} property and threaded to the write
> cache, while a repository that has its own notion of where a storage unit
> ends has no way to declare it. The two values must be kept consistent by
> hand. While participation from Process Session is necessary for the max
> appendable size to work properly, it should be the responsibility of the
> pluggable {{ContentRepository}} to dictate what max size should be used.
> h2. Proposed Solution
> Three backward-compatible additions to the repository SPI:
> * *Component/Connector context.* Pass a small, immutable context object on
> each create/update describing the component and Connector the session is
> acting for. It is populated by the framework and passed straight through;
> stock repositories ignore it.
> * *Public stream abstractions.* Expose the content-claim input-stream
> abstraction and the output-stream claim-reuse contract in nifi-framework-api,
> and have {{StandardProcessSession}} and the write cache depend on those
> public types rather than the framework-internal classes.
> * *Repository-owned appendable size.* Let {{ContentRepository}} report its
> maximum appendable claim size, and have the framework source the rollover
> size from the repository. The existing property remains the default the stock
> repository reports. All additions are default methods or newly exposed types;
> no existing type or method is removed, changed, or deprecated. The stock
> FileSystemRepository and WriteAheadFlowFileRepository behave exactly as they
> do today.
> h2. Framework API Changes (nifi-framework-api)
> *Component/Connector context*
> * New {{LossTolerance}} enum ({{{}LOSS_TOLERANT{}}},
> {{{}LOSS_INTOLERANT{}}}). While we could keep with the existing {{boolean
> lossTolerant}} the general pattern that NiFi has tended toward is to drop
> booleans in interfaces/APIs in favor of more descriptive enums.
> * New {{ContentClaimCreationContext}} (component id, Connector id, loss
> tolerance) and {{FlowFileUpdateContext}} (component id, Connector id).
> * New {{default}} {{ContentRepository.create(ContentClaimCreationContext)}}
> and {{{}FlowFileRepository.updateRepository(Collection<RepositoryRecord>,
> FlowFileUpdateContext){}}}, each delegating to the existing method.{*}{{*}}
>
> *Public stream abstractions*
> * Relocate {{ContentClaimInputStream}} and {{ContentClaimOutputStream}} from
> {{nifi-framework-components}} to {{nifi-framework-api}}
> *Repository-owned appendable size*
> * New {{default}} {{ContentRepository.getMaxAppendableClaimBytes()}}
> returning the configured default (so stock behavior is unchanged); the
> framework sources the rollover size from the repository. The value must be a
> cheap in-memory read,
> since it is consulted on the write hot path. The existing
> {{{}create(boolean){}}}, {{{}updateRepository(Collection){}}}, and the
> {{nifi.content.claim.max.appendable.size}} property are all retained and not
> deprecated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
