[
https://issues.apache.org/jira/browse/NIFI-16322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt updated NIFI-16322:
----------------------------
Status: Patch Available (was: Open)
> GenerateFlowFile should stream File Size instead of allocating a byte array
> ---------------------------------------------------------------------------
>
> Key: NIFI-16322
> URL: https://issues.apache.org/jira/browse/NIFI-16322
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Joe Witt
> Assignee: Joe Witt
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> GenerateFlowFile builds the entire File Size in a heap byte[] before
> session.write.
> generateData() does asDataSize(DataUnit.B).intValue() then new
> byte[byteCount].
> Double.intValue() saturates at Integer.MAX_VALUE, so a legal value such as 3
> GB is
> silently generated as 2,147,483,647 bytes. DATA_SIZE_VALIDATOR has no upper
> bound.
> The array is then written to the content repository, so the payload exists
> twice.
> Unique FlowFiles = false builds that array in @OnScheduled and holds it for
> the
> run. Unique FlowFiles = true allocates a new array per FlowFile in the batch.
> OutOfMemoryError is uncaught; the processor only has success.
> There is no protocol reason for a contiguous buffer. Generate random or text
> bytes into a small buffer and stream them through session.write until File
> Size
> bytes are written. Unique = false can reuse a seed or a modest buffer so
> FlowFiles
> still share the same content without caching a full-size array.
> GenerateFlowFile has no @SystemResourceConsideration(MEMORY) today. Do not add
> one that claims the whole File Size lives on heap. After streaming, leave the
> annotation off.
> Expected:
> - File Size is written in chunks via session.write; do not hold a byte[] of
> File Size.
> - Values greater than Integer.MAX_VALUE bytes are invalid or rejected without
> allocating.
> - Do not truncate 3 GB to 2 GiB - 1.
> - Unique FlowFiles and Batch Size behavior stay the same.
> - Custom Text is unchanged (File Size is already ignored on that path).
> - Small sizes (default 0B, existing tests) still produce the exact length.
> Tests (surefire TestGenerateFlowFile, no multi-GB fixture):
> - Size within cap produces a FlowFile of that exact length (unique and
> non-unique).
> - Size above Integer.MAX_VALUE is invalid at configuration or fails without
> NegativeArraySizeException / a huge array.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
