[
https://issues.apache.org/jira/browse/MINIFICPP-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi updated MINIFICPP-1515:
------------------------------------
Description:
*Background:*
Issues with big flowfiles were reported on stack overflow:
[https://stackoverflow.com/questions/66330866/minifi-getfile-processor-fails-to-get-large-files/66334615?noredirect=1#comment117275399_66334615]
Seems like this is related to twi issues, one a [narrowing exception
happens|https://github.com/apache/nifi-minifi-cpp/blob/minifi-cpp-0.9.0-RC2/libminifi/src/core/ContentSession.cpp#L75]
when trying to determine the length of the file to be written into the content
repository, and another is that we use _stat on windows even though we should
be using _stat64 to query files larger than 2GB.
*Proposal:*
MiNiFi should be tested with large files.
*Acceptance criteria:*
{code:python|title=Feature definition proposal}
Scenario Outline: MiNiFi is capable of manipulating flowfiles of different
sizes
Given a GetFile processor with the "Input Directory" property set to
"/tmp/input"
And a file with <file size> of content is present in "/tmp/input"
And a PutFile processor with the "Directory" property set to "/tmp/output"
And the "success" relationship of the GetFile processor is connected to the
PutFile
When the MiNiFi instance starts up
Then a flowfile with matching content is placed in the monitored directory
in less than 20 seconds
Examples: File size
| file size |
| 10 B |
| 1.5 KiB |
| 10 MiB |
| 1.0 GiB |
| 2.1 GiB |
{code}
was:
*Background:*
Issues with big flowfiles were reported here:
https://stackoverflow.com/questions/66330866/minifi-getfile-processor-fails-to-get-large-files/66334615?noredirect=1#comment117275399_66334615
Seems like this is related to twi issues, one a [narrowing exception
happens|https://github.com/apache/nifi-minifi-cpp/blob/minifi-cpp-0.9.0-RC2/libminifi/src/core/ContentSession.cpp#L75]
when trying to determine the length of the file to be written into the content
repository, and another is that we use _stat on windows even though we should
be using _stat64 to query files larger than 2GB.
*Proposal:*
MiNiFi should be tested with large files.
*Acceptance criteria:*
{code:python|title=Feature definition proposal}
Scenario Outline: MiNiFi is capable of manipulating flowfiles of different
sizes
Given a GetFile processor with the "Input Directory" property set to
"/tmp/input"
And a file with <file size> of content is present in "/tmp/input"
And a PutFile processor with the "Directory" property set to "/tmp/output"
And the "success" relationship of the GetFile processor is connected to the
PutFile
When the MiNiFi instance starts up
Then a flowfile with matching content is placed in the monitored directory
in less than 20 seconds
Examples: File size
| file size |
| 10 B |
| 1.5 KiB |
| 10 MiB |
| 1.0 GiB |
| 2.1 GiB |
{code}
> Add integration tests verifying that MiNiFi handles large flowfiles as
> expected
> -------------------------------------------------------------------------------
>
> Key: MINIFICPP-1515
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1515
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
> Affects Versions: 1.0.0, 0.9.0
> Reporter: Adam Hunyadi
> Assignee: Adam Hunyadi
> Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
> Fix For: 1.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Background:*
> Issues with big flowfiles were reported on stack overflow:
>
> [https://stackoverflow.com/questions/66330866/minifi-getfile-processor-fails-to-get-large-files/66334615?noredirect=1#comment117275399_66334615]
> Seems like this is related to twi issues, one a [narrowing exception
> happens|https://github.com/apache/nifi-minifi-cpp/blob/minifi-cpp-0.9.0-RC2/libminifi/src/core/ContentSession.cpp#L75]
> when trying to determine the length of the file to be written into the
> content repository, and another is that we use _stat on windows even though
> we should be using _stat64 to query files larger than 2GB.
> *Proposal:*
> MiNiFi should be tested with large files.
> *Acceptance criteria:*
> {code:python|title=Feature definition proposal}
> Scenario Outline: MiNiFi is capable of manipulating flowfiles of different
> sizes
> Given a GetFile processor with the "Input Directory" property set to
> "/tmp/input"
> And a file with <file size> of content is present in "/tmp/input"
> And a PutFile processor with the "Directory" property set to "/tmp/output"
> And the "success" relationship of the GetFile processor is connected to
> the PutFile
> When the MiNiFi instance starts up
> Then a flowfile with matching content is placed in the monitored
> directory in less than 20 seconds
> Examples: File size
> | file size |
> | 10 B |
> | 1.5 KiB |
> | 10 MiB |
> | 1.0 GiB |
> | 2.1 GiB |
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
