[
https://issues.apache.org/jira/browse/NIFI-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648696#comment-16648696
]
ASF GitHub Bot commented on NIFI-5629:
--------------------------------------
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r224947166
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGetFile.java
---
@@ -151,6 +151,90 @@ public void testFilePickedUp() throws IOException {
assertEquals(absTargetPathStr, absolutePath);
}
+ @Test
+ public void testFilePickedUpArrivalSinceListing() throws IOException {
+ final File directory = new File("target/test/data/in");
+ deleteDirectory(directory);
+ assertTrue("Unable to create test data directory " +
directory.getAbsolutePath(), directory.exists() || directory.mkdirs());
+
+ final TestRunner runner = TestRunners.newTestRunner(new GetFile());
+ runner.setProperty(GetFile.DIRECTORY, directory.getAbsolutePath());
+ runner.setProperty(GetFile.KEEP_SOURCE_FILE, "false");
+
+ final File inFile = new File("src/test/resources/hello.txt");
+ {
+ final Path inPath = inFile.toPath();
+ final File destFile = new File(directory, inFile.getName());
+ final Path targetPath = destFile.toPath();
+ Files.copy(inPath, targetPath);
+ }
+ {
+ final Path inPath = inFile.toPath();
+ final File destFile = new File(directory, inFile.getName() +
"1");
+ final Path targetPath = destFile.toPath();
+ Files.copy(inPath, targetPath);
+
+ runner.run(1);
+ }
+ final Path inPath = inFile.toPath();
+ final File destFile = new File(directory, inFile.getName() + "2");
+ final Path targetPath = destFile.toPath();
+ final Path absTargetPath = targetPath.toAbsolutePath();
+ final String absTargetPathStr = absTargetPath.getParent() + "/";
+ Files.copy(inPath, targetPath);
+
+ runner.run(1);
+ runner.run(1);
+
+ runner.assertAllFlowFilesTransferred(GetFile.REL_SUCCESS, 3);
+ final List<MockFlowFile> successFiles =
runner.getFlowFilesForRelationship(GetFile.REL_SUCCESS);
+ successFiles.get(0).assertContentEquals("Hello,
World!".getBytes("UTF-8"));
+ successFiles.get(1).assertContentEquals("Hello,
World!".getBytes("UTF-8"));
+ successFiles.get(2).assertContentEquals("Hello,
World!".getBytes("UTF-8"));
+
+ final String path = successFiles.get(2).getAttribute("path");
+ assertEquals("/", path);
+ final String absolutePath =
successFiles.get(2).getAttribute(CoreAttributes.ABSOLUTE_PATH.key());
+ assertEquals(absTargetPathStr, absolutePath);
+ }
+
+ @Test
--- End diff --
I still have more testing to do, but so far this test looks good at
demonstrating the issue. When i run this with the unmodified GetFile it runs
out steam around 9993 files.
> GetFile becomes slow listing vast directories
> ---------------------------------------------
>
> Key: NIFI-5629
> URL: https://issues.apache.org/jira/browse/NIFI-5629
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 1.6.0
> Reporter: Adam
> Priority: Minor
>
> GetFile repeatedly lists entire directories before applying batching, meaning
> for vast directories it spends a long time listing directories.
>
> Pull request to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
