Hello all,
I'm encountering an error in the TestListFile.java class and in the
testFilterAge() specifically. This unit test and associated class can be
found in the Nifi-Standard-Bundle. I've noticed a couple issues have been
reported on ListFile before. My issue is most closely related to this one:
https://issues.apache.org/jira/browse/NIFI-3831, although it may be an
issue altogether separate. I have never seen this particular test pass
before and I've attempted building several versions of nifi now.
I'm doing the nifi build on a new model mac. My OS Version is:
18.2.0 Darwin Kernel Version 18.2.0: Fri Oct 5 19:41:49 PDT 2018;
root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64
I noticed that my root file system is apfs via mount -v */dev/disk1s1 on /
(apfs, local, journaled)*
In looking at the ListFiles.testFilterAge() and related methods I found a
comment above the getTestModifiedTime talking about hfs+ only having
granularity to one second. For APFS this no longer seems to be the case
based on the the link tagged below. I've modified this method removing the
minus 1 and it doesn't seem to have any affect on the failures I get.
/*
* HFS+, default for OS X, only has granularity to one second,
accordingly, we go back in time to establish consistent test cases
*
* Provides "now" minus 1 second in millis
*/
private static long getTestModifiedTime() {
final long nowMillis = System.currentTimeMillis();
// Subtract a second to avoid possible rounding issues
final long nowSeconds = TimeUnit.SECONDS.convert(nowMillis,
TimeUnit.MILLISECONDS) - 1;
return TimeUnit.MILLISECONDS.convert(nowSeconds, TimeUnit.SECONDS);
}
https://www.gillware.com/data-recovery-lab/apfs-apple-file-system/
"Namely, APFS addresses file checksum, nanosecond time stamps, volume
snapshots, and other features which were lacking under HFS+."
All the file ranges seem shifted, but I'm surprised that nobody else has
encountered this same issue. I haven't raised this issue as a story yet
because no one else has noticed the same problem.
In trying to gain additional understanding I've been modifying the unit
test for testFilterAge and I found incrementing the boundaries by one time
range unit has allowed me to build a passing test case.
// exclude oldest
runner.setProperty(ListFile.MIN_AGE, age0);
- runner.setProperty(ListFile.MAX_AGE, age3);
+ runner.setProperty(ListFile.MAX_AGE, age4);
runNext.apply(true);
runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
final List<MockFlowFile> successFiles2 =
runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
@@ -362,7 +362,7 @@ public class TestListFile {
assertEquals(file1.getName(),
successFiles2.get(1).getAttribute("filename"));
// exclude newest
- runner.setProperty(ListFile.MIN_AGE, age1);
+ runner.setProperty(ListFile.MIN_AGE, age2);
runner.setProperty(ListFile.MAX_AGE, age5);
runNext.apply(true);
runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
@@ -372,8 +372,8 @@ public class TestListFile {
assertEquals(file2.getName(),
successFiles3.get(1).getAttribute("filename"));
// exclude oldest and newest
- runner.setProperty(ListFile.MIN_AGE, age1);
- runner.setProperty(ListFile.MAX_AGE, age3);
+ runner.setProperty(ListFile.MIN_AGE, age2);
+ runner.setProperty(ListFile.MAX_AGE, age4);
runNext.apply(true);
runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
final List<MockFlowFile> successFiles4 =
runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
Has anyone else encountered this? Are there special platform specific
flags I need to pass when running on a mac? It makes me wonder if the
generated timing is conflicting with the following line in ListFile.java
final long fileAge = System.currentTimeMillis() -
attributes.lastModifiedTime().toMillis();
That line can be found on line 622 in the ListFile class in the
createFileFilter method. I'm thinking that utilizing
System.currentTImeMillis() instead of providing some kind of shim where the
current clock time can be reliably mocked is causing some inconsistencies
here. I'm not 100% confident in this because I would think that others
would be encountering the same or similar issues.
Any help would be greatly appreciated. I'm also happy to clarify if
anything seems unclear.
Thanks in advance,
--
Ryan Withers
Senior Software Developer / Analyst
http://www.linkedin.com/in/ryanwithers
