Hi Joe
I know what you are thinking but that’s not the case.
Check my very short description of my test flow.
In my loop the PutSFTP process is using default settings which means it’s
uploading files as .filename and rename it when done. The next process is the
FetchSFTP which will load the file as filename. If PutSFTP is not finished
uploading the file it will have the wrong filename and the flow file will not
go from the PutSFTP -> FetchSFTP and therefore the FetchSFTP can’t fetch the
file. So in my test flow it is not the case.
In our production flow, after nifi gets its data it calculates the sha256.
uploads the data to a sftp server as .filename and rename it when done. Default
settings for PutSFTP. Next it create a new file with the value of the hash and
save it as filename.sha256.
At that sftp server a bash script is looking for NOT hidden files every 2
seconds with a ls command. If there are files the bash script does a cp
filename /archive/filename and sends the data to server 3 via a data diode. At
the other side another nifi server reads the filename.sha256, reads in the hash
value and reads in the original data. Calculate a new sha256 and compare the
two hashes.
Yesterday there was a corruption again and we checked the file at the first
sftp server where the first nifi saved it after creating the first hash.
Running a sha256sum at the /archive/filename produced a different hash than
nifi. So after the PutSFTP and a Linux cp command the file was corrupted.
It have been less than 1 file pr. 1.000.000 files where we have seen theses
issues. But we see them.
Now we try to investigate that course the issue. Therefore I created the small
test flow and already after nearly 9000 iteration in the loop the file has been
corrupted just being uploaded and downloaded again.
Are we facing a network issue where a data packed is corrupted?
Are there a very rare cases where the sftp implementation is doing something
wrong?
We don’t know yet but we are running some more tests and at different systems
to narrow it down
Kind regards
Jens M. Kofoed
> Den 12. okt. 2021 kl. 19.39 skrev Joe Witt <joe.w...@gmail.com>:
>
> Hello
>
> How does nifi grab the data from the file system? It sounds like it is
> doing partial reads due to a competing consumer (data still being written)
> scenario.
>
> Thanks
>
> On Mon, Oct 11, 2021 at 10:36 PM Jens M. Kofoed <jmkofoed....@gmail.com>
> wrote:
>
>> Dear Developers
>>
>> We have a situation where we see corrupted file after using PutSFTP and
>> FetchSFTP in NIFI 1.13.2 with openjdk version "1.8.0_292", OpenJDK Runtime
>> Environment (build 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10), OpenJDK 64-Bit
>> Server VM (build 25.292-b10, mixed mode) running on a Ubuntu Server 20.04
>>
>> We have a flow between 2 separated systems where we use a PUTSFTP to export
>> data from one NIFI instance to a datadiode and use FetchSFTP to grep data
>> on the other end. To be sure data is not corrupted we calculate a SHA256 on
>> each side, and transfer the flowfile metadata in a seperate file. In rare
>> cases have see that the SHA256 doesn't match on both sides and are
>> investigation where the errors happens. We see 2 errors. Manually
>> calculation a SHA256 on both side of the diodes the file is OK and we have
>> found that the errors at happens between NIFI and the SFTP servers. And it
>> can happens at both sides.
>> So for testing I created this little flow:
>> GeneratingFlowFile (size 100MB) (Run once) ->
>> CryptographicHashContent (SHA256) ->
>> UpdateAttribute ( hash.root = ${content_SHA-256} , iteration=1) ->
>> PutSFTP ->
>> FetchSFTP ->
>> CryptographicHashContent (SHA256) ->
>> routeOnAttribute (compare root.hash vs.content_SHA-256)
>> If unmatch ->
>> Going to a disabled process for placeholding the corrupted file in
>> a file queue
>> If match ->
>> UpdateAttribute ( iteration= ${iteration:plus(1)} ) -> looping back
>> to PutSFTP
>>
>> After 8992 iteration the file is corrupted. To test if the errors are in
>> the calculation of the SHA256 I have a copy of the flow without the
>> PUT/FETCH SFTP processors which haven't got any errors yet.
>>
>> It is very rare that we see these errors, millions of files are going
>> through without any issues but some time it happens which is not good.
>>
>> Can any one please help? Maybe trying to setup the same test and see if you
>> also have a corrupted file after some days.
>>
>> Kind regards
>> Jens M. Kofoed
>>
