Artur M. created NIFI-13805:
-------------------------------
Summary: ListSFTP issue - single file is listed multiple times
with incorrect size
Key: NIFI-13805
URL: https://issues.apache.org/jira/browse/NIFI-13805
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.18.0
Environment: NiFi - v1.18.0
OS - RedHat 4.8.5-44
3 nodes cluster with external Zookeeper
Reporter: Artur M.
Attachments: image-2024-09-26-15-34-24-604.png,
image-2024-09-26-15-35-06-389.png, image-2024-09-26-15-35-43-616.png,
image-2024-09-26-15-35-56-677.png, image-2024-09-26-15-38-04-850.png,
image-2024-09-26-15-46-01-227.png, image-2024-09-26-15-48-02-174.png,
image-2024-09-26-15-48-11-008.png
We've noticed that in environment with high data load, from time to time files
are listed multiple times with incorrect size. File upload to SFTP server is
completed and confirmed on another VM but after successfull upload NiFi's
ListSFTP processor see those files as incomplete. It takes few ListSFTP
iterations for NiFI to see file's full size, in each iteration file's size
seems to be increasing. It's quite confusing, because as I've described files
are already fully uploaded, so NiFi should see their final size. In the case of
another environment with a similar configuration but less load we don't observe
this issue.
Our current settings for ListSFTP processor are visible on following
screenshots. Sensitive data were hide.
!image-2024-09-26-15-34-24-604.png!
!image-2024-09-26-15-35-06-389.png!
!image-2024-09-26-15-35-43-616.png!
!image-2024-09-26-15-35-56-677.png!
Current flow is presented on following screenshot (this flow is used in
multiple processor groups for different data).
!image-2024-09-26-15-38-04-850.png!
Files has declared filesize in their name, value from filename is compared in
"Ensure File Completion" with size returned from ListSFTP processor.
Following screenshots present files collected by queue "unmatched" with
incorrect sizes.
!image-2024-09-26-15-46-01-227.png!
For files presented above, behaviour of ListSFTP is like on example below.
File's size changes in each iteration, even though file has been fully uploaded
and final size of 966 370 266 bytes should be returned in the first iteration.
* 153 248 000 bytes
* 294 464 000 bytes
* 460 224 000 bytes
* 618 592 000 bytes
* 777 472 000 bytes
* 959 936 000 bytes
* 966 370 266 bytes
In other processor groups which uses ListSFTP, situation is same as it was
described above.
!image-2024-09-26-15-48-02-174.png!
!image-2024-09-26-15-48-11-008.png!
We believe that described behaviour of ListSFTP is unexpected. Expected
behaviour is single listing for single file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)