File corruption with Put/Fetch SFTP

[Jens M. Kofoed]

Dear Developers

We have a situation where we see corrupted file after using PutSFTP and
FetchSFTP in NIFI 1.13.2 with openjdk version "1.8.0_292", OpenJDK Runtime
Environment (build 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10), OpenJDK 64-Bit
Server VM (build 25.292-b10, mixed mode) running on a Ubuntu Server 20.04

We have a flow between 2 separated systems where we use a PUTSFTP to export
data from one NIFI instance to a datadiode and use FetchSFTP to grep data
on the other end. To be sure data is not corrupted we calculate a SHA256 on
each side, and transfer the flowfile metadata in a seperate file. In rare
cases have see that the SHA256 doesn't match on both sides and are
investigation where the errors happens. We see 2 errors. Manually
calculation a SHA256 on both side of the diodes the file is OK and we have
found that the errors at  happens between NIFI and the SFTP servers. And it
can happens at both sides.
So for testing I created this little flow:
GeneratingFlowFile (size 100MB) (Run once) ->
CryptographicHashContent (SHA256) ->
UpdateAttribute ( hash.root = ${content_SHA-256} , iteration=1) ->
PutSFTP ->
FetchSFTP ->
CryptographicHashContent (SHA256) ->
routeOnAttribute (compare root.hash vs.content_SHA-256)
If unmatch ->
Going to a disabled process for placeholding the corrupted file in
a file queue
If match ->
UpdateAttribute ( iteration= ${iteration:plus(1)} ) -> looping back
to PutSFTP

After 8992 iteration the file is corrupted. To test if the errors are in
the calculation of the SHA256 I have a copy of the flow without the
PUT/FETCH SFTP processors which haven't got any errors yet.

It is very rare that we see these errors, millions of files are going
through without any issues but some time it happens which is not good.

Can any one please help? Maybe trying to setup the same test and see if you
also have a corrupted file after some days.

Kind regards
Jens M. Kofoed

Re: Build failure

[Mike Thomsen]

Did you run `mvn clean install` or just have an IDE do whatever it
considers a build to be after importing the Maven POMs?

On Fri, Oct 1, 2021 at 11:11 AM tobin  wrote:
>
> Hi,
>
> I'm trying to build NiFi for the first time.
>
> The build completes but when I browse to https://127.0.0.1:8443/nifi/
> the page doesn't finish loading.
>
> Inspecting the page shows a lot of missing resources such as jquery
> libraries. All the missing items come from the nifi/assets
>
> Which module in Maven is responsible for populating the assets folder?
>
> Regards,
> Tobin

Re: Issue with PuMongoRecord

[Lehel Boér]

Hi Deepak,

Currently, in case of any error, MongoDB processors transfer flowfiles to
the failure relationship. I believe it is a good idea to enhance the error
handling and either rollback if the processor is unable to connect to the
database or move the flowfiles to a retry relationship. I'm not sure
whether there are any technical limitations, however I'd suggest opening a
ticket on NiFi JIRA .

Kind Regards,
Lehel Boér

Chirthani, Deepak Reddy  ezt írta
(időpont: 2021. okt. 10., V, 21:11):

> Hi Nifi Devs,
>
>
>
> We are having an issue with PutMongoRecord Processor(same for PutMongo as
> well) which is routing the flowfiles to the failure relationship when the
> Mongo database(or the server hosting the db) is down. We have a set-up
> which looks like below on a high-level overview.
>
>
>
>
>
> As you can clearly see, all the failure relationships of different
> processors in the dataflow are connected to an Error Handling Process
> Group. Now, when the Mongodb is down, the PutMongoRecord processor is
> routing all the flowfiles to the failure relationship. Now, I know that a
> processor could route a flowfile to the failure relationship for multitude
> of reasons and I believe PutMongoRecord routing the flowfiles to failure
> when the database is down is not ideal for our scenario. Our dataflow is
> real-time streaming and when mongo database is down and flowfiles get
> routed to failure relationship, we get hundreds of emails because of the
> PutEmail processor in the Error Handling Process Group. We configured the
> Error Handling process group to check if we get any erroneous data which is
> against our schema or whatever maybe the reason.
>
> Is there any logic you guys help us in this scenario?
>
> I also observed that when we load the data to a RDBMS, lets say SQL
> Server, using the PutSql processor, it doesn’t route the flowfiles to the
> failure relationship when the sql server is down. I remember the flowfiles
> were held up in the connector above it until the db is up and running. I
> also noticed that PutSql has retry and PutMongo(and Record) doesn’t have
> any retry.
>
> Please help us what could be best thing for us to as this is critical to
> us.
>
> Thanks in Advance. Please reach out to me if you have any questions.
>
> Thanks and Regards
>
> *[image: image005]*
>
> *Deepak Reddy* | Data Engineer
> ​IT Centers of Excellence
> 13736 Riverport Dr., Maryland Heights, MO 63043
>
>
> The contents of this e-mail message and
> any attachments are intended solely for the
> addressee(s) and may contain confidential
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you
> in error, please immediately alert the sender
> by reply e-mail and then delete this message
> and any attachments. If you are not the
> intended recipient, you are notified that
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment
> is strictly prohibited.
>

Re: Minifi Docker Image

[Marton Szasz]

Hi,

For MiNiFi C++, I think it was simply forgotten since 0.7.0 and not
documented. At least since 0.9.0, we're following the release guide on
the confluence wiki. [1]
I'm not too familiar with docker-related processes, but added a stub
point to the guide in the finalize section. If someone knows the
workflow and could contribute it to the guide, that would be awesome
and would ensure that a docker image is published with every new
release.

Thanks,
Marton

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70254849

On Mon, 11 Oct 2021 at 15:48, Pierre Villard
 wrote:
>
> Hi Daniel, the images are available here:
> https://hub.docker.com/r/apache/nifi-minifi (MiNiFi Java)
> https://hub.docker.com/r/apache/nifi-minifi-cpp (MiNiFi C++)
>
> MiNiFi Java has not been updated in a while because we merged the MiNiFi
> Java code into NiFi and it's now a NiFi headless version. I believe the
> existing image could be used or a dedicated one could be created for
> headless.
>
> For MiNiFi C++, not sure why the latest versions have not been published.
> Someone here may know the answer.
>
> Thanks,
> Pierre
>
> Le lun. 11 oct. 2021 à 17:37, Daniel Beicht  a
> écrit :
>
> > Hello,
> >
> > since I couldn't find the Minifi Docker Image on Docker Hub I wanted to
> > ask why it is not available and if you plan to do so.
> >
> > Best regards,
> > Daniel Beicht
> >

Re: Minifi Docker Image

[Pierre Villard]

Hi Daniel, the images are available here:
https://hub.docker.com/r/apache/nifi-minifi (MiNiFi Java)
https://hub.docker.com/r/apache/nifi-minifi-cpp (MiNiFi C++)

MiNiFi Java has not been updated in a while because we merged the MiNiFi
Java code into NiFi and it's now a NiFi headless version. I believe the
existing image could be used or a dedicated one could be created for
headless.

For MiNiFi C++, not sure why the latest versions have not been published.
Someone here may know the answer.

Thanks,
Pierre

Le lun. 11 oct. 2021 à 17:37, Daniel Beicht  a
écrit :

> Hello,
>
> since I couldn't find the Minifi Docker Image on Docker Hub I wanted to
> ask why it is not available and if you plan to do so.
>
> Best regards,
> Daniel Beicht
>

Minifi Docker Image

[Daniel Beicht]

Hello,

since I couldn't find the Minifi Docker Image on Docker Hub I wanted to ask why 
it is not available and if you plan to do so.

Best regards,
Daniel Beicht

Re: Jira contributor access

[Matt Burgess]

Seda,

I have added you as a contributor to the NiFi Jira projects, looking
forward to your contribution(s)!

Regards,
Matt

On Mon, Oct 11, 2021 at 9:07 AM Seda Dogan  wrote:
>
> Hello;
>
>
> I would like to be contributor to the Nifi project for a small improvement in 
> AMQP Processor. I created a new account on Jira.
>
> Username: sedad. Can you please activate?
>
>
> Best Regards
>
> Seda Dogan
>
> [ABAS ERP]
> abas Software GmbH | Gartenstraße 67 | 76135 Karlsruhe | Germany
> Phone: +49(0)721-96723-0 | Fax: +49(0)721-96723-100
> E-Mail: seda.do...@abas.de | Web: 
> https://abas-erp.com/de
>
> [ABAS ERP 20 
> Release]
>
> Managing Directors / Geschäftsführer: Richard Furby, Paul Smolinski
> Registered Office / Sitz der Gesellschaft: Karlsruhe
> Commercial Register / Handelsregister: HRB 734651 Amtsgericht Mannheim
>
> [ISO 9001][BITMi Software Made in Germany]
>
> Informationen zum Umgang mit Ihren personenbezogenen Daten nach den Art. 13 
> und 14 EU-DSGVO
> finden Sie unter https://abas-erp.com/de/datenschutz.
> Further information according to Art. 13 and 14 GDPR about processing your 
> personal data you can find
> here: https://abas-erp.com/en/privacy-copyright.

Re: Jira contributor access

[Matt Burgess]

Bengt,

I have added you as a contributor to the NiFi Jira projects, looking
forward to your contribution(s)!

Regards,
Matt

On Mon, Oct 11, 2021 at 5:39 AM Bengt Håård
 wrote:
>
> Hello,
> my Jira username is bhaard .
>
> /Bengt

Jira contributor access

[Seda Dogan]

Hello;


I would like to be contributor to the Nifi project for a small improvement in 
AMQP Processor. I created a new account on Jira.

Username: sedad. Can you please activate?


Best Regards

Seda Dogan

[ABAS ERP]
abas Software GmbH | Gartenstraße 67 | 76135 Karlsruhe | Germany
Phone: +49(0)721-96723-0 | Fax: +49(0)721-96723-100
E-Mail: seda.do...@abas.de | Web: 
https://abas-erp.com/de

[ABAS ERP 20 
Release]

Managing Directors / Geschäftsführer: Richard Furby, Paul Smolinski
Registered Office / Sitz der Gesellschaft: Karlsruhe
Commercial Register / Handelsregister: HRB 734651 Amtsgericht Mannheim

[ISO 9001][BITMi Software Made in Germany]

Informationen zum Umgang mit Ihren personenbezogenen Daten nach den Art. 13 und 
14 EU-DSGVO
finden Sie unter https://abas-erp.com/de/datenschutz.
Further information according to Art. 13 and 14 GDPR about processing your 
personal data you can find
here: https://abas-erp.com/en/privacy-copyright.

Jira contributor access

[Bengt Håård]

Hello,
my Jira username is bhaard .

/Bengt