Data duplication When NIFI is restarted

Hi all, I notice that, if the NIFI instance gets terminated while a processor is processing a flow file, that processor starts to process the flow file again from the beginning when NIFI is restarted. I'm using the PutKudu processor and the PutParquet processor to write data into kudu and parquet

Re: Data duplication When NIFI is restarted

Hi Vibhath, How is NiFi terminated / restarted ? Thanks, Pierre Le mer. 17 mars 2021 à 15:04, Vibhath Ileperuma a écrit : > Hi all, > > I notice that, if the NIFI instance gets terminated while a processor is > processing a flow file, that processor starts to process the flow file > again from

Re: Data duplication When NIFI is restarted

Hi Pierre, The NIFI flow I'm implementing can be run for a long time continuously(maybe a couple of weeks/months). During this time period it can be terminated due to memory issue or some other system issue, can't it be? In such a case, I may need to restart NIFi manually and run the flow from whe

Re: Data duplication When NIFI is restarted

I’m just jumping in, we are seeing this issue as well when we are restarting the nifi process from time. We are aware of the nifi.properties “nifi.flowcontroller.graceful.shutdown.period=10 sec” parameter, but to be honest we didn’t try to raise it up yet. Maybe it takes more than 10s to fully

Re: Data duplication When NIFI is restarted

If a processor uses the session to take a flow file from the incoming queues, and then nifi crashes before session.commit is called, then that flow file will be back in the original queue when nifi starts again since the session never updated the repositories. So it is possible that a destination

Any known issue on SplitRecord?

Hi all, I am using latest nifi version and SplitRecord works only once and then hangs: [image: image.png] I cannot stop it also. Juan

Re: Any known issue on SplitRecord?

Juan We found a bug in 1.13.1 today as reported here https://issues.apache.org/jira/browse/NIFI-8337 and https://issues.apache.org/jira/browse/NIFI-8334. We will have a 1.13.2 out asap to fix this and the regression now has tests to prevent it in the future. Thanks Joe On Wed, Mar 17, 2021 at 8

Regression in 1.13.1 - creating 1.13.2

All Based on two reports received today we've found a regression in process session handling in NiFi 1.13.1. This is fixed on main and we will have a 1.13.2 RC1 up for vote in a couple hours. The core issue found is https://issues.apache.org/jira/browse/NIFI-8337. This is the same root proble

Re: Any known issue on SplitRecord?

Wow that is fast! You are awesome, thanks Joe. I will test it. Juan On Thu, 18 Mar 2021 at 00:47, Joe Witt wrote: > Juan > > We found a bug in 1.13.1 today as reported here > https://issues.apache.org/jira/browse/NIFI-8337 and > https://issues.apache.org/jira/browse/NIFI-8334. > > We will have

Re: Any known issue on SplitRecord?

I should clarify that I am not positive it is the same issue but it is certainly possible especially if this worked in 1.13.0 Thanks On Wed, Mar 17, 2021 at 8:46 PM Joe Witt wrote: > Juan > > We found a bug in 1.13.1 today as reported here > https://issues.apache.org/jira/browse/NIFI-8337 and >

Re: Any known issue on SplitRecord?

Thanks Juan - that would be very valuable actually. I'll send you a link to a build here in an hour or so. If you can test that and let us know that will help us with the release candidate voting process quite a bit. Thanks On Wed, Mar 17, 2021 at 8:49 PM Juan Pablo Gardella < gardellajuanpa...@