Hi Peter, Thanks for trying that and getting back to me. At least we have confirmed the root cause. I am sorry that suggested workaround does not seem feasible. I explored the code for other workarounds and none proved usable.
Does anyone else have a suggested workaround for Peter’s issue? Given the uncertainty of a workaround, Peter, I suggest you enter a Jira ticket describing the situation and what you have learned about root cause,. Then hopefully someone will make a change before the final NiFi 2.0 release. Best, Jim Steinebrey > On Jul 11, 2024, at 12:44 PM, Sharp, Peter <[email protected]> wrote: > > Hi Jim, > > Thanks for the response. > > You are correct the local state management referenced in the > state-management.xml has partitions set. > > The local state directory is partitioned to about 20 partitions, each with > its own .journal file. > > I went ahead and concatenated all of the .journal files from the partition > directories along with the .journal file under /state/local/journals/ and > renamed the new file to follow the numbered sequence of journal files in > /state/local/journals. > > I tried placing the new .journal file first in /state/local and restarting > nifi, it was not picked up. I then placed it in /state/local/journals where > it was picked up but is throwing errors stating the .journal file does not > appear to be a valid journal file. > > My guess is theres some metadata or header information that is being > corrupted when the files are combined. Hence not sure if this is the best > approach for restoring the processor state for the ListSFTP processors. > > Any further suggestions would be greatly appreciated! > > Cheers ~ > > Peter > > From: Jim Steinebrey <[email protected] <mailto:[email protected]>> > Sent: Wednesday, July 10, 2024 11:00 AM > To: [email protected] <mailto:[email protected]> > Subject: Re: ListSFTP Processors losing state after upgrading from 1.26.0 to > 2.0.0-M3 > > > IMPORTANT!! - External Content - Please use caution. > > > Hi Peter, > Can you review the configuration for your local state provider from > state-management.xml for 1.26.0 and 2.0.0-M3 configurations? > > From comparing the code in 1.26.0 vs 2.x, > My working theory is that you have partitions set to the default of 16 for > both old and new Nifi configurations. > NiFI 1.26.0 uses > <property name="Partitions">16</property> > to decide how many partition subdirectories to create under your state base > path (./state/local) directory. > Partition directories for 1.26.0 are in your ./state/local/ and begin with > prefix partition- > > > But NiFi 2.x ignores the number specified by > <property name="Partitions">16</property> > and never creates any partition subdirectories as far I can see in the code. > > > Therefore NiFI 2.x does NOT look for any partition subdirectories. > > > This partition directory change in WriteAheadLocalStateProvider.java was a > side effect of this change. > MinimalLockingWriteAheadLog.java was the deprecated class that was removed. > MinimalLockingWriteAheadLog had the logic for interacting with partition > directories. > [NIFI-11833] Remove deprecated classes in nifi-commons - ASF JIRA > <https://issues.apache.org/jira/browse/NIFI-11833> > issues.apache.org <https://issues.apache.org/jira/browse/NIFI-11833> > <image001.png> <https://issues.apache.org/jira/browse/NIFI-11833> > > Not sure if this is easy or not because I do not have local state partition > directories handy to inspect, but as a workaround can you manually > move/combine all the state information from the partition directories' files > and merge them into a single file that you place directly in ./state/local > > Please let know what you find out, Peter, > Jim Steinebrey > > > > > On Jul 9, 2024, at 4:13 PM, Sharp, Peter <[email protected] > <mailto:[email protected]>> wrote: > > Hello, > > I am reaching out with an issue I am facing when attempting to upgrade Nifi > from 1.26.0 to 2.0.0-M3. > > After migrating from 1.26.0 to 2.0.0-M3 all of the ListSFTP processors lose > their stored state. > > We are running Nifi as a standalone instance and are using local state > management. > > What I did and what has worked for me in the past with minor version upgrades > (1.23.2 to 1.26.0) is copying over the existing /state/local directory to the > new version. > > I validated that the state-mangement.xml file is referencing the local state > provider with the correct path to the state directory. I also validated the > state properties in the nifi.properties file are consistent with the old > version. > > When checking the app logs between the old running version (1.26.0) and the > new running version (2.0.0-M3) I am seeing some differences between where its > trying to load state. (see below screenshots of logs) > > 1.26.0 nifi-app-log - > <image002.png> > > 2.0.0-M3 nifi-app-log - > <image003.png> > > It appears that its unable to pick the snapshot file successfully even though > it exists in the /state/local directory. > > Any help on this would be greatly appreciated! > > Thanks ~ > > Peter Sharp > Software Engineer > Data & Analytics | ICE Mortgage Technology > M: +1 760-390-4895 > [email protected] <mailto:[email protected]> > www.ice.com <http://www.ice.com/> > <image001.png> > > The information contained in this message is proprietary and/or confidential. > If you are not the intended recipient, please: (i) delete the message and all > copies; (ii) do not disclose, distribute or use the message in any manner; > and (iii) notify the sender immediately. In addition, please be aware that > any message addressed to our domain is subject to archiving and review by > persons other than the intended recipient. Thank you. > > The information contained in this message is proprietary and/or confidential. > If you are not the intended recipient, please: (i) delete the message and all > copies; (ii) do not disclose, distribute or use the message in any manner; > and (iii) notify the sender immediately. In addition, please be aware that > any message addressed to our domain is subject to archiving and review by > persons other than the intended recipient. Thank you.
