Hi Jim, Thanks for the response.
You are correct the local state management referenced in the state-management.xml has partitions set. The local state directory is partitioned to about 20 partitions, each with its own .journal file. I went ahead and concatenated all of the .journal files from the partition directories along with the .journal file under /state/local/journals/ and renamed the new file to follow the numbered sequence of journal files in /state/local/journals. I tried placing the new .journal file first in /state/local and restarting nifi, it was not picked up. I then placed it in /state/local/journals where it was picked up but is throwing errors stating the .journal file does not appear to be a valid journal file. My guess is theres some metadata or header information that is being corrupted when the files are combined. Hence not sure if this is the best approach for restoring the processor state for the ListSFTP processors. Any further suggestions would be greatly appreciated! Cheers ~ Peter From: Jim Steinebrey <[email protected]> Sent: Wednesday, July 10, 2024 11:00 AM To: [email protected] Subject: Re: ListSFTP Processors losing state after upgrading from 1.26.0 to 2.0.0-M3 IMPORTANT!! - External Content - Please use caution. Hi Peter, Can you review the configuration for your local state provider from state-management.xml for 1.26.0 and 2.0.0-M3 configurations? From comparing the code in 1.26.0 vs 2.x, My working theory is that you have partitions set to the default of 16 for both old and new Nifi configurations. NiFI 1.26.0 uses <property name="Partitions">16</property> to decide how many partition subdirectories to create under your state base path (./state/local) directory. Partition directories for 1.26.0 are in your ./state/local/ and begin with prefix partition- But NiFi 2.x ignores the number specified by <property name="Partitions">16</property> and never creates any partition subdirectories as far I can see in the code. Therefore NiFI 2.x does NOT look for any partition subdirectories. This partition directory change in WriteAheadLocalStateProvider.java was a side effect of this change. MinimalLockingWriteAheadLog.java was the deprecated class that was removed. MinimalLockingWriteAheadLog had the logic for interacting with partition directories. [NIFI-11833] Remove deprecated classes in nifi-commons - ASF JIRA<https://issues.apache.org/jira/browse/NIFI-11833> issues.apache.org<https://issues.apache.org/jira/browse/NIFI-11833> [cid:[email protected]]<https://issues.apache.org/jira/browse/NIFI-11833> Not sure if this is easy or not because I do not have local state partition directories handy to inspect, but as a workaround can you manually move/combine all the state information from the partition directories' files and merge them into a single file that you place directly in ./state/local Please let know what you find out, Peter, Jim Steinebrey On Jul 9, 2024, at 4:13 PM, Sharp, Peter <[email protected]<mailto:[email protected]>> wrote: Hello, I am reaching out with an issue I am facing when attempting to upgrade Nifi from 1.26.0 to 2.0.0-M3. After migrating from 1.26.0 to 2.0.0-M3 all of the ListSFTP processors lose their stored state. We are running Nifi as a standalone instance and are using local state management. What I did and what has worked for me in the past with minor version upgrades (1.23.2 to 1.26.0) is copying over the existing /state/local directory to the new version. I validated that the state-mangement.xml file is referencing the local state provider with the correct path to the state directory. I also validated the state properties in the nifi.properties file are consistent with the old version. When checking the app logs between the old running version (1.26.0) and the new running version (2.0.0-M3) I am seeing some differences between where its trying to load state. (see below screenshots of logs) 1.26.0 nifi-app-log - <image002.png> 2.0.0-M3 nifi-app-log - <image003.png> It appears that its unable to pick the snapshot file successfully even though it exists in the /state/local directory. Any help on this would be greatly appreciated! Thanks ~ Peter Sharp Software Engineer Data & Analytics | ICE Mortgage Technology M: +1 760-390-4895 [email protected]<mailto:[email protected]> www.ice.com<http://www.ice.com/> <image001.png> The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
