Sanjeet, For this use case, you should not be using the zk-migrator.sh tool. That tool is not intended to be used when upgrading nifi. Rather, the tool is to be used if you’re migrating nifi away from one zookeeper and onto another. For example, if you have a ZooKeeper instance that is shared by many other services, and you decide that you want to run a separate ZooKeeper instance purely for NiFi, then you could use that tool to copy the nifi state from one zookeeper instance to another.
NiFi has two types of state: Local and Cluster state. ListS3, for example, would use Cluster state because if the Primary Node changes, the new node in the cluster needs to have that same state. So the state must be shared across the cluster. So the state itself is stored in ZooKeeper. However, you could also have something like ListFile running on every node in the cluster, listing files on the local disk. In such a case, if Node 1 performs some listing, it does not make sense to share that state with Node 2, because Node 2 has a completely different listing (a completely different disk). So the state is then stored in the ./state/local directory. So when you upgrade from 1.8 to 1.12.1 you should also copy over the state directory to avoid losing any local state. But as long as the 1.12.1 cluster is pointing at the same zookeeper, there is no need to migrate the zookeeper state. Hope this helps! -Mark On Feb 24, 2021, at 8:56 AM, sanjeet rath <rath.sanj...@gmail.com<mailto:rath.sanj...@gmail.com>> wrote: Hi, My use case is to upgrade the nifi cluster from 1.8 to 1.12.1 with state (we are using external zookeeper and its 3 node cluster ). So the approach I followed , -> created 3 node linux box and installed nifi 1.12.1 & zookeeper 3.5.8 -> brought the flow.xml.gz, users.xml and authorization.xml from old 1.8 env to newly created 1.12 cluster.(Both cluster are in different linux box) -> Then followed with the use of zk-migrator.sh utility to create a zk-source-data.json in dev 1.8 env and applying it in 1.12 cluster to apply the states. Everything is fine with the above approach and state is captured in the processor. I am seeing another wired behaviour which is , without using zk-migrator.sh the state is captured properly by bringing just the flow.xml.gz Steps: -> created 3 node linux box and installed nifi 1.12.1 & zookeeper 3.5.8 -> brought the flow.xml.gz, users.xml and authorization.xml from old 1.8 env to newly created 1.12 cluster. and i have not used zk-migrator.sh utility in in newly created 1.12.1 env. -> When i am seeing processors like ListS3, Listsftp, the state id is captured properly in newly created 1.12.1 env, also when i run the processor it is pulling the data after the captured timestamp only ,which is perfect. Could someone help me to understand how the state id is captured in the lists3 processor in 1.12.1 env, because i am seeing its not present in flow.xml.gz Does it mean we can migrate the state by just bringing the flow.xml file without using zk-migrator.sh utilyty ? one more question where does the zk-migrator.sh utility write the states in the destination cluster in the external zookeper configuration Is it inside /nifi/state/locale/* ? Thanks in advance, -- Sanjeet Kumar Rath,
