Thanks Mark for the detailed explanation. Along with nifi i am also upgrading zookeeper 3.4 version to 3.5 version. So My nifi1.8 version runs on 3.4 version & new 1.12.1 version nifi runs on 3.5 version of zookeeper.and both zookeper are in differnet linux server running.
That is why i need to transfer the state from zookeeper 3.4 instance to 3.5 instance. But my original question is when i just copy the flow.xml.gz file from 1.8 version nifi instance to new 1.12 nifi instance and please note this new 1.12 nifi instance runs on different zookeeper instance, then how the state id is present in the 1.12 version of nifi lists3 processor and it starts prcessing where it left in 1.8 environment.I have not applied zk-migrator.sh to this new 3.5 zookeeper instance.So my expection the state id should not be prensent here. On Wed, 24 Feb 2021, 8:03 pm Mark Payne, <marka...@hotmail.com> wrote: > Sanjeet, > > For this use case, you should not be using the zk-migrator.sh tool. That > tool is not intended to be used when upgrading nifi. Rather, the tool is to > be used if you’re migrating nifi away from one zookeeper and onto another. > For example, if you have a ZooKeeper instance that is shared by many other > services, and you decide that you want to run a separate ZooKeeper instance > purely for NiFi, then you could use that tool to copy the nifi state from > one zookeeper instance to another. > > NiFi has two types of state: Local and Cluster state. ListS3, for example, > would use Cluster state because if the Primary Node changes, the new node > in the cluster needs to have that same state. So the state must be shared > across the cluster. So the state itself is stored in ZooKeeper. > > However, you could also have something like ListFile running on every node > in the cluster, listing files on the local disk. In such a case, if Node 1 > performs some listing, it does not make sense to share that state with Node > 2, because Node 2 has a completely different listing (a completely > different disk). So the state is then stored in the ./state/local directory. > > So when you upgrade from 1.8 to 1.12.1 you should also copy over the state > directory to avoid losing any local state. But as long as the 1.12.1 > cluster is pointing at the same zookeeper, there is no need to migrate the > zookeeper state. > > Hope this helps! > -Mark > > > On Feb 24, 2021, at 8:56 AM, sanjeet rath <rath.sanj...@gmail.com> wrote: > > Hi, > > My use case is to upgrade the nifi cluster from 1.8 to 1.12.1 with state (we > are using external zookeeper and its 3 node cluster ). > > So the approach I followed , > -> created 3 node linux box and installed nifi 1.12.1 & zookeeper 3.5.8 > -> brought the flow.xml.gz, users.xml and authorization.xml from old 1.8 > env to newly created 1.12 cluster.(Both cluster are in different linux box) > -> Then followed with the use of zk-migrator.sh utility to create a > zk-source-data.json in dev 1.8 env and applying it in 1.12 cluster to apply > the states. > > Everything is fine with the above approach and state is captured in the > processor. > > > I am seeing another wired behaviour which is , *without* > using zk-migrator.sh the state is captured properly by bringing just the > flow.xml.gz > > Steps: > -> created 3 node linux box and installed nifi 1.12.1 & zookeeper 3.5.8 > -> brought the flow.xml.gz, users.xml and authorization.xml from old 1.8 > env to newly created 1.12 cluster. and i have * not* used zk-migrator.sh > utility in in newly created 1.12.1 env. > -> When i am seeing processors like ListS3, Listsftp, the state id is > captured properly in newly created 1.12.1 env, also when i run > the processor it is pulling the data after the captured timestamp only > ,which is perfect. > > Could someone help me to understand how the state id is captured in > the lists3 processor in 1.12.1 env, because i am seeing its not present in > flow.xml.gz > Does it mean we can migrate the state by just bringing the flow.xml file > without using zk-migrator.sh utilyty ? > one more question where does the zk-migrator.sh utility write the states > in the destination cluster in the external zookeper configuration > Is it inside /nifi/state/locale/* ? > > Thanks in advance, > -- > Sanjeet Kumar Rath, > > >
