Re: List processors
Ah I see, the providers say which scopes they support, and then the framework only lets them be used for the appropriate scope, which makes sense. I suppose the WriteAheadLocalStateProvider could be modified to support CLUSTER scope, although this seems like a bad idea to give people an option that we know causes data loss. You could make your own version of the WriteAheadLocalStateProvider with this change though. If you are running a cluster on 0.7, don't you already have ZooKeeper in order to run the cluster? Just trying to see what the issue with using ZooKeeper is. On Fri, Apr 21, 2017 at 2:32 PM, Juan Sequeiros <helloj...@gmail.com> wrote: > Thanks Bryan just tried and NIFI does not start because of this: > > " Cannot use Cluster State Provider ( WriteAheadLocalStateProvider ) as it > only supports scope(s) {LOCAL} but instance is configured to use scope > CLUSTER" > > > On Fri, Apr 21, 2017 at 2:12 PM Bryan Bende <bbe...@gmail.com> wrote: >> >> Juan, >> >> I believe from the processor side of the things, when a processor >> calls save/retrieve on the state manager, the processor has to specify >> a context like CLUSTER or LOCAL. If you specify CLUSTER, and no >> clustered state provider exists, then it will save it to the local >> provider. This allows a processor to work seamlessly across a >> standalone Nifi and a clustered Nifi. >> >> The issue is that NiFi is not allowed to start up in clustered mode >> without a clustered state provider, generally that is the ZooKeeper >> provider, although it is an extension point and someone can implement >> their own. >> >> I would think you could do the following... >> >> The normal clustered provider looks like this in state-management.xml >> >> >> zk-provider >> >> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider >> >> /nifi >> 10 seconds >> Open >> >> >> If you take the config from the local provider and drop it in the >> cluster provider... >> >> >> local-cluster-provider >> >> org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider >>./state/local >>false >>16 >>2 mins >> >> >> Basically defining another instance of the local state provider as the >> cluster provider. >> >> Not totally sure if this works, but theoretically it should. >> >> -Bryan >> >> >> On Fri, Apr 21, 2017 at 1:54 PM, Juan Sequeiros <helloj...@gmail.com> >> wrote: >> > To add more to the issue I see this on my log: >> > >> > "Failed to restore processor state; yielding java.io.IOException; Failed >> > to >> > obtain value from Zookeeper for component " with exception code >> > CONNECTIONLOSS >> > >> > So this is confirming what I expected and at this point not sure if this >> > is >> > a bug or working as expected Feels like the dataflow manager should >> > configure how to handle state and be able to use ListS3 in cluster even >> > though I do not have zookeeper? >> > >> > >> > >> > On Fri, Apr 21, 2017 at 10:57 AM Juan Sequeiros <helloj...@gmail.com> >> > wrote: >> >> >> >> Hello all, >> >> >> >> My preliminary testing shows that if I run ListS3 ( maybe all list >> >> processors? ) processor on a cluster and that cluster is not running >> >> or >> >> configured to talk to zookeeper that he does not maintain state at all >> >> even >> >> though I would expect him to maintain state locally. >> >> >> >> EX: ListProcessor ( run on primary node ) > distribute to cluster and >> >> use >> >> ConsumeProcessor, >> >> >> >> * We accept the fact that if primary node changes it would lose state, >> >> but >> >> I want to maintain local state on cluster. >> >> >> >> I have tried changing the >> >> nifi.state.management.provider.cluster=local-provider and he fails >> >> since I >> >> am clustered. >> >> >> >> I am certainly going to do more testing unless its definitely true that >> >> if >> >> clustered it only maintains zookeeper state. >> >> >> >> Or it might be processor dependent? My tests have been with ListS3 and >> >> i >> >> am on Apache NIFI 0.7
Re: List processors
Thanks Bryan just tried and NIFI does not start because of this: " Cannot use Cluster State Provider ( WriteAheadLocalStateProvider ) as it only supports scope(s) {LOCAL} but instance is configured to use scope CLUSTER" On Fri, Apr 21, 2017 at 2:12 PM Bryan Bende <bbe...@gmail.com> wrote: > Juan, > > I believe from the processor side of the things, when a processor > calls save/retrieve on the state manager, the processor has to specify > a context like CLUSTER or LOCAL. If you specify CLUSTER, and no > clustered state provider exists, then it will save it to the local > provider. This allows a processor to work seamlessly across a > standalone Nifi and a clustered Nifi. > > The issue is that NiFi is not allowed to start up in clustered mode > without a clustered state provider, generally that is the ZooKeeper > provider, although it is an extension point and someone can implement > their own. > > I would think you could do the following... > > The normal clustered provider looks like this in state-management.xml > > > zk-provider > > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider > > /nifi > 10 seconds > Open > > > If you take the config from the local provider and drop it in the > cluster provider... > > > local-cluster-provider > > > org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider >./state/local >false >16 >2 mins > > > Basically defining another instance of the local state provider as the > cluster provider. > > Not totally sure if this works, but theoretically it should. > > -Bryan > > > On Fri, Apr 21, 2017 at 1:54 PM, Juan Sequeiros <helloj...@gmail.com> > wrote: > > To add more to the issue I see this on my log: > > > > "Failed to restore processor state; yielding java.io.IOException; Failed > to > > obtain value from Zookeeper for component " with exception code > > CONNECTIONLOSS > > > > So this is confirming what I expected and at this point not sure if this > is > > a bug or working as expected Feels like the dataflow manager should > > configure how to handle state and be able to use ListS3 in cluster even > > though I do not have zookeeper? > > > > > > > > On Fri, Apr 21, 2017 at 10:57 AM Juan Sequeiros <helloj...@gmail.com> > wrote: > >> > >> Hello all, > >> > >> My preliminary testing shows that if I run ListS3 ( maybe all list > >> processors? ) processor on a cluster and that cluster is not running or > >> configured to talk to zookeeper that he does not maintain state at all > even > >> though I would expect him to maintain state locally. > >> > >> EX: ListProcessor ( run on primary node ) > distribute to cluster and > use > >> ConsumeProcessor, > >> > >> * We accept the fact that if primary node changes it would lose state, > but > >> I want to maintain local state on cluster. > >> > >> I have tried changing the > >> nifi.state.management.provider.cluster=local-provider and he fails > since I > >> am clustered. > >> > >> I am certainly going to do more testing unless its definitely true that > if > >> clustered it only maintains zookeeper state. > >> > >> Or it might be processor dependent? My tests have been with ListS3 and i > >> am on Apache NIFI 0.7 >
Re: List processors
Juan, I believe from the processor side of the things, when a processor calls save/retrieve on the state manager, the processor has to specify a context like CLUSTER or LOCAL. If you specify CLUSTER, and no clustered state provider exists, then it will save it to the local provider. This allows a processor to work seamlessly across a standalone Nifi and a clustered Nifi. The issue is that NiFi is not allowed to start up in clustered mode without a clustered state provider, generally that is the ZooKeeper provider, although it is an extension point and someone can implement their own. I would think you could do the following... The normal clustered provider looks like this in state-management.xml zk-provider org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider /nifi 10 seconds Open If you take the config from the local provider and drop it in the cluster provider... local-cluster-provider org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider ./state/local false 16 2 mins Basically defining another instance of the local state provider as the cluster provider. Not totally sure if this works, but theoretically it should. -Bryan On Fri, Apr 21, 2017 at 1:54 PM, Juan Sequeiros <helloj...@gmail.com> wrote: > To add more to the issue I see this on my log: > > "Failed to restore processor state; yielding java.io.IOException; Failed to > obtain value from Zookeeper for component " with exception code > CONNECTIONLOSS > > So this is confirming what I expected and at this point not sure if this is > a bug or working as expected Feels like the dataflow manager should > configure how to handle state and be able to use ListS3 in cluster even > though I do not have zookeeper? > > > > On Fri, Apr 21, 2017 at 10:57 AM Juan Sequeiros <helloj...@gmail.com> wrote: >> >> Hello all, >> >> My preliminary testing shows that if I run ListS3 ( maybe all list >> processors? ) processor on a cluster and that cluster is not running or >> configured to talk to zookeeper that he does not maintain state at all even >> though I would expect him to maintain state locally. >> >> EX: ListProcessor ( run on primary node ) > distribute to cluster and use >> ConsumeProcessor, >> >> * We accept the fact that if primary node changes it would lose state, but >> I want to maintain local state on cluster. >> >> I have tried changing the >> nifi.state.management.provider.cluster=local-provider and he fails since I >> am clustered. >> >> I am certainly going to do more testing unless its definitely true that if >> clustered it only maintains zookeeper state. >> >> Or it might be processor dependent? My tests have been with ListS3 and i >> am on Apache NIFI 0.7
Re: List processors
To add more to the issue I see this on my log: "Failed to restore processor state; yielding java.io.IOException; Failed to obtain value from Zookeeper for component " with exception code CONNECTIONLOSS So this is confirming what I expected and at this point not sure if this is a bug or working as expected Feels like the dataflow manager should configure how to handle state and be able to use ListS3 in cluster even though I do not have zookeeper? On Fri, Apr 21, 2017 at 10:57 AM Juan Sequeiros <helloj...@gmail.com> wrote: > Hello all, > > My preliminary testing shows that if I run ListS3 ( maybe all list > processors? ) processor on a cluster and that cluster is not running or > configured to talk to zookeeper that he does not maintain state at all even > though I would expect him to maintain state locally. > > EX: ListProcessor ( run on primary node ) > distribute to cluster and use > ConsumeProcessor, > > * We accept the fact that if primary node changes it would lose state, but > I want to maintain local state on cluster. > > I have tried changing the > nifi.state.management.provider.cluster=local-provider and he fails since I > am clustered. > > I am certainly going to do more testing unless its definitely true that if > clustered it only maintains zookeeper state. > > Or it might be processor dependent? My tests have been with ListS3 and i > am on Apache NIFI 0.7 >
List processors
Hello all, My preliminary testing shows that if I run ListS3 ( maybe all list processors? ) processor on a cluster and that cluster is not running or configured to talk to zookeeper that he does not maintain state at all even though I would expect him to maintain state locally. EX: ListProcessor ( run on primary node ) > distribute to cluster and use ConsumeProcessor, * We accept the fact that if primary node changes it would lose state, but I want to maintain local state on cluster. I have tried changing the nifi.state.management.provider.cluster=local-provider and he fails since I am clustered. I am certainly going to do more testing unless its definitely true that if clustered it only maintains zookeeper state. Or it might be processor dependent? My tests have been with ListS3 and i am on Apache NIFI 0.7