Thanks Bryan just tried and NIFI does not start because of this:
" Cannot use Cluster State Provider ( WriteAheadLocalStateProvider ) as it
only supports scope(s) {LOCAL} but instance is configured to use scope
CLUSTER"
On Fri, Apr 21, 2017 at 2:12 PM Bryan Bende <bbe...@gmail.com> wrote:
> Juan,
>
> I believe from the processor side of the things, when a processor
> calls save/retrieve on the state manager, the processor has to specify
> a context like CLUSTER or LOCAL. If you specify CLUSTER, and no
> clustered state provider exists, then it will save it to the local
> provider. This allows a processor to work seamlessly across a
> standalone Nifi and a clustered Nifi.
>
> The issue is that NiFi is not allowed to start up in clustered mode
> without a clustered state provider, generally that is the ZooKeeper
> provider, although it is an extension point and someone can implement
> their own.
>
> I would think you could do the following...
>
> The normal clustered provider looks like this in state-management.xml....
>
> <cluster-provider>
> <id>zk-provider</id>
>
> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
> <property name="Connect String"></property>
> <property name="Root Node">/nifi</property>
> <property name="Session Timeout">10 seconds</property>
> <property name="Access Control">Open</property>
> </cluster-provider>
>
> If you take the config from the local provider and drop it in the
> cluster provider...
>
> <cluster-provider>
> <id>local-cluster-provider</id>
>
>
> <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class>
> <property name="Directory">./state/local</property>
> <property name="Always Sync">false</property>
> <property name="Partitions">16</property>
> <property name="Checkpoint Interval">2 mins</property>
> </cluster-provider>
>
> Basically defining another instance of the local state provider as the
> cluster provider.
>
> Not totally sure if this works, but theoretically it should.
>
> -Bryan
>
>
> On Fri, Apr 21, 2017 at 1:54 PM, Juan Sequeiros <helloj...@gmail.com>
> wrote:
> > To add more to the issue I see this on my log:
> >
> > "Failed to restore processor state; yielding java.io.IOException; Failed
> to
> > obtain value from Zookeeper for component ...." with exception code
> > CONNECTIONLOSS
> >
> > So this is confirming what I expected and at this point not sure if this
> is
> > a bug or working as expected .... Feels like the dataflow manager should
> > configure how to handle state and be able to use ListS3 in cluster even
> > though I do not have zookeeper?
> >
> >
> >
> > On Fri, Apr 21, 2017 at 10:57 AM Juan Sequeiros <helloj...@gmail.com>
> wrote:
> >>
> >> Hello all,
> >>
> >> My preliminary testing shows that if I run ListS3 ( maybe all list
> >> processors? ) processor on a cluster and that cluster is not running or
> >> configured to talk to zookeeper that he does not maintain state at all
> even
> >> though I would expect him to maintain state locally.
> >>
> >> EX: ListProcessor ( run on primary node ) > distribute to cluster and
> use
> >> ConsumeProcessor,
> >>
> >> * We accept the fact that if primary node changes it would lose state,
> but
> >> I want to maintain local state on cluster.
> >>
> >> I have tried changing the
> >> nifi.state.management.provider.cluster=local-provider and he fails
> since I
> >> am clustered.
> >>
> >> I am certainly going to do more testing unless its definitely true that
> if
> >> clustered it only maintains zookeeper state.
> >>
> >> Or it might be processor dependent? My tests have been with ListS3 and i
> >> am on Apache NIFI 0.7
>
