Christian,
It looks like you are attempting to use a DistributedMapCacheClientService with
a DistributedSetCacheServer. I.e., you’re using a MAP-based client with a
SET-based server. You need to use a DistributedMapCacheServer, not a
DistributedSetCacheServer.
Thanks
-Mark
On Jan 10, 2022, a
Hi guys,
we got a problem using DistributedMapCacheClientService with
DistributedSetCacheServer in NiFi 1.15.1.
After setting up both services we got the following exception:
2022-01-10 16:00:15,519 ERROR [Timer-Driven Process Thread-7]
org.apache.nifi.processors.standard.Wait
Wait[id=4a04023
on't understand what it's doing.
1. I can use mburgess's dcache.groovy
script<https://community.cloudera.com/t5/Community-Articles/Working-with-a-NiFi-DistributedMapCache/ta-p/248370>
to list and remove cache entries from the DMC. If I stop a particular
ListFile, delete its
Yes from a quick look at the code, ListSFTP should be able to work fine
with out the distributed cache.
If you are interested, the relevant code is in the updateState method of
this class:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/sr
Thanks for the explanation Bryan. Do you know offhand if ListSFTP
specifically has the logic to check for the distributed cache first, and
then will fail over to use the more recent state management? If so, I
should be able to remove the reference to the distributed map cache client
service and sti
I believe that Pierre's last point about this processor being developed
before NiFi's built in state management feature is correct. Many processors
would originally store state in a local file as well as in the distributed
cache, they were still meant to run only on the primary node, but this way
Not sure I'm following you on "So, the DMC is just so you won't duplicate
fetches if you're listing faster than you're fetching... got it". :)
Let's say the DMC is just here to store the state of the List processor
across the cluster in case the node goes down and a new primary node is
elected. Bu
Pierre,
Thank you for the quick response. So, the DMC is just so you won't
duplicate fetches if you're listing faster than you're fetching... got it.
The usage documentation is kinda vague about that, so I made it out to be
more magical than it is. Thanks for pointing me in the right direction!
-
Hi Nicholas,
You need to configure your ListSFTP processor to only run on the primary
node (scheduling strategy in processor configuration), then to send the
flow files to a RPG that points to an input port in the cluster itself (so
that flow files are distributed over the cluster and do not stay
I'm testing a simple List/Fetch setup on a 3 node cluster. I created a
DistributedMapCacheServer controller service with the default settings (no
SSL) and then created a DistributedMapCacheClientService that points at one
of the cluster hostnames. The ListSFTP processor is set to use the
Distribute
s overkill for NiFi. If one needs such level of
>> availability, then I'd recommend to use other NoSQL databases.
>>
>> How do you think about that? I'd like to hear from others, too, to see
>> if it's worth for trying.
>>
>> Thanks,
>> Koji
&g
gt; mailto:yari.marche...@buongiorno.com>> wrote:
> > Hello,
> > I'm running a 3 nodes cluster and I've been trying to implement a
> > deduplication workflow using the DetectDuplicate but, on my first try, I
> > noticed that there were always 3 messages marked
ster and I've been trying to implement a
> > deduplication workflow using the DetectDuplicate but, on my first try, I
> > noticed that there were always 3 messages marked as non-duplicates. After
> > some investigation I tracked down this issue to be related to a
> > co
deduplication workflow using the DetectDuplicate but, on my first try, I
> noticed that there were always 3 messages marked as non-duplicates. After
> some investigation I tracked down this issue to be related to a
> configuration I did for DistributedMapCache server address which was set to
figuration I did for DistributedMapCache server address which was set to
localhost: if instead I set it to the IP of one of the nodes than
everything's working as expected.
My concern with this approach is of reliability: if that specific node goes
down, than the workflow will not work pro
15 matches
Mail list logo
