Re: Need help with DetectDuplicate
William, DistributedMapCacheClientService works in a standalone NiFi. Do you have a DistributedMapCacheServer configured for localhost:4557 and enabled? On Thu, Jan 2, 2020 at 4:39 PM William Gosse wrote: > > Will the DetectDuplicate with DistributedMapCacheClientService only work in a > cluster? I’m trying to use it in a standalone NiFi. > > > > From: Emanuel Oliveira > Sent: Tuesday, December 24, 2019 4:29 PM > To: users@nifi.apache.org > Subject: Re: Need help with DetectDuplicate > > > > [CAUTION: This email originated from outside of Kodak Alaris. Do not click > links or open attachments unless you recognize the sender and know the > content is safe.] > > > > Hi, > > > > Depending on how your cluster setup you may need to add/setup ssl controller > service? > > > > Emanuel > > > > On Tue 24 Dec 2019, 18:16 William Gosse, wrote: > > I’m trying to use DetectDuplicate processor but not having much luck. Here > the config: > > Cache Entry Identifier > > ${resourceId} > > FlowFile Description > > Ingestion > > Age Off Duration > > 60 sec > > Cache The Entry Identifier > > true > > Distributed Cache Service > > DistributedMapCacheClientService > > > > I created abd enabled a DistributedMapCacheClientService. Here’s its config: > > Server Hostname > > localhost > > Server Port > > 4557 > > SSL Context ServiceNo value setCommunications Timeout > > 30 secs > > > > When I run it I get the following error: > > 2019-12-24 13:14:05,355 ERROR [Timer-Driven Process Thread-9] > o.a.n.p.standard.DetectDuplicate > DetectDuplicate[id=38cb8a64-016f-1000-b55b-f6c4e0f69f61] Unable to > communicate with cache when processing > StandardFlowFileRecord[uuid=7f049d8e-1d04-4fee-9f04-320d6980bc55,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1577202782598-43, container=default, > section=43], offset=74111, > length=4528],offset=0,name=84068ffb-69b1-4471-abbd-29243d3be39e,size=4528] > due to java.net.ConnectException: Connection refused: no further information: > java.net.ConnectException: Connection refused: no further information > > java.net.ConnectException: Connection refused: no further information > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) > > at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) > > at > org.apache.nifi.distributed.cache.client.StandardCommsSession.(StandardCommsSession.java:52) > > at > org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.createCommsSession(DistributedMapCacheClientService.java:410) > > at > org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:425) > > at > org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:491) > > at > org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:174) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87) > > at com.sun.proxy.$Proxy142.getAndPutIfAbsent(Unknown Source) > > at > org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) > > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) > > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) > > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > > at > org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > > at > j
RE: Need help with DetectDuplicate
Will the DetectDuplicate with DistributedMapCacheClientService only work in a cluster? I’m trying to use it in a standalone NiFi. From: Emanuel Oliveira Sent: Tuesday, December 24, 2019 4:29 PM To: users@nifi.apache.org Subject: Re: Need help with DetectDuplicate [CAUTION: This email originated from outside of Kodak Alaris. Do not click links or open attachments unless you recognize the sender and know the content is safe.] Hi, Depending on how your cluster setup you may need to add/setup ssl controller service? Emanuel On Tue 24 Dec 2019, 18:16 William Gosse, mailto:william.go...@aifoundry.com>> wrote: I’m trying to use DetectDuplicate processor but not having much luck. Here the config: Cache Entry Identifier ${resourceId} FlowFile Description Ingestion Age Off Duration 60 sec Cache The Entry Identifier true Distributed Cache Service DistributedMapCacheClientService I created abd enabled a DistributedMapCacheClientService. Here’s its config: Server Hostname localhost Server Port 4557 SSL Context ServiceNo value setCommunications Timeout 30 secs When I run it I get the following error: 2019-12-24 13:14:05,355 ERROR [Timer-Driven Process Thread-9] o.a.n.p.standard.DetectDuplicate DetectDuplicate[id=38cb8a64-016f-1000-b55b-f6c4e0f69f61] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=7f049d8e-1d04-4fee-9f04-320d6980bc55,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1577202782598-43, container=default, section=43], offset=74111, length=4528],offset=0,name=84068ffb-69b1-4471-abbd-29243d3be39e,size=4528] due to java.net.ConnectException: Connection refused: no further information: java.net.ConnectException: Connection refused: no further information java.net.ConnectException: Connection refused: no further information at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) at org.apache.nifi.distributed.cache.client.StandardCommsSession.(StandardCommsSession.java:52) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.createCommsSession(DistributedMapCacheClientService.java:410) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:425) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:491) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:174) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87) at com.sun.proxy.$Proxy142.getAndPutIfAbsent(Unknown Source) at org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Not sure whats missing?
Re: Need help with DetectDuplicate
Did you create the actual map cache server in controller services? I couldn’t tell. All I saw was the client service. Thanks Shawn Sent from my iPhone On Dec 24, 2019, at 12:16 PM, William Gosse wrote: I’m trying to use DetectDuplicate processor but not having much luck. Here the config: Cache Entry Identifier ${resourceId} FlowFile Description Ingestion Age Off Duration 60 sec Cache The Entry Identifier true Distributed Cache Service DistributedMapCacheClientService I created abd enabled a DistributedMapCacheClientService. Here’s its config: Server Hostname localhost Server Port 4557 SSL Context ServiceNo value setCommunications Timeout 30 secs When I run it I get the following error: 2019-12-24 13:14:05,355 ERROR [Timer-Driven Process Thread-9] o.a.n.p.standard.DetectDuplicate DetectDuplicate[id=38cb8a64-016f-1000-b55b-f6c4e0f69f61] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=7f049d8e-1d04-4fee-9f04-320d6980bc55,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1577202782598-43, container=default, section=43], offset=74111, length=4528],offset=0,name=84068ffb-69b1-4471-abbd-29243d3be39e,size=4528] due to java.net.ConnectException: Connection refused: no further information: java.net.ConnectException: Connection refused: no further information java.net.ConnectException: Connection refused: no further information at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) at org.apache.nifi.distributed.cache.client.StandardCommsSession.(StandardCommsSession.java:52) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.createCommsSession(DistributedMapCacheClientService.java:410) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:425) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:491) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:174) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87) at com.sun.proxy.$Proxy142.getAndPutIfAbsent(Unknown Source) at org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Not sure whats missing?
Re: Need help with DetectDuplicate
Hi, Depending on how your cluster setup you may need to add/setup ssl controller service? Emanuel On Tue 24 Dec 2019, 18:16 William Gosse, wrote: > I’m trying to use DetectDuplicate processor but not having much luck. Here > the config: > > Cache Entry Identifier > > ${resourceId} > > FlowFile Description > > Ingestion > > Age Off Duration > > 60 sec > > Cache The Entry Identifier > > true > > Distributed Cache Service > > DistributedMapCacheClientService > > > > I created abd enabled a DistributedMapCacheClientService. Here’s its > config: > > Server Hostname > > localhost > > Server Port > > 4557 > > SSL Context ServiceNo value setCommunications Timeout > > 30 secs > > > > When I run it I get the following error: > > 2019-12-24 13:14:05,355 ERROR [Timer-Driven Process Thread-9] > o.a.n.p.standard.DetectDuplicate > DetectDuplicate[id=38cb8a64-016f-1000-b55b-f6c4e0f69f61] Unable to > communicate with cache when processing > StandardFlowFileRecord[uuid=7f049d8e-1d04-4fee-9f04-320d6980bc55,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1577202782598-43, > container=default, section=43], offset=74111, > length=4528],offset=0,name=84068ffb-69b1-4471-abbd-29243d3be39e,size=4528] > due to java.net.ConnectException: Connection refused: no further > information: java.net.ConnectException: Connection refused: no further > information > > java.net.ConnectException: Connection refused: no further information > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) > > at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) > > at > org.apache.nifi.distributed.cache.client.StandardCommsSession.(StandardCommsSession.java:52) > > at > org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.createCommsSession(DistributedMapCacheClientService.java:410) > > at > org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:425) > > at > org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:491) > > at > org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:174) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87) > > at com.sun.proxy.$Proxy142.getAndPutIfAbsent(Unknown > Source) > > at > org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) > > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) > > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) > > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > > at > org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > Not sure whats missing? >
Need help with DetectDuplicate
I’m trying to use DetectDuplicate processor but not having much luck. Here the config: Cache Entry Identifier ${resourceId} FlowFile Description Ingestion Age Off Duration 60 sec Cache The Entry Identifier true Distributed Cache Service DistributedMapCacheClientService I created abd enabled a DistributedMapCacheClientService. Here’s its config: Server Hostname localhost Server Port 4557 SSL Context ServiceNo value setCommunications Timeout 30 secs When I run it I get the following error: 2019-12-24 13:14:05,355 ERROR [Timer-Driven Process Thread-9] o.a.n.p.standard.DetectDuplicate DetectDuplicate[id=38cb8a64-016f-1000-b55b-f6c4e0f69f61] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=7f049d8e-1d04-4fee-9f04-320d6980bc55,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1577202782598-43, container=default, section=43], offset=74111, length=4528],offset=0,name=84068ffb-69b1-4471-abbd-29243d3be39e,size=4528] due to java.net.ConnectException: Connection refused: no further information: java.net.ConnectException: Connection refused: no further information java.net.ConnectException: Connection refused: no further information at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) at org.apache.nifi.distributed.cache.client.StandardCommsSession.(StandardCommsSession.java:52) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.createCommsSession(DistributedMapCacheClientService.java:410) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:425) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:491) at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.getAndPutIfAbsent(DistributedMapCacheClientService.java:174) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87) at com.sun.proxy.$Proxy142.getAndPutIfAbsent(Unknown Source) at org.apache.nifi.processors.standard.DetectDuplicate.onTrigger(DetectDuplicate.java:183) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Not sure whats missing?