Request: Add my account to the JIRA contributors list
Hello, I'd like to be able to assign JIRA issues to myself. My JIRA account uses this particular email address. Thanks in advance.
use case similar to Wait/Notify
Good morning Everyone, I am trying to see if I could make my scenario work with Wait/Notify processors. Basically, I require the following implementation: 1. Pass a single flow file into a processor group 2. Keep subsequent all flow flies in a wait queue until the first flow file is processed and sent out of the processor group to Notify processor, which updates the cache. Would I be able to accomplish this scenario without writing a custom processor? Thanks much, Yuri LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this e-mail and any files transmitted with it to be protected, proprietary or privileged information intended solely for the use of the named recipient(s). Any disclosure of this material or the information contained herein, in whole or in part, to anyone outside of the intended recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts no liability for the content of this e-mail or for the consequences of any actions taken on the basis of the information contained in it, unless that information is subsequently confirmed in writing. Employees of M.C. Dean, Inc. are instructed not to infringe on any rights of the recipient; any such communication violates company policy. If you are not the intended recipient, any disclosure, copying, distribution, or action taken or omitted in reliance on this information is strictly prohibited by M.C. Dean, Inc.; please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
Re: evaluateJsonPath processor
Hi Matt, Thanks. ³$" works well. On 7/5/17, 4:09 PM, "Matt Burgess" <mattyb...@apache.org> wrote: >Yuri, > >If your Return Type is set to "json" then you should be able to use >"$" as the JSON Path rather than "$." > >Also to put the entire content into an attribute you could use >ExtractText to match the entire body, this works whether the file is >JSON or not. > >Regards, >Matt > >On Wed, Jul 5, 2017 at 4:02 PM, Yuri Krysko <yuri.kry...@mcdean.com> >wrote: >> Hello Everyone, >> >> Could anyone please advise why $. expression is considered invalid in >>EvaluateJsonPath processor. The idea is to copy the entire content of a >>JSON-formatted flowfile into its attribute. >> >> Thanks, >> Yuri >> >> >> >> LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this >>e-mail and any files transmitted with it to be protected, proprietary or >>privileged information intended solely for the use of the named >>recipient(s). Any disclosure of this material or the information >>contained herein, in whole or in part, to anyone outside of the intended >>recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts >>no liability for the content of this e-mail or for the consequences of >>any actions taken on the basis of the information contained in it, >>unless that information is subsequently confirmed in writing. Employees >>of M.C. Dean, Inc. are instructed not to infringe on any rights of the >>recipient; any such communication violates company policy. If you are >>not the intended recipient, any disclosure, copying, distribution, or >>action taken or omitted in reliance on this information is strictly >>prohibited by M.C. Dean, Inc.; please notify the sender immediately by >>return e-mail, delete this communication and destroy all copies. LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this e-mail and any files transmitted with it to be protected, proprietary or privileged information intended solely for the use of the named recipient(s). Any disclosure of this material or the information contained herein, in whole or in part, to anyone outside of the intended recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts no liability for the content of this e-mail or for the consequences of any actions taken on the basis of the information contained in it, unless that information is subsequently confirmed in writing. Employees of M.C. Dean, Inc. are instructed not to infringe on any rights of the recipient; any such communication violates company policy. If you are not the intended recipient, any disclosure, copying, distribution, or action taken or omitted in reliance on this information is strictly prohibited by M.C. Dean, Inc.; please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
evaluateJsonPath processor
Hello Everyone, Could anyone please advise why $. expression is considered invalid in EvaluateJsonPath processor. The idea is to copy the entire content of a JSON-formatted flowfile into its attribute. Thanks, Yuri LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this e-mail and any files transmitted with it to be protected, proprietary or privileged information intended solely for the use of the named recipient(s). Any disclosure of this material or the information contained herein, in whole or in part, to anyone outside of the intended recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts no liability for the content of this e-mail or for the consequences of any actions taken on the basis of the information contained in it, unless that information is subsequently confirmed in writing. Employees of M.C. Dean, Inc. are instructed not to infringe on any rights of the recipient; any such communication violates company policy. If you are not the intended recipient, any disclosure, copying, distribution, or action taken or omitted in reliance on this information is strictly prohibited by M.C. Dean, Inc.; please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
Re: NiFi processor with custom UI
Will do. Thank you! On 6/28/17, 12:51 PM, "Matt Gilman" <matt.c.gil...@gmail.com> wrote: >Yuri, > >There is no maven archetype but there is some documentation in the >Developer Guide [1] with links to examples. Let us know if you have any >specific questions. > >Thanks! > >Matt > >[1] >https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#ui-extens >ions > >On Wed, Jun 28, 2017 at 12:48 PM, Yuri Krysko <yuri.kry...@mcdean.com> >wrote: > >> Hi Everyone, >> >> I am looking into developing a custom processor with custom UI. I am >> currently reviewing UpdateAttribute processor as an example of this. >>Could >> anyone please advise whether there's a maven archetype for generating >> boilerplate structure and files to support this type of processor? >> >> Thanks, >> Yuri >> >> >> >> LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this >> e-mail and any files transmitted with it to be protected, proprietary or >> privileged information intended solely for the use of the named >> recipient(s). Any disclosure of this material or the information >>contained >> herein, in whole or in part, to anyone outside of the intended >>recipient or >> affiliates is strictly prohibited. M. C. Dean, Inc. accepts no liability >> for the content of this e-mail or for the consequences of any actions >>taken >> on the basis of the information contained in it, unless that >>information is >> subsequently confirmed in writing. Employees of M.C. Dean, Inc. are >> instructed not to infringe on any rights of the recipient; any such >> communication violates company policy. If you are not the intended >> recipient, any disclosure, copying, distribution, or action taken or >> omitted in reliance on this information is strictly prohibited by M.C. >> Dean, Inc.; please notify the sender immediately by return e-mail, >>delete >> this communication and destroy all copies. >> LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this e-mail and any files transmitted with it to be protected, proprietary or privileged information intended solely for the use of the named recipient(s). Any disclosure of this material or the information contained herein, in whole or in part, to anyone outside of the intended recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts no liability for the content of this e-mail or for the consequences of any actions taken on the basis of the information contained in it, unless that information is subsequently confirmed in writing. Employees of M.C. Dean, Inc. are instructed not to infringe on any rights of the recipient; any such communication violates company policy. If you are not the intended recipient, any disclosure, copying, distribution, or action taken or omitted in reliance on this information is strictly prohibited by M.C. Dean, Inc.; please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
NiFi processor with custom UI
Hi Everyone, I am looking into developing a custom processor with custom UI. I am currently reviewing UpdateAttribute processor as an example of this. Could anyone please advise whether there's a maven archetype for generating boilerplate structure and files to support this type of processor? Thanks, Yuri LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this e-mail and any files transmitted with it to be protected, proprietary or privileged information intended solely for the use of the named recipient(s). Any disclosure of this material or the information contained herein, in whole or in part, to anyone outside of the intended recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts no liability for the content of this e-mail or for the consequences of any actions taken on the basis of the information contained in it, unless that information is subsequently confirmed in writing. Employees of M.C. Dean, Inc. are instructed not to infringe on any rights of the recipient; any such communication violates company policy. If you are not the intended recipient, any disclosure, copying, distribution, or action taken or omitted in reliance on this information is strictly prohibited by M.C. Dean, Inc.; please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
Re: periodic exception
Mark, Joe,
Thanks for your input. I am about to deploy my processor to another server
that has standard logging configured. I will give it a run in the
environment mimicking our future production. I will let you know if there
are any unknown exceptions and post the logs then.
Thank you for such quick responses
On 5/23/17, 5:06 PM, "Mark Payne" <marka...@hotmail.com> wrote:
>Yuri,
>
>All sessions get rolled back when they are garbage collected. Just in
>case the session wasn't properly handled.
>It looks like you've got DEBUG logging turned on for
>StandardProcessSession,
>so it is logging a stack trace just to show you exactly how rollback()
>was called. You'll get some pretty verbose
>logging about the internal workings of NiFi so you may want to not have
>DEBUG logging on for framework components
>if not debugging the framework itself :)
>
>Thanks
>-Mark
>
>> On May 23, 2017, at 4:57 PM, Yuri Krysko <yuri.kry...@mcdean.com> wrote:
>>
>> Hi Devs,
>>
>> I am developing a custom processor for nifi. My first one. I am seeing
>>the below periodically in the app log. To my knowledge, there should not
>>be any rollbacks, as my events are processed as expected. Overall, the
>>code seems to work fine. I would like to understand how to t-shoot this
>>better. Thanks!
>>
>>
>>
>> 2017-05-23 16:51:01,931 INFO [Provenance Repository Rollover Thread-1]
>>o.a.n.p.PersistentProvenanceRepository Successfully Rolled over
>>Provenance Event file containing 2 records. In the past 5 minutes, 1
>>events have been written to the Provenance Repository, totaling 1.13 KB
>>
>> 2017-05-23 16:51:20,046 DEBUG [Finalizer]
>>o.a.n.c.r.StandardProcessSession StandardProcessSession[id=0] session
>>rollback called, FlowFile records are [none] {}
>>
>> java.lang.Throwable: Stack Trace on rollback
>>
>> at
>>org.apache.nifi.controller.repository.StandardProcessSession.rollback(Sta
>>ndardProcessSession.java:900)
>>
>> at
>>org.apache.nifi.controller.repository.StandardProcessSession.rollback(Sta
>>ndardProcessSession.java:894)
>>
>> at
>>org.apache.nifi.controller.repository.StandardProcessSession.rollback(Sta
>>ndardProcessSession.java:889)
>>
>> at
>>org.apache.nifi.controller.repository.StandardProcessSession.finalize(Sta
>>ndardProcessSession.java:3021)
>>
>> at java.lang.System$2.invokeFinalize(System.java:1270)
>>
>> at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
>>
>> at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
>>
>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
>>
>>
>>
>> LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this
>>e-mail and any files transmitted with it to be protected, proprietary or
>>privileged information intended solely for the use of the named
>>recipient(s). Any disclosure of this material or the information
>>contained herein, in whole or in part, to anyone outside of the intended
>>recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts
>>no liability for the content of this e-mail or for the consequences of
>>any actions taken on the basis of the information contained in it,
>>unless that information is subsequently confirmed in writing. Employees
>>of M.C. Dean, Inc. are instructed not to infringe on any rights of the
>>recipient; any such communication violates company policy. If you are
>>not the intended recipient, any disclosure, copying, distribution, or
>>action taken or omitted in reliance on this information is strictly
>>prohibited by M.C. Dean, Inc.; please notify the sender immediately by
>>return e-mail, delete this communication and destroy all copies.
>
LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this e-mail
and any files transmitted with it to be protected, proprietary or privileged
information intended solely for the use of the named recipient(s). Any
disclosure of this material or the information contained herein, in whole or in
part, to anyone outside of the intended recipient or affiliates is strictly
prohibited. M. C. Dean, Inc. accepts no liability for the content of this
e-mail or for the consequences of any actions taken on the basis of the
information contained in it, unless that information is subsequently confirmed
in writing. Employees of M.C. Dean, Inc. are instructed not to infringe on any
rights of the recipient; any such communication violates company policy. If you
are not the intended recipient, any disclosure, copying, distribution, or
action taken or omitted in reliance on this information is strictly prohibited
by M.C. Dean, Inc.; please notify the sender immediately by return e-mail,
delete this communication and destroy all copies.
periodic exception
Hi Devs,
I am developing a custom processor for nifi. My first one. I am seeing the
below periodically in the app log. To my knowledge, there should not be any
rollbacks, as my events are processed as expected. Overall, the code seems to
work fine. I would like to understand how to t-shoot this better. Thanks!
2017-05-23 16:51:01,931 INFO [Provenance Repository Rollover Thread-1]
o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance
Event file containing 2 records. In the past 5 minutes, 1 events have been
written to the Provenance Repository, totaling 1.13 KB
2017-05-23 16:51:20,046 DEBUG [Finalizer] o.a.n.c.r.StandardProcessSession
StandardProcessSession[id=0] session rollback called, FlowFile records are
[none] {}
java.lang.Throwable: Stack Trace on rollback
at
org.apache.nifi.controller.repository.StandardProcessSession.rollback(StandardProcessSession.java:900)
at
org.apache.nifi.controller.repository.StandardProcessSession.rollback(StandardProcessSession.java:894)
at
org.apache.nifi.controller.repository.StandardProcessSession.rollback(StandardProcessSession.java:889)
at
org.apache.nifi.controller.repository.StandardProcessSession.finalize(StandardProcessSession.java:3021)
at java.lang.System$2.invokeFinalize(System.java:1270)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this e-mail
and any files transmitted with it to be protected, proprietary or privileged
information intended solely for the use of the named recipient(s). Any
disclosure of this material or the information contained herein, in whole or in
part, to anyone outside of the intended recipient or affiliates is strictly
prohibited. M. C. Dean, Inc. accepts no liability for the content of this
e-mail or for the consequences of any actions taken on the basis of the
information contained in it, unless that information is subsequently confirmed
in writing. Employees of M.C. Dean, Inc. are instructed not to infringe on any
rights of the recipient; any such communication violates company policy. If you
are not the intended recipient, any disclosure, copying, distribution, or
action taken or omitted in reliance on this information is strictly prohibited
by M.C. Dean, Inc.; please notify the sender immediately by return e-mail,
delete this communication and destroy all copies.
Re: Custom processor UI exception
Hi Koji, Thank you for your response. Indeed, you are right. I have sorted it out already :) I had a custom Abstract processor class which I was extending, and misused init method. Thanks again! On 5/21/17, 9:38 PM, "Koji Kawamura" <ijokaruma...@gmail.com> wrote: >Hello Yuri, > >From the stack trace (line numbers) I assume you are using NiFi 1.2.0. >If so, I think NullPointer can be thrown if your custom processor >returns null from its getRelationships() method. A list of possible >relationships from a processor should be returned, if the processor >doesn't have outgoing relationship, then return >Collections.emptySet(). > >Otherwise, please let us know which version you are using. > >Thanks, >Koji > >On Sat, May 20, 2017 at 12:41 AM, Yuri Krysko <yuri.kry...@mcdean.com> >wrote: >> Hello Devs, >> >> I am working on a custom processor for NiFi and I am getting the below >>exception in the nifi-user.log when I try to add the processor to the >>flow canvas. Could you please advise what could I be doing wrong. No >>errors in the nifi-app.log though. Thanks! >> >> >> 2017-05-19 11:31:07,983 ERROR [NiFi Web Server-22] >>o.a.nifi.web.api.config.ThrowableMapper An unexpected error has >>occurred: java.lang.NullPointerException. Returning Internal Server >>Error response. >> >> java.lang.NullPointerException: null >> >> at >>org.apache.nifi.web.api.dto.DtoFactory.createProcessorDto(DtoFactory.java >>:2226) >> >> at >>org.apache.nifi.web.api.dto.DtoFactory.createFlowDto(DtoFactory.java:1884 >>) >> >> at >>org.apache.nifi.web.api.dto.DtoFactory.createProcessGroupFlowDto(DtoFacto >>ry.java:1726) >> >> at >>org.apache.nifi.web.StandardNiFiServiceFacade.getProcessGroupFlow(Standar >>dNiFiServiceFacade.java:3055) >> >> at >>org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$35 >>8780e0.invoke() >> >> at >>org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) >> >> at >>org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.inv >>okeJoinpoint(CglibAopProxy.java:720) >> >> at >>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl >>ectiveMethodInvocation.java:157) >> >> at >>org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proce >>ed(MethodInvocationProceedingJoinPoint.java:85) >> >> at >>org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithReadLock(NiFiService >>FacadeLock.java:137) >> >> at >>org.apache.nifi.web.NiFiServiceFacadeLock.getLock(NiFiServiceFacadeLock.j >>ava:108) >> >> at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source) >> >> at >>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI >>mpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:498) >> >> at >>org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodW >>ithGivenArgs(AbstractAspectJAdvice.java:621) >> >> at >>org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod( >>AbstractAspectJAdvice.java:610) >> >> at >>org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundA >>dvice.java:68) >> >> at >>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl >>ectiveMethodInvocation.java:179) >> >> at >>org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Ex >>poseInvocationInterceptor.java:92) >> >> at >>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl >>ectiveMethodInvocation.java:179) >> >> at >>org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor >>.intercept(CglibAopProxy.java:655) >> >> at >>org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$c6b >>bb5bc.getProcessGroupFlow() >> >> at org.apache.nifi.web.api.FlowResource.getFlow(FlowResource.java:375) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java >>:62) >> >> at >>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI >>mpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:498) >> >> at >>com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethod &g
Controller service question
Hello Devs, I discovered Apache NiFi several days ago, parsed through all the docs and would like to ask for your opinion on my use-case. I plan to have several custom processors to do HTTP POSTs to a 3rd party ticketing system via its REST API based on the Flowfiles fed by other processors. In order to compose JSON payload messages and POST to this ticketing system, in addition to the FlowFiles, these custom processors require metadata info from the ticketing system. The metadata may or may not change over time, as it is potentially configurable by the ticketing system admins. I am wondering whether I should create a custom Controller service to pull the metadata and make it available to all interested processors? Can I include a Runnable task in the Controller service and have a configurable parameter of how often to pull the data? Thank you, Yuri LEGAL DISCLAIMER: M.C. Dean, Inc. and its subsidiaries considers this e-mail and any files transmitted with it to be protected, proprietary or privileged information intended solely for the use of the named recipient(s). Any disclosure of this material or the information contained herein, in whole or in part, to anyone outside of the intended recipient or affiliates is strictly prohibited. M. C. Dean, Inc. accepts no liability for the content of this e-mail or for the consequences of any actions taken on the basis of the information contained in it, unless that information is subsequently confirmed in writing. Employees of M.C. Dean, Inc. are instructed not to infringe on any rights of the recipient; any such communication violates company policy. If you are not the intended recipient, any disclosure, copying, distribution, or action taken or omitted in reliance on this information is strictly prohibited by M.C. Dean, Inc.; please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
