Re: Nifi question

[Jim Wagoner]

Thanks Joe

On Tue, Apr 26, 2016 at 5:48 PM, Joe Witt  wrote:

> We're a gentle bunch.  Fire away and ask questions as needed.  You can
> reply to this to get this back on list.  All good.
>
> So yeah we need to handle these cases better.  You should not have to
> worry about this so that is on us to improve.  Let's go ahead and
> start this dialogue.
>
> Thanks
> Joe
>
> On Tue, Apr 26, 2016 at 1:31 PM, Jim Wagoner  wrote:
> > It is a few hundred thousand so that definitely sounds like the issue I
> am
> > seeing. On a previous project I had the same issue, but had much more
> > powerful machine compared to the EC2 instance I am running now so I could
> > just throw hardware at the problem.
> >
> > The dev list is fine. Just wasn't sure of the etiquette when submitting
> to
> > the dev list.
> >
> > On Tue, Apr 26, 2016 at 5:27 PM, Joe Witt  wrote:
> >>
> >> Jim,
> >>
> >> I suspect the issue lies elsewhere really unless there are hundreds of
> >> thousands of objects in each zip file.  We don't load the data in
> >> memory while compressing or decompressing because we have a streaming
> >> API for operating on the data.  However, we do keep pointers and
> >> attributes of those flow files so after hundreds of thousands those
> >> could become problematic.
> >>
> >> Any reason to be off-list or is it ok if I put this on the dev list?
> >>
> >> Thanks
> >> Joe
> >>
> >> On Tue, Apr 26, 2016 at 1:21 PM, Jim Wagoner 
> wrote:
> >> > Joe,
> >> >
> >> >  I have been using the unpack processor to uncompress some fairly
> >> > large
> >> > files and have been running into memory and performance issues. I
> wrote
> >> > my
> >> > own unpacker that transfers files as they are uncompress (clearly not
> >> > side
> >> > affect free) and it seems to help with memory usage, performance and
> >> > gives
> >> > me the option of handling corrupt zips. I wanted see if you could this
> >> > of
> >> > any issues within the framework with taking this approach. Is there a
> >> > pattern I should look at when creating a large number of flow files
> in a
> >> > single processor call within Nifi?
> >> >
> >> > Thanks,
> >> > Jim
> >
> >
>

Re: nifi question

[Joe Witt]

Hello,

Could you share a screenshot.  Also, which web browser (and version)
are you using ?  Finally, which version of NiFi?

Thanks
JOe

On Wed, Feb 17, 2016 at 10:24 AM, yonatan kebede  wrote:
> hi,
> i am new to nifi and have got problems on the UI, that its not displaying
> the icons on top of the processors and also in the connections/relations,
> it was working in windows 8 but when i update it to win 10 , sth like this
> happen?
>
> what do you think is the problem? or any helpis appreciated
>
> Regards,
>
> Yonatan Kebede.

nifi question

[yonatan kebede]

hi,
i am new to nifi and have got problems on the UI, that its not displaying
the icons on top of the processors and also in the connections/relations,
it was working in windows 8 but when i update it to win 10 , sth like this
happen?

what do you think is the problem? or any helpis appreciated

Regards,

Yonatan Kebede.

Re: nifi question

[Mark Payne]

Sumo,

Generally, the approach that we take is to perform that type of operation in a 
method that has the @OnScheduled annotation.

Then you do it only once when the Processor is scheduled.

If you want to ensure that it happens only one time for the lifecycle of the 
JVM, you could use a boolean to keep track of whether or
not the action has been performed. For example:

private volatile boolean expensiveActionPerformed = false;

@OnScheduled
public void doExpensiveSetup(final ProcessContext context) {
if (!expensiveActionPerformed) {

// do expensive action

expensiveActionPerformed = true;
}
}


Thanks
-Mark


> On Sep 28, 2015, at 5:38 PM, Sumanth Chinthagunta  wrote:
> 
> Thanks Mark for pointing me to GetTwitter example. I will try to follow this 
> pattern. 
> PropertyDescriptor is only available in onTrigger method via ProcessContext. 
> I needed to get hold of Service from  initialize method using 
> ProcessorInitializationContexts as documented in the Developer guide. this 
> helps me to do expensive Service calls once not doing every time onTrigger 
> method is invoked. 
> 
> Thanks 
> Sumo
> 
> 
>> On Sep 28, 2015, at 5:32 AM, Mark Payne  wrote:
>> 
>> Sumo,
>> 
>> The preferred mechanism for obtaining a Controller Service is to do so via a 
>> PropertyDescriptor. You would
>> specify that the property represents a Controller Service by using the 
>> identifiesControllerService(Class) method.
>> This is discussed in the "Interacting with a Controller Service" section of 
>> the Developer's Guide [1].
>> 
>> In terms of communicating with some asynchronous process, generally the best 
>> solution is to use a queue and
>> then read from that queue in the onTrigger method. You will not be able to 
>> call the onTrigger method properly
>> yourself, as it is designed to be called by the framework at the appropriate 
>> time. I would recommend using
>> a BlockingQueue with a small capacity, as you do not want your Java Heap to 
>> fill up with objects from this
>> Processor if the Processor is stopped for a while. You can look at how this 
>> is done in the GetTwitter processor,
>> if you would like to have an example to look at.
>> 
>> Let us know if you have any more questions, or if anything is still not 
>> clear!
>> 
>> Thanks
>> -Mark
>> 
>> 
>> [1] 
>> http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#interacting-with-controller-service
>> 
>> 
>> 
>>> On Sep 27, 2015, at 3:53 PM, Sumanth Chinthagunta  wrote:
>>> 
>>> 
>>> Hi All,
>>> I am new to NiFi and   I'm stuck with couple issues:  
>>> 
>>> 1. Unable to get hold of ControllerService  from Processor’s init method. 
>>> I wanted to pre-set some dependencies in my Processor during init phase 
>>> instead of  querying them repeatedly in onTrigger method. 
>>> I am getting null for service and not sure what I have to pass for 
>>> 'serviceIdentifier’ . I couldn't find documentation or examples on how to 
>>> give Identifier to a service.  
>>> 
>>> 
>>> final VertxServiceInterface vertxService = (VertxServiceInterface) 
>>> context.getControllerServiceLookup().getControllerService("VertxService”)
>>> 
>>> 
>>> https://github.com/xmlking/nifi-websocket/blob/master/src/main/java/com/crossbusiness/nifi/processors/PutEventBus.java#L55
>>>  
>>> 
>>> 
>>> 2. for my usecase I get data published to a topic from EventBus with 
>>> following code. 
>>> 
>>> EventBus eb = vertx.eventBus();
>>> 
>>> eb.consumer("news.uk.sport", message -> {
>>> System.out.println("I have received a message: " + 
>>> message.body());
>>> });
>>> 
>>> I am working on a date ingest processor (push based) that needs to 
>>> listen for new messages on a topic and send them to flow as FlowFile. 
>>> In my case data source is EvenBus that expose emit messages via 
>>> callback API. 
>>> I am looking for ideas on how to call Processor’s onTrigger method when 
>>> the above callback is evoked. 
>>> Should I have to use my own intermediate queue and poll it in onTrigger 
>>> method? 
>>> is there a better way to trigger the  onTrigger method programmatically 
>>> ? 
>>> 
>>> Thanks 
>>> Sumo
>> 
> 

Re: nifi question

[Mark Payne]

Sumo,

The preferred mechanism for obtaining a Controller Service is to do so via a 
PropertyDescriptor. You would
specify that the property represents a Controller Service by using the 
identifiesControllerService(Class) method.
This is discussed in the "Interacting with a Controller Service" section of the 
Developer's Guide [1].

In terms of communicating with some asynchronous process, generally the best 
solution is to use a queue and
then read from that queue in the onTrigger method. You will not be able to call 
the onTrigger method properly
yourself, as it is designed to be called by the framework at the appropriate 
time. I would recommend using
a BlockingQueue with a small capacity, as you do not want your Java Heap to 
fill up with objects from this
Processor if the Processor is stopped for a while. You can look at how this is 
done in the GetTwitter processor,
if you would like to have an example to look at.

Let us know if you have any more questions, or if anything is still not clear!

Thanks
-Mark


[1] 
http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#interacting-with-controller-service



> On Sep 27, 2015, at 3:53 PM, Sumanth Chinthagunta  wrote:
> 
> 
> Hi All,
> I am new to NiFi and   I'm stuck with couple issues:  
> 
> 1. Unable to get hold of ControllerService  from Processor’s init method. 
>   I wanted to pre-set some dependencies in my Processor during init phase 
> instead of  querying them repeatedly in onTrigger method. 
>   I am getting null for service and not sure what I have to pass for 
> 'serviceIdentifier’ . I couldn't find documentation or examples on how to 
> give Identifier to a service.  
> 
> 
>   final VertxServiceInterface vertxService = (VertxServiceInterface) 
> context.getControllerServiceLookup().getControllerService("VertxService”)
> 
>   
> https://github.com/xmlking/nifi-websocket/blob/master/src/main/java/com/crossbusiness/nifi/processors/PutEventBus.java#L55
>  
> 
> 
> 2. for my usecase I get data published to a topic from EventBus with 
> following code. 
> 
>   EventBus eb = vertx.eventBus();
> 
>   eb.consumer("news.uk.sport", message -> {
>   System.out.println("I have received a message: " + 
> message.body());
>   });
>   
>   I am working on a date ingest processor (push based) that needs to 
> listen for new messages on a topic and send them to flow as FlowFile. 
>   In my case data source is EvenBus that expose emit messages via 
> callback API. 
>   I am looking for ideas on how to call Processor’s onTrigger method when 
> the above callback is evoked. 
>   Should I have to use my own intermediate queue and poll it in onTrigger 
> method? 
>   is there a better way to trigger the  onTrigger method programmatically 
> ? 
> 
> Thanks 
> Sumo

nifi question

[Sumanth Chinthagunta]

Hi All,
I am new to NiFi and   I'm stuck with couple issues:  

1. Unable to get hold of ControllerService  from Processor’s init method. 
I wanted to pre-set some dependencies in my Processor during init phase 
instead of  querying them repeatedly in onTrigger method. 
I am getting null for service and not sure what I have to pass for 
'serviceIdentifier’ . I couldn't find documentation or examples on how to give 
Identifier to a service.  


final VertxServiceInterface vertxService = (VertxServiceInterface) 
context.getControllerServiceLookup().getControllerService("VertxService”)


https://github.com/xmlking/nifi-websocket/blob/master/src/main/java/com/crossbusiness/nifi/processors/PutEventBus.java#L55
 


2. for my usecase I get data published to a topic from EventBus with following 
code. 

EventBus eb = vertx.eventBus();

eb.consumer("news.uk.sport", message -> {
System.out.println("I have received a message: " + 
message.body());
});

I am working on a date ingest processor (push based) that needs to 
listen for new messages on a topic and send them to flow as FlowFile. 
In my case data source is EvenBus that expose emit messages via 
callback API. 
I am looking for ideas on how to call Processor’s onTrigger method when 
the above callback is evoked. 
Should I have to use my own intermediate queue and poll it in onTrigger 
method? 
is there a better way to trigger the  onTrigger method programmatically 
? 

Thanks 
Sumo