Issues with creating and connecting RPG programmatically

[Pushkara R]

Hi,

I am trying to write a function that takes two parameters, a processor ID
in machine A and an input port in machine B and creates an RPG in machine B
which connects the above to.

Now, how I am doing that is
1. A POST to the
*/nifi-api/process-groups//remote-process-groups*
endpoint to create an RPG and retrieve the ID of that RPG
2. A POST to the */nifi-api/process-groups//connections
*endpoint to create a connection between the processor and the input port.
the processorID and the ID of the input port are being provided along with
the list of the relationships.
3. A final PUT to */nifi-api/remote-process-groups/ *to enable the
transmission between the machines*.*

Now, the function always throws errors in step 2. A 409 is thrown for the
POST request with the error being 'Unable to find specified destination'.
(though refreshing the canvas on machine 1 shows the rpg having been
created)
However, when I manually run the steps 2 and 3 afterwards, with the same
rpgid, the connection happens.

Now, I'm not sure if this is a synchronization issue or not, but I want to
figure it out because I would not want to separate out steps 1 2 and 3.
Could somebody point out what could be the issue here?

Pushkar

PS - the post messages for steps 2 are the same when the api is called from
within the function and manually.

Re: Deep Dive into NiFi's Site2Site

[Pushkara R]

Thanks a lot Joseph. This is very helpful.

On Tue, Apr 4, 2017 at 5:06 PM, Joseph Niemiec <josephx...@gmail.com> wrote:

> Not all of what your looking for but there are a number of details on the
> HTTP implementation here. -
> https://cwiki.apache.org/confluence/display/NIFI/Support+HTTP%28S%29+as+a+
> transport+mechanism+for+Site-to-Site
>
>
> On Tue, Apr 4, 2017 at 5:32 AM, Pushkara R <pushkar1...@gmail.com> wrote:
>
> > Hi,
> >
> > I wanted to understand the site-to-site architecture of NiFi, mainly the
> > differences between using HTTP and RAW protocols, the architecture of the
> > site-to-site client and the input/output ports and how the NiFi REST API
> > comes into picture in site-to-site.
> >
> > Could someone please point me to architecture documents or any other
> media
> > that could clarify these things for me? If not, I will go through the
> code
> > to understand these but I would really appreciate some pointers in that
> as
> > well.
> >
> > Thanks a lot.
> >
>
>
>
> --
> Joseph
>

Deep Dive into NiFi's Site2Site

[Pushkara R]

Hi,

I wanted to understand the site-to-site architecture of NiFi, mainly the
differences between using HTTP and RAW protocols, the architecture of the
site-to-site client and the input/output ports and how the NiFi REST API
comes into picture in site-to-site.

Could someone please point me to architecture documents or any other media
that could clarify these things for me? If not, I will go through the code
to understand these but I would really appreciate some pointers in that as
well.

Thanks a lot.

Re: Nifi on Raspberry pi - jetty issues

[Pushkara R]

Hi,

We have taken a look at MiNiFi, and the reason why we want to use NiFi
still is because of the REST API that NiFi provides which we want to use to
dynamically rewire the processors whenever needed without needing to halt
all the processors in the graph.

>From what I understand of MiNiFi, this is not possible. Any changes to the
processor graph will involve a restart. So if there are disconnected
components of the processor graph, then they will be restarted
unnecessarily. This can be avoided in NiFi. However, let me know if I am
missing anything here.

Thanks for the link though. I'll see if I can find anything there.

Pushkar

On Fri, Mar 3, 2017 at 3:32 PM, Andrew Psaltis <psaltis.and...@gmail.com>
wrote:

> Pushkar,
> I just realized after sending the email, that I did not provide any
> resources where you can learn more about MiNiFi outside of the Apache site.
> Here are some resources that may make your experience a little smoother:
>
> Edge Intelligence with MiNiFi --
> https://hortonworks.com/webinar/edge-intelligence-iot-apache-minifi/
> Reading Sense HAT data -
> https://community.hortonworks.com/articles/85984/using-
> minifi-to-read-data-from-a-sense-hat-on-a-ra.html
> NiFi on a Rasp. PI Best Practices -
> https://community.hortonworks.com/articles/32605/running-
> nifi-on-raspberry-pi-best-practices.html
> -- this does not cover MiNiFi, thus it may have some pointers for solving
> the current problem you are having.
>
> Hope that helps.
>
> On Fri, Mar 3, 2017 at 3:27 PM, Andrew Psaltis <psaltis.and...@gmail.com>
> wrote:
>
> > Pushkar,
> > While it certainly should and has been possible to run regular NiFi on a
> > PI. Perhaps the best way forward is to actually use MiNiFi [1]. MiNiFi is
> > available in as a C++ and Java agent and designed explicitly to run on
> > devices.
> >
> > I'm sorry I could not be of more assistance with the actually config
> > changes you may want to make, I am traveling at the present time and do
> not
> > have access to a PI.
> >
> >
> > [1] https://nifi.apache.org/minifi/index.html
> >
> > Thanks,
> > Andrew
> >
> > On Fri, Mar 3, 2017 at 3:21 PM, Pushkara R <pushkar1...@gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> I'm getting NiFi to run on a raspberry pi. the bootstrap takes a very
> long
> >> time. But at the end, the port doesn't open because of a jetty issue and
> >> NiFi shuts down. I've pasted the exception below.
> >>
> >> A quick look at stackoverflow tells me I need to insrease the timeout
> >> value
> >> to go around this. But I do not know how to change it, I'm not sure if
> it
> >> is picking it up at runtime or not.
> >>
> >> Any assistance would be helpful.
> >>
> >> Thanks
> >> Pushkar
> >>
> >> PS : the stacktrace
> >>
> >> java.lang.Exception: Timeout scanning annotations
> >> at
> >> org.eclipse.jetty.annotations.AnnotationConfiguration.scanFo
> >> rAnnotations(AnnotationConfiguration.java:575)
> >> ~[jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
> >> at
> >> org.eclipse.jetty.annotations.AnnotationConfiguration.config
> >> ure(AnnotationConfiguration.java:447)
> >> ~[jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
> >> at
> >> org.eclipse.jetty.webapp.WebAppContext.configure(
> WebAppContext.java:494)
> >> ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
> >> at
> >> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppCo
> >> ntext.java:1361)
> >> ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
> >> at
> >> org.eclipse.jetty.server.handler.ContextHandler.doStart(
> >> ContextHandler.java:772)
> >> ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
> >> at
> >> org.eclipse.jetty.servlet.ServletContextHandler.doStart(Serv
> >> letContextHandler.java:262)
> >> ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
> >> at
> >> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520)
> >> ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
> >> at
> >> org.eclipse.jetty.util.component.AbstractLifeCycle.start(
> >> AbstractLifeCycle.java:68)
> >> ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
> >> at
> >> org.eclipse.jetty.util.component.ContainerLifeCycle.start(
> >> ContainerLifeCycle.java:132)
> >> ~[jetty-util-9.3.9.v20160517.jar:9.3

Re: Need help with Custom Processor

[Pushkara R]

Hi Andrew,

The particular class that isn't being found is part of an artifact (let's
call it A) which is a dependency of another artifact B that my processors
depend on. A is in B's pom file and B is in my processors' pom file. Yet I
am getting the class not found issue.

I am currently working by adding A's jar in the lib folder as I am only
doing a few benchmarks on NiFi. But please do let me know how I could fix
this problem.

Regards
Pushkar

On Tue, Feb 28, 2017 at 12:40 PM, Andrew Psaltis <psaltis.and...@gmail.com>
wrote:

> Hi Pushkar,
> The preferred approach is to add the external dependency to the pom file
> for your processor. After doing that when you build your nar bundle, the
> resulting nar file will contain all of the 3rd party dependencies required
> by your processor. Can you double check that you are indeed including it as
> a maven dependency?
>
> Thanks,
> Andrew
>
>
> On Tue, Feb 28, 2017 at 12:31 PM, Pushkara R <pushkar1...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I'm writing a new processor for NiFi which has a dependency on an
> external
> > library. When I build the NAR and add it to the lib folder and restart
> > NiFi, the external library classes that my processor depends on fail
> > getting loaded with a "ClassNotFound" exception.
> >
> > How do I go about fixing this issue? Do I add the jar for the external
> > library separately in the lib folder of NiFi? Or is there something else
> to
> > it?
> >
> > Thanks
> > Pushkar
> >
>
>
>
> --
> Thanks,
> Andrew
>
> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>

Need help with Custom Processor

[Pushkara R]

Hi,

I'm writing a new processor for NiFi which has a dependency on an external
library. When I build the NAR and add it to the lib folder and restart
NiFi, the external library classes that my processor depends on fail
getting loaded with a "ClassNotFound" exception.

How do I go about fixing this issue? Do I add the jar for the external
library separately in the lib folder of NiFi? Or is there something else to
it?

Thanks
Pushkar

Questions on Topology and custom-processor deployment on NiFi and MiNiFi

[Pushkara R]

Hi,

I am a part of a Research Lab in Indian Institute of Science and we are
currently evaluating MiNiFi as candidate for out IoT deployments. There are
two things I wanted get clarified as I could not find relavant
documentation.

1. It is mentioned in multiple places that using NiFi we can update the
processor Topology of multiple instances of MiNiFi that are connected to
it. However, I am not able to figure out how. I assume it would involve
communication with the MiNiFi bootstrap and passing it the relavant yml
file, but I don't know how to do it. Could someone please elaborate?

2. On NiFi, the processor topology can be updated in the UI without
stopping the entire topology. Only the concerned processors need to be
stopped. Though MiNiFi does not have a UI, is it necessary that every minor
edit to the topology needs a restart? Is there a way to go around the
necessity to restart MiNiFi entirely?

Regards
Pushkar

local NiFi and MiNiFi - yml generation errors

[Pushkara R]

Hi,

I was using this tutorial

to get started with NiFi and MiNiFi. At step 11 in the linked tutorial, I
tried creating the yml for the given template. I couldn't find the
config.sh file in the latest source of minifi-0.2.0
 which is why I downloaded
minifi-toolkit-0.1.0

which has the config.sh.

Executing 'bin/config.sh transform  ' gave the
following error
-
config.sh: JAVA_HOME not set; results may vary

Java home:
MiNiFi Toolkit home: /home/pushkar/workplace/minifi-toolkit-0.1.0


There are validation errors with the template, still outputting YAML but it
will need to be edited.
'Input Ports' in section 'top level' because must be empty in root group as
external input/output ports are currently unsupported
-
Could someone please help me debug this error? I'd also like to ask if the
linked tutorial is still valid or is it dated?

Thanks
Pushkar

Re: NiFi installation issues

[Pushkara R]

I notice that the server takes a minute or so to come up after running
nifi.sh start. I think the first few times I tried running nifi, I tried
connecting to the server immediately after running nifi.sh. Obviously I
couldn't connect as the server wasn't up yet. Maybe I tried to rerun the
script which could have caused clashes for the 8080 port.

This is just a guess, I'm not sure if this is the reason. But everything
seems to be working fine now. Thanks a lot Oleg.

Pushkar

On Mon, Jan 9, 2017 at 8:23 PM, Pushkara R <pushkar1...@gmail.com> wrote:

> Thanks for pointing that out Oleg. I'll see what's occupying that port.
>
> Pushkar
>
> On Mon, Jan 9, 2017 at 8:10 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
>
>> Pushkar
>>
>> It appears that something is running on port 8080 so Jetty can not start.
>> You can see at the bottom of the log the following (see below)
>>
>>
>>   1.
>> 2017-01-09 17:24:23,606 WARN [main] org.apache.nifi.web.server.JettyServer
>> Failed to start web server... shutting down.
>>   2.
>> java.net.BindException: Address already in use
>>   3.
>> at sun.nio.ch.Net.bind0(Native Method) ~[na:1.8.0_112]
>>   4.
>> at sun.nio.ch.Net.bind(Net.java:433) ~[na:1.8.0_112]
>>   5.
>> at sun.nio.ch.Net.bind(Net.java:425) ~[na:1.8.0_112]
>>   6.
>>     at 
>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
>> ~[na:1.8.0_112]
>>
>> On Jan 9, 2017, at 9:37 AM, Pushkara R <pushkar1...@gmail.com> pushkar1...@gmail.com>> wrote:
>>
>> Oleg
>>
>> I've pasted the app log here http://pastebin.com/YXkEvNZJ
>> and the bootstrap log here http://pastebin.com/iPJeJvwX.
>>
>> I built the project using 'maven clean install' (after setting
>> MAVEN_OPTS="-Xmx2048m -XX:MaxPermSize=128m" because of OutOfMemory
>> exceptions).
>> Then I extracted the nifi-1.2.0-snapshot-bin.tar.gz.
>> Followed by '$ ./bin/nifi.sh start'
>>
>> I then try to connect to 'http://localhost:8080/nifi/' which gives me the
>> screen I pasted in my previous mail.
>>
>> These were the documentation as available in the README.md in the github
>> clone of the project.
>>
>> Thanks
>> Pushkar
>>
>>
>

Re: NiFi installation issues

[Pushkara R]

Thanks for pointing that out Oleg. I'll see what's occupying that port.

Pushkar

On Mon, Jan 9, 2017 at 8:10 PM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:

> Pushkar
>
> It appears that something is running on port 8080 so Jetty can not start.
> You can see at the bottom of the log the following (see below)
>
>
>   1.
> 2017-01-09 17:24:23,606 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
>   2.
> java.net.BindException: Address already in use
>   3.
> at sun.nio.ch.Net.bind0(Native Method) ~[na:1.8.0_112]
>   4.
> at sun.nio.ch.Net.bind(Net.java:433) ~[na:1.8.0_112]
>   5.
> at sun.nio.ch.Net.bind(Net.java:425) ~[na:1.8.0_112]
>   6.
> at sun.nio.ch.ServerSocketChannelImpl.bind(
> ServerSocketChannelImpl.java:223) ~[na:1.8.0_112]
>
> On Jan 9, 2017, at 9:37 AM, Pushkara R <pushkar1...@gmail.com pushkar1...@gmail.com>> wrote:
>
> Oleg
>
> I've pasted the app log here http://pastebin.com/YXkEvNZJ
> and the bootstrap log here http://pastebin.com/iPJeJvwX.
>
> I built the project using 'maven clean install' (after setting
> MAVEN_OPTS="-Xmx2048m -XX:MaxPermSize=128m" because of OutOfMemory
> exceptions).
> Then I extracted the nifi-1.2.0-snapshot-bin.tar.gz.
> Followed by '$ ./bin/nifi.sh start'
>
> I then try to connect to 'http://localhost:8080/nifi/' which gives me the
> screen I pasted in my previous mail.
>
> These were the documentation as available in the README.md in the github
> clone of the project.
>
> Thanks
> Pushkar
>
>

Re: NiFi installation issues

[Pushkara R]

Oleg

I've pasted the app log here http://pastebin.com/YXkEvNZJ
and the bootstrap log here http://pastebin.com/iPJeJvwX.

I built the project using 'maven clean install' (after setting
MAVEN_OPTS="-Xmx2048m -XX:MaxPermSize=128m" because of OutOfMemory
exceptions).
Then I extracted the nifi-1.2.0-snapshot-bin.tar.gz.
Followed by '$ ./bin/nifi.sh start'

I then try to connect to 'http://localhost:8080/nifi/' which gives me the
screen I pasted in my previous mail.

These were the documentation as available in the README.md in the github
clone of the project.

Thanks
Pushkar

NiFi installation issues

[Pushkara R]

Hi,

I'm installing NiFi on my arch linux machine following the instructions in
https://github.com/apache/nifi.

After I built the project, I executed the NiFi startup script and tried
connecting to the local server but I got a 404 error with the following
message.

HTTP ERROR: 404

Problem accessing /nifi/. Reason:

Not Found

--
Powered by Jetty:// 9.3.9.v20160517 

Could someone please help me debug this?

Thank You
Pushkar