GitHub user bbende opened a pull request:
https://github.com/apache/incubator-apex-malhar/pull/133
MLHR-1936 Adding NiFi operators to contrib
This pull request adds NiFi operators to the contrib module. The
integration is based off the NiFi site-to-site client which is the mechanism
for pulling/pushing data to/from a NiFi instance or cluster. It is used
internally when a NiFi instance communicates with another instance, and also
between other third party frameworks. In a cluster, the site-to-site client
knows about the nodes in the cluster and will pull/push to/from all nodes.
To test the operators against a running NiFi instance you can do the
following:
* Download the latest NiFi release and extract it:
https://nifi.apache.org/download.html
* Edit conf/nifi.properties and set nifi.remote.input.socket.port to any
available port, and nifi.remote.input.secure=false
* Start NiFi with bin/nifi.sh start
* Go the NiFi UI in your browser http://localhost:8080/nifi/
* Import the NiFi_Apex template that will be attached to the JIRA, template
button in top-right toolbar
* Drag the template on to the canvas, button near top-left of toolbar
* Start all of the processors and ports
* Run either TestNiFiInputApplication or TestNiFiOutputApplication in
contrib/src/test/java/com.datatorrent.contrib.nifi.demo
The left side of the flow generates fake data to a NiFi output port, the
NiFi Site-To-Site client will pull data from this output port in the case of
the Malhar NiFi input operator. The right side has an a NiFi input port waiting
for data to be pushed from the Malhar NiFi output operator.
Some questions/comments....
* Is using activate/deactivate preferred over setup/teardown?
* I wasn't sure if calling Thread.sleep in emitTuples was appropriate here.
We did this in other integrations such as storm and spark, but in those cases
we were with in a while loop so we didn't want to make a ton of requests when
no data was available. I'm not familiar with how Malhar will call emitTuples,
so let me know if this doesn't sense here.
* Wasn't sure about the preferred approach for exception handling, seems
wrapping a checked exception with RuntimeException is appropriate?
* I noticed some other operators used a WindowDataManager which I think was
to be able to replay data, didn't really know enough about that to know if
that was needed here.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bbende/incubator-apex-malhar
MLHR-1936.nifi-connector
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-apex-malhar/pull/133.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #133
----
commit c58087a47ddeb19fd917b767307ae3562a04f90b
Author: Bryan Bende <[email protected]>
Date: 2015-12-14T22:34:59Z
MLHR-1936 Adding NiFi operators to contrib
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
