Adding properties
Howdy, I've been away from NiFi for awhile and have forgotten how to do some things. I'm using the GetMongo processor (1.2) and I want to use the 'Mongo Collection Name' property later . How do I do that? I've tried adding a property to the GetMongo processor but it always complaints that it isn't legal. I added the property myproperty and set the value to collectionname. I get the error: "'myproperty' validated against 'collectionname' is invalid because 'myproperty' is not a supported property" Eventually I would like to save the results from the GetMongo to a file. I would like the filename to be the collectionname_date_time. How should I be doing this. Thank for your help -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Adding-properties-tp16413.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Run on ec2
Howdy, I'm trying to run NiFi on an ec2 instance. In nifi.properties if I leave blank nifi.web.http.host= and nifi.remote.input.socket.host= I get in the log nifi-bootstrap.log: 2016-01-22 14:57:06,592 WARN [main] org.apache.nifi.bootstrap.RunNiFi Failed to obtain hostname for notification due to: java.net.UnknownHostException: ip-172-16-6-26: ip-172-16-6-26: unknown error at java.net.InetAddress.getLocalHost(InetAddress.java:1505) ~[na:1.8.0_65] at org.apache.nifi.bootstrap.RunNiFi.getHostname(RunNiFi.java:765) [nifi-bootstrap-0.4.1.jar:0.4.1] at org.apache.nifi.bootstrap.RunNiFi.start(RunNiFi.java:947) [nifi-bootstrap-0.4.1.jar:0.4.1] at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:200) [nifi-bootstrap-0.4.1.jar:0.4.1] If I put in the hosts name or the ip address for the host I get the same thing. Any thoughts on what i'm doing wrong? thank you, Patrick Jones -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Run-on-ec2-tp6664.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Run on ec2
Thanks for the quick response.
I solved the problem by following the advice @
http://deploymentzone.com/2014/01/06/aws-instances-and-java-net-unknownhostexception/
I modified my /etc/hosts file
#!/bin/bash
HOSTNAME=`hostname`
ETC_HOSTS=${1:-/etc/hosts}
if ! grep $HOSTNAME $ETC_HOSTS > /dev/null ; then
if grep "127.0.0.1" $ETC_HOSTS > /dev/null ; then
sed -i "s/127\.0\.0\.1.*$/& $HOSTNAME/" $ETC_HOSTS
else
echo "127.0.0.1 localhost $HOSTNAME" >> $ETC_HOSTS
fi
fi
thank you,
Patrick Jones
--
View this message in context:
http://apache-nifi-developer-list.39713.n7.nabble.com/Run-on-ec2-tp6664p6686.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
S3 configuration
Greetings, I'm trying to do a PutS3Object. When I try to use it I get : 2016-01-08 15:43:17,518 INFO [Timer-Driven Process Thread-6] com.amazonaws.http.AmazonHttpClient Unable to execute HTTP request: Connection refused: conjava.net.ConnectException: Connection refused: connect at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_66] ... Can someone give me a clue as to what I'm doing wrong? I set the bucket, access key, secret key, and the region. Everything else I left as the default. Is there more I have to do? thank you, Patrick Jones -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/S3-configuration-tp6122.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Cluster Setup
Thank you very much. That solved the problem. I changed from the default blank value to the name of the machine on the NCM but I still got errors. When I filled in the value on the node machine it started working. -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Cluster-Setup-tp5853p5911.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Cluster Setup
Howdy, I'm trying to set up a cluster for the 1st time. I 1st tried to setup a NCM and a node on the same machine. In nifi.properties I set: nifi.web.http.port=8081 nifi.cluster.is.node=true nifi.cluster.node.address= nifi.cluster.node.protocol.port=8083 nifi.cluster.is.manager=true nifi.cluster.manager.address= nifi.cluster.manager.protocol.port=8082 I got the error: 2015-12-18 11:14:56,827 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: ... nested exception is java.lang.IllegalStateException: Application may be configured as a cluster manager or a node, but not both. What did I do wrong? The admin guild says : "it is also perfectly fine to install the NCM and one of the nodes on the same server, as the NCM is very lightweight". So I decided I could figure that out later and set nifi.cluster.is.node=false I started my NCM and then started a node on another machine. I got the following error. 2015-12-18 11:06:45,112 INFO [Handle Controller Startup Failure Message from [id=24b57ab2-1853-4814-8c31-4e467d3364e3, apiAddress=localhost, apiPort=8081, socketAddress=localhost, socketPort=8083]] o.a.n.c.manager.impl.WebClusterManager Node Event: [id=24b57ab2-1853-4814-8c31-4e467d3364e3, apiAddress=localhost, apiPort=8081, socketAddress=localhost, socketPort=8083] -- 'Node could not join cluster because it failed to start up properly. Setting node to Disconnected. Node reported the following error: Failed to connect node to cluster because local flow is different than cluster flow.' How do I get the cluster flow and the local flow to be the same? thank you -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Cluster-Setup-tp5853.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Cluster Setup
Thank you for your help. I deleted the flow.xml.gz . So now I have the NCM on machine 'b' and a node on machine 'c'. I start up a and then start up b. In b's logs I see logs of heartbeats: 2015-12-18 14:41:05,807 INFO [Process NCM Request-10] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 790c06c7-6857-4a90-9d4f-76ec99159369 (type=HEARTBEAT, length=3250 bytes) in 2 millis On machine c is also see heartbeats: 2015-12-18 14:39:55,741 INFO [Clustering Tasks Thread-3] org.apache.nifi.cluster.heartbeat Heartbeat created at 2015-12-18 14:39:55,664 and sent at 2015-12-18 14:39:55,741; send took 6 millis Everything seems fine. Then I try to view NiFi on the NCM which I set the web to: nifi.web.http.port=8089 http://machine-b.mitre.org:8089/nifi/ I see: An unexpected error has occurred No nodes were able to process this request. I don't see anything in the logs to suggest what went wrong. Thoughts? -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Cluster-Setup-tp5853p5860.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Keep Files
Is there a way for GetFile to not delete a file but only read it once? I have a directory with files in it. I only want the new files that are added to the to be processed. It seems that if I set GetFile to not delete the files, the same files get read over and over. thoughts? -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Keep-Files-tp4864.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Merge Binary
I have written a processor that will take an image and tile it into 512x512 images. I now want to write a processor to merge them back together again. I see MergeContent and I would like to extend it. 1st off does this seem reasonable? If so: I'm working in my own bundle and my current hold up is I can't figure out how to put in the correct dependancy's in maven (nifi-mitre-bundle/mitre-nifi-processors/pom.xml) I tried org.apache.nifi nifi-standard-bundle 0.3.0 Maven couldn't find any of the references to from my MergeImageTiles to the MergeContent that it extends [ERROR] /home/plj/nifi-0.3.0/nifi-mitre-bundle/nifi-mitre-processors/src/main/java/org/mitre/nifi/MergeImageTiles.java:[13,42] error: package org.apache.nifi.processors.standard does not exist So I tried org.apache.nifi nifi-standard-processors 0.3.0 this compiled but then wouldn't run as it looked for dependencies at RunTime Thoughts? -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Merge-Binary-tp4872.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
help with adding a process
Howdy, I'm trying to add my own process to NiFi. So far I have written the process and it compiles. What I want now is actually see my process show up in NiFis list web GUI of processors. What I did, I tried to follow the Developers Guild modified by what the code looks like (I sort of used nifi-geo-bundle as an example. So in: root/nifi-nar-bundles I added nifi-image-bundles |---pom.xml |--nifi-image-processor |--src |-- main |--java |-- org.mitre.nitfimages |--resources |... |--nifi-images-nar |--pom.xml So it all compiles in maven but I don't know what I have to do to make my process show up. Clearly I'm missing a step or 3 to get my stuff added. Can you please enlighten me? thanks in advance, Pat -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/help-with-adding-a-process-tp2579.html Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
RE: help with adding a process
I had not set the org.apache.nifi.processor.Processor file but I have now. No change. Pat From: Aldrin Piri [via Apache NiFi (incubating) Developer List] [mailto:ml-node+s39713n2580...@n7.nabble.com] Sent: Wednesday, August 26, 2015 3:36 PM To: Jones, Patrick L. p...@mitre.org Subject: Re: help with adding a process Pat, As a quick check, did you add the fully qualified package name of your processor (org.mitre.nitfimages.Processor Classname) to the org.apache.nifi.processor.Processor file within the nifi-image-processors src/main/resources/META-INF/services directory? The overall structure provided seems to be appropriate. As an aside, if you are creating a bundle for your organization, you may be better suited creating a separate bundle to include in NiFi releases. We have an archetype available to aid in this process, the documentation of which is available on the NiFi Confluence wiki [1]. [1] https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-MavenProcessorArchetype On Wed, Aug 26, 2015 at 3:25 PM, plj [hidden email]/user/SendEmail.jtp?type=nodenode=2580i=0 wrote: Howdy, I'm trying to add my own process to NiFi. So far I have written the process and it compiles. What I want now is actually see my process show up in NiFis list web GUI of processors. What I did, I tried to follow the Developers Guild modified by what the code looks like (I sort of used nifi-geo-bundle as an example. So in: root/nifi-nar-bundles I added nifi-image-bundles |---pom.xml |--nifi-image-processor |--src |-- main |--java |-- org.mitre.nitfimages |--resources |... |--nifi-images-nar |--pom.xml So it all compiles in maven but I don't know what I have to do to make my process show up. Clearly I'm missing a step or 3 to get my stuff added. Can you please enlighten me? thanks in advance, Pat -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/help-with-adding-a-process-tp2579.html Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com. If you reply to this email, your message will be added to the discussion below: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/help-with-adding-a-process-tp2579p2580.html To unsubscribe from help with adding a process, click herehttp://apache-nifi-incubating-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_codenode=2579code=cGxqQG1pdHJlLm9yZ3wyNTc5fDU1MTMzODA1. NAMLhttp://apache-nifi-incubating-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/help-with-adding-a-process-tp2579p2584.html Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
RE: Process to create multiple files
Ok that makes sense. What processor would I extend/implement to allow me to send out multiple files from the process that unpacks the NITF bundle? Any examples? Thank, Pat From: Joe Witt [via Apache NiFi (incubating) Developer List] [mailto:ml-node+s39713n2513...@n7.nabble.com] Sent: Monday, August 17, 2015 1:56 PM To: Jones, Patrick L. p...@mitre.org Subject: Re: Process to create multiple files Hello Pat, Yeah makes sense. You would need a custom processor to support the NITF format [1] There does appear to be a Java library to deal with it but its licensing isn't ASL v2 compatible so we're not likely to be able to play along anytime soon. But that should be a good way to get started building what you need. The processor would unpack the items out of the NITF bundle and then could send out the full size images if that is how it works. Those full size images can then be resized to thumbnails. Tons of ways to play this one and it is a very common style of use case. Thanks Joe [1] https://en.wikipedia.org/wiki/National_Imagery_Transmission_Format [2] https://github.com/codice/imaging-nitf On Mon, Aug 17, 2015 at 1:45 PM, plj [hidden email]/user/SendEmail.jtp?type=nodenode=2513i=0 wrote: Howdy, Thanks for the reply. I think my situation is different than you suggest. I have an image file in NITF format. That file may have multiple images and multiple bands inside that one file. The software that I have that reads that file does create a thumbnail for each image and each band inside that file. I don't know how many thumbnails I need until I read the files metadata. Thank Pat From: Dan Bress [via Apache NiFi (incubating) Developer List] [mailto:[hidden email]/user/SendEmail.jtp?type=nodenode=2513i=1] Sent: Monday, August 17, 2015 1:40 PM To: Jones, Patrick L. [hidden email]/user/SendEmail.jtp?type=nodenode=2513i=2 Subject: Re: Process to create multiple files plj, I would not recommend having this processor create multiple thumbnails. What I would recommend is the following: Create a new processor called CreateThumbnail or RescaleImage Then have a configuration on the processor that says what size the output image should be(e.g. 128x128, or 1/X of original size). Your new processor will read in the incoming image, and rescale it down to the user specified size and pass it forward. Now if you want to create a 128x128 64x64 and 32x32 sized images you would do the following. (GetFile)-(RescaleImage configured to 128x128)-(PutFile) |-(RescaleImage configured to 64x64)-(PutFile) \-(RescaleImage configured to 32x32)-(PutFile) Where GetFile has 3 success relationships, each going to a different RescaleImage processor. I think it makes more sense to have one processor create one file, then you can use the flow to visually configure how many copies of the file you want to make. This should make this processor simpler and more reusable. Dan Bress Software Engineer ONYX Consulting Services From: plj [hidden email]/user/SendEmail.jtp?type=nodenode=2511i=0 Sent: Monday, August 17, 2015 1:27 PM To: [hidden email]/user/SendEmail.jtp?type=nodenode=2511i=1 Subject: Process to create multiple files Howdy, I'm new to NiFi so please bear with me. What I want to accomplish is: read an image file process the file to create one or more thumbnails from the image. Send the resulting thumbnails along the flow So I can use GetFile to read the file and then send it along. I think I need to write a custom java processor that will process the image file and then send each of the thumbnail files (say .jpg for now) on to the next thing in the flow (say PutFile for example). Are there suggestions on what I should implement or extend to create my custom processor? It will take in one file and output multiple files. Would extending PutFile so that it processed and then puts each thumbnail on the flow be a good strategy? Other ideas? Thank you, -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Process-to-create-multiple-files-tp2510.html Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com. If you reply to this email, your message will be added to the discussion below: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Process-to-create-multiple-files-tp2510p2511.html To unsubscribe from Process to create multiple files, click here NAMLhttp://apache-nifi-incubating-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers
