Re: Fetch change list

Where is this 1TB dataset living today? [anup] Resides in a filesystem - What is the current nature of the dataset? Is it already in large bundles as files or is it a series of tiny messages, etc..? Does it need to be split/merged/etc.. [anup] Archived files of size 3MB each collected over

Re: Fetch change list

Suppose, I have 1 TB of data that I need to backup/sync to a HDFS location and then be passed onto a Kafka, is there a way out to do that? -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Fetch-change-list-tp1351p1706.html Sent from the Apache N

RE: send JSON format to kafka and avoid duplication

Hi Mark, Great! That works for now until we the next release. Thanks, anup -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/send-JSON-format-to-kafka-and-avoid-duplication-tp1624p1651.html Sent from the Apache NiFi (incubating) Developer List

RE: send JSON format to kafka and avoid duplication

Mark, That doesn't help. I'm getting the same results with the extra "$" too.. Observation: If I have the curly braces around it doesn't replace the actual value (for the below ones) - {filename:${filename}} - {filename:$${filename}} But, if I omit the curly braces, the evaluation goes fine file

Re: send JSON format to kafka and avoid duplication

That works, but what if I have to replace it with the JSON format, how do I send the values? For the below replacement, the whole JSON text goes as it is, without replacing the values! { filename=${filename}, ? property=${property} } -- View this message in context: http://apac

Re: send JSON format to kafka and avoid duplication

Thanks Mark! But what if I have to replace it with the JSON format, how do I send the values? For the below replacement, the whole JSON text goes as it is, without replacing the values! { filename=${filename}, property=${property} } -- View this message in context: http://apac

send JSON format to kafka and avoid duplication

I have a set of files for which I need to send its filename and another property in a JSON format (shown below) to Kafka. /{ filename=${filename}, property=${property} } / I tried to replace the entire content with the above JSON content and send it to Kafka. But instead of obtaining v

Re: Process group id

Thanks Matt! -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Process-group-id-tp1621p1623.html Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.

Process group id

I have created a custom process group and I need to use the process-group id to query with REST api .. I see the process group id by right-clicking on the Process group and selecting "stats". But there's no way I can select this and copy. Is there another way out? -- View this message in contex

Re: Fetch change list

Hi Mark, I downloaded the latest version and I see that the FetchHDFS processor could be used for my delta files that have arrived to the HDFS. But how do I maintain a *sync * from a local file system to my HDFS. I cannot move files from the local filesystem. It needs to be copied. I'm facing

NiFi restart on windows failing

I've faced this problem quite often and always ended up restarting my system, deleting the war manually and starting up NiFi. Is there an easy way out to deal with the permission issue? 15-05-19 12:22:31,631 INFO [main] org.eclipse.jetty.server.ServerConnector Started ServerConnector@7b5a12ae{HTT

RE: losing property

Thanks Mark! That worked.. -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/losing-property-tp1565p1589.html Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.

RE: losing property

Thanks Mark that worked. I got another problem while writing it to HDFS. I don't think this is related to that. PutHDFS[id=104d8f8d-f5b3-35fa-81b2-1e761064a89b] Failed to write to HDFS due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutHDFS[id=104d8f8d-f5b3-3

RE: losing property

That's right Mark! I want to retain the directory structure. -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/losing-property-tp1565p1577.html Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.

RE: losing property

Hi Mark, I am auto-terminating the orginal zip flow file. And on success (viz. uncompression) the unzipped file goes to HDFS and puts the file at the new location. But the directory structure isn't created, althougth the Put HDFS file location is mentioned as /${path}. In the case, where the so

losing property

Hi, I have the below workflow. I am intentionally failing the original zip file, but when I put the uncompressed file into HDFS I'm losing on the ${path} property which is lost along with the original flow file. Is there any way I can retain this property?

Recovering NiFi

How do I recover from the below error. My Queue got filled up and my whole drive was consumed. I had to manually delete the files in content repository. And I'm getting the below error continuously .. 2015-05-15 16:12:39,577 ERROR [Swap In FlowFiles Thread-1] o.a.n.controller.FileSystemSwapManage

Re: Invoke HTTP POST

I tried doing that with the below entries.. but its failing. Regular ExpressionInfo = (.*) Replacement ValueInfo = *appName=myApp&filename=${path}${filename}* Character SetInfo = UTF-8 Maximum Buffer SizeInfo = 1 MB Evaluation ModeInfo = Entire text How would these be treated as form parameters?

Invoke HTTP POST

Hi, I have a REST service that needs to be invoked with a few input parameters like filename and appName. I was able to call this REST service from NiFi through 'InvokeHTTP' processor by sending the parameters as part of the URL: "http://:/app/context?filename=&appName= Now I need to use POST

Re: configuring site-to-site

Hi Mark, The Secure Configuration section isn't updated yet. Could you point me to some other location. Regards, anup -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/configuring-site-to-site-tp1253p1280.html Sent from the Apache NiFi (incub

RE: configuring site-to-site

roup? Were you able to find the 'refresh' button in the context menu? -- Original Message -- From: "anup s" <[hidden email]> To: [hidden email] Sent: 4/28/2015 10:55:29 AM Subject: RE: configuring site-to-site >Mark, > Yes, That works. But I don't

RE: configuring site-to-site

f you right-click on the RPG and click "Go to" does it take you to the other instance? -- Original Message ------ From: "anup s" <[hidden email]> To: [hidden email] Sent: 4/28/2015 10:49:09 AM Subject: Re: configuring site-to-site >Hi Corey, >I don'

Re: configuring site-to-site

Hi Corey, I don't see any input ports been shown in Remote Process Group Ports :( -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/configuring-site-to-site-tp1253p1265.html Sent from the Apache NiFi (incubating) Developer List mailing list

Re: configuring site-to-site

Thanks Mark, I tried that initially connecting the GetFile to the Remote Process Group. But it throws me an error "NiFi Flow does not have any input ports', I configured the input port, but it doesn't show up any entries when I right - click on the RPG. Also, I am not able to find the "Refresh Fl

configuring site-to-site

Hi, I am trying to perform a site to site configuration. I am trying to do a GetFile at one site and do a PutFile in another. What I did: Configured NiFi in machine A and set the socket input port number and set secure to false in nifi.properties - Ran the first NiFi instance in machine A