Re: How to lock getfile upto putfile write into same file?

I have running multiple instances of PutFile writing to same file that leads fail in Strange ways. I have tried MergeContent processor but it not yields same results for all time. For one time it correctly merges files according to the filename but sometimes it could not merge. In my use case i

How to process files sequentially?

I have approximately 1000 files in local drive.I need to move that files into SQL Server accordingly one after another. Since local drive having files like file1.csv,file2.csv,..upto file1000.csv.I am sure that number of files in local drive may change dynamically. I can able to created template

Re: Nifi Cluster fails to disconnect node when node was killed

Neil, Want to make sure I understand what you're saying. What are stating is a single point of failure? Thanks Joe On Thu, May 18, 2017 at 5:27 PM, Neil Derraugh wrote: > Thanks for the insight Matt. > > It's a disaster recovery issue. It's not something

Re: how to set version for NIFI customized Processor?

Thanks Bryan and Joe, I managed to set the specific version for my processor with the properties. 2017-05-18 20:50 GMT+08:00 Bryan Bende : > As Joe mentioned, the default behavior is to use the Maven group, > artifact, and version, which will happen by default if you build your

Re: Nifi Cluster fails to disconnect node when node was killed

Thanks for the insight Matt. It's a disaster recovery issue. It's not something I plan on doing on purpose. It seems it is a single point of failure unfortunately. I can see no other way to resolve the issue other than to blow everything away and start a new cluster. On Thu, May 18, 2017 at

Re: Nifi Cluster fails to disconnect node when node was killed

Neil, Disconnecting a node prior to removal is the correct process. It appears that the check was lost going from 0.x to 1.x. Folks reported this JIRA [1] indicating that deleting a connected node did not work. This process does not work because the node needs to be disconnected first. The JIRA

Re: Nifi Cluster fails to disconnect node when node was killed

Pretty sure this is the problem I was describing in the "Phantom Node" thread recently. If I kill non-primary nodes the cluster remains healthy despite the lost nodes. The terminated nodes end up with a DISCONNECTED status. If I kill the primary it winds up with a CONNECTED status, but a new

Re: [EXT] Parsing Email Attachments

@pwicks no luck on escaping \n. Thanks for the suggestion. I even tried hardcoding the boundary value, in case it had something to do with using nifi expressions in regex but that didn't work either. On Wed, May 17, 2017 at 10:54 PM, Peter Wicks (pwicks) wrote: > Nick, > > >

Re: Nifi Cluster fails to disconnect node when node was killed

Hello, Just looking through this thread now. I believe that I understand the problem. I have updated the JIRA with details about what I think is the problem and a potential remedy for the problem. Thanks -Mark > On May 18, 2017, at 9:49 AM, Matt Gilman wrote: > >

Re: Archive selected "feeds"

Thank you. I've created this JIRA feature request: https://issues.apache.org/jira/browse/NIFI-3934 On Wed, May 17, 2017 at 12:51 PM Joe Witt wrote: > The concept of being able to provide hints to the underlying content > repository about which content is more important to

Re: Nifi Cluster fails to disconnect node when node was killed

Hi, Just wanted to point out that the newly appointed coordinator (centos-b) does end up sending heartbeats to itself as you described. 2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3] o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from centos-b:8080 It

Re: Nifi Cluster fails to disconnect node when node was killed

Thanks for the additional details. They will be helpful when working the JIRA. All nodes, including the coordinator, heartbeat to the active coordinator. This means that the coordinator effectively heartbeats to itself. It appears, based on your log messages, that this is not happening. Because

Re: how to set version for NIFI customized Processor?

As Joe mentioned, the default behavior is to use the Maven group, artifact, and version, which will happen by default if you build your NAR with the latest NAR Maven plugin (version 1.2.0 at this time). If you prefer to set different values than the Maven fields, you can override them by

Re: Nifi Cluster fails to disconnect node when node was killed

Found something interesting in the centos-b debug logging after centos-a (the coordinator) is killed centos-b takes over. Notice how it "Will not disconnect any nodes due to lack of heartbeat" and how it still sees centos-a as connected despite the fact that there are no heartbeats anymore.

Re: Nifi Cluster fails to disconnect node when node was killed

Thank you for the confirmation. I've filed this JIRA [1] to track the issue. Matt [1] https://issues.apache.org/jira/browse/NIFI-3933 On Thu, May 18, 2017 at 8:21 AM, ddewaele wrote: > Thanks for the response. > > When killing a non-coordinator node, it does take 8 * 5

Re: Nifi Cluster fails to disconnect node when node was killed

Thanks for the response. When killing a non-coordinator node, it does take 8 * 5 seconds before I see this : nifi-app.log:2017-05-18 12:04:29,644 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Status of centos-b:8080 changed from

Re: Nifi Cluster fails to disconnect node when node was killed

I can reproduce the issue by killing the java processes associated with the cluster coordinator node. The NiFi UI will not be accessible anymore until that particular node is brought up again, or until the node entry is removed from the cluster (via the REST API). Killing non-coordinator nodes

Re: Nifi Cluster fails to disconnect node when node was killed

Hi, Once the new coordinator was elected, it is responsible for disconnecting nodes due to lack of heartbeat. It will wait 8 times the configured nifi.cluster.protocol.heartbeat.interval before the node is disconnected. Can you confirm that this amount of time has elapsed? Did you see any

Re: is it possible for a NIFI processor to be both timer driven and event driven?

Hello It is certainly possible to have a process act as a listening daemon and you can see this in various Listen processors. Basically once they are scheduled they open up some port under some protocol and listen for incoming data. The custom processor you're mentioning can spawn a thread to

Re: How to lock getfile upto putfile write into same file?

PutFile while writing automatically writes with a dot prepended and once the write is complete it removes the dot. If you have multiple instances of PutFile writing to the same file with the intent of appending to that file it will like fail in strange ways. I would recommend you simply build up

Re: how to set version for NIFI customized Processor?

Just rebuild it with the latest nifi nar maven plugin and it will get the version info at that time. thanks On Thu, May 18, 2017 at 4:20 AM, 尹文才 wrote: > Thanks Joe, is it possible to set a specific version for a customized NIFI > processor and show the version in that

Nifi Cluster fails to disconnect node when node was killed

Hi, I have a NiFi cluster up and running and I'm testing various failover scenarios. I have 2 nodes in the cluster : - centos-a : Coordinator node / primary - centos-b : Cluster node I noticed in 1 of the scenarios when I killed the Cluster Coordinator node, that the following happened :

Re: how to set version for NIFI customized Processor?

Thanks Joe, is it possible to set a specific version for a customized NIFI processor and show the version in that processor selection dialog? 2017-05-18 10:42 GMT+08:00 Joe Witt : > Hello > > They will remain unversioned until they are built with the latest nar > plugin.

is it possible for a NIFI processor to be both timer driven and event driven?

Hi guys, actually I have 2 questions, what I need to do is like this: I have a Java program outside NIFI and inside NIFI I have a Customized processor inside NIFI, I need to send some data from my Java program to my NIFI processor and the processor is timer-driven by now. The processor needs to do

Re: Kafka - Hadoop Kerberos issue

mail was sent before completion: Security Protocol: SASL_SSL Kerberos Service Name: myuser (please note that its mandatory to put something here or the process fail) SSL Context Service: KafkaSSLcontext sasl.jaas.config: org.apache.kafka.common.security.plain.PlainSaslServer

Re: Kafka - Hadoop Kerberos issue

Hi again, Here are more details: 1) We are redoing the test on a test cluster which is plain 1.2 vanilla (upgraded from 1.1 but without any additional JAR, bootstrap is the package version. 2) The org.apache.kafka.common.security.plain.PlainSaslServer is coming from our JAAS configuration and

Re: How to lock getfile upto putfile write into same file?

Ok. I will append the '.' before Putfile using UpdateAttribute. Is this name(without '.') will automatically changed by Putfile when its done? Since I am appending each rows from html content into proper csv using Putfile processor. Is this works fine? How putfile knows complete html flowfiles