Re: Clustered Site-toSite

On Tue, Nov 24, 2015 at 1:38 PM, Edgardo Vega wrote: > Yeah the S2S port is set on all node. > > What should the host be set to on each machine? I first set it to the NCM > ip on each machine in the cluster. Then I set the host to be the ip of each > individual machine without luck. > > The S2S p

Re: Clustered Site-toSite

let me explain first how S2S works when connecting from one cluster to another cluster. I will start with the source cluster (this would be the cluster where you are adding the Remote Process Group (RPG) to the graph). The NCM has no role in this cluster. Every Node in a cluster works independent

Re: Clustered Site-toSite

Thank you for the info. I was working with Edgardo on this. We ended up having to set the SAME hostname on each of the source nodes, as the destination NCM uses for each of its nodes and of course open up the firewall rules so all source nodes can talk to each destination node. This seems to jiv

Re: Clustered Site-toSite

I am not following why you set all your Nodes (source and destination) to use the same hostname(s). Each hostname resolves to a single IP and by doing so doesn't all data get sent to a single end-point? The idea behind spreading out the connections when using S2S is for smart load balancing purpo

[ANNOUNCE] CFP open for ApacheCon North America 2016

Community growth starts by talking with those interested in your project. ApacheCon North America is coming, are you? We are delighted to announce that the Call For Presentations (CFP) is now open for ApacheCon North America. You can submit your proposed sessions at http://events.linuxfoundation.o

[GitHub] nifi pull request: NIFI-1192 added support for dynamic properties ...

Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/131 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[ANNOUNCE] New Apache NiFi PMC Member (and Committer) Sean Busbey

On behalf of the Apache NiFI PMC, I am very pleased to announce that Sean Busbey has accepted the PMC's invitation to become a PMC Member and Committer on the Apache NiFi project. We greatly appreciate all of Sean's hard work and generous contributions to the project. In addition to his contributi

Re: [ANNOUNCE] New Apache NiFi PMC Member (and Committer) Sean Busbey

Congrats, Busbey!! On Nov 25, 2015 1:14 PM, "Tony Kurc" wrote: > On behalf of the Apache NiFI PMC, I am very pleased to announce that Sean > Busbey has accepted the PMC's invitation to become a PMC Member and > Committer on the Apache NiFi project. We greatly appreciate all of Sean's > hard work

[GitHub] nifi pull request: Fixes NIFI-1220

GitHub user gilday opened a pull request: https://github.com/apache/nifi/pull/133 Fixes NIFI-1220 `MockProcessSession` returns a new FlowFile from its `penalty` method instead of mutating then returning the given FlowFile You can merge this pull request into a Git repository by run

absolute.path vs path for FetchFile/ListFile

All, Joe and I commented on NIFI-631 that it didn't "just work" when wiring the processors together. ListFile was populating the attributes as described in CoreAttributes.java [1] (path being relative to the input directory, and absolute being the full path). FetchFile was using ${path}/${filename}

Re: Clustered Site-toSite

Ok, that all makes sense. The main reason, we like doing it strictly as S2S is to maintain the flowfile attributes, so we would like to avoid HTTP. Otherwise we would have to rebuild some of these attributes from the content, which isn't the end of the world, but still no fun. We may consider th

Re: absolute.path vs path for FetchFile/ListFile

It sounds like ListFile kept logic similar to GetFile which I can understand that approach. However, I do believe it makes more sense to follow the behavior of ListHDFS where the path would be absolute. Thanks Joe On Wed, Nov 25, 2015 at 1:56 PM, Tony Kurc wrote: > All, > Joe and I commented on

Re: absolute.path vs path for FetchFile/ListFile

Tony, I would recommend that ListFile add both 'path' and 'absolute.path'. The 'path' would be relative to the base directory being listed. For example, if ListFile is configured to list files from /data/nifi/in and recurse subdirectories, and it finds a file named: /data/nifi/in/123/myfile.txt

Re: absolute.path vs path for FetchFile/ListFile

I certainly cannot argue with that, either. > On Nov 25, 2015, at 1:59 PM, Joe Witt wrote: > > It sounds like ListFile kept logic similar to GetFile which I can > understand that approach. > > However, I do believe it makes more sense to follow the behavior of > ListHDFS where the path would be

Re: absolute.path vs path for FetchFile/ListFile

+1 to Mark's idea. Having both attributes might be nice and probably doesn't hurt anything. The convenience of having a relative path is that you can munge it or add prefixes to it more easily via the expression language. An absolute path would somewhat be more difficult to work with, if you wan

Re: absolute.path vs path for FetchFile/ListFile

I am reading the ListHDFS code. I can't tell if the description is wrong, the code is wrong, or I'm missing something. Description: The path is set to the absolute path of the file's directory on HDFS. For example, if the Directory property is set to /tmp then files picked up from /tmp will have t

Re: absolute.path vs path for FetchFile/ListFile

Mark, What you described is the behavior of ListFile (in spite of confusing doc info). JoeW, Consistency with ListHDFS makes sense, and if that is the desired behavior it's easy to change ListFile. But CoreAttributres state "The flowfile's path indicates the relative directory" and if that's n

Re: remote command execution via SSH?

I have first-cut implementation of ExecuteRemoteProcess processor at: https://github.com/xmlking/nifi-scripting/releases I tried to provide all capabilities offed by groovy-ssh (https://gradle-ssh-plugin.github.io/docs/

Re: absolute.path vs path for FetchFile/ListFile

I am 100% in favor of keeping the relative path (I brought up out of band the value if the Lister and Fetcher were different machines with different mount points). I think is just a matter of what attribute to fill with what value. On Wed, Nov 25, 2015 at 2:09 PM, Joe Skora wrote: > Mark, > > W

Re: absolute.path vs path for FetchFile/ListFile

Okay, since we don't have consensus, here is what I propose: ListFile 1. absolute.path will be absolute, path will be relative to input directory FetchFile: change default property to ${absolute.path}/${filename}. Don't have a windows machine at the ready - will / work as a path separator? Revisi

Re: absolute.path vs path for FetchFile/ListFile

i'm fine with your proposal which merges mark's concept but aligns consistency of Fetch(File/HDFS) We should fix the docs for the CoreAttribute.PATH The concept of it being relative is simply too vague. We should just be honest that it is unspecified - subject to the meaning of whichever process

Re: PRs

Things that make me feel better: The persistence mechanism is very similar to that of ListHDFS. https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L417 On Tue, Nov 24, 2015 at 10:56 PM,

[GitHub] nifi pull request: Nifi 631

Github user jskora closed the pull request at: https://github.com/apache/nifi/pull/113 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] nifi pull request: Nifi 631

Github user jskora commented on the pull request: https://github.com/apache/nifi/pull/113#issuecomment-159769035 Closed by commit 226ac64ef95f3d755dfbb3d5288ba98052855473 and 4c4d62c61f7c828dbcb124090992b91d631cb22e. --- If your project is set up for it, you can reply to this email a

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Github user trkurc commented on a diff in the pull request: https://github.com/apache/nifi/pull/132#discussion_r45942561 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java --- @@ -102,6 +177,94 @@ protect

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/132#discussion_r45942649 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java --- @@ -102,6 +177,94 @@ protec

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Github user trkurc commented on a diff in the pull request: https://github.com/apache/nifi/pull/132#discussion_r45943160 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java --- @@ -102,6 +177,94 @@ protect

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Github user trkurc commented on a diff in the pull request: https://github.com/apache/nifi/pull/132#discussion_r45944159 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java --- @@ -143,23 +316,223 @@ publi

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Github user trkurc commented on the pull request: https://github.com/apache/nifi/pull/132#issuecomment-159804534 My major concern is with the persistence file. I would think scrubbing the file periodically, removing older stale entries would be a must, otherwise the file will begin ac

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Github user trkurc commented on the pull request: https://github.com/apache/nifi/pull/132#issuecomment-159806672 Another, major secondary concern is what to do with failed partial multipart puts and maybe having to do bucket cleanup. --- If your project is set up for it, you can repl

Re: PRs

I recommend we push NIFI-1107 to next release. We discovered some unfun issues the S3 Multipart "API" creates, notably, leaving dangling pieces around [1]: "Once you initiate a multipart upload there is no expiry; you must explicitly complete or abort the multipart upload" And charging while they

Re: PRs

Understood tony - thanks for digging into the review so thoroughly and Joe thank you. This is a very non-trivial contrib. On Thu, Nov 26, 2015 at 12:12 AM, Tony Kurc wrote: > I recommend we push NIFI-1107 to next release. We discovered some unfun > issues the S3 Multipart "API" creates, notably,

Re: Clustered Site-toSite

The postHTTP processor has an option to send as a FlowFile to a listenHTTP processor on another NiFi. This allows you to keep the FlowFile attributes across multiple NiFis just like S2S. On Nov 25, 2015 1:58 PM, "Matthew Gaulin" wrote: > Ok, that all makes sense. The main reason, we like doing i

Re: PRs

I'll second what joe said. non-trivial and a less than ideal API to work with. seriously?! no expiry! On Thu, Nov 26, 2015 at 12:17 AM, Joe Witt wrote: > Understood tony - thanks for digging into the review so thoroughly and > Joe thank you. This is a very non-trivial contrib. > > On Thu, Nov