Re: absolute.path vs path for FetchFile/ListFile

It sounds like ListFile kept logic similar to GetFile which I can understand that approach. However, I do believe it makes more sense to follow the behavior of ListHDFS where the path would be absolute. Thanks Joe On Wed, Nov 25, 2015 at 1:56 PM, Tony Kurc wrote: > All, > Joe

Re: absolute.path vs path for FetchFile/ListFile

I am reading the ListHDFS code. I can't tell if the description is wrong, the code is wrong, or I'm missing something. Description: The path is set to the absolute path of the file's directory on HDFS. For example, if the Directory property is set to /tmp then files picked up from /tmp will have

Re: absolute.path vs path for FetchFile/ListFile

Tony, I would recommend that ListFile add both 'path' and 'absolute.path'. The 'path' would be relative to the base directory being listed. For example, if ListFile is configured to list files from /data/nifi/in and recurse subdirectories, and it finds a file named: /data/nifi/in/123/myfile.txt

Re: absolute.path vs path for FetchFile/ListFile

I certainly cannot argue with that, either. > On Nov 25, 2015, at 1:59 PM, Joe Witt wrote: > > It sounds like ListFile kept logic similar to GetFile which I can > understand that approach. > > However, I do believe it makes more sense to follow the behavior of > ListHDFS

Re: absolute.path vs path for FetchFile/ListFile

Okay, since we don't have consensus, here is what I propose: ListFile 1. absolute.path will be absolute, path will be relative to input directory FetchFile: change default property to ${absolute.path}/${filename}. Don't have a windows machine at the ready - will / work as a path separator?

Re: absolute.path vs path for FetchFile/ListFile

i'm fine with your proposal which merges mark's concept but aligns consistency of Fetch(File/HDFS) We should fix the docs for the CoreAttribute.PATH The concept of it being relative is simply too vague. We should just be honest that it is unspecified - subject to the meaning of whichever

absolute.path vs path for FetchFile/ListFile

All, Joe and I commented on NIFI-631 that it didn't "just work" when wiring the processors together. ListFile was populating the attributes as described in CoreAttributes.java [1] (path being relative to the input directory, and absolute being the full path). FetchFile was using