Mark,
What you described is the behavior of ListFile (in spite of confusing doc
info).
JoeW,
Consistency with ListHDFS makes sense, and if that is the desired behavior
it's easy to change ListFile. But CoreAttributres state "The flowfile's
path indicates the relative directory" and if that's not true, does
CoreAttributes need revision too?
Thanks,
Joe
On Wed, Nov 25, 2015 at 2:00 PM, Mark Payne <marka...@hotmail.com> wrote:
> Tony,
>
> I would recommend that ListFile add both 'path' and 'absolute.path'. The
> 'path' would be relative to the base directory being listed.
> For example, if ListFile is configured to list files from /data/nifi/in
> and recurse subdirectories, and it finds a file named:
> /data/nifi/in/123/myfile.txt
> then i would expect the following attributes:
>
> absolute.path = /data/nifi/in/123
> path = ./123
> filename = myfile.txt
>
> Thanks
> -Mark
>
>
> > On Nov 25, 2015, at 1:56 PM, Tony Kurc <trk...@gmail.com> wrote:
> >
> > All,
> > Joe and I commented on NIFI-631 that it didn't "just work" when wiring
> the
> > processors together. ListFile was populating the attributes as
> > described in CoreAttributes.java
> > [1] (path being relative to the input directory, and absolute being the
> > full path). FetchFile was using ${path}/${filename} as the default, which
> > wouldn't grab the directory. I'm puzzled as to what the correct behavior
> > should be. The description of path said it is relative ... relative to
> > what? ListHDFS appears to state path is absolute [2] [3], and I expect we
> > should have consistent behavior between ListHDFS and ListFile.
> >
> > So, I guess I'm not sure what guidance to give on a review of NIFI-631.
> > Should the default of FetchFile be changed to
> ${absolute.path}/${filename}
> > (which may be inconsistent with other List/Fetch processor combos), or
> > should ListFile be changed to have path be absolute?
> >
> > [1]
> >
> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/flowfile/attributes/CoreAttributes.java
> > [2]
> >
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L79
> > [3]
> >
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L442
>
>
