i'm fine with your proposal which merges mark's concept but aligns
consistency of Fetch(File/HDFS)
We should fix the docs for the CoreAttribute.PATH The concept of it
being relative is simply too vague. We should just be honest that it
is unspecified - subject to the meaning of whichever processors
sets/updates that value.
Mark will have to address the question of ListHDFS path description I think.
Thanks
Joe
On Wed, Nov 25, 2015 at 2:36 PM, Tony Kurc <trk...@gmail.com> wrote:
> Okay, since we don't have consensus, here is what I propose:
> ListFile
> 1. absolute.path will be absolute, path will be relative to input directory
>
> FetchFile:
> change default property to ${absolute.path}/${filename}. Don't have a
> windows machine at the ready - will / work as a path separator?
>
> Revisit consistency of List/Fetch when we can do a breaking change (1.0)
>
> If someone can confirm that they get the same read on ListHDFS path
> description as I do and we fix it before 0.4.0, I'd like that.
>
>
>
>
> On Wed, Nov 25, 2015 at 2:12 PM, Tony Kurc <trk...@gmail.com> wrote:
>
>> I am 100% in favor of keeping the relative path (I brought up out of band
>> the value if the Lister and Fetcher were different machines with different
>> mount points). I think is just a matter of what attribute to fill with what
>> value.
>>
>>
>> On Wed, Nov 25, 2015 at 2:09 PM, Joe Skora <jsk...@gmail.com> wrote:
>>
>>> Mark,
>>>
>>> What you described is the behavior of ListFile (in spite of confusing doc
>>> info).
>>>
>>> JoeW,
>>>
>>> Consistency with ListHDFS makes sense, and if that is the desired
>>> behavior
>>> it's easy to change ListFile. But CoreAttributres state "The flowfile's
>>> path indicates the relative directory" and if that's not true, does
>>> CoreAttributes need revision too?
>>>
>>> Thanks,
>>> Joe
>>>
>>> On Wed, Nov 25, 2015 at 2:00 PM, Mark Payne <marka...@hotmail.com> wrote:
>>>
>>> > Tony,
>>> >
>>> > I would recommend that ListFile add both 'path' and 'absolute.path'. The
>>> > 'path' would be relative to the base directory being listed.
>>> > For example, if ListFile is configured to list files from /data/nifi/in
>>> > and recurse subdirectories, and it finds a file named:
>>> > /data/nifi/in/123/myfile.txt
>>> > then i would expect the following attributes:
>>> >
>>> > absolute.path = /data/nifi/in/123
>>> > path = ./123
>>> > filename = myfile.txt
>>> >
>>> > Thanks
>>> > -Mark
>>> >
>>> >
>>> > > On Nov 25, 2015, at 1:56 PM, Tony Kurc <trk...@gmail.com> wrote:
>>> > >
>>> > > All,
>>> > > Joe and I commented on NIFI-631 that it didn't "just work" when wiring
>>> > the
>>> > > processors together. ListFile was populating the attributes as
>>> > > described in CoreAttributes.java
>>> > > [1] (path being relative to the input directory, and absolute being
>>> the
>>> > > full path). FetchFile was using ${path}/${filename} as the default,
>>> which
>>> > > wouldn't grab the directory. I'm puzzled as to what the correct
>>> behavior
>>> > > should be. The description of path said it is relative ... relative to
>>> > > what? ListHDFS appears to state path is absolute [2] [3], and I
>>> expect we
>>> > > should have consistent behavior between ListHDFS and ListFile.
>>> > >
>>> > > So, I guess I'm not sure what guidance to give on a review of
>>> NIFI-631.
>>> > > Should the default of FetchFile be changed to
>>> > ${absolute.path}/${filename}
>>> > > (which may be inconsistent with other List/Fetch processor combos), or
>>> > > should ListFile be changed to have path be absolute?
>>> > >
>>> > > [1]
>>> > >
>>> >
>>> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/flowfile/attributes/CoreAttributes.java
>>> > > [2]
>>> > >
>>> >
>>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L79
>>> > > [3]
>>> > >
>>> >
>>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L442
>>> >
>>> >
>>>
>>
>>
