File names are ending with .part (.part is getting suffixed to filename, not
prefixed).
In case of files from linux file system, “.” (dot) getting prefixed while
transfer is in progress.
Thanks,
Ravi Papisetti
On 06/12/17, 1:47 AM, "Joe Witt" <joe.w...@gmail.com> wrote:
Imagine a filename construct where you wanted to pick up any file that
begins with the phrase 'start' but does NOT end in the phrase 'part'.
The name is of a form 'begin.middle.end'.
This filename start.middle.ok would get picked up.
This filename start.middle.part would not.
The pattern for that example would be
start\..+\.(?!part)
The key part of that is the negative lookahead for ensuring it does
not end in part.
Thanks
On Wed, Dec 6, 2017 at 2:29 AM, Ravi Papisetti (rpapiset)
<rpapi...@cisco.com> wrote:
> Yeah..that is good idea, but we are already using this option to copy
file with certain prefix. Not sure how I can use this field to meet both
exclusion and inclusion criterion.
>
> Any thoughts.
>
> Thanks,
> Ravi Papisetti
>
> On 06/12/17, 1:26 AM, "Joe Witt" <joe.w...@gmail.com> wrote:
>
> Ravi
>
> Please use the 'File Filter' property of ListFile to control ignoring
> filenames until they no longer end in 'part'.
>
> Thanks
>
> On Wed, Dec 6, 2017 at 2:14 AM, Ravi Papisetti (rpapiset)
> <rpapi...@cisco.com> wrote:
> > Hi,
> >
> >
> >
> > We are using Apache NiFi 1.3.0
> >
> >
> >
> > We have a process flow to copy files from NFS to HDFS (with
processors
> > ListFile, FetchFile and PutHDFS)
> >
> >
> >
> > In the NiFi process flow, ListFile is configured to listen to a
directory on
> > NFS. When a file (ex: x.csv) is being copied from a windows machine
to NFS
> > (while transfer is in the middle), a part file(x.csv.part) is
created at NFS
> > until transfer is complete.
> >
> >
> >
> > ListFile has picked up this x.csv.part file and fetchFile picked up
this to
> > transfer to HDFS, didn’t update the file name back to x.csv in HDFS
when
> > transfer is complete.
> >
> >
> >
> > But, in case a file from linux file system, while file copy to NFS
is in
> > progress it created (.x.csv) and when transfer is complete, at both
NFS and
> > HDFS, filename is updated to x.csv (from .x.csv).
> >
> >
> >
> > Any thought how we can configure ListFile not to pickup these part
files or
> > any configurations in NiFi that fixes file names for these windows
part
> > files?
> >
> >
> >
> > Appreciate your help.
> >
> >
> >
> > Thanks,
> >
> > Ravi Papisetti
>
>
