Re: Question on ingesting HDFS batches

2018-07-30 Thread Sudhindra Tirupati Nagaraj
and probably at this point should since it is a common ask, have a JIRA to add a filter to only select files in a folder if we see a file meeting a certain name such as 'success'. Thanks Joe On Mon, Jul 30, 2018 at 6:34 PM, Sudhindra Tirupat

Question on ingesting HDFS batches

2018-07-30 Thread Sudhindra Tirupati Nagaraj
Hi, We just came across NIFI as a possible option for backing up our data lake periodically into S3. We have our pipelines that dump batches of data at some granularity. For example, our one-minute dumps are of the form “201807210617”, “201807210618”, “201807210619” etc. We are looking for a