if the file is local, we don't want
> > > >> to save state across the cluster, because each node needs its own
> > state.
> > > >> So that would likely just be an extra property
> > > >> on the processor.
> >
>> (recommend you allow user to specify the state file
> > >> and default it to conf/ListFile-.state or something like
> > >> that.
> > >>
> > >> I have not documented this pattern. Specifically because we've been
> > >> talking for a while about implementing the Simple
> > >> State Manag
thing like
> >> that.
> >>
> >> I have not documented this pattern. Specifically because we've been
> >> talking for a while about implementing the Simple
> >> State Management but we just haven't gotten there yet. I expected that
> we
&
would get, though. Just a
> thought.
>
> Hopefully this helped to clear things up, rather than muddy them up :)
> Feel free to fire back any questions.
>
> Thanks
> -Mark
>
>
>
> > Date: Wed, 29 Jul 2015 06:42:39 -0400
> > Subject: Re: Fetch change list
> > Fro
processors. That will radically change how we
>> handle all of this.
>>
>> But since it is not there... it may actually make sense to just refactor
>> the ListHDFS processor into an AbstractListFileProcessor
>> that is responsible for handling the state managem
, 29 Jul 2015 06:42:39 -0400
> Subject: Re: Fetch change list
> From: joe.w...@gmail.com
> To: dev@nifi.apache.org
>
> JoeS
>
> Sounds great. I'd ignore my provenance comment as that was really
> more about how something external could keep tabs on progress, etc..
> Mar
JoeS
Sounds great. I'd ignore my provenance comment as that was really
more about how something external could keep tabs on progress, etc..
Mark Payne designed/built the List/Fetch HDFS one so I'll defer to him
for the good bits. But the logic to follow for saving state you'll
want is probably t
Joe,
I'm interested in working on List/FetchFile. It seems like starting with
[NIFI-631|https://issues.apache.org/jira/browse/NIFI-631] makes sense.
I'll look at List/FetchHDFS, but is there any further detail on how this
functionality should differ from GetFile? As for keeping state,
provenanc
Anup,
The two tickets in question appear to be:
https://issues.apache.org/jira/browse/NIFI-631
https://issues.apache.org/jira/browse/NIFI-673
Neither have been claimed as of yet. Anybody interested in taking one
or both of these on? It would be a lot like List/Fetch HDFS so you'll
have good exa
Can I expect this functionality in the upcoming releases of Nifi ?
On 13/07/15 9:13 am, "Sethuram, Anup" wrote:
>Where is this 1TB dataset living today?
>[anup] Resides in a filesystem
>
>- What is the current nature of the dataset? Is it already in large
>bundles as files or is it a series of
10 matches
Mail list logo
