Re: Nifi processor last execution time

2015-11-13 Thread Andrew Grande 
There are 2 approached NiFi processors follow today.

E.g. a new TailFile saves it in a local state file (relative to NiFi root, in 
the ./conf/ directory).

Others leverage a DistributedCache service (it's actually more like a 
RemoteCache, not a distributed/replicated), which they declare as a requirement 
in the properties. This also has a benefit of being able to share the state 
between nodes and, optionally, persist the cache to the disk to be able to 
re-hydrate it on restart.

Andrew



On 11/13/15, 10:09 AM, "Sean Busbey"  wrote:

>Sam,
>
>For now I would store this information in HDFS. Alternatively, if you have
>it ZooKeeper would also work.
>
>On Thu, Nov 12, 2015 at 10:45 PM, Sam Kamau  wrote:
>
>> I am working on a processor that loads data to HDFS for an RDBMS using
>> ExecuteSQL processor. The data has a last update time that I would like to
>> use to load records that have changed since the last load.
>>
>> How best can I go about saving the last execution time so I can use it as a
>> filter in the query.
>>
>> Thanks
>>
>
>
>
>-- 
>Sean

Re: Nifi processor last execution time

2015-11-13 Thread Sean Busbey 
Sam,

For now I would store this information in HDFS. Alternatively, if you have
it ZooKeeper would also work.

On Thu, Nov 12, 2015 at 10:45 PM, Sam Kamau  wrote:

> I am working on a processor that loads data to HDFS for an RDBMS using
> ExecuteSQL processor. The data has a last update time that I would like to
> use to load records that have changed since the last load.
>
> How best can I go about saving the last execution time so I can use it as a
> filter in the query.
>
> Thanks
>



-- 
Sean

Re: Nifi processor last execution time

2015-11-13 Thread Joe Witt 
Sam,

The framework itself doesn't offer a state management mechanism for
extensions other than on FlowFiles themselves (which won't help for
this case).  So you can leverage any sort of database or storage
mechanism (even simple local files) you like.  You can see more about
some thoughts on how to address this moving forward [1].

Thanks
Joe

[1] https://cwiki.apache.org/confluence/display/NIFI/State+Management

On Thu, Nov 12, 2015 at 11:45 PM, Sam Kamau  wrote:
> I am working on a processor that loads data to HDFS for an RDBMS using
> ExecuteSQL processor. The data has a last update time that I would like to
> use to load records that have changed since the last load.
>
> How best can I go about saving the last execution time so I can use it as a
> filter in the query.
>
> Thanks

Nifi processor last execution time

2015-11-13 Thread Sam Kamau 
I am working on a processor that loads data to HDFS for an RDBMS using
ExecuteSQL processor. The data has a last update time that I would like to
use to load records that have changed since the last load.

How best can I go about saving the last execution time so I can use it as a
filter in the query.

Thanks