Hi Manee,
It depends on the Client API on how to tell what the next response
data set should be.
That may be an additional query parameter such as last fetch
timestamp, or something like HTTP etag header in most APIs.
You can pass FlowFiles to InvokeHTTP to tell such parameters.
Also, I recommend using Record processors over splitting dataset to process.
Because Record processors work more efficiently and the data set unit
will be more meaningful in your case.
I think your flow would be something like:
InvokeHTTP (Assuming the API result contains value to make next
incremental request)
-> Do some JSON transformation (QueryRecord, UpdateRecord or
JoltTransformRecord)
-> PutDatabaseRecord
-> Then connect success back to InvokeHTTP to fetch next dataset
Hope this helps.
Thanks,
Koji
On Mon, Nov 12, 2018 at 9:06 PM Manee wrote:
>
> Hi Team,
>
> I am new to NiFi .I have a task like .We need to fetch a data from Client
> API and stored into Postgresql
> My Flow ;
>
> InvokeHTTP -->>--SplitJson-->>--EvaluateJsonPath
> -->>ConvertJSONTosql--->>PutSql
>
> This is my flow is working fine but i need to make this as a incremental
> fetch from API ..whenever client API made changes it should reflect in our
> databases .How i can process for incremental fetch in API call ?.Please
> guide me to fix this problem .
>
>
> Thanks in Advance,
> Manikandan K
>
>
>
>
>
>
> -
> Thanks,
> Manee
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
