[
https://issues.apache.org/jira/browse/NIFI-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdullah Aqeeli resolved NIFI-9477.
-----------------------------------
Resolution: Resolved
> data loss doing Incremental load based on a timestamp from SQL server using
> QueryDatabaseTableRecord
> -----------------------------------------------------------------------------------------------------
>
> Key: NIFI-9477
> URL: https://issues.apache.org/jira/browse/NIFI-9477
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.13.2
> Environment: ubuntu 20
> 2-node cluster using an embedded zk
> Reporter: Abdullah Aqeeli
> Priority: Critical
>
> hello all,
>
> We are using Nifi to do an incremental load from SQL Server using
> QueryDatabaseTableRecord where max-value column is updated_at column with a
> [Datetime2
> |https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
> type. The source tables get updated in milliseconds. We noticed that we are
> losing few events with each QueryDatabaseTableRecord run. As a work-around we
> created a custom processor that subtracts few seconds from the max-value in
> the state fetching overlapping events and then we handle any duplicates later
> on in the pipeline and it seems to solve the issue. However, this is not
> optimal. We'd like to know what's causing the skips. Our assumption so far is
> that timestamp datatype accuracy is causing the skips in the incremental load
> however we're not sure.
>
> Thanks!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
