[ https://issues.apache.org/jira/browse/NIFI-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15468533#comment-15468533 ]
ASF subversion and git services commented on NIFI-2712: ------------------------------------------------------- Commit 2afc739ab72fa2bce8f64c8833f3eae1399966ce in nifi's branch refs/heads/master from [~mattyb149] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=2afc739 ] NIFI-2712: Fixed Fetch processors for multiple max-value columns. This closes #976 > Database Fetch processors' max-value columns don't work as expected > ------------------------------------------------------------------- > > Key: NIFI-2712 > URL: https://issues.apache.org/jira/browse/NIFI-2712 > Project: Apache NiFi > Issue Type: Bug > Reporter: Matt Burgess > Assignee: Matt Burgess > > Currently, for QueryDatabaseTable and GenerateTableFetch, the user can enter > any number of maximum-value columns, which are used to generate a SQL query > that will fetch all records whose values are greater than the last-observed > maximum values for those columns. > However this makes multiple max-value columns not very useful, since they > will both have to increase in lockstep or records will be lost/skipped. In > such a case, using one or the other (but not both) would suffice, making > multiple max-value columns useless. > The more likely use case is that there are multiple columns whose values are > strictly increasing, but at different rates. This is common with very large > tables where a column could be for "date_created" and also a "bucket number" > that strictly increases once a day. Queries for a day's worth of data are > more efficient if they can be filtered on "bucket" (in this case), then on > timestamp. However the generated SQL queries would have to reflect that > "bucket" may remain the same as timestamp is increasing, but once the bucket > value has increased, then only the (new) timestamps for that bucket should be > fetched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)