[ https://issues.apache.org/jira/browse/NIFI-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122656#comment-16122656 ]
Joseph Witt commented on NIFI-4283: ----------------------------------- NIFI-3335 got into a funky state that probably would have been solved by asking for integrity checker run against NIFI. Instead just cloning and deleting old ticket. History from NIFI-3335 is here Back to previous view [NIFI-3335] GenerateTableFetch should allow you to specify an initial Max Value Created: 12/Jan/17 Updated: 11/Aug/17 Status: Open Project: Apache NiFi Component/s: Extensions Affects Version/s: None Fix Version/s: 1.4.0 Type: Improvement Priority: Major Reporter: Matt Burgess Assignee: Joseph Witt Resolution: Unresolved Votes: 0 Labels: None Remaining Estimate: Not Specified Time Spent: Not Specified Original Estimate: Not Specified Description NIFI-2583 added the ability (via dynamic properties) to specify initial Max Values for columns, to enable the user to "pick up where they left off" if something happened with a flow, a NiFi instance, etc. where the state was stored but the processing did not complete successfully. This feature would also be helpful in GenerateTableFetch, which also supports max-value columns. Since NIFI-2881 adds incoming flow file support, it's more useful if Initial max values can be specified via flow file attributes. Because if a table name is dynamically passed via flow file attribute and Expression Language, user won't be able to configure dynamic processor attribute in advance for each possible table. Add dynamic properties ('initial.maxvalue.<max_value_column>' same as QueryDatabaseTable) to specify initial max values statically, and also use incoming flow file attributes named 'initial.maxvalue.<max_value_column>' if any. Comments Comment by Matt Burgess [ 17/Jul/17 ] Peter Wicks Are you still working on this? If not, do you mind if I unassign it? Thanks! Comment by Peter Wicks [ 19/Jul/17 ] Matt Burgess I'm not working on this. I've unassigned myself. Comment by ASF GitHub Bot [ 25/Jul/17 ] GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/2039 NIFI-3335: Add initial.maxvalue support to GenerateTableFetch Moved the common code back to the abstract base class For all changes: [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. [x] Has your PR been rebased against the latest commit within the target branch (typically master)? [x] Is your initial contribution a single, squashed commit? For code changes: [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? [x] Have you written or updated unit tests to verify your changes? [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? For documentation related changes: [x] Have you ensured that format looks appropriate for the output in which it is rendered? Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-3335 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2039.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2039 commit 8eea477c02fdb2ca19ec77da9168ceec5e8eae7c Author: Matt Burgess <mattyb...@apache.org> Date: 2017-07-25T20:07:52Z NIFI-3335: Add initial.maxvalue support to GenerateTableFetch Comment by ASF GitHub Bot [ 26/Jul/17 ] Github user pvillard31 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2039#discussion_r129494131 — Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java — @@ -198,6 +201,25 @@ public void onTrigger(final ProcessContext context, final ProcessSessionFactory // set as the current state map (after the session has been committed) final Map<String, String> statePropertyMap = new HashMap<>(stateMap.toMap()); + // If an initial max value for column(s) has been specified using properties, and this column is not in the state manager, sync them to the state property map + final Map<String,String> maxValueProperties = getDefaultMaxValueProperties(context.getProperties()); — End diff – Would it make sense to do it in the ``@OnScheduled`` method of the Abstract class? You could have the same code in common for QueryDatabaseTable and GenerateTableFetch. Besides not sure it needs to be done each time the processor is triggered? Thoughts? Comment by ASF GitHub Bot [ 26/Jul/17 ] Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2039#discussion_r129648705 — Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java — @@ -198,6 +201,25 @@ public void onTrigger(final ProcessContext context, final ProcessSessionFactory // set as the current state map (after the session has been committed) final Map<String, String> statePropertyMap = new HashMap<>(stateMap.toMap()); + // If an initial max value for column(s) has been specified using properties, and this column is not in the state manager, sync them to the state property map + final Map<String,String> maxValueProperties = getDefaultMaxValueProperties(context.getProperties()); — End diff – Yeah that's a better idea, should've been done in the original PR but I missed it during my review and just cut-pasted it back to the abstract class to share. I won't be able to move all that common code block as it needs the table name which can come from a flow file. But I can call getDefaultMaxValueProperties() at schedule-time vs trigger time. Comment by ASF subversion and git services [ 27/Jul/17 ] Commit dc4006f423b83e7819981569d17e8d90b9a9d1a4 in nifi's branch refs/heads/master from Matt Burgess [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=dc4006f ] NIFI-3335: Add initial.maxvalue support to GenerateTableFetch Signed-off-by: Pierre Villard <pierre.villard...@gmail.com> This closes #2039. Comment by ASF GitHub Bot [ 27/Jul/17 ] Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2039 Comment by Pierre Villard [ 27/Jul/17 ] OK, don't know why there is an issue with this JIRA specifically... but fact is: I cannot resolve/close it. I think that's the same for you Matt Burgess. Could you have a look Joseph Witt? Comment by Joseph Witt [ 11/Aug/17 ] i dont see an obvious way to close it either. will try some things...weird Generated at Fri Aug 11 01:28:09 UTC 2017 by Joseph Witt using JIRA 6.4.14#64029-sha1:ae256fe0fbb912241490ff1cecfb323ea0905ca5. > GenerateTableFetch should allow you to specify an initial Max Value (clone of > defunct NIFI-3335) > ------------------------------------------------------------------------------------------------ > > Key: NIFI-4283 > URL: https://issues.apache.org/jira/browse/NIFI-4283 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions > Reporter: Matt Burgess > Assignee: Joseph Witt > Fix For: 1.4.0 > > > NIFI-2583 added the ability (via dynamic properties) to specify initial Max > Values for columns, to enable the user to "pick up where they left off" if > something happened with a flow, a NiFi instance, etc. where the state was > stored but the processing did not complete successfully. > This feature would also be helpful in GenerateTableFetch, which also supports > max-value columns. > Since NIFI-2881 adds incoming flow file support, it's more useful if Initial > max values can be specified via flow file attributes. Because if a table name > is dynamically passed via flow file attribute and Expression Language, user > won't be able to configure dynamic processor attribute in advance for each > possible table. > Add dynamic properties ('initial.maxvalue.<max_value_column>' same as > QueryDatabaseTable) to specify initial max values statically, and also use > incoming flow file attributes named 'initial.maxvalue.<max_value_column>' if > any. -- This message was sent by Atlassian JIRA (v6.4.14#64029)