[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025545#comment-16025545 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak closed the pull request at: https://github.com/apache/nifi/pull/1856 > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > Fix For: 1.3.0 > > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025544#comment-16025544 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1856 Thanks @jfrazee ! > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > Fix For: 1.3.0 > > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024965#comment-16024965 ] ASF GitHub Bot commented on NIFI-3404: -- Github user jfrazee commented on the issue: https://github.com/apache/nifi/pull/1856 This was a +1 btw, obviously. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > Fix For: 1.3.0 > > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024936#comment-16024936 ] ASF GitHub Bot commented on NIFI-3404: -- Github user jfrazee closed the pull request at: https://github.com/apache/nifi/pull/1830 > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024935#comment-16024935 ] ASF GitHub Bot commented on NIFI-3404: -- Github user jfrazee commented on the issue: https://github.com/apache/nifi/pull/1830 Superseded by #1856 > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024917#comment-16024917 ] ASF subversion and git services commented on NIFI-3404: --- Commit 4d0667380a62510e9a10896be41ce1b2c5538db9 in nifi's branch refs/heads/master from [~ijokarumawak] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=4d06673 ] NIFI-3404: Improved UX of LookupAttributes. - Added dependency notice. - Added EL evaluation at SimpleKeyValueLookupService. - Updated documentation. - Updated CommonsConfigurationLookupService to throw LookupFailureException if it fails to get configuration so that error messages can be displayed at each processor bulletin. - Added calling getConfiguration at OnEnabled of CommonsConfigurationLookupService, so that the service will stay in Enabling state if there is any issue. Signed-off-by: Joey Frazee > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024916#comment-16024916 ] ASF subversion and git services commented on NIFI-3404: --- Commit 46e2420d7425a26f5ec6bc787f375b741a0d8ca2 in nifi's branch refs/heads/master from [~jfrazee] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=46e2420 ] NIFI-3404 Added LookupAttribute processor and lookup controller services Signed-off-by: Joey Frazee > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024267#comment-16024267 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1830 @jfrazee @mattyb149 I reviewed the updated commit and found few points to improve. I was trying to comment all of those, but since it's easier to explain with code instead of comment, I created another PR based on this one. @jfrazee Would you take a look on #1856 to see if those changes are reasonable? Please find the [commit comments](https://github.com/apache/nifi/pull/1856/commits/4f81af0961c3d65c4df37b9de3934c8fed1a70ff) for what I've changed. If my additional commit looks ok, then I'd give +1 to this PR. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024259#comment-16024259 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118425516 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- I think supporting EL is useful, because user can refer variable registries or do some text processing with EL. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024250#comment-16024250 ] ASF GitHub Bot commented on NIFI-3404: -- GitHub user ijokarumawak opened a pull request: https://github.com/apache/nifi/pull/1856 NIFI-3404: LookupAttribute, SimpleCsvFileLookupService, PropertiesFileLookupService, XMLFileLookupService Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijokarumawak/nifi nifi-3404 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1856.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1856 commit 86d4198ce5537387a375e23003c8265be2c1e14d Author: Joey Frazee Date: 2017-05-19T16:45:14Z NIFI-3404 Added LookupAttribute processor and lookup controller services commit 4f81af0961c3d65c4df37b9de3934c8fed1a70ff Author: Koji Kawamura Date: 2017-05-25T04:22:54Z NIFI-3404: Improved UX of LookupAttributes. - Added dependency notice. - Added EL evaluation at SimpleKeyValueLookupService. - Updated documentation. - Updated CommonsConfigurationLookupService to throw LookupFailureException if it fails to get configuration so that error messages can be displayed at each processor bulletin. - Added calling getConfiguration at OnEnabled of CommonsConfigurationLookupService, so that the service will stay in Enabling state if there is any issue. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024105#comment-16024105 ] ASF GitHub Bot commented on NIFI-3404: -- Github user jfrazee commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118408976 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- I meant to remove the expressionLanguageSupported(true). I don't think I had an especially good reason to have added the EL support to the dynamic property for the CS in the first place. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024094#comment-16024094 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118408138 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- `cacheConfiguredValues` is still not evaluating expression. I can add EL evaluation when I merge. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024043#comment-16024043
]
ASF GitHub Bot commented on NIFI-3404:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r118404133
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleCsvFileLookupService.java
---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.lookup;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.file.monitor.LastModifiedMonitor;
+import org.apache.nifi.util.file.monitor.SynchronousFileWatcher;
+
+@Tags({"lookup", "cache", "enrich", "join", "csv", "reloadable", "key",
"value"})
+@CapabilityDescription("A reloadable properties file-based lookup service")
--- End diff --
"CSV file-based" instead of "properties".
> Add lookup processor for enrichments/joins to reference data
>
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining
> against reference data sets or performing attribute lookups against external
> data sources.
> Since enrichments and joins are basic streaming use cases, and since
> attributes and EL are often used to parameterize processor properties, there
> is a need for an easy way to do enrichments, joins and lookups without having
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started
> work on a LookupAttribute [2] processor that delegates the work to controller
> services.
> 1.
> https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E
> 2. https://github.com/jfrazee/nifi-lookup-service
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024025#comment-16024025 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1830 @jfrazee I understand, thank you! I am reviewing the updated commit now. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024021#comment-16024021 ] ASF GitHub Bot commented on NIFI-3404: -- Github user jfrazee commented on the issue: https://github.com/apache/nifi/pull/1830 @ijokarumawak The changes from [NIFI-3339](https://issues.apache.org/jira/browse/NIFI-3339)/#1450 have been removed from this PR along with the DatabaseLookupService. Will re-submit that stuff in another PR so we can move ahead with this and work out the details separately. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023479#comment-16023479
]
ASF GitHub Bot commented on NIFI-3404:
--
Github user jfrazee commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r118341314
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java
---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
--- End diff --
Holdover from the interfaces this was originally written against. There was
a LookupService and a MutableLookupService. Will remove.
> Add lookup processor for enrichments/joins to reference data
>
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining
> against reference data sets or performing attribute lookups against external
> data sources.
> Since enrichments and joins are basic streaming use cases, and since
> attributes and EL are often used to parameterize processor properties, there
> is a need for an easy way to do enrichments, joins and lookups without having
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started
> work on a LookupAttribute [2] processor that delegates the work to controller
> services.
> 1.
> https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023304#comment-16023304
]
ASF GitHub Bot commented on NIFI-3404:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r118318860
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java
---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
--- End diff --
What does "mutable" refer to here?
> Add lookup processor for enrichments/joins to reference data
>
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining
> against reference data sets or performing attribute lookups against external
> data sources.
> Since enrichments and joins are basic streaming use cases, and since
> attributes and EL are often used to parameterize processor properties, there
> is a need for an easy way to do enrichments, joins and lookups without having
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started
> work on a LookupAttribute [2] processor that delegates the work to controller
> services.
> 1.
> https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E
> 2. https://github.com/jfrazee/nifi-lookup-service
--
This message
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023303#comment-16023303
]
ASF GitHub Bot commented on NIFI-3404:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r118318520
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java
---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
+@CapabilityDescription("Lookup attributes from a lookup service")
+@DynamicProperty(name = "The name of the attribute to add to the FlowFile",
+value = "The name of the key or property to retrieve from the lookup
service",
--- End diff --
The value of the dynamic property is the _value_ to match as the key
against the lookup service right? I wonder if saying "name of the key" is
confusing vs "value for the key". Nothing required here, just thinking aloud :)
> Add lookup processor for enrichments/joins to reference data
>
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining
> against reference data sets or performing attribute lookups against external
> data sources.
> Since enrichments and joins are basic streaming use cases, and since
> attributes and EL are often used to parameterize processor properties, there
> is a need for an easy way to do enrichments, joins and lookups with
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023299#comment-16023299
]
ASF GitHub Bot commented on NIFI-3404:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r118317913
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java
---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
+@CapabilityDescription("Lookup attributes from a lookup service")
+@DynamicProperty(name = "The name of the attribute to add to the FlowFile",
+value = "The name of the key or property to retrieve from the lookup
service",
+supportsExpressionLanguage = true,
+description = "Adds a FlowFile attribute specified by the dynamic
property's key with the value found in the lookup service using the the dynamic
property's value")
+@WritesAttribute(attribute = "See additional details", description = "This
processor may write zero or more attributes as described in additional details")
+public class LookupAttribute extends AbstractProcessor {
+
+public static final String MULTI_STRATEGY = "multi-criteria";
+
+public static final String KEY_VALUE_STRATEGY = "key-value";
+
+public static final PropertyDescriptor LOOKUP_SERVICE =
+new PropertyDescriptor.Builder()
+.name("lookup-service")
+.displayName("Lookup Service")
+.description("The lookup service to use for attribute lookups")
+.identifiesControllerService(StringLookupService.class)
+.required(true)
+.build();
+
+
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019937#comment-16019937 ] ASF GitHub Bot commented on NIFI-3404: -- Github user jfrazee commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r117810505 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -41,7 +48,9 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .name(propertyDescriptorName) .required(false) .dynamic(true) -.addValidator(Validator.VALID) + .addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING, true)) + .addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR) +.expressionLanguageSupported(true) --- End diff -- You're right, I was trying to merge changes in from a VolatileLookupService I wrote elsewhere and didn't get that updated. The use for this is a little marginal so I'll remove it and we can see if the need comes back. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019929#comment-16019929
]
ASF GitHub Bot commented on NIFI-3404:
--
Github user jfrazee commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117809682
--- Diff:
nifi-nar-bundles/nifi-lookup-bundle/nifi-lookup-processors/src/main/java/org/apache/nifi/processors/lookup/LookupAttribute.java
---
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.lookup;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
+@CapabilityDescription("Lookup attributes from a lookup service")
+@DynamicProperty(name = "The name of the attribute to add to the FlowFile",
+value = "The name of the key or property to retrieve from the lookup
service",
+supportsExpressionLanguage = true,
+description = "Adds a FlowFile attribute specified by the dynamic
property's key with the value found in the lookup service using the the dynamic
property's value")
+@WritesAttribute(attribute = "See additional details", description = "This
processor may write zero or more attributes as described in additional details")
+public class LookupAttribute extends AbstractProcessor {
+
+public static final PropertyDescriptor LOOKUP_SERVICE =
+new PropertyDescriptor.Builder()
+.name("lookup-service")
+.displayName("Lookup Service")
+.description("The lookup service to use for attribute lookups")
+.identifiesControllerService(StringLookupService.class)
+.required(true)
+.build();
+
+public static final PropertyDescriptor INCLUDE_EMPTY_VALUES =
+new PropertyDescriptor.Builder()
+.name("include-empty-values")
+.displayName("Include Empty Values")
+.description("Include null or blank values for keys that are
null or blank")
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019928#comment-16019928
]
ASF GitHub Bot commented on NIFI-3404:
--
Github user jfrazee commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117809657
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-lookup-service-api/src/main/java/org/apache/nifi/lookup/LookupService.java
---
@@ -17,25 +17,47 @@
package org.apache.nifi.lookup;
+import java.util.Map;
import java.util.Optional;
+import java.util.Set;
import org.apache.nifi.controller.ControllerService;
public interface LookupService extends ControllerService {
/**
- * Looks up a value that corresponds to the given key
+ * Looks up a value that corresponds to the given map of information,
referred to as lookup coordinates
*
- * @param key the key to lookup
- * @return a value that corresponds to the given key
+ * @param coordinates a Map of key/value pairs that indicate the
information that should be looked up
+ * @return a value that corresponds to the given coordinates
*
- * @throws LookupFailureException if unable to lookup a value for the
given key
+ * @throws LookupFailureException if unable to lookup a value for the
given coordinates
*/
-Optional lookup(String key) throws LookupFailureException;
+Optional lookup(Map coordinates) throws
LookupFailureException;
/**
- * @return the Class that represents the type of value that will be
returned by {@link #lookup(String)}
+ * Returns the entire lookup table as a {@link Map}
+ *
+ * @return the lookup table stored in the backing service
+ * @throws LookupFailureException if the backing service is
unavailable or
+ * the table cannot be loaded
+ */
+default Map asMap() throws LookupFailureException {
--- End diff --
I think/know there's a use case for this but I get the worry, so I'll drop
it.
> Add lookup processor for enrichments/joins to reference data
>
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining
> against reference data sets or performing attribute lookups against external
> data sources.
> Since enrichments and joins are basic streaming use cases, and since
> attributes and EL are often used to parameterize processor properties, there
> is a need for an easy way to do enrichments, joins and lookups without having
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started
> work on a LookupAttribute [2] processor that delegates the work to controller
> services.
> 1.
> https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E
> 2. https://github.com/jfrazee/nifi-lookup-service
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019261#comment-16019261
]
ASF GitHub Bot commented on NIFI-3404:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117684959
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/DatabaseLookupService.java
---
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.lookup;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import javax.sql.DataSource;
+
+import org.apache.commons.configuration2.Configuration;
+import org.apache.commons.configuration2.DatabaseConfiguration;
+import org.apache.commons.configuration2.ex.ConfigurationException;
+import org.apache.commons.configuration2.builder.BasicConfigurationBuilder;
+import org.apache.commons.configuration2.builder.fluent.Parameters;
+import
org.apache.commons.configuration2.builder.fluent.DatabaseBuilderParameters;
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+@Tags({"lookup", "cache", "enrich", "join", "jdbc", "database", "key",
"value"})
+@CapabilityDescription("A reloadable properties file-based lookup service")
+public class DatabaseLookupService extends AbstractControllerService
implements StringLookupService {
--- End diff --
While this lookup single column service is powerful, I think it can be more
powerful if we also provide lookup-record implementation like IPLookupService
which lookups Record in the future.
> Add lookup processor for enrichments/joins to reference data
>
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining
> against reference data sets or performing attribute lookups against external
> data sources.
> Since enrichments and joins are basic streaming use cases, and since
> attributes and EL are often used to parameterize processor properties, there
> is a need for an easy way to do enrichments, joins and lookups without having
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started
> work on a LookupAttribute [2] processor that delegates the work to controller
> services.
> 1.
> https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E
> 2. https://github.com/jfrazee/nifi-lookup-service
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019263#comment-16019263
]
ASF GitHub Bot commented on NIFI-3404:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117681788
--- Diff:
nifi-nar-bundles/nifi-lookup-bundle/nifi-lookup-processors/src/main/java/org/apache/nifi/processors/lookup/LookupAttribute.java
---
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.lookup;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
+@CapabilityDescription("Lookup attributes from a lookup service")
+@DynamicProperty(name = "The name of the attribute to add to the FlowFile",
+value = "The name of the key or property to retrieve from the lookup
service",
+supportsExpressionLanguage = true,
+description = "Adds a FlowFile attribute specified by the dynamic
property's key with the value found in the lookup service using the the dynamic
property's value")
+@WritesAttribute(attribute = "See additional details", description = "This
processor may write zero or more attributes as described in additional details")
+public class LookupAttribute extends AbstractProcessor {
+
+public static final PropertyDescriptor LOOKUP_SERVICE =
+new PropertyDescriptor.Builder()
+.name("lookup-service")
+.displayName("Lookup Service")
+.description("The lookup service to use for attribute lookups")
+.identifiesControllerService(StringLookupService.class)
+.required(true)
+.build();
+
+public static final PropertyDescriptor INCLUDE_EMPTY_VALUES =
+new PropertyDescriptor.Builder()
+.name("include-empty-values")
+.displayName("Include Empty Values")
+.description("Include null or blank values for keys that are
null or blank")
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019259#comment-16019259
]
ASF GitHub Bot commented on NIFI-3404:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117685550
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/DatabaseLookupService.java
---
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.lookup;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import javax.sql.DataSource;
+
+import org.apache.commons.configuration2.Configuration;
+import org.apache.commons.configuration2.DatabaseConfiguration;
+import org.apache.commons.configuration2.ex.ConfigurationException;
+import org.apache.commons.configuration2.builder.BasicConfigurationBuilder;
+import org.apache.commons.configuration2.builder.fluent.Parameters;
+import
org.apache.commons.configuration2.builder.fluent.DatabaseBuilderParameters;
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+@Tags({"lookup", "cache", "enrich", "join", "jdbc", "database", "key",
"value"})
+@CapabilityDescription("A reloadable properties file-based lookup service")
+public class DatabaseLookupService extends AbstractControllerService
implements StringLookupService {
+
+static final PropertyDescriptor CONNECTION_POOL = new
PropertyDescriptor.Builder()
+.name("connection-pool")
+.displayName("Connection Pool")
+.description("Specifices the JDBC connection pool used to connect
to the database.")
+.identifiesControllerService(DBCPService.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor LOOKUP_TABLE_NAME = new
PropertyDescriptor.Builder()
+.name("lookup-table-name")
+.displayName("Lookup Table Name")
+.description("Lookup table name.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+static final PropertyDescriptor LOOKUP_KEY_COLUMN =
+new PropertyDescriptor.Builder()
+.name("lookup-key-column")
+.displayName("Lookup Key Column")
+.description("Lookup key column.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+static final PropertyDescriptor LOOKUP_VALUE_COLUMN =
+new PropertyDescriptor.Builder()
+.name("lookup-value-column")
+.displayName("Lookup Value Column")
+.description("Lookup value column.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+static final PropertyDescriptor LOOKUP_NAME_COLUMN =
+new PropertyDesc
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019262#comment-16019262
]
ASF GitHub Bot commented on NIFI-3404:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117681251
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-lookup-service-api/src/main/java/org/apache/nifi/lookup/LookupService.java
---
@@ -17,25 +17,47 @@
package org.apache.nifi.lookup;
+import java.util.Map;
import java.util.Optional;
+import java.util.Set;
import org.apache.nifi.controller.ControllerService;
public interface LookupService extends ControllerService {
/**
- * Looks up a value that corresponds to the given key
+ * Looks up a value that corresponds to the given map of information,
referred to as lookup coordinates
*
- * @param key the key to lookup
- * @return a value that corresponds to the given key
+ * @param coordinates a Map of key/value pairs that indicate the
information that should be looked up
+ * @return a value that corresponds to the given coordinates
*
- * @throws LookupFailureException if unable to lookup a value for the
given key
+ * @throws LookupFailureException if unable to lookup a value for the
given coordinates
*/
-Optional lookup(String key) throws LookupFailureException;
+Optional lookup(Map coordinates) throws
LookupFailureException;
/**
- * @return the Class that represents the type of value that will be
returned by {@link #lookup(String)}
+ * Returns the entire lookup table as a {@link Map}
+ *
+ * @return the lookup table stored in the backing service
+ * @throws LookupFailureException if the backing service is
unavailable or
+ * the table cannot be loaded
+ */
+default Map asMap() throws LookupFailureException {
--- End diff --
Returning entire lookup table as a Map would cause issue with a large
number of entries. I think this should be avoided.
> Add lookup processor for enrichments/joins to reference data
>
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining
> against reference data sets or performing attribute lookups against external
> data sources.
> Since enrichments and joins are basic streaming use cases, and since
> attributes and EL are often used to parameterize processor properties, there
> is a need for an easy way to do enrichments, joins and lookups without having
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started
> work on a LookupAttribute [2] processor that delegates the work to controller
> services.
> 1.
> https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E
> 2. https://github.com/jfrazee/nifi-lookup-service
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019258#comment-16019258 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r117680294 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -41,7 +48,9 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .name(propertyDescriptorName) .required(false) .dynamic(true) -.addValidator(Validator.VALID) + .addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING, true)) + .addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR) +.expressionLanguageSupported(true) --- End diff -- This change enabled EL support, however, cacheConfiguredValues method is not updated to evaluate EL. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019260#comment-16019260 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r117677184 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -41,7 +48,9 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .name(propertyDescriptorName) .required(false) .dynamic(true) -.addValidator(Validator.VALID) + .addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING, true)) + .addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR) --- End diff -- I think we can remove ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR. The dynamic properties of this processor are not representing FlowFile attribute names. It is a little bit misleading. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019154#comment-16019154 ] ASF GitHub Bot commented on NIFI-3404: -- Github user jfrazee commented on the issue: https://github.com/apache/nifi/pull/1830 @ijokarumawak FYI, I think we need to make sure we see what happens to [NIFI-3946](https://issues.apache.org/jira/browse/NIFI-3946)/#1833 before we merge this. By all means start having a look though. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019147#comment-16019147 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1830 Reviewing... > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017895#comment-16017895 ] ASF GitHub Bot commented on NIFI-3404: -- GitHub user jfrazee opened a pull request: https://github.com/apache/nifi/pull/1830 NIFI-3404 Add lookup processor for enrichments/joins to reference data Note: This is overlaid on [NIFI-3339](https://issues.apache.org/jira/browse/NIFI-3339)/#1450. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jfrazee/nifi NIFI-3404 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1830.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1830 commit 449f33ca5379b22f8dc4b7ba0042481c46129045 Author: Joey Frazee Date: 2017-05-19T16:45:14Z NIFI-3404 Added properties and csv file lookup services commit f2cda5c47c19b1d95a8813c2c79329f49537da1a Author: Joey Frazee Date: 2017-05-19T17:49:24Z NIFI-3404 Added LookupAttribute processor, L&N fixes commit 64e03e344432665b7b4184405335f62221c3e93a Author: Toivo Adams Date: 2017-01-29T11:40:37Z NIFI-3339 Add getDataSource() to DBCPService This closes #1450 Signed-off-by: Joey Frazee commit c3b7e2e76eb38500c02e42676cfe1bcaa0b2291d Author: Joey Frazee Date: 2017-05-19T18:32:41Z NIFI-3404 Added database lookup service > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
