[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user jfrazee closed the pull request at: https://github.com/apache/nifi/pull/1830 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118425516 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- I think supporting EL is useful, because user can refer variable registries or do some text processing with EL. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user jfrazee commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118408976 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- I meant to remove the expressionLanguageSupported(true). I don't think I had an especially good reason to have added the EL support to the dynamic property for the CS in the first place. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118408138 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- `cacheConfiguredValues` is still not evaluating expression. I can add EL evaluation when I merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r118404133
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleCsvFileLookupService.java
---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.lookup;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.file.monitor.LastModifiedMonitor;
+import org.apache.nifi.util.file.monitor.SynchronousFileWatcher;
+
+@Tags({"lookup", "cache", "enrich", "join", "csv", "reloadable", "key",
"value"})
+@CapabilityDescription("A reloadable properties file-based lookup service")
--- End diff --
"CSV file-based" instead of "properties".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user jfrazee commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r118341314
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java
---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
--- End diff --
Holdover from the interfaces this was originally written against. There was
a LookupService and a MutableLookupService. Will remove.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r118318860
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java
---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
--- End diff --
What does "mutable" refer to here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r118318520
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java
---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
+@CapabilityDescription("Lookup attributes from a lookup service")
+@DynamicProperty(name = "The name of the attribute to add to the FlowFile",
+value = "The name of the key or property to retrieve from the lookup
service",
--- End diff --
The value of the dynamic property is the _value_ to match as the key
against the lookup service right? I wonder if saying "name of the key" is
confusing vs "value for the key". Nothing required here, just thinking aloud :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r118317913
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java
---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
+@CapabilityDescription("Lookup attributes from a lookup service")
+@DynamicProperty(name = "The name of the attribute to add to the FlowFile",
+value = "The name of the key or property to retrieve from the lookup
service",
+supportsExpressionLanguage = true,
+description = "Adds a FlowFile attribute specified by the dynamic
property's key with the value found in the lookup service using the the dynamic
property's value")
+@WritesAttribute(attribute = "See additional details", description = "This
processor may write zero or more attributes as described in additional details")
+public class LookupAttribute extends AbstractProcessor {
+
+public static final String MULTI_STRATEGY = "multi-criteria";
+
+public static final String KEY_VALUE_STRATEGY = "key-value";
+
+public static final PropertyDescriptor LOOKUP_SERVICE =
+new PropertyDescriptor.Builder()
+.name("lookup-service")
+.displayName("Lookup Service")
+.description("The lookup service to use for attribute lookups")
+.identifiesControllerService(StringLookupService.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor LOOKUP_STRATEGY = new
PropertyDescriptor.Builder()
+.name("lookup-strategy")
+.displayName("Lookup Strategy")
+.description("The strategy to use to retrieve results from the " +
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user jfrazee commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r117810505 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -41,7 +48,9 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .name(propertyDescriptorName) .required(false) .dynamic(true) -.addValidator(Validator.VALID) + .addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING, true)) + .addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR) +.expressionLanguageSupported(true) --- End diff -- You're right, I was trying to merge changes in from a VolatileLookupService I wrote elsewhere and didn't get that updated. The use for this is a little marginal so I'll remove it and we can see if the need comes back. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user jfrazee commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117809682
--- Diff:
nifi-nar-bundles/nifi-lookup-bundle/nifi-lookup-processors/src/main/java/org/apache/nifi/processors/lookup/LookupAttribute.java
---
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.lookup;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
+@CapabilityDescription("Lookup attributes from a lookup service")
+@DynamicProperty(name = "The name of the attribute to add to the FlowFile",
+value = "The name of the key or property to retrieve from the lookup
service",
+supportsExpressionLanguage = true,
+description = "Adds a FlowFile attribute specified by the dynamic
property's key with the value found in the lookup service using the the dynamic
property's value")
+@WritesAttribute(attribute = "See additional details", description = "This
processor may write zero or more attributes as described in additional details")
+public class LookupAttribute extends AbstractProcessor {
+
+public static final PropertyDescriptor LOOKUP_SERVICE =
+new PropertyDescriptor.Builder()
+.name("lookup-service")
+.displayName("Lookup Service")
+.description("The lookup service to use for attribute lookups")
+.identifiesControllerService(StringLookupService.class)
+.required(true)
+.build();
+
+public static final PropertyDescriptor INCLUDE_EMPTY_VALUES =
+new PropertyDescriptor.Builder()
+.name("include-empty-values")
+.displayName("Include Empty Values")
+.description("Include null or blank values for keys that are
null or blank")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.allowableValues("true", "false")
+.defaultValue("true")
+.required(true)
+.build();
+
+public static final Rel
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user jfrazee commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117809657
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-lookup-service-api/src/main/java/org/apache/nifi/lookup/LookupService.java
---
@@ -17,25 +17,47 @@
package org.apache.nifi.lookup;
+import java.util.Map;
import java.util.Optional;
+import java.util.Set;
import org.apache.nifi.controller.ControllerService;
public interface LookupService extends ControllerService {
/**
- * Looks up a value that corresponds to the given key
+ * Looks up a value that corresponds to the given map of information,
referred to as lookup coordinates
*
- * @param key the key to lookup
- * @return a value that corresponds to the given key
+ * @param coordinates a Map of key/value pairs that indicate the
information that should be looked up
+ * @return a value that corresponds to the given coordinates
*
- * @throws LookupFailureException if unable to lookup a value for the
given key
+ * @throws LookupFailureException if unable to lookup a value for the
given coordinates
*/
-Optional lookup(String key) throws LookupFailureException;
+Optional lookup(Map coordinates) throws
LookupFailureException;
/**
- * @return the Class that represents the type of value that will be
returned by {@link #lookup(String)}
+ * Returns the entire lookup table as a {@link Map}
+ *
+ * @return the lookup table stored in the backing service
+ * @throws LookupFailureException if the backing service is
unavailable or
+ * the table cannot be loaded
+ */
+default Map asMap() throws LookupFailureException {
--- End diff --
I think/know there's a use case for this but I get the worry, so I'll drop
it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117681251
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-lookup-service-api/src/main/java/org/apache/nifi/lookup/LookupService.java
---
@@ -17,25 +17,47 @@
package org.apache.nifi.lookup;
+import java.util.Map;
import java.util.Optional;
+import java.util.Set;
import org.apache.nifi.controller.ControllerService;
public interface LookupService extends ControllerService {
/**
- * Looks up a value that corresponds to the given key
+ * Looks up a value that corresponds to the given map of information,
referred to as lookup coordinates
*
- * @param key the key to lookup
- * @return a value that corresponds to the given key
+ * @param coordinates a Map of key/value pairs that indicate the
information that should be looked up
+ * @return a value that corresponds to the given coordinates
*
- * @throws LookupFailureException if unable to lookup a value for the
given key
+ * @throws LookupFailureException if unable to lookup a value for the
given coordinates
*/
-Optional lookup(String key) throws LookupFailureException;
+Optional lookup(Map coordinates) throws
LookupFailureException;
/**
- * @return the Class that represents the type of value that will be
returned by {@link #lookup(String)}
+ * Returns the entire lookup table as a {@link Map}
+ *
+ * @return the lookup table stored in the backing service
+ * @throws LookupFailureException if the backing service is
unavailable or
+ * the table cannot be loaded
+ */
+default Map asMap() throws LookupFailureException {
--- End diff --
Returning entire lookup table as a Map would cause issue with a large
number of entries. I think this should be avoided.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r117680294 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -41,7 +48,9 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .name(propertyDescriptorName) .required(false) .dynamic(true) -.addValidator(Validator.VALID) + .addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING, true)) + .addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR) +.expressionLanguageSupported(true) --- End diff -- This change enabled EL support, however, cacheConfiguredValues method is not updated to evaluate EL. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117685550
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/DatabaseLookupService.java
---
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.lookup;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import javax.sql.DataSource;
+
+import org.apache.commons.configuration2.Configuration;
+import org.apache.commons.configuration2.DatabaseConfiguration;
+import org.apache.commons.configuration2.ex.ConfigurationException;
+import org.apache.commons.configuration2.builder.BasicConfigurationBuilder;
+import org.apache.commons.configuration2.builder.fluent.Parameters;
+import
org.apache.commons.configuration2.builder.fluent.DatabaseBuilderParameters;
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+@Tags({"lookup", "cache", "enrich", "join", "jdbc", "database", "key",
"value"})
+@CapabilityDescription("A reloadable properties file-based lookup service")
+public class DatabaseLookupService extends AbstractControllerService
implements StringLookupService {
+
+static final PropertyDescriptor CONNECTION_POOL = new
PropertyDescriptor.Builder()
+.name("connection-pool")
+.displayName("Connection Pool")
+.description("Specifices the JDBC connection pool used to connect
to the database.")
+.identifiesControllerService(DBCPService.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor LOOKUP_TABLE_NAME = new
PropertyDescriptor.Builder()
+.name("lookup-table-name")
+.displayName("Lookup Table Name")
+.description("Lookup table name.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+static final PropertyDescriptor LOOKUP_KEY_COLUMN =
+new PropertyDescriptor.Builder()
+.name("lookup-key-column")
+.displayName("Lookup Key Column")
+.description("Lookup key column.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+static final PropertyDescriptor LOOKUP_VALUE_COLUMN =
+new PropertyDescriptor.Builder()
+.name("lookup-value-column")
+.displayName("Lookup Value Column")
+.description("Lookup value column.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+static final PropertyDescriptor LOOKUP_NAME_COLUMN =
+new PropertyDescriptor.Builder()
+.name("lookup-name-column")
+.displayName("Lookup Name Column")
+.description("Lookup name column.")
+.required(false)
+.addValidator(StandardValidators.NO
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117681788
--- Diff:
nifi-nar-bundles/nifi-lookup-bundle/nifi-lookup-processors/src/main/java/org/apache/nifi/processors/lookup/LookupAttribute.java
---
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.lookup;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes",
"Attribute Expression Language"})
+@CapabilityDescription("Lookup attributes from a lookup service")
+@DynamicProperty(name = "The name of the attribute to add to the FlowFile",
+value = "The name of the key or property to retrieve from the lookup
service",
+supportsExpressionLanguage = true,
+description = "Adds a FlowFile attribute specified by the dynamic
property's key with the value found in the lookup service using the the dynamic
property's value")
+@WritesAttribute(attribute = "See additional details", description = "This
processor may write zero or more attributes as described in additional details")
+public class LookupAttribute extends AbstractProcessor {
+
+public static final PropertyDescriptor LOOKUP_SERVICE =
+new PropertyDescriptor.Builder()
+.name("lookup-service")
+.displayName("Lookup Service")
+.description("The lookup service to use for attribute lookups")
+.identifiesControllerService(StringLookupService.class)
+.required(true)
+.build();
+
+public static final PropertyDescriptor INCLUDE_EMPTY_VALUES =
+new PropertyDescriptor.Builder()
+.name("include-empty-values")
+.displayName("Include Empty Values")
+.description("Include null or blank values for keys that are
null or blank")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.allowableValues("true", "false")
+.defaultValue("true")
+.required(true)
+.build();
+
+public static fina
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r117677184 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -41,7 +48,9 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .name(propertyDescriptorName) .required(false) .dynamic(true) -.addValidator(Validator.VALID) + .addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING, true)) + .addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR) --- End diff -- I think we can remove ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR. The dynamic properties of this processor are not representing FlowFile attribute names. It is a little bit misleading. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1830#discussion_r117684959
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/DatabaseLookupService.java
---
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.lookup;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import javax.sql.DataSource;
+
+import org.apache.commons.configuration2.Configuration;
+import org.apache.commons.configuration2.DatabaseConfiguration;
+import org.apache.commons.configuration2.ex.ConfigurationException;
+import org.apache.commons.configuration2.builder.BasicConfigurationBuilder;
+import org.apache.commons.configuration2.builder.fluent.Parameters;
+import
org.apache.commons.configuration2.builder.fluent.DatabaseBuilderParameters;
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+@Tags({"lookup", "cache", "enrich", "join", "jdbc", "database", "key",
"value"})
+@CapabilityDescription("A reloadable properties file-based lookup service")
+public class DatabaseLookupService extends AbstractControllerService
implements StringLookupService {
--- End diff --
While this lookup single column service is powerful, I think it can be more
powerful if we also provide lookup-record implementation like IPLookupService
which lookups Record in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
GitHub user jfrazee opened a pull request: https://github.com/apache/nifi/pull/1830 NIFI-3404 Add lookup processor for enrichments/joins to reference data Note: This is overlaid on [NIFI-3339](https://issues.apache.org/jira/browse/NIFI-3339)/#1450. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jfrazee/nifi NIFI-3404 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1830.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1830 commit 449f33ca5379b22f8dc4b7ba0042481c46129045 Author: Joey Frazee Date: 2017-05-19T16:45:14Z NIFI-3404 Added properties and csv file lookup services commit f2cda5c47c19b1d95a8813c2c79329f49537da1a Author: Joey Frazee Date: 2017-05-19T17:49:24Z NIFI-3404 Added LookupAttribute processor, L&N fixes commit 64e03e344432665b7b4184405335f62221c3e93a Author: Toivo Adams Date: 2017-01-29T11:40:37Z NIFI-3339 Add getDataSource() to DBCPService This closes #1450 Signed-off-by: Joey Frazee commit c3b7e2e76eb38500c02e42676cfe1bcaa0b2291d Author: Joey Frazee Date: 2017-05-19T18:32:41Z NIFI-3404 Added database lookup service --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
