[GitHub] nifi issue #2876: NIFI-5406: Adding new listing strategy to List processors.
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/2876 @markap14 I believe all review comments are addressed now. Also, fixed few things found during tests under clustered environment. And changed default initial listing target from 'window' to 'all' to make it behave the same as existing listing does. I hope it's ready to be merged. Thanks for reviewing and insightful feedback! ---
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[ https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546089#comment-16546089 ] ASF GitHub Bot commented on NIFI-5406: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/2876 @markap14 I believe all review comments are addressed now. Also, fixed few things found during tests under clustered environment. And changed default initial listing target from 'window' to 'all' to make it behave the same as existing listing does. I hope it's ready to be merged. Thanks for reviewing and insightful feedback! > Add new listing strategy by tracking listed entities to List processors > --- > > Key: NIFI-5406 > URL: https://issues.apache.org/jira/browse/NIFI-5406 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > > Current List processors (ListFile, ListFTP, ListSFTP ... etc) implementation > relies on file last modified timestamp to pick new or updated files. This > approach is efficient and lightweight in terms of state management, because > it only tracks latest modified timestamp and last executed timestamp. > However, timestamps do not work as expected in some file systems, causing > List processors missing files periodically. See NIFI-3332 comments for > details. > In order to pick every entity that has not seen before or has been updated > since it had seen last time, we need another set of processors using > different approach, that is by tracking listed entities: > * Add new abstract processor AbstractWatchEntries similar to > AbstractListProcessor but uses different approach > * Target entities have: name (path), size and last-modified-timestamp > * Implementation Processors have following properties: > ** 'Watch Time Window' to limit the maximum time period to hold the already > listed entries. E.g. if set as '30min', the processor keeps entities listed > in the last 30 mins. > ** 'Minimum File Age' to defer listing entities potentially being written > * Any entity added but not listed ever having last-modified-timestamp older > than configured 'Watch Time Window' will not be listed. If user needs to pick > these items, they have to make 'Watch Time Window' longer. It also increases > the size of data the processor has to persist in the K/V store. Efficiency vs > reliability trade-off. > * The already-listed entities are persisted into one of supported K/V store > through DistributedMapCacheClient service. User can chose what KVS to use > from HBase, Redis, Couchbase and File (DistributedMapCacheServer with > persistence file). > * The reason to use KVS instead of ManagedState is, to avoid hammering > Zookeeper too much with frequently updating Zk node with large amount of > data. The number of already-listed entries can be huge depending on > use-cases. Also, we can compress entities with DistributedMapCacheClient as > it supports putting byte array, while ManagedState only supports Map String>. > * On each onTrigger: > ** Processor performs listing. Listed entries meeting any of the following > condition will be written to the 'success' output FlowFile: > *** Not exists in the already-listed entities > *** Having newer last-modified-timestamp > *** Having different size > ** Already listed entries those are old enough compared to 'Watch Time > Window' are discarded from the already-listed entries. > * Initial supporting target is Local file system, FTP and SFTP -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546087#comment-16546087
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202906360
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCa
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202906360
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCacheClient.class)
+.build();
+
+public static final PropertyDescriptor TRACKING_TIME_WINDOW = new
PropertyDescriptor.Builder()
+.name("et-time-window")
+.displayName("Entity Tracking Time Window
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546074#comment-16546074
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202902757
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCa
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202902757
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCacheClient.class)
+.build();
+
+public static final PropertyDescriptor TRACKING_TIME_WINDOW = new
PropertyDescriptor.Builder()
+.name("et-time-window")
+.displayName("Entity Tracking Time Window
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546070#comment-16546070
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202902116
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntity.java
---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+public class ListedEntity {
+/**
+ * Milliseconds.
+ */
+private long timestamp;
+/**
+ * Bytes.
+ */
+private long size;
+
+public void setTimestamp(long timestamp) {
--- End diff --
You are right. I was using final fields and constructor, but when it's
deserialized, I got an Exception around Jackson, so I added setters. I've added
Jackson annotations to use custom constructor for deserialization. Now it's
clear those are immutable.
> Add new listing strategy by tracking listed entities to List processors
> ---
>
> Key: NIFI-5406
> URL: https://issues.apache.org/jira/browse/NIFI-5406
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> Current List processors (ListFile, ListFTP, ListSFTP ... etc) implementation
> relies on file last modified timestamp to pick new or updated files. This
> approach is efficient and lightweight in terms of state management, because
> it only tracks latest modified timestamp and last executed timestamp.
> However, timestamps do not work as expected in some file systems, causing
> List processors missing files periodically. See NIFI-3332 comments for
> details.
> In order to pick every entity that has not seen before or has been updated
> since it had seen last time, we need another set of processors using
> different approach, that is by tracking listed entities:
> * Add new abstract processor AbstractWatchEntries similar to
> AbstractListProcessor but uses different approach
> * Target entities have: name (path), size and last-modified-timestamp
> * Implementation Processors have following properties:
> ** 'Watch Time Window' to limit the maximum time period to hold the already
> listed entries. E.g. if set as '30min', the processor keeps entities listed
> in the last 30 mins.
> ** 'Minimum File Age' to defer listing entities potentially being written
> * Any entity added but not listed ever having last-modified-timestamp older
> than configured 'Watch Time Window' will not be listed. If user needs to pick
> these items, they have to make 'Watch Time Window' longer. It also increases
> the size of data the processor has to persist in the K/V store. Efficiency vs
> reliability trade-off.
> * The already-listed entities are persisted into one of supported K/V store
> through DistributedMapCacheClient service. User can chose what KVS to use
> from HBase, Redis, Couchbase and File (DistributedMapCacheServer with
> persistence file).
> * The reason to use KVS instead of ManagedState is, to avoid hammering
> Zookeeper too much with frequently updating Zk node with large amount of
> data. The number of already-listed entries can be huge depending on
> use-cases. Also, we can compress entities with DistributedMapCacheClient as
> it supports putting byte array, while ManagedState only supports Map String>.
> * On each onTrigger:
> ** Processor performs listing. Listed entries meeting any of the following
> condition will be written to the 'success' output FlowFile:
> *** Not exists in the already-listed entities
> *** Having newer last-modified-timestamp
> *** Having different size
> ** Already listed entries those are ol
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202902116
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntity.java
---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+public class ListedEntity {
+/**
+ * Milliseconds.
+ */
+private long timestamp;
+/**
+ * Bytes.
+ */
+private long size;
+
+public void setTimestamp(long timestamp) {
--- End diff --
You are right. I was using final fields and constructor, but when it's
deserialized, I got an Exception around Jackson, so I added setters. I've added
Jackson annotations to use custom constructor for deserialization. Now it's
clear those are immutable.
---
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546068#comment-16546068
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901813
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
--- End diff --
Updated to use ConcurrentHashMap. Thanks.
> Add new listing strategy by tracking listed entities to List processors
> ---
>
> Key: NIFI-5406
> URL: https://issues.apache.org/jira/browse/NIFI-5406
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> Current List processors (ListFile, ListFTP, ListSFTP ... etc) implementation
> relies on file last modified timestamp to pick new or updated files. This
> approach is efficient and lightweight in terms of state management, because
> it only tracks latest modified timestamp and last executed timestamp.
> However, timestamps do not work as expected in some file systems, causing
> List processors missing files periodically. See NIFI-3332 comments for
> details.
> In order to pick every entity that has not seen before or has been updated
> since it had seen last time, we need another set of processors using
> different approach, that is by tracking listed entities:
> * Add new abstract processor AbstractWatchEntries similar to
> AbstractListProcessor but uses different approach
> * Target entities have: name (path), size and last-modified-timestamp
> * Implementation Processors have following properties:
> ** 'Watch Time Window' to limit the maximum time period to hold the already
> listed entries. E.g. if set as '30
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546067#comment-16546067
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901766
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
--- End diff --
Done.
> Add new listing strategy by tracking listed entities to List processors
> ---
>
> Key: NIFI-5406
> URL: https://issues.apache.org/jira/browse/NIFI-5406
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> Current List processors (ListFile, ListFTP, ListSFTP ... etc) implementation
> relies on file last modified timestamp to pick new or updated files. This
> approach is efficient and lightweight in terms of state management, because
> it only tracks latest modified timestamp and last executed timestamp.
> However, timestamps do not work as expected in some file systems, causing
> List processors missing files periodically. See NIFI-3332 comments for
> details.
> In order to pick every entity that has not seen before or has been updated
> since it had seen last time, we need another set of processors using
> different approach, that is by tracking listed entities:
> * Add new abstract processor AbstractWatchEntries similar to
> AbstractListProcessor but uses different approach
> * Target entities have: name (path), size and last-modified-timestamp
> * Implementation Processors have following properties:
> ** 'Watch Time Window' to limit the maximum time period to hold the already
> listed entries. E.g. if set as '30min', the processor keeps entities listed
> in the last 30 mins.
> ** 'Minimum File Age
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546066#comment-16546066
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901745
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCa
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901813
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
--- End diff --
Updated to use ConcurrentHashMap. Thanks.
---
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901766
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
--- End diff --
Done.
---
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546064#comment-16546064
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901658
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCa
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901676
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCacheClient.class)
+.build();
+
+public static final PropertyDescriptor TRACKING_TIME_WINDOW = new
PropertyDescriptor.Builder()
+.name("et-time-window")
+.displayName("Entity Tracking Time Window
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546065#comment-16546065
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901676
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCa
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901658
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCacheClient.class)
+.build();
+
+public static final PropertyDescriptor TRACKING_TIME_WINDOW = new
PropertyDescriptor.Builder()
+.name("et-time-window")
+.displayName("Entity Tracking Time Window
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901745
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCacheClient.class)
+.build();
+
+public static final PropertyDescriptor TRACKING_TIME_WINDOW = new
PropertyDescriptor.Builder()
+.name("et-time-window")
+.displayName("Entity Tracking Time Window
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546063#comment-16546063
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901585
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCa
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546062#comment-16546062
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901570
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCa
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901585
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCacheClient.class)
+.build();
+
+public static final PropertyDescriptor TRACKING_TIME_WINDOW = new
PropertyDescriptor.Builder()
+.name("et-time-window")
+.displayName("Entity Tracking Time Window
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901570
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCacheClient.class)
+.build();
+
+public static final PropertyDescriptor TRACKING_TIME_WINDOW = new
PropertyDescriptor.Builder()
+.name("et-time-window")
+.displayName("Entity Tracking Time Window
[jira] [Created] (NIFI-5433) Global Menu Doesn't open on Chrome
Mark Jackson created NIFI-5433: -- Summary: Global Menu Doesn't open on Chrome Key: NIFI-5433 URL: https://issues.apache.org/jira/browse/NIFI-5433 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.7.0 Reporter: Mark Jackson After installing nifi 1.7.0, you start up nifi (no changes). Then you click the global menu at the top right (the kebab menu; aka the one with the 3 horizontal lines) and nothing happens. There are no messages in the console or any network requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546061#comment-16546061
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901279
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java
---
@@ -169,6 +179,28 @@
.description("All FlowFiles that are received are routed to
success")
.build();
+public static final AllowableValue BY_TIMESTAMPS = new
AllowableValue("timestamps", "Tracking Timestamps",
+"This strategy tracks the latest timestamp of listed entity to
determine new/updated entities." +
+" Since it only tracks few timestamps, it can manage
listing state efficiently." +
+" However, any newly added, or updated entity having
timestamp older than the tracked latest timestamp can not be picked by this
strategy." +
+" For example, such situation can happen in a file
system if a file with old timestamp" +
+" is copied or moved into the target directory without
its last modified timestamp being updated.");
+
+public static final AllowableValue BY_ENTITIES = new
AllowableValue("entities", "Tracking Entities",
+"This strategy tracks information of all the listed entities
within the latest 'Entity Tracking Time Window' to determine new/updated
entities." +
+" See 'Entity Tracking Time Window' description for
detail on how it works." +
--- End diff --
I wanted to guide user to look at 'Entity Tracking Time Window' property's
description. Updated the text.
> Add new listing strategy by tracking listed entities to List processors
> ---
>
> Key: NIFI-5406
> URL: https://issues.apache.org/jira/browse/NIFI-5406
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> Current List processors (ListFile, ListFTP, ListSFTP ... etc) implementation
> relies on file last modified timestamp to pick new or updated files. This
> approach is efficient and lightweight in terms of state management, because
> it only tracks latest modified timestamp and last executed timestamp.
> However, timestamps do not work as expected in some file systems, causing
> List processors missing files periodically. See NIFI-3332 comments for
> details.
> In order to pick every entity that has not seen before or has been updated
> since it had seen last time, we need another set of processors using
> different approach, that is by tracking listed entities:
> * Add new abstract processor AbstractWatchEntries similar to
> AbstractListProcessor but uses different approach
> * Target entities have: name (path), size and last-modified-timestamp
> * Implementation Processors have following properties:
> ** 'Watch Time Window' to limit the maximum time period to hold the already
> listed entries. E.g. if set as '30min', the processor keeps entities listed
> in the last 30 mins.
> ** 'Minimum File Age' to defer listing entities potentially being written
> * Any entity added but not listed ever having last-modified-timestamp older
> than configured 'Watch Time Window' will not be listed. If user needs to pick
> these items, they have to make 'Watch Time Window' longer. It also increases
> the size of data the processor has to persist in the K/V store. Efficiency vs
> reliability trade-off.
> * The already-listed entities are persisted into one of supported K/V store
> through DistributedMapCacheClient service. User can chose what KVS to use
> from HBase, Redis, Couchbase and File (DistributedMapCacheServer with
> persistence file).
> * The reason to use KVS instead of ManagedState is, to avoid hammering
> Zookeeper too much with frequently updating Zk node with large amount of
> data. The number of already-listed entries can be huge depending on
> use-cases. Also, we can compress entities with DistributedMapCacheClient as
> it supports putting byte array, while ManagedState only supports Map String>.
> * On each onTrigger:
> ** Processor performs listing. Listed entries meeting any of the following
> condition will be written to the 'success' output FlowFile:
> *** Not exists in the already-listed entities
> *** Having newer last-modified-timestamp
> *** Having different size
> ** Already listed entries those are old enough compared to 'Watch Time
> Window' are discarded from the al
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202901279
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java
---
@@ -169,6 +179,28 @@
.description("All FlowFiles that are received are routed to
success")
.build();
+public static final AllowableValue BY_TIMESTAMPS = new
AllowableValue("timestamps", "Tracking Timestamps",
+"This strategy tracks the latest timestamp of listed entity to
determine new/updated entities." +
+" Since it only tracks few timestamps, it can manage
listing state efficiently." +
+" However, any newly added, or updated entity having
timestamp older than the tracked latest timestamp can not be picked by this
strategy." +
+" For example, such situation can happen in a file
system if a file with old timestamp" +
+" is copied or moved into the target directory without
its last modified timestamp being updated.");
+
+public static final AllowableValue BY_ENTITIES = new
AllowableValue("entities", "Tracking Entities",
+"This strategy tracks information of all the listed entities
within the latest 'Entity Tracking Time Window' to determine new/updated
entities." +
+" See 'Entity Tracking Time Window' description for
detail on how it works." +
--- End diff --
I wanted to guide user to look at 'Entity Tracking Time Window' property's
description. Updated the text.
---
[jira] [Commented] (NIFI-5431) Authorization with LDAP does work for first launch
[
https://issues.apache.org/jira/browse/NIFI-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545984#comment-16545984
]
Tony Mao commented on NIFI-5431:
Hi Bryan,
I've attached my xml file to this ticket. Thanks.
> Authorization with LDAP does work for first launch
> --
>
> Key: NIFI-5431
> URL: https://issues.apache.org/jira/browse/NIFI-5431
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.0
> Environment: Ubuntu 16.04
>Reporter: Tony Mao
>Priority: Major
> Attachments: authorizers.xml
>
>
> I'm trying to make authorization works with our LDAP server. I have installed
> a new 1.7.0 instance without any legacy things on my Ubuntu. But when I tried
> to start the nifi by nifi.sh start command, I got the error from
> nifi-bootstrap.log like following
> {code:java}
> 2018-07-16 01:34:48,267 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> Failed to start web server: Error creating bean with name
> 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
> Unsatisfied dependency expressed through method
> 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is
> org.springframework.beans.factory.BeanExpressionException: Expression parsing
> failed; nested exception is
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name
> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency
> expressed through method 'setJwtAuthenticationProvider' parameter 0; nested
> exception is org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'jwtAuthenticationProvider' defined in class path
> resource [nifi-web-security-context.xml]: Cannot resolve reference to bean
> 'authorizer' while setting constructor argument; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'authorizer': FactoryBean threw exception on object creation;
> nested exception is
> org.apache.nifi.authorization.exception.AuthorizerCreationException:
> org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable
> to locate initial admin cn=Tony Mao,ou=Users,ou=Accounts,dc=abc,dc=local to
> seed policies
> {code}
> Let me attach the config file as well.
> Need your help please.
> Thanks a lot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (NIFI-5431) Authorization with LDAP does work for first launch
[
https://issues.apache.org/jira/browse/NIFI-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545984#comment-16545984
]
Tony Mao edited comment on NIFI-5431 at 7/17/18 3:15 AM:
-
Hi [~bende],
I've attached my xml file to this ticket. Thanks.
was (Author: tony0918):
Hi Bryan,
I've attached my xml file to this ticket. Thanks.
> Authorization with LDAP does work for first launch
> --
>
> Key: NIFI-5431
> URL: https://issues.apache.org/jira/browse/NIFI-5431
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.0
> Environment: Ubuntu 16.04
>Reporter: Tony Mao
>Priority: Major
> Attachments: authorizers.xml
>
>
> I'm trying to make authorization works with our LDAP server. I have installed
> a new 1.7.0 instance without any legacy things on my Ubuntu. But when I tried
> to start the nifi by nifi.sh start command, I got the error from
> nifi-bootstrap.log like following
> {code:java}
> 2018-07-16 01:34:48,267 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> Failed to start web server: Error creating bean with name
> 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
> Unsatisfied dependency expressed through method
> 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is
> org.springframework.beans.factory.BeanExpressionException: Expression parsing
> failed; nested exception is
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name
> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency
> expressed through method 'setJwtAuthenticationProvider' parameter 0; nested
> exception is org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'jwtAuthenticationProvider' defined in class path
> resource [nifi-web-security-context.xml]: Cannot resolve reference to bean
> 'authorizer' while setting constructor argument; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'authorizer': FactoryBean threw exception on object creation;
> nested exception is
> org.apache.nifi.authorization.exception.AuthorizerCreationException:
> org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable
> to locate initial admin cn=Tony Mao,ou=Users,ou=Accounts,dc=abc,dc=local to
> seed policies
> {code}
> Let me attach the config file as well.
> Need your help please.
> Thanks a lot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-site pull request #27: Added GPG Guide
Github user asfgit closed the pull request at: https://github.com/apache/nifi-site/pull/27 ---
[jira] [Commented] (NIFI-5414) Release Apache NiFi 1.7.1
[ https://issues.apache.org/jira/browse/NIFI-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545943#comment-16545943 ] ASF subversion and git services commented on NIFI-5414: --- Commit e959630c22c9a52ec717141f6cf9f018830a38bf in nifi's branch refs/heads/support/nifi-1.7.x from [~alopresto] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=e959630 ] NIFI-5414 Fixed checkstyle error due to unused import. > Release Apache NiFi 1.7.1 > - > > Key: NIFI-5414 > URL: https://issues.apache.org/jira/browse/NIFI-5414 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Blocker > Labels: release > Fix For: 1.7.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5399) Improve documentation to indicate that wildcard certificates are not recommended for cluster communications
[ https://issues.apache.org/jira/browse/NIFI-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545939#comment-16545939 ] ASF subversion and git services commented on NIFI-5399: --- Commit 1836076 from alopre...@apache.org in branch 'site/trunk' [ https://svn.apache.org/r1836076 ] Added note about wildcard certificates from NIFI-5399. > Improve documentation to indicate that wildcard certificates are not > recommended for cluster communications > --- > > Key: NIFI-5399 > URL: https://issues.apache.org/jira/browse/NIFI-5399 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Affects Versions: 1.7.0 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Major > Labels: certificate, cluster, documentation, security, tls > Fix For: 1.8.0, 1.7.1 > > > Based on the research from NIFI-5370 and potential changes in NIFI-5398, the > Admin Guide should be improved to explain why wildcard certificates without a > unique SAN are not recommended for secure cluster configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5414) Release Apache NiFi 1.7.1
[ https://issues.apache.org/jira/browse/NIFI-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545924#comment-16545924 ] ASF subversion and git services commented on NIFI-5414: --- Commit 1836075 from alopre...@apache.org in branch 'site/trunk' [ https://svn.apache.org/r1836075 ] NIFI-5414 Added 1.7.1 docs to site. > Release Apache NiFi 1.7.1 > - > > Key: NIFI-5414 > URL: https://issues.apache.org/jira/browse/NIFI-5414 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Blocker > Labels: release > Fix For: 1.7.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[ https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545785#comment-16545785 ] ASF GitHub Bot commented on NIFI-5432: -- GitHub user ottobackwards opened a pull request: https://github.com/apache/nifi/pull/2900 NIFI-5432 Add Syslog Record Reader legacy Syslog This PR adds a Record reader of syslog messages. It follows the older ParseSyslog rules, and uses that parser and event object. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [-] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [-] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/nifi syslog-legacy-reader Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2900.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2900 commit a18278d357a13a5b434db77a3e44f39dfd39e062 Author: Otto Fowler Date: 2018-07-16T21:58:13Z NIFI-5432 Add Syslog Record Reader legacy Syslog > Add Syslog Record Reader legacy Syslog > -- > > Key: NIFI-5432 > URL: https://issues.apache.org/jira/browse/NIFI-5432 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Create a record reader, based on nifi-syslog-utils for reading syslog without > support for structured data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[ https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545786#comment-16545786 ] ASF GitHub Bot commented on NIFI-5432: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2900 @bbende > Add Syslog Record Reader legacy Syslog > -- > > Key: NIFI-5432 > URL: https://issues.apache.org/jira/browse/NIFI-5432 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Create a record reader, based on nifi-syslog-utils for reading syslog without > support for structured data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2900: NIFI-5432 Add Syslog Record Reader legacy Syslog
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2900 @bbende ---
[GitHub] nifi pull request #2900: NIFI-5432 Add Syslog Record Reader legacy Syslog
GitHub user ottobackwards opened a pull request: https://github.com/apache/nifi/pull/2900 NIFI-5432 Add Syslog Record Reader legacy Syslog This PR adds a Record reader of syslog messages. It follows the older ParseSyslog rules, and uses that parser and event object. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [-] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [-] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/nifi syslog-legacy-reader Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2900.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2900 commit a18278d357a13a5b434db77a3e44f39dfd39e062 Author: Otto Fowler Date: 2018-07-16T21:58:13Z NIFI-5432 Add Syslog Record Reader legacy Syslog ---
[jira] [Commented] (NIFI-4535) Set the Page Title to the name of the Root Process Group
[
https://issues.apache.org/jira/browse/NIFI-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545712#comment-16545712
]
ASF GitHub Bot commented on NIFI-4535:
--
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2899#discussion_r202814854
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-canvas.js
---
@@ -167,12 +167,14 @@
nfNgBridge.injector.get('breadcrumbsCtrl').resetScrollPosition();
// set page title to the name of the root processor group
-var rootBreadcrumb = breadcrumb;
-while(rootBreadcrumb.parentBreadcrumb != null) {
-rootBreadcrumb = rootBreadcrumb.parentBreadcrumb
-}
+if (breadcrumb.permissions.canRead) {
+var rootBreadcrumb = breadcrumb;
+while(rootBreadcrumb.parentBreadcrumb != null) {
+rootBreadcrumb = rootBreadcrumb.parentBreadcrumb
+}
-document.title = rootBreadcrumb.breadcrumb.name;
+document.title = rootBreadcrumb.breadcrumb.name;
--- End diff --
@patricker Thanks for the PR!
I haven't run this yet, but I think there may still be an issue. We need to
ensure we have permissions to the `rootBreadcrumb` here. If the user navigated
to a sub-Process Group but did not have permissions to the root Process Group I
think we'd still have the error.
Additionally, if the user navigated to a sub-Process Group which they
didn't have access to but they did have access to the root Process Group we
would unnecessarily avoid setting the document title.
I don't believe there was an issue with identifying the `rootBreadcrumb`.
The issue was attempting to extract the breadcrumb's name through the
`breadcrumb` field which is `null`ed out when a user doesn't have permissions
to the corresponding Process Group.
> Set the Page Title to the name of the Root Process Group
>
>
> Key: NIFI-4535
> URL: https://issues.apache.org/jira/browse/NIFI-4535
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core UI
>Reporter: Peter Wicks
>Assignee: Peter Wicks
>Priority: Minor
> Fix For: 1.8.0
>
>
> NiFi's UI has a hard coded page title of NiFi. I have many servers and it's
> hard to keep track of the tabs in Chrome.
> Please change the Title of the page so it matches the name of the Root
> processor group. This way I can name the root group by server/instance, and
> easily identify the right tab/window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2899: NIFI-4535 Only update Page Title to root flow name ...
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2899#discussion_r202814854
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-canvas.js
---
@@ -167,12 +167,14 @@
nfNgBridge.injector.get('breadcrumbsCtrl').resetScrollPosition();
// set page title to the name of the root processor group
-var rootBreadcrumb = breadcrumb;
-while(rootBreadcrumb.parentBreadcrumb != null) {
-rootBreadcrumb = rootBreadcrumb.parentBreadcrumb
-}
+if (breadcrumb.permissions.canRead) {
+var rootBreadcrumb = breadcrumb;
+while(rootBreadcrumb.parentBreadcrumb != null) {
+rootBreadcrumb = rootBreadcrumb.parentBreadcrumb
+}
-document.title = rootBreadcrumb.breadcrumb.name;
+document.title = rootBreadcrumb.breadcrumb.name;
--- End diff --
@patricker Thanks for the PR!
I haven't run this yet, but I think there may still be an issue. We need to
ensure we have permissions to the `rootBreadcrumb` here. If the user navigated
to a sub-Process Group but did not have permissions to the root Process Group I
think we'd still have the error.
Additionally, if the user navigated to a sub-Process Group which they
didn't have access to but they did have access to the root Process Group we
would unnecessarily avoid setting the document title.
I don't believe there was an issue with identifying the `rootBreadcrumb`.
The issue was attempting to extract the breadcrumb's name through the
`breadcrumb` field which is `null`ed out when a user doesn't have permissions
to the corresponding Process Group.
---
[jira] [Resolved] (MINIFICPP-557) Segfault when including C2 AgentInformation
[ https://issues.apache.org/jira/browse/MINIFICPP-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFICPP-557. --- Resolution: Fixed Fix Version/s: 0.6.0 > Segfault when including C2 AgentInformation > --- > > Key: MINIFICPP-557 > URL: https://issues.apache.org/jira/browse/MINIFICPP-557 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: marco polo >Priority: Major > Fix For: 0.6.0 > > Attachments: minifi-app.log > > > When running an instance with AgentInformation of one of the > nifi.c2.root.classes, the instance fails to startup with a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-557) Segfault when including C2 AgentInformation
[ https://issues.apache.org/jira/browse/MINIFICPP-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-557: -- Summary: Segfault when including C2 AgentInformation (was: Segfault when including C2 AgentInformation in Alpine) > Segfault when including C2 AgentInformation > --- > > Key: MINIFICPP-557 > URL: https://issues.apache.org/jira/browse/MINIFICPP-557 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: marco polo >Priority: Major > Fix For: 0.6.0 > > Attachments: minifi-app.log > > > When running an instance with AgentInformation of one of the > nifi.c2.root.classes, the instance fails to startup with a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-557) Segfault when including C2 AgentInformation in Alpine
[ https://issues.apache.org/jira/browse/MINIFICPP-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545669#comment-16545669 ] ASF GitHub Bot commented on MINIFICPP-557: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/372 > Segfault when including C2 AgentInformation in Alpine > - > > Key: MINIFICPP-557 > URL: https://issues.apache.org/jira/browse/MINIFICPP-557 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: marco polo >Priority: Major > Fix For: 0.6.0 > > Attachments: minifi-app.log > > > When running an instance with AgentInformation of one of the > nifi.c2.root.classes, the instance fails to startup with a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #372: MINIFICPP-557: Previously the empty flag ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/372 ---
[jira] [Commented] (MINIFICPP-557) Segfault when including C2 AgentInformation in Alpine
[ https://issues.apache.org/jira/browse/MINIFICPP-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545666#comment-16545666 ] ASF GitHub Bot commented on MINIFICPP-557: -- Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/372 these changes look good. was able to verify the issue was remedied. will merge. thanks! > Segfault when including C2 AgentInformation in Alpine > - > > Key: MINIFICPP-557 > URL: https://issues.apache.org/jira/browse/MINIFICPP-557 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: marco polo >Priority: Major > Attachments: minifi-app.log > > > When running an instance with AgentInformation of one of the > nifi.c2.root.classes, the instance fails to startup with a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #372: MINIFICPP-557: Previously the empty flag was not...
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/372 these changes look good. was able to verify the issue was remedied. will merge. thanks! ---
[jira] [Commented] (NIFI-5431) Authorization with LDAP does work for first launch
[
https://issues.apache.org/jira/browse/NIFI-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545664#comment-16545664
]
Bryan Bende commented on NIFI-5431:
---
Can you show your authorizers.xml?
> Authorization with LDAP does work for first launch
> --
>
> Key: NIFI-5431
> URL: https://issues.apache.org/jira/browse/NIFI-5431
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.0
> Environment: Ubuntu 16.04
>Reporter: Tony Mao
>Priority: Major
> Attachments: authorizers.xml
>
>
> I'm trying to make authorization works with our LDAP server. I have installed
> a new 1.7.0 instance without any legacy things on my Ubuntu. But when I tried
> to start the nifi by nifi.sh start command, I got the error from
> nifi-bootstrap.log like following
> {code:java}
> 2018-07-16 01:34:48,267 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> Failed to start web server: Error creating bean with name
> 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
> Unsatisfied dependency expressed through method
> 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is
> org.springframework.beans.factory.BeanExpressionException: Expression parsing
> failed; nested exception is
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name
> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency
> expressed through method 'setJwtAuthenticationProvider' parameter 0; nested
> exception is org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'jwtAuthenticationProvider' defined in class path
> resource [nifi-web-security-context.xml]: Cannot resolve reference to bean
> 'authorizer' while setting constructor argument; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'authorizer': FactoryBean threw exception on object creation;
> nested exception is
> org.apache.nifi.authorization.exception.AuthorizerCreationException:
> org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable
> to locate initial admin cn=Tony Mao,ou=Users,ou=Accounts,dc=abc,dc=local to
> seed policies
> {code}
> Let me attach the config file as well.
> Need your help please.
> Thanks a lot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFI-5430) NiFi CLI tool extension for cluster summary
[ https://issues.apache.org/jira/browse/NIFI-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende resolved NIFI-5430. --- Resolution: Fixed Fix Version/s: 1.8.0 > NiFi CLI tool extension for cluster summary > --- > > Key: NIFI-5430 > URL: https://issues.apache.org/jira/browse/NIFI-5430 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Peter Wilcsinszky >Assignee: Peter Wilcsinszky >Priority: Minor > Fix For: 1.8.0 > > > Cluster summary is required in use cases like in a Kubernetes deployment to > get information about cluster status periodically. As soon as a node reports > itself to be part of the cluster it can be considered ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5430) NiFi CLI tool extension for cluster summary
[ https://issues.apache.org/jira/browse/NIFI-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende reassigned NIFI-5430: - Assignee: Peter Wilcsinszky > NiFi CLI tool extension for cluster summary > --- > > Key: NIFI-5430 > URL: https://issues.apache.org/jira/browse/NIFI-5430 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Peter Wilcsinszky >Assignee: Peter Wilcsinszky >Priority: Minor > > Cluster summary is required in use cases like in a Kubernetes deployment to > get information about cluster status periodically. As soon as a node reports > itself to be part of the cluster it can be considered ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5430) NiFi CLI tool extension for cluster summary
[ https://issues.apache.org/jira/browse/NIFI-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545648#comment-16545648 ] ASF subversion and git services commented on NIFI-5430: --- Commit b191f6a62a717cb37a318e1a4180aa5ebcaae29e in nifi's branch refs/heads/master from pepov [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b191f6a ] NIFI-5430 CLI tool extension for cluster summary This closes #2894. Signed-off-by: Bryan Bende > NiFi CLI tool extension for cluster summary > --- > > Key: NIFI-5430 > URL: https://issues.apache.org/jira/browse/NIFI-5430 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Peter Wilcsinszky >Priority: Minor > > Cluster summary is required in use cases like in a Kubernetes deployment to > get information about cluster status periodically. As soon as a node reports > itself to be part of the cluster it can be considered ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5430) NiFi CLI tool extension for cluster summary
[ https://issues.apache.org/jira/browse/NIFI-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545651#comment-16545651 ] ASF GitHub Bot commented on NIFI-5430: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2894 > NiFi CLI tool extension for cluster summary > --- > > Key: NIFI-5430 > URL: https://issues.apache.org/jira/browse/NIFI-5430 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Peter Wilcsinszky >Priority: Minor > > Cluster summary is required in use cases like in a Kubernetes deployment to > get information about cluster status periodically. As soon as a node reports > itself to be part of the cluster it can be considered ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2894: NIFI-5430 CLI tool extension for cluster summary
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2894 ---
[jira] [Commented] (NIFI-5430) NiFi CLI tool extension for cluster summary
[ https://issues.apache.org/jira/browse/NIFI-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545647#comment-16545647 ] ASF GitHub Bot commented on NIFI-5430: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2894 No problem, looks good and going to merge, thanks! > NiFi CLI tool extension for cluster summary > --- > > Key: NIFI-5430 > URL: https://issues.apache.org/jira/browse/NIFI-5430 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Peter Wilcsinszky >Priority: Minor > > Cluster summary is required in use cases like in a Kubernetes deployment to > get information about cluster status periodically. As soon as a node reports > itself to be part of the cluster it can be considered ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2894: NIFI-5430 CLI tool extension for cluster summary
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2894 No problem, looks good and going to merge, thanks! ---
[jira] [Commented] (NIFI-4535) Set the Page Title to the name of the Root Process Group
[ https://issues.apache.org/jira/browse/NIFI-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545631#comment-16545631 ] ASF GitHub Bot commented on NIFI-4535: -- GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/2899 NIFI-4535 Only update Page Title to root flow name when user has perm… …ission. Updated to handle scenario where user does not have read access. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-4535 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2899.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2899 commit 0bb229b3e0d26fdef2fc3f6287f3e228de6b21d2 Author: patricker Date: 2018-07-16T19:03:05Z NIFI-4535 Only update Page Title to root flow name when user has permission. > Set the Page Title to the name of the Root Process Group > > > Key: NIFI-4535 > URL: https://issues.apache.org/jira/browse/NIFI-4535 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Minor > Fix For: 1.8.0 > > > NiFi's UI has a hard coded page title of NiFi. I have many servers and it's > hard to keep track of the tabs in Chrome. > Please change the Title of the page so it matches the name of the Root > processor group. This way I can name the root group by server/instance, and > easily identify the right tab/window. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2899: NIFI-4535 Only update Page Title to root flow name ...
GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/2899 NIFI-4535 Only update Page Title to root flow name when user has perm⦠â¦ission. Updated to handle scenario where user does not have read access. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-4535 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2899.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2899 commit 0bb229b3e0d26fdef2fc3f6287f3e228de6b21d2 Author: patricker Date: 2018-07-16T19:03:05Z NIFI-4535 Only update Page Title to root flow name when user has permission. ---
[jira] [Commented] (NIFI-4407) Non-EL statement processed as expression language
[
https://issues.apache.org/jira/browse/NIFI-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545627#comment-16545627
]
Mark Payne commented on NIFI-4407:
--
I reviewed the documentation in the Expression Language Guide, and
unfortunately it doesn't really call out when $$ will be escaped and when it
won't. So I started by updating the Expression Language Guide to explain
exactly when the $$ will and won't be escaped. I then updated unit tests to
include the examples that I added to the EL Guide and to include the
${literal('SNAP$')} type examples provided by [~Absolutesantaja]
(thanks!). I then updated the code to ensure that all unit tests pass. PR is
now ready to be reviewed.
> Non-EL statement processed as expression language
> -
>
> Key: NIFI-4407
> URL: https://issues.apache.org/jira/browse/NIFI-4407
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.1.1, 1.0.1, 1.3.0
>Reporter: Pierre Villard
>Priority: Critical
> Fix For: 1.8.0
>
>
> If you take a GFF with custom text: {{test$$foo}}
> The generated text will be: {{test$foo}}
> The property supports expression language and one $ is removed during the EL
> evaluation step. This can be an issue if a user wants to use a value
> containing to consecutive $$ (such as in password fields).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-4407) Non-EL statement processed as expression language
[
https://issues.apache.org/jira/browse/NIFI-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-4407:
-
Assignee: Mark Payne
Fix Version/s: 1.8.0
Status: Patch Available (was: Open)
> Non-EL statement processed as expression language
> -
>
> Key: NIFI-4407
> URL: https://issues.apache.org/jira/browse/NIFI-4407
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.0.1, 1.1.1, 1.2.0, 1.1.0, 1.0.0
>Reporter: Pierre Villard
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.8.0
>
>
> If you take a GFF with custom text: {{test$$foo}}
> The generated text will be: {{test$foo}}
> The property supports expression language and one $ is removed during the EL
> evaluation step. This can be an issue if a user wants to use a value
> containing to consecutive $$ (such as in password fields).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4407) Non-EL statement processed as expression language
[
https://issues.apache.org/jira/browse/NIFI-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545623#comment-16545623
]
ASF GitHub Bot commented on NIFI-4407:
--
GitHub user markap14 opened a pull request:
https://github.com/apache/nifi/pull/2898
NIFI-4407: Updated Expression Language Guide to provide details about…
… how escaping $ is accomplished, and when the $ character should and
should not be escaped. Fixed bug in the Query compiler that mistakenly would
blindly replace 484 with $ even when the 484 did not precede an Expression.
Added additional test cases.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markap14/nifi NIFI-4407
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2898.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2898
commit 8a9ff6884253b3294598772fad60347605257b0d
Author: Mark Payne
Date: 2018-07-16T17:27:46Z
NIFI-4407: Updated Expression Language Guide to provide details about how
escaping $ is accomplished, and when the $ character should and should not be
escaped. Fixed bug in the Query compiler that mistakenly would blindly replace
484 with $ even when the 484 did not precede an Expression. Added additional
test cases.
> Non-EL statement processed as expression language
> -
>
> Key: NIFI-4407
> URL: https://issues.apache.org/jira/browse/NIFI-4407
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.1.1, 1.0.1, 1.3.0
>Reporter: Pierre Villard
>Priority: Critical
>
> If you take a GFF with custom text: {{test$$foo}}
> The generated text will be: {{test$foo}}
> The property supports expression language and one $ is removed during the EL
> evaluation step. This can be an issue if a user wants to use a value
> containing to consecutive $$ (such as in password fields).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2898: NIFI-4407: Updated Expression Language Guide to pro...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/2898 NIFI-4407: Updated Expression Language Guide to provide details about⦠⦠how escaping $ is accomplished, and when the $ character should and should not be escaped. Fixed bug in the Query compiler that mistakenly would blindly replace 484 with $ even when the 484 did not precede an Expression. Added additional test cases. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-4407 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2898.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2898 commit 8a9ff6884253b3294598772fad60347605257b0d Author: Mark Payne Date: 2018-07-16T17:27:46Z NIFI-4407: Updated Expression Language Guide to provide details about how escaping $ is accomplished, and when the $ character should and should not be escaped. Fixed bug in the Query compiler that mistakenly would blindly replace 484 with $ even when the 484 did not precede an Expression. Added additional test cases. ---
[jira] [Commented] (NIFI-5430) NiFi CLI tool extension for cluster summary
[ https://issues.apache.org/jira/browse/NIFI-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545620#comment-16545620 ] ASF GitHub Bot commented on NIFI-5430: -- Github user pepov commented on the issue: https://github.com/apache/nifi/pull/2894 had a hard time triggering a build with the import fixes due to github issues, sorry @bbende , it should be better now > NiFi CLI tool extension for cluster summary > --- > > Key: NIFI-5430 > URL: https://issues.apache.org/jira/browse/NIFI-5430 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Peter Wilcsinszky >Priority: Minor > > Cluster summary is required in use cases like in a Kubernetes deployment to > get information about cluster status periodically. As soon as a node reports > itself to be part of the cluster it can be considered ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2894: NIFI-5430 CLI tool extension for cluster summary
Github user pepov commented on the issue: https://github.com/apache/nifi/pull/2894 had a hard time triggering a build with the import fixes due to github issues, sorry @bbende , it should be better now ---
[jira] [Commented] (MINIFICPP-557) Segfault when including C2 AgentInformation in Alpine
[ https://issues.apache.org/jira/browse/MINIFICPP-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545590#comment-16545590 ] ASF GitHub Bot commented on MINIFICPP-557: -- Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/372 reviewing > Segfault when including C2 AgentInformation in Alpine > - > > Key: MINIFICPP-557 > URL: https://issues.apache.org/jira/browse/MINIFICPP-557 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: marco polo >Priority: Major > Attachments: minifi-app.log > > > When running an instance with AgentInformation of one of the > nifi.c2.root.classes, the instance fails to startup with a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #372: MINIFICPP-557: Previously the empty flag was not...
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/372 reviewing ---
[GitHub] nifi pull request #2897: NIFI-4407: Updated Expression Language Guide to pro...
Github user markap14 closed the pull request at: https://github.com/apache/nifi/pull/2897 ---
[jira] [Commented] (NIFI-4407) Non-EL statement processed as expression language
[
https://issues.apache.org/jira/browse/NIFI-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545586#comment-16545586
]
ASF GitHub Bot commented on NIFI-4407:
--
Github user markap14 closed the pull request at:
https://github.com/apache/nifi/pull/2897
> Non-EL statement processed as expression language
> -
>
> Key: NIFI-4407
> URL: https://issues.apache.org/jira/browse/NIFI-4407
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.1.1, 1.0.1, 1.3.0
>Reporter: Pierre Villard
>Priority: Critical
>
> If you take a GFF with custom text: {{test$$foo}}
> The generated text will be: {{test$foo}}
> The property supports expression language and one $ is removed during the EL
> evaluation step. This can be an issue if a user wants to use a value
> containing to consecutive $$ (such as in password fields).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-557) Segfault when including C2 AgentInformation in Alpine
[ https://issues.apache.org/jira/browse/MINIFICPP-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545583#comment-16545583 ] ASF GitHub Bot commented on MINIFICPP-557: -- Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/372 @apiri feel free to try this > Segfault when including C2 AgentInformation in Alpine > - > > Key: MINIFICPP-557 > URL: https://issues.apache.org/jira/browse/MINIFICPP-557 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: marco polo >Priority: Major > Attachments: minifi-app.log > > > When running an instance with AgentInformation of one of the > nifi.c2.root.classes, the instance fails to startup with a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-557) Segfault when including C2 AgentInformation in Alpine
[ https://issues.apache.org/jira/browse/MINIFICPP-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545582#comment-16545582 ] ASF GitHub Bot commented on MINIFICPP-557: -- GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/372 MINIFICPP-557: Previously the empty flag was not a necessary check as… … we didn't have the case where empty ValueNodes could or should occur ( and these were dealt with elsewhere) but recent changes have made this a new requirement thus we can add the empty check for validation of lists. Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-557 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/372.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #372 commit 15ab090b3148456ca5e52aa1b79de1df5cf8badc Author: Marc Parisi Date: 2018-07-16T18:22:24Z MINIFICPP-557: Previously the empty flag was not a necessary check as we didn't have the case where empty ValueNodes could or should occur ( and these were dealt with elsewhere) but recent changes have made this a new requirement thus we can add the empty check for validation of lists. > Segfault when including C2 AgentInformation in Alpine > - > > Key: MINIFICPP-557 > URL: https://issues.apache.org/jira/browse/MINIFICPP-557 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: marco polo >Priority: Major > Attachments: minifi-app.log > > > When running an instance with AgentInformation of one of the > nifi.c2.root.classes, the instance fails to startup with a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (MINIFICPP-557) Segfault when including C2 AgentInformation in Alpine
[ https://issues.apache.org/jira/browse/MINIFICPP-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545412#comment-16545412 ] marco polo edited comment on MINIFICPP-557 at 7/16/18 6:25 PM: --- [~aldrin] Changes in -MINIFICPP-5+00+- caused this. The issue is not inherently AgentInformation. It's the mismanagement of string memory during copies and moves. I will fix appropriately. was (Author: phrocker): [~aldrin] Changes in MINIFICPP-515 caused this. The issue is not inherently AgentInformation. It's the mismanagement of string memory during copies and moves. I will fix appropriately. > Segfault when including C2 AgentInformation in Alpine > - > > Key: MINIFICPP-557 > URL: https://issues.apache.org/jira/browse/MINIFICPP-557 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: marco polo >Priority: Major > Attachments: minifi-app.log > > > When running an instance with AgentInformation of one of the > nifi.c2.root.classes, the instance fails to startup with a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #372: MINIFICPP-557: Previously the empty flag was not...
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/372 @apiri feel free to try this ---
[GitHub] nifi-minifi-cpp pull request #372: MINIFICPP-557: Previously the empty flag ...
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/372 MINIFICPP-557: Previously the empty flag was not a necessary check as⦠⦠we didn't have the case where empty ValueNodes could or should occur ( and these were dealt with elsewhere) but recent changes have made this a new requirement thus we can add the empty check for validation of lists. Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-557 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/372.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #372 commit 15ab090b3148456ca5e52aa1b79de1df5cf8badc Author: Marc Parisi Date: 2018-07-16T18:22:24Z MINIFICPP-557: Previously the empty flag was not a necessary check as we didn't have the case where empty ValueNodes could or should occur ( and these were dealt with elsewhere) but recent changes have made this a new requirement thus we can add the empty check for validation of lists. ---
[jira] [Commented] (NIFI-5430) NiFi CLI tool extension for cluster summary
[
https://issues.apache.org/jira/browse/NIFI-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545580#comment-16545580
]
ASF GitHub Bot commented on NIFI-5430:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2894#discussion_r202777026
--- Diff:
nifi-toolkit/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/client/nifi/impl/JerseyFlowClient.java
---
@@ -217,4 +217,12 @@ public ActivateControllerServicesEntity
activateControllerServices(final Activat
ActivateControllerServicesEntity.class);
});
}
+
+@Override
+public ClusteSummaryEntity getClusterSummary() throws
NiFiClientException, IOException {
--- End diff --
Same import issue as above, also I assume the message should be "Error
retrieving cluster summary" or something like that
> NiFi CLI tool extension for cluster summary
> ---
>
> Key: NIFI-5430
> URL: https://issues.apache.org/jira/browse/NIFI-5430
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
>Reporter: Peter Wilcsinszky
>Priority: Minor
>
> Cluster summary is required in use cases like in a Kubernetes deployment to
> get information about cluster status periodically. As soon as a node reports
> itself to be part of the cluster it can be considered ready.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2894: NIFI-5430 CLI tool extension for cluster summary
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2894#discussion_r202777026
--- Diff:
nifi-toolkit/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/client/nifi/impl/JerseyFlowClient.java
---
@@ -217,4 +217,12 @@ public ActivateControllerServicesEntity
activateControllerServices(final Activat
ActivateControllerServicesEntity.class);
});
}
+
+@Override
+public ClusteSummaryEntity getClusterSummary() throws
NiFiClientException, IOException {
--- End diff --
Same import issue as above, also I assume the message should be "Error
retrieving cluster summary" or something like that
---
[jira] [Commented] (NIFI-5430) NiFi CLI tool extension for cluster summary
[ https://issues.apache.org/jira/browse/NIFI-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545579#comment-16545579 ] ASF GitHub Bot commented on NIFI-5430: -- Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/2894#discussion_r202776797 --- Diff: nifi-toolkit/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/client/nifi/FlowClient.java --- @@ -97,4 +97,10 @@ VersionedFlowSnapshotMetadataSetEntity getVersions(String registryId, String buc */ ActivateControllerServicesEntity activateControllerServices(ActivateControllerServicesEntity activateControllerServicesEntity) throws NiFiClientException, IOException; +/** + * + * @return cluster summary response + */ +ClusteSummaryEntity getClusterSummary() throws NiFiClientException, IOException; --- End diff -- This was still showing as missing import for me > NiFi CLI tool extension for cluster summary > --- > > Key: NIFI-5430 > URL: https://issues.apache.org/jira/browse/NIFI-5430 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Peter Wilcsinszky >Priority: Minor > > Cluster summary is required in use cases like in a Kubernetes deployment to > get information about cluster status periodically. As soon as a node reports > itself to be part of the cluster it can be considered ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2894: NIFI-5430 CLI tool extension for cluster summary
Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/2894#discussion_r202776797 --- Diff: nifi-toolkit/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/client/nifi/FlowClient.java --- @@ -97,4 +97,10 @@ VersionedFlowSnapshotMetadataSetEntity getVersions(String registryId, String buc */ ActivateControllerServicesEntity activateControllerServices(ActivateControllerServicesEntity activateControllerServicesEntity) throws NiFiClientException, IOException; +/** + * + * @return cluster summary response + */ +ClusteSummaryEntity getClusterSummary() throws NiFiClientException, IOException; --- End diff -- This was still showing as missing import for me ---
[jira] [Commented] (NIFI-5430) NiFi CLI tool extension for cluster summary
[ https://issues.apache.org/jira/browse/NIFI-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1654#comment-1654 ] ASF GitHub Bot commented on NIFI-5430: -- Github user pepov commented on the issue: https://github.com/apache/nifi/pull/2894 Pushed a revert and fixed missing imports, however github seems to have issues, will trigger a new build later. > NiFi CLI tool extension for cluster summary > --- > > Key: NIFI-5430 > URL: https://issues.apache.org/jira/browse/NIFI-5430 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Peter Wilcsinszky >Priority: Minor > > Cluster summary is required in use cases like in a Kubernetes deployment to > get information about cluster status periodically. As soon as a node reports > itself to be part of the cluster it can be considered ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2894: NIFI-5430 CLI tool extension for cluster summary
Github user pepov commented on the issue: https://github.com/apache/nifi/pull/2894 Pushed a revert and fixed missing imports, however github seems to have issues, will trigger a new build later. ---
[jira] [Commented] (NIFI-4407) Non-EL statement processed as expression language
[
https://issues.apache.org/jira/browse/NIFI-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545522#comment-16545522
]
ASF GitHub Bot commented on NIFI-4407:
--
GitHub user markap14 opened a pull request:
https://github.com/apache/nifi/pull/2897
NIFI-4407: Updated Expression Language Guide to provide details about…
… how escaping $ is accomplished, and when the $ character should and
should not be escaped. Fixed bug in the Query compiler that mistakenly would
blindly replace 484 with $ even when the 484 did not precede an Expression.
Added additional test cases.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markap14/nifi NIFI-4407
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2897.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2897
commit bad1d94291ebb7ec64a15094f7c7f938a3973b49
Author: Mark Payne
Date: 2018-07-16T17:27:46Z
NIFI-4407: Updated Expression Language Guide to provide details about how
escaping $ is accomplished, and when the $ character should and should not be
escaped. Fixed bug in the Query compiler that mistakenly would blindly replace
484 with $ even when the 484 did not precede an Expression. Added additional
test cases.
> Non-EL statement processed as expression language
> -
>
> Key: NIFI-4407
> URL: https://issues.apache.org/jira/browse/NIFI-4407
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.1.1, 1.0.1, 1.3.0
>Reporter: Pierre Villard
>Priority: Critical
>
> If you take a GFF with custom text: {{test$$foo}}
> The generated text will be: {{test$foo}}
> The property supports expression language and one $ is removed during the EL
> evaluation step. This can be an issue if a user wants to use a value
> containing to consecutive $$ (such as in password fields).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2897: NIFI-4407: Updated Expression Language Guide to pro...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/2897 NIFI-4407: Updated Expression Language Guide to provide details about⦠⦠how escaping $ is accomplished, and when the $ character should and should not be escaped. Fixed bug in the Query compiler that mistakenly would blindly replace 484 with $ even when the 484 did not precede an Expression. Added additional test cases. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-4407 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2897.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2897 commit bad1d94291ebb7ec64a15094f7c7f938a3973b49 Author: Mark Payne Date: 2018-07-16T17:27:46Z NIFI-4407: Updated Expression Language Guide to provide details about how escaping $ is accomplished, and when the $ character should and should not be escaped. Fixed bug in the Query compiler that mistakenly would blindly replace 484 with $ even when the 484 did not precede an Expression. Added additional test cases. ---
[jira] [Created] (NIFI-5432) Add Syslog Record Reader legacy Syslog
Otto Fowler created NIFI-5432: - Summary: Add Syslog Record Reader legacy Syslog Key: NIFI-5432 URL: https://issues.apache.org/jira/browse/NIFI-5432 Project: Apache NiFi Issue Type: Sub-task Reporter: Otto Fowler Assignee: Otto Fowler Create a record reader, based on nifi-syslog-utils for reading syslog without support for structured data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5430) NiFi CLI tool extension for cluster summary
[
https://issues.apache.org/jira/browse/NIFI-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545515#comment-16545515
]
ASF GitHub Bot commented on NIFI-5430:
--
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2894#discussion_r202762452
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ClusterSummaryEntity.java
---
@@ -24,7 +24,7 @@
* A serialized representation of this class can be placed in the entity
body of a request or response to or from the API. This particular entity holds
a reference to a ClusterSummaryDTO.
*/
@XmlRootElement(name = "clusterSummaryEntity")
-public class ClusteSummaryEntity extends Entity {
+public class ClusterSummaryEntity extends Entity {
--- End diff --
I wasn't sure about that, so I created in a separate commit, reverting.
> NiFi CLI tool extension for cluster summary
> ---
>
> Key: NIFI-5430
> URL: https://issues.apache.org/jira/browse/NIFI-5430
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
>Reporter: Peter Wilcsinszky
>Priority: Minor
>
> Cluster summary is required in use cases like in a Kubernetes deployment to
> get information about cluster status periodically. As soon as a node reports
> itself to be part of the cluster it can be considered ready.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2894: NIFI-5430 CLI tool extension for cluster summary
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2894#discussion_r202762452
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ClusterSummaryEntity.java
---
@@ -24,7 +24,7 @@
* A serialized representation of this class can be placed in the entity
body of a request or response to or from the API. This particular entity holds
a reference to a ClusterSummaryDTO.
*/
@XmlRootElement(name = "clusterSummaryEntity")
-public class ClusteSummaryEntity extends Entity {
+public class ClusterSummaryEntity extends Entity {
--- End diff --
I wasn't sure about that, so I created in a separate commit, reverting.
---
[GitHub] nifi issue #2894: CLI tool extension for cluster summary
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2894 Hi @pepov can you please update the title of this PR to reference the NiFi Jira it resolves? Thanks. ---
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545456#comment-16545456
]
ASF GitHub Bot commented on NIFI-5406:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202748684
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCacheC
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2876#discussion_r202748684
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/ListedEntityTracker.java
---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processor.util.list;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static java.lang.String.format;
+import static
org.apache.nifi.processor.util.list.AbstractListProcessor.REL_SUCCESS;
+
+public class ListedEntityTracker {
+
+private ObjectMapper objectMapper = new ObjectMapper();
+private volatile Map alreadyListedEntities;
+
+private static final String NOTE = "Used by 'Tracking Entities'
strategy.";
+public static final PropertyDescriptor TRACKING_STATE_CACHE = new
PropertyDescriptor.Builder()
+.name("et-state-cache")
+.displayName("Entity Tracking State Cache")
+.description(format("Listed entities are stored in the
specified cache storage" +
+" so that this processor can resume listing across
NiFi restart or in case of primary node change." +
+" 'Tracking Entities' strategy require tracking
information of all listed entities within the last 'Tracking Time Window'." +
+" To support large number of entities, the strategy
uses DistributedMapCache instead of managed state." +
+" Cache key format is
'ListedEntityTracker::{processorId}(::{nodeId})'." +
+" If it tracks per node listed entities, then the
optional '::{nodeId}' part is added to manage state separately." +
+" E.g. cluster wide cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b'," +
+" per node cache key =
'ListedEntityTracker::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3'" +
+" The stored cache content is Gzipped JSON string." +
+" The cache key will be deleted when target listing
configuration is changed." +
+" %s", NOTE))
+.identifiesControllerService(DistributedMapCacheClient.class)
+.build();
+
+public static final PropertyDescriptor TRACKING_TIME_WINDOW = new
PropertyDescriptor.Builder()
+.name("et-time-window")
+.displayName("Entity Tracking Time Window")
[jira] [Commented] (NIFI-5370) Cluster request replication failing with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545455#comment-16545455
]
Andy LoPresto commented on NIFI-5370:
-
[~prashanv] yes, there are multiple ways to avoid/correct the behavior you are
seeing here. The error appears because the certificate authority (CA) you are
using to sign one set of certs is not the same as the one that signs the next
set. The toolkit generates a truststore which contains that CA to allow for
cross-node verification. In order for all nodes (added at different times) to
verify each other, they must have all the CAs used in every truststore -- this
can be accomplished by signing all certs with the same CA, or adding multiple
CAs to the same truststore.
1. Run the TLS toolkit from the same directory and use the same output
directory. On the first run, the TLS toolkit will create a CA if one is not
present, and store the public key in {{nifi-cert.pem}} and the private key in
{{nifi-key.key}}. On subsequent runs, the toolkit will detect that a CA is
present and use it to sign the additional certs.
1. Run the TLS toolkit in an arbitrary location, and import the newly-generated
{{nifi-cert.pem}} into the already existing {{truststore.jks}} deployed on the
existing nodes. You will also need to import the original {{nifi-cert.pem}}
into the *new* {{truststore.jks}} for the new node.
1. Run the TLS toolkit in *server* mode on an instance, and have the other
nodes connect using *client* mode to receive their certificates. These certs
will all be signed by the same CA, and new nodes can come online and have certs
generated whenever you like. The *server* component runs indefinitely.
More details on all of this are available in the [Admin
Guide|https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit].
If you have further questions, I would recommend asking on the
*us...@nifi.apache.org* mailing list, as this issue is closed and community
members are unlikely to see future comments here.
> Cluster request replication failing with wildcard certs
> ---
>
> Key: NIFI-5370
> URL: https://issues.apache.org/jira/browse/NIFI-5370
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, cluster, security, tls, wildcard
> Fix For: 1.8.0, 1.7.1
>
>
> From the users mailing list:
> {quote}
> Team,
>
> NiFi secured cluster throws below error with wildcarded self-signed
> standalone certificate. Just a brief background, we are deploying nifi in
> Kubernetes where we have to use wildcarded certificates. Till nifi 1.6.0, it
> was working fine.
> Also I tried bringing up NiFi in linux VM in secured cluster mode with
> wildcarded certs, I am getting same error.
>
> Toolkit command to generate certs:
> bin/tls-toolkit.sh standalone -n
> '*.mynifi-nifi-headless.default.svc.cluster.local’ -C 'CN=admin, OU=NIFI' -o
>
>
> Logs:
> 2018-07-02 12:40:32,369 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local:8443 due to
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> 2018-07-02 12:40:32,370 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> at
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:316)
>
> Please help me in resolving this.
>
> Note: Same certificates is working for single mode setup.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (NIFI-5370) Cluster request replication failing with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545455#comment-16545455
]
Andy LoPresto edited comment on NIFI-5370 at 7/16/18 4:33 PM:
--
[~prashanv] yes, there are multiple ways to avoid/correct the behavior you are
seeing here. The error appears because the certificate authority (CA) you are
using to sign one set of certs is not the same as the one that signs the next
set. The toolkit generates a truststore which contains that CA to allow for
cross-node verification. In order for all nodes (added at different times) to
verify each other, they must have all the CAs used in every truststore -- this
can be accomplished by signing all certs with the same CA, or adding multiple
CAs to the same truststore.
# Run the TLS toolkit from the same directory and use the same output
directory. On the first run, the TLS toolkit will create a CA if one is not
present, and store the public key in {{nifi-cert.pem}} and the private key in
{{nifi-key.key}}. On subsequent runs, the toolkit will detect that a CA is
present and use it to sign the additional certs.
# Run the TLS toolkit in an arbitrary location, and import the newly-generated
{{nifi-cert.pem}} into the already existing {{truststore.jks}} deployed on the
existing nodes. You will also need to import the original {{nifi-cert.pem}}
into the *new* {{truststore.jks}} for the new node.
# Run the TLS toolkit in *server* mode on an instance, and have the other nodes
connect using *client* mode to receive their certificates. These certs will all
be signed by the same CA, and new nodes can come online and have certs
generated whenever you like. The *server* component runs indefinitely.
More details on all of this are available in the [Admin
Guide|https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit].
If you have further questions, I would recommend asking on the
*us...@nifi.apache.org* mailing list, as this issue is closed and community
members are unlikely to see future comments here.
was (Author: alopresto):
[~prashanv] yes, there are multiple ways to avoid/correct the behavior you are
seeing here. The error appears because the certificate authority (CA) you are
using to sign one set of certs is not the same as the one that signs the next
set. The toolkit generates a truststore which contains that CA to allow for
cross-node verification. In order for all nodes (added at different times) to
verify each other, they must have all the CAs used in every truststore -- this
can be accomplished by signing all certs with the same CA, or adding multiple
CAs to the same truststore.
1. Run the TLS toolkit from the same directory and use the same output
directory. On the first run, the TLS toolkit will create a CA if one is not
present, and store the public key in {{nifi-cert.pem}} and the private key in
{{nifi-key.key}}. On subsequent runs, the toolkit will detect that a CA is
present and use it to sign the additional certs.
1. Run the TLS toolkit in an arbitrary location, and import the newly-generated
{{nifi-cert.pem}} into the already existing {{truststore.jks}} deployed on the
existing nodes. You will also need to import the original {{nifi-cert.pem}}
into the *new* {{truststore.jks}} for the new node.
1. Run the TLS toolkit in *server* mode on an instance, and have the other
nodes connect using *client* mode to receive their certificates. These certs
will all be signed by the same CA, and new nodes can come online and have certs
generated whenever you like. The *server* component runs indefinitely.
More details on all of this are available in the [Admin
Guide|https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit].
If you have further questions, I would recommend asking on the
*us...@nifi.apache.org* mailing list, as this issue is closed and community
members are unlikely to see future comments here.
> Cluster request replication failing with wildcard certs
> ---
>
> Key: NIFI-5370
> URL: https://issues.apache.org/jira/browse/NIFI-5370
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, cluster, security, tls, wildcard
> Fix For: 1.8.0, 1.7.1
>
>
> From the users mailing list:
> {quote}
> Team,
>
> NiFi secured cluster throws below error with wildcarded self-signed
> standalone certificate. Just a brief background, we are deploying nifi in
> Kubernetes where we have to use wildcarded certificates. Till nifi 1.6.0, it
> was working fine.
> Also I tried bringing up NiFi
[GitHub] nifi pull request #2894: CLI tool extension for cluster summary
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2894#discussion_r202741535
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ClusterSummaryEntity.java
---
@@ -24,7 +24,7 @@
* A serialized representation of this class can be placed in the entity
body of a request or response to or from the API. This particular entity holds
a reference to a ClusterSummaryDTO.
*/
@XmlRootElement(name = "clusterSummaryEntity")
-public class ClusteSummaryEntity extends Entity {
+public class ClusterSummaryEntity extends Entity {
--- End diff --
I know this seems a bit nit-picky but I think we need to hold off renaming
this class since we try to keep the nifi-client-dto backwards compatible
between minor releases, and changing the name means any code written against
this class would need to be updated, so I think we can hold off on that part
and make that change during a major release like 2.0.0.
---
[jira] [Commented] (MINIFICPP-557) Segfault when including C2 AgentInformation in Alpine
[ https://issues.apache.org/jira/browse/MINIFICPP-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545414#comment-16545414 ] marco polo commented on MINIFICPP-557: -- Note that this is not alpine specific. > Segfault when including C2 AgentInformation in Alpine > - > > Key: MINIFICPP-557 > URL: https://issues.apache.org/jira/browse/MINIFICPP-557 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: marco polo >Priority: Major > Attachments: minifi-app.log > > > When running an instance with AgentInformation of one of the > nifi.c2.root.classes, the instance fails to startup with a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-557) Segfault when including C2 AgentInformation in Alpine
[ https://issues.apache.org/jira/browse/MINIFICPP-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545412#comment-16545412 ] marco polo commented on MINIFICPP-557: -- [~aldrin] Changes in MINIFICPP-515 caused this. The issue is not inherently AgentInformation. It's the mismanagement of string memory during copies and moves. I will fix appropriately. > Segfault when including C2 AgentInformation in Alpine > - > > Key: MINIFICPP-557 > URL: https://issues.apache.org/jira/browse/MINIFICPP-557 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Priority: Major > Attachments: minifi-app.log > > > When running an instance with AgentInformation of one of the > nifi.c2.root.classes, the instance fails to startup with a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (MINIFICPP-557) Segfault when including C2 AgentInformation in Alpine
[ https://issues.apache.org/jira/browse/MINIFICPP-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] marco polo reassigned MINIFICPP-557: Assignee: marco polo > Segfault when including C2 AgentInformation in Alpine > - > > Key: MINIFICPP-557 > URL: https://issues.apache.org/jira/browse/MINIFICPP-557 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Aldrin Piri >Assignee: marco polo >Priority: Major > Attachments: minifi-app.log > > > When running an instance with AgentInformation of one of the > nifi.c2.root.classes, the instance fails to startup with a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5337) Add Syslog 5424 Record Reader and create nifi-syslog-utils
[ https://issues.apache.org/jira/browse/NIFI-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende resolved NIFI-5337. --- Resolution: Fixed Fix Version/s: 1.8.0 > Add Syslog 5424 Record Reader and create nifi-syslog-utils > -- > > Key: NIFI-5337 > URL: https://issues.apache.org/jira/browse/NIFI-5337 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 1.8.0 > > > Create Record Reader for Syslog 5424 messages ( including structured data ) > built on the ParseSyslog5424 processor infrastructure. > > Create nifi-syslog-utils to centralize syslog support to be shared better the > standard processors and the serialization library > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5239) Make MongoDBControllerService able to act as a configuration source for MongoDB processors
[ https://issues.apache.org/jira/browse/NIFI-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545387#comment-16545387 ] ASF GitHub Bot commented on NIFI-5239: -- GitHub user MikeThomsen opened a pull request: https://github.com/apache/nifi/pull/2896 NIFI-5239 Made a client service an optional source of connection pool… …ing in Mongo processors. NIFI-5239 Updated two driver versions. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MikeThomsen/nifi NIFI-5239 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2896.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2896 commit c9085c8c552ba13acded3e0e79fb645e865df6bd Author: Mike Thomsen Date: 2018-07-15T12:07:21Z NIFI-5239 Made a client service an optional source of connection pooling in Mongo processors. NIFI-5239 Updated two driver versions. > Make MongoDBControllerService able to act as a configuration source for > MongoDB processors > -- > > Key: NIFI-5239 > URL: https://issues.apache.org/jira/browse/NIFI-5239 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > The MongoDBControllerService should be able to provide the getDatabase and > getCollection functionality that are built into the MongoDB processors > through AbstractMongoDBProcessor. Using the controller service with the > processors should be optional in the first release it's added and then > mandatory going forward. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5337) Add Syslog 5424 Record Reader and create nifi-syslog-utils
[ https://issues.apache.org/jira/browse/NIFI-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545388#comment-16545388 ] ASF subversion and git services commented on NIFI-5337: --- Commit b10220439cfddd876e01f1a2279b33552e47c7ac in nifi's branch refs/heads/master from [~ottobackwards] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b102204 ] NIFI-5337 Syslog 5424 Record Reader and nifi-syslog-utils - Create nifi-syslog-utils to move syslog parsing functionalty to a central location shared by the processors and serialization/record system. - Refactor Processors to use these utils - Update 5424 syslog classes using simple-syslog-5424 to pick up new changes to support this work, as well as keep dependencies/types from bleeding out to the processors or readers - Refactor Syslog5424Event and Parser - Create Syslog5424RecordReader - per review, handle blank message differently from eof - name schema per review This closes #2816. Signed-off-by: Bryan Bende > Add Syslog 5424 Record Reader and create nifi-syslog-utils > -- > > Key: NIFI-5337 > URL: https://issues.apache.org/jira/browse/NIFI-5337 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Create Record Reader for Syslog 5424 messages ( including structured data ) > built on the ParseSyslog5424 processor infrastructure. > > Create nifi-syslog-utils to centralize syslog support to be shared better the > standard processors and the serialization library > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5337) Add Syslog 5424 Record Reader and create nifi-syslog-utils
[ https://issues.apache.org/jira/browse/NIFI-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545389#comment-16545389 ] ASF GitHub Bot commented on NIFI-5337: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2816 > Add Syslog 5424 Record Reader and create nifi-syslog-utils > -- > > Key: NIFI-5337 > URL: https://issues.apache.org/jira/browse/NIFI-5337 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Create Record Reader for Syslog 5424 messages ( including structured data ) > built on the ParseSyslog5424 processor infrastructure. > > Create nifi-syslog-utils to centralize syslog support to be shared better the > standard processors and the serialization library > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2816: NIFI-5337 Syslog 5424 Record Reader and nifi-syslog...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2816 ---
[jira] [Commented] (NIFI-5329) Add ability to "empty queue" on an entire process group
[ https://issues.apache.org/jira/browse/NIFI-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545385#comment-16545385 ] ASF GitHub Bot commented on NIFI-5329: -- Github user MikeThomsen closed the pull request at: https://github.com/apache/nifi/pull/2893 > Add ability to "empty queue" on an entire process group > --- > > Key: NIFI-5329 > URL: https://issues.apache.org/jira/browse/NIFI-5329 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Mark Bean >Priority: Major > > It is desirable to clear all queues in a section of the flow. Specifically, a > Process Group can be taken as the unit to which 'empty queue' would be > applied. All connections within the Process Group are emptied. > If the user does not have permission to empty a given queue, it should be > excluded. However, all connections which the user has permission should > complete successfully. A dialog at the end of the operation should indicate > "X FlowFiles (Y bytes) were removed from Z queues". If appropriate, the > dialog should also include "X queues were not emptied due to insufficient > permission" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5337) Add Syslog 5424 Record Reader and create nifi-syslog-utils
[ https://issues.apache.org/jira/browse/NIFI-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545386#comment-16545386 ] ASF GitHub Bot commented on NIFI-5337: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2816 Everything looks good, going to merge shortly, thanks! > Add Syslog 5424 Record Reader and create nifi-syslog-utils > -- > > Key: NIFI-5337 > URL: https://issues.apache.org/jira/browse/NIFI-5337 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Create Record Reader for Syslog 5424 messages ( including structured data ) > built on the ParseSyslog5424 processor infrastructure. > > Create nifi-syslog-utils to centralize syslog support to be shared better the > standard processors and the serialization library > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2896: NIFI-5239 Made a client service an optional source ...
GitHub user MikeThomsen opened a pull request: https://github.com/apache/nifi/pull/2896 NIFI-5239 Made a client service an optional source of connection pool⦠â¦ing in Mongo processors. NIFI-5239 Updated two driver versions. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MikeThomsen/nifi NIFI-5239 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2896.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2896 commit c9085c8c552ba13acded3e0e79fb645e865df6bd Author: Mike Thomsen Date: 2018-07-15T12:07:21Z NIFI-5239 Made a client service an optional source of connection pooling in Mongo processors. NIFI-5239 Updated two driver versions. ---
[GitHub] nifi pull request #2893: NIFI-5329 Made a client service an optional source ...
Github user MikeThomsen closed the pull request at: https://github.com/apache/nifi/pull/2893 ---
[GitHub] nifi issue #2816: NIFI-5337 Syslog 5424 Record Reader and nifi-syslog-utils
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2816 Everything looks good, going to merge shortly, thanks! ---
[jira] [Commented] (NIFI-5329) Add ability to "empty queue" on an entire process group
[ https://issues.apache.org/jira/browse/NIFI-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545381#comment-16545381 ] ASF GitHub Bot commented on NIFI-5329: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2893 Thanks. Ill redo this then.. > Add ability to "empty queue" on an entire process group > --- > > Key: NIFI-5329 > URL: https://issues.apache.org/jira/browse/NIFI-5329 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Mark Bean >Priority: Major > > It is desirable to clear all queues in a section of the flow. Specifically, a > Process Group can be taken as the unit to which 'empty queue' would be > applied. All connections within the Process Group are emptied. > If the user does not have permission to empty a given queue, it should be > excluded. However, all connections which the user has permission should > complete successfully. A dialog at the end of the operation should indicate > "X FlowFiles (Y bytes) were removed from Z queues". If appropriate, the > dialog should also include "X queues were not emptied due to insufficient > permission" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2893: NIFI-5329 Made a client service an optional source of conn...
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2893 Thanks. Ill redo this then.. ---
