[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344307#comment-17344307 ] thang nguyen dinh commented on NIFI-8541: - thank [~joewitt], I will create new Jira to describe the ListFTP problem i got. > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! > > LINK lsof output and flow.xml.gz: > [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 commented on pull request #5059: NIFI-8519 Adding HDFS support for NAR autoload
markap14 commented on pull request #5059: URL: https://github.com/apache/nifi/pull/5059#issuecomment-840846222 One last thing that I think is necessary: please make sure that the admin guide is updated to describe the new properties. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-8519) HDFS support for Hot Loading
[
https://issues.apache.org/jira/browse/NIFI-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-8519:
-
Attachment: 0001-NIFI-8519-Support-RequiresInstanceClassLoading-annot.patch
> HDFS support for Hot Loading
>
>
> Key: NIFI-8519
> URL: https://issues.apache.org/jira/browse/NIFI-8519
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
> Attachments:
> 0001-NIFI-8519-Support-RequiresInstanceClassLoading-annot.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In NiFi, we have
> {code:java}
> nifi.nar.library.autoload.directory=./extensions
> {code}
> This directory is where NiFi would be looking for automatically loading NARs
> without the need for a restart of the service. It would be comfortable in
> multiple environments to support loading NARs from object stores, like HDFS.
> In this story I am willing to extend autoload with this capability and
> provide HDFS support for it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] markap14 commented on pull request #5059: NIFI-8519 Adding HDFS support for NAR autoload
markap14 commented on pull request #5059: URL: https://github.com/apache/nifi/pull/5059#issuecomment-840845848 https://issues.apache.org/jira/secure/attachment/13025442/0001-NIFI-8519-Support-RequiresInstanceClassLoading-annot.patch is the patch that I attached. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5059: NIFI-8519 Adding HDFS support for NAR autoload
markap14 commented on a change in pull request #5059:
URL: https://github.com/apache/nifi/pull/5059#discussion_r631964569
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-nar-loading-utils/src/main/java/org/apache/nifi/nar/PropertyBasedNarProviderInitializationContext.java
##
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.nar;
+
+import org.apache.nifi.util.NiFiProperties;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A facade at front of {@code NiFiProperties} for auto loader extensions.
Also limits the scope of the reachable properties.
+ */
+public class PropertyBasedNarProviderInitializationContext implements
NarProviderInitializationContext {
+private static Set GUARDED_PROPERTIES = new
HashSet<>(Arrays.asList("implementation"));
+static final String BASIC_PREFIX = "nifi.nar.library.provider.";
+
+private final Map parameters;
+private final String name;
+
+public PropertyBasedNarProviderInitializationContext(final NiFiProperties
properties, final String name) {
+this.parameters = extractParameters(properties, name);
+this.name = name;
+}
+
+@Override
+public Map getParameters() {
Review comment:
I think we should call this `getProperties()` rather than
`getParameters()` - within the NiFi parlance, both of these have very specific
meanings. These really are configuration elements for the provider, coming from
NiFiProperties, so I think `getProperties()` is more appropriate here.
##
File path:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/nar/hadoop/HDFSNarProvider.java
##
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.nar.hadoop;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.net.NetUtils;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.nifi.hadoop.SecurityUtil;
+import org.apache.nifi.nar.NarProvider;
+import org.apache.nifi.nar.NarProviderInitializationContext;
+import org.apache.nifi.nar.hadoop.util.ExtensionFilter;
+import org.apache.nifi.processors.hadoop.ExtendedConfiguration;
+import org.apache.nifi.processors.hadoop.HdfsResources;
+import org.apache.nifi.security.krb.KerberosKeytabUser;
+import org.apache.nifi.security.krb.KerberosPasswordUser;
+import org.apache.nifi.security.krb.KerberosUser;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.SocketFactory;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.net.URI;
+import java.security.PrivilegedExceptionAction;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+public class HDFSNarProvider implements NarProvider {
+private static final Logger LOGGER =
LoggerFactory.getLogger(HDFSNarProvider.class);
+
+private static final String RESOURCES_PARAMETER = "resources";
+private static final String
[GitHub] [nifi] mattyb149 commented on pull request #5076: NIFI-8542: When returning content via TriggerResult.readContent(FlowF…
mattyb149 commented on pull request #5076: URL: https://github.com/apache/nifi/pull/5076#issuecomment-840820441 Reviewing... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-8527) LivySessionController validator issue
[ https://issues.apache.org/jira/browse/NIFI-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344087#comment-17344087 ] Matt Burgess commented on NIFI-8527: Does NIFI-8206 fix this issue? > LivySessionController validator issue > - > > Key: NIFI-8527 > URL: https://issues.apache.org/jira/browse/NIFI-8527 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.13.1, 1.13.2 >Reporter: Celso Marques >Assignee: Matt Burgess >Priority: Major > Attachments: nifi-livy.png > > > In > [NIFI-5906|https://github.com/apache/nifi/commit/1955e7ac617c2ce962a388076b4d8b0f646fe449] > the FILE_EXISTS_VALIDATOR was removed but new issue was introduced as > attached image shows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8527) LivySessionController validator issue
[ https://issues.apache.org/jira/browse/NIFI-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344084#comment-17344084 ] Matt Burgess commented on NIFI-8527: Actually it appears to be replaced by .identifiesExternalResource(ResourceCardinality.MULTIPLE, ResourceType.FILE) > LivySessionController validator issue > - > > Key: NIFI-8527 > URL: https://issues.apache.org/jira/browse/NIFI-8527 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.13.1, 1.13.2 >Reporter: Celso Marques >Assignee: Matt Burgess >Priority: Major > Attachments: nifi-livy.png > > > In > [NIFI-5906|https://github.com/apache/nifi/commit/1955e7ac617c2ce962a388076b4d8b0f646fe449] > the FILE_EXISTS_VALIDATOR was removed but new issue was introduced as > attached image shows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8603) Use non-heap collections in StandardProcessSession
[ https://issues.apache.org/jira/browse/NIFI-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Secules updated NIFI-8603: --- Description: *Use Case:* A processor is splitting one flowfile into thousands or millions of flowfiles. It's preferred to keep all those flowfiles in the process session in case there is some error and we need to roll back. However, the process session stores the FlowFiles in Maps on the heap while the session is waiting to commit. One workaround is to commit the session early while processing splits and require the user to use wait/notify to hold the split portions until the whole input file is successfully split. This takes advantage of the fact that NiFi queues are swapped to disk after they reach a certain size. Another workaround is to use an off-heap collection in the processor code to store flowfiles to be transferred and only transfer them to the session and commit in a loop when you otherwise would have just committed. It would be great if the process session had a similar behaviour to NiFi queues between processors so that it is able to hold a large amount of flowfiles in the session without consuming the whole heap. *Acceptance Criteria:* * NiFi StandardProcessSession can handle arbitrarily large transactions by holding flowfiles off-heap or partially-off heap was: *Use Case:* A processor is splitting one flowfile into thousands or millions of flowfiles. It's preferred to keep all those flowfiles in the process session in case there is some error and we need to roll back. However, the process session stores the FlowFiles in Maps on the heap while the session is waiting to commit. One workaround is to commit the session early while processing splits and require the user to use wait/notify to hold the split portions until the whole input file is successfully split. This takes advantage of the fact that NiFi queues are swapped to disk after they reach a certain size. It would be great if the process session had a similar behaviour to NiFi queues between processors so that it is able to hold a large amount of flowfiles in the session without consuming the whole heap. *Acceptance Criteria:* * NiFi StandardProcessSession can handle arbitrarily large transactions by holding flowfiles off-heap or partially-off heap > Use non-heap collections in StandardProcessSession > -- > > Key: NIFI-8603 > URL: https://issues.apache.org/jira/browse/NIFI-8603 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: Eric Secules >Priority: Minor > > *Use Case:* > A processor is splitting one flowfile into thousands or millions of > flowfiles. It's preferred to keep all those flowfiles in the process session > in case there is some error and we need to roll back. However, the process > session stores the FlowFiles in Maps on the heap while the session is waiting > to commit. > One workaround is to commit the session early while processing splits and > require the user to use wait/notify to hold the split portions until the > whole input file is successfully split. This takes advantage of the fact that > NiFi queues are swapped to disk after they reach a certain size. > Another workaround is to use an off-heap collection in the processor code to > store flowfiles to be transferred and only transfer them to the session and > commit in a loop when you otherwise would have just committed. > It would be great if the process session had a similar behaviour to NiFi > queues between processors so that it is able to hold a large amount of > flowfiles in the session without consuming the whole heap. > > *Acceptance Criteria:* > * NiFi StandardProcessSession can handle arbitrarily large transactions by > holding flowfiles off-heap or partially-off heap -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] gresockj commented on a change in pull request #5068: NIFI-8516 Enabled HTTPS and Single User Authentication by default
gresockj commented on a change in pull request #5068: URL: https://github.com/apache/nifi/pull/5068#discussion_r632018895 ## File path: nifi-docs/src/main/asciidoc/administration-guide.adoc ## @@ -3527,7 +3527,7 @@ For example, to provide two additional network interfaces, a user could also spe + Providing three total network interfaces, including `nifi.web.http.network.interface.default`. |`nifi.web.https.host`|The HTTPS host. It is blank by default. Review comment: Should change the language for nifi.web.https.host since it's no longer blank by default. Reverse comment for nifi.web.http.host. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-8603) Use non-heap collections in StandardProcessSession
Eric Secules created NIFI-8603: -- Summary: Use non-heap collections in StandardProcessSession Key: NIFI-8603 URL: https://issues.apache.org/jira/browse/NIFI-8603 Project: Apache NiFi Issue Type: New Feature Components: Core Framework Affects Versions: 1.13.2 Reporter: Eric Secules *Use Case:* A processor is splitting one flowfile into thousands or millions of flowfiles. It's preferred to keep all those flowfiles in the process session in case there is some error and we need to roll back. However, the process session stores the FlowFiles in Maps on the heap while the session is waiting to commit. One workaround is to commit the session early while processing splits and require the user to use wait/notify to hold the split portions until the whole input file is successfully split. This takes advantage of the fact that NiFi queues are swapped to disk after they reach a certain size. It would be great if the process session had a similar behaviour to NiFi queues between processors so that it is able to hold a large amount of flowfiles in the session without consuming the whole heap. *Acceptance Criteria:* * NiFi StandardProcessSession can handle arbitrarily large transactions by holding flowfiles off-heap or partially-off heap -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] JonathanKessler commented on pull request #4780: NIFI-8126 Include Total Queued Duration in ConnectionStatus metrics
JonathanKessler commented on pull request #4780: URL: https://github.com/apache/nifi/pull/4780#issuecomment-840756785 @markap14 @markobean I just pushed a commit that I believe covers all outstanding items. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-8527) LivySessionController validator issue
[ https://issues.apache.org/jira/browse/NIFI-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344050#comment-17344050 ] Matt Burgess commented on NIFI-8527: When the File Exists Validator was removed, it should have been replaced with Validator.VALID > LivySessionController validator issue > - > > Key: NIFI-8527 > URL: https://issues.apache.org/jira/browse/NIFI-8527 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.13.1, 1.13.2 >Reporter: Celso Marques >Assignee: Matt Burgess >Priority: Major > Attachments: nifi-livy.png > > > In > [NIFI-5906|https://github.com/apache/nifi/commit/1955e7ac617c2ce962a388076b4d8b0f646fe449] > the FILE_EXISTS_VALIDATOR was removed but new issue was introduced as > attached image shows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-8527) LivySessionController validator issue
[ https://issues.apache.org/jira/browse/NIFI-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-8527: -- Assignee: Matt Burgess > LivySessionController validator issue > - > > Key: NIFI-8527 > URL: https://issues.apache.org/jira/browse/NIFI-8527 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.13.1, 1.13.2 >Reporter: Celso Marques >Assignee: Matt Burgess >Priority: Major > Attachments: nifi-livy.png > > > In > [NIFI-5906|https://github.com/apache/nifi/commit/1955e7ac617c2ce962a388076b4d8b0f646fe449] > the FILE_EXISTS_VALIDATOR was removed but new issue was introduced as > attached image shows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8536) PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port
[
https://issues.apache.org/jira/browse/NIFI-8536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-8536:
---
Component/s: (was: Variable Registry)
> PrometheusReportingTask and PrometheusRecordSink do not support variable
> registry for port
> --
>
> Key: NIFI-8536
> URL: https://issues.apache.org/jira/browse/NIFI-8536
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matthew Clarke
>Assignee: Nadeem
>Priority: Minor
> Fix For: 1.14.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The documentation for both PrometheusReportingTask [1] (reporting task) and
> PrometheusRecordSink [2] (controller service) state that NiFi Variable
> Registry can be used to define the port. However, configuring NiFi EL in the
> port property results in:
> java.lang.NumberFormatException: For input string: "${ property name>}"
> Support variable registry here would allow user to set a unique port on each
> host in a NiFi cluster.
> [1]
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html
> [2]
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusRecordSink/index.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-8536) PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port
[
https://issues.apache.org/jira/browse/NIFI-8536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344043#comment-17344043
]
ASF subversion and git services commented on NIFI-8536:
---
Commit 73a2283c279390a8d31d67eec8d0d190a2115131 in nifi's branch
refs/heads/main from Mohammed Nadeem
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=73a2283 ]
NIFI-8536: PrometheusReportingTask and PrometheusRecordSink do not support
variable registry for port property
Signed-off-by: Matthew Burgess
This closes #5075
> PrometheusReportingTask and PrometheusRecordSink do not support variable
> registry for port
> --
>
> Key: NIFI-8536
> URL: https://issues.apache.org/jira/browse/NIFI-8536
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions, Variable Registry
>Affects Versions: 1.12.1, 1.13.2
>Reporter: Matthew Clarke
>Assignee: Nadeem
>Priority: Minor
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The documentation for both PrometheusReportingTask [1] (reporting task) and
> PrometheusRecordSink [2] (controller service) state that NiFi Variable
> Registry can be used to define the port. However, configuring NiFi EL in the
> port property results in:
> java.lang.NumberFormatException: For input string: "${ property name>}"
> Support variable registry here would allow user to set a unique port on each
> host in a NiFi cluster.
> [1]
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html
> [2]
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusRecordSink/index.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8536) PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port
[
https://issues.apache.org/jira/browse/NIFI-8536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-8536:
---
Fix Version/s: 1.14.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> PrometheusReportingTask and PrometheusRecordSink do not support variable
> registry for port
> --
>
> Key: NIFI-8536
> URL: https://issues.apache.org/jira/browse/NIFI-8536
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions, Variable Registry
>Affects Versions: 1.12.1, 1.13.2
>Reporter: Matthew Clarke
>Assignee: Nadeem
>Priority: Minor
> Fix For: 1.14.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The documentation for both PrometheusReportingTask [1] (reporting task) and
> PrometheusRecordSink [2] (controller service) state that NiFi Variable
> Registry can be used to define the port. However, configuring NiFi EL in the
> port property results in:
> java.lang.NumberFormatException: For input string: "${ property name>}"
> Support variable registry here would allow user to set a unique port on each
> host in a NiFi cluster.
> [1]
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html
> [2]
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusRecordSink/index.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8536) PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port
[
https://issues.apache.org/jira/browse/NIFI-8536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-8536:
---
Affects Version/s: (was: 1.13.2)
(was: 1.12.1)
> PrometheusReportingTask and PrometheusRecordSink do not support variable
> registry for port
> --
>
> Key: NIFI-8536
> URL: https://issues.apache.org/jira/browse/NIFI-8536
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions, Variable Registry
>Reporter: Matthew Clarke
>Assignee: Nadeem
>Priority: Minor
> Fix For: 1.14.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The documentation for both PrometheusReportingTask [1] (reporting task) and
> PrometheusRecordSink [2] (controller service) state that NiFi Variable
> Registry can be used to define the port. However, configuring NiFi EL in the
> port property results in:
> java.lang.NumberFormatException: For input string: "${ property name>}"
> Support variable registry here would allow user to set a unique port on each
> host in a NiFi cluster.
> [1]
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html
> [2]
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusRecordSink/index.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] mattyb149 closed pull request #5075: NIFI-8536: PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port property
mattyb149 closed pull request #5075: URL: https://github.com/apache/nifi/pull/5075 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #5075: NIFI-8536: PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port property
mattyb149 commented on pull request #5075: URL: https://github.com/apache/nifi/pull/5075#issuecomment-840733896 +1 LGTM, thanks for the fix! Merging to main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] scottyaslan commented on pull request #5060: NIFI-8520 - Parameter Contexts - Show the wrong information of referencing compon…
scottyaslan commented on pull request #5060: URL: https://github.com/apache/nifi/pull/5060#issuecomment-840733345 reviewing... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] scottyaslan commented on a change in pull request #5069: NIFI-6685: Add align and distribute UI actions
scottyaslan commented on a change in pull request #5069:
URL: https://github.com/apache/nifi/pull/5069#discussion_r632001034
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-context-menu.js
##
@@ -844,9 +853,21 @@
{id: 'show-source-menu-item', condition: isConnection, menuItem:
{clazz: 'fa fa-long-arrow-left', text: 'Go to source', action: 'showSource'}},
{id: 'show-destination-menu-item', condition: isConnection, menuItem:
{clazz: 'fa fa-long-arrow-right', text: 'Go to destination', action:
'showDestination'}},
{separator: true},
-{id: 'align-menu-item', groupMenuItem: {clazz: 'fa', text: 'Align'},
menuItems: [
-{id: 'align-horizontal-menu-item', condition: canAlign, menuItem:
{ clazz: 'fa fa-align-center fa-rotate-90', text: 'Horizontally', action:
'alignHorizontal'}},
-{id: 'align-vertical-menu-item', condition: canAlign, menuItem:
{clazz: 'fa fa-align-center', text: 'Vertically', action: 'alignVertical'}}
+{id: 'align-menu-item', groupMenuItem: {clazz: 'carbon
carbon-align--horizontal-left', text: 'Align'}, menuItems: [
Review comment:
All other icons in the context menu have a color of #004849. Please
update these icons to also have this color applied.
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/pom.xml
##
@@ -246,6 +246,11 @@
moment/README.md
moment/LICENSE
+
+
@carbon/icons/svg/**/*
Review comment:
It would be great if we did not need to add yet another dependency here.
Maybe we can use one of the available icons from font awesome or the custom
flowfont created for this project?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344033#comment-17344033 ] Joe Witt commented on NIFI-8541: Ok.lets maybe slow down a bit and focus on the specific problem statement of what you need to solve rather than the symptoms you might be seeing. "ListFTP processor on centos 7 cluster. It work well before can not list file." That appears to be the root issue that led you to look at other aspects. I see no sign of a real leak yet. The lsof -p output is pretty authoritative and looks normal as far as I can tell. The flow/logs you show reflect nothing abnormal at all but that makes sense since it is an empty flow/default started node. Lets go back to ListSFTP.before centos7 it worked great. Now it doesn't. What does NiFi see when listing now? Nothing? What errors happen? If it is not working there should be errors or there should be nothing which means it is connecting and looking and getting no results. So...please share what you're seeing for that issue itself. > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! > > LINK lsof output and flow.xml.gz: > [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] naddym commented on pull request #5075: NIFI-8536: PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port property
naddym commented on pull request #5075: URL: https://github.com/apache/nifi/pull/5075#issuecomment-840713162 Thanks @mattyb149 for the review, I have made requested changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[
https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344019#comment-17344019
]
thang nguyen dinh commented on NIFI-8541:
-
we have a problem with ListFTP processor on centos 7 cluster. It work well
before can not list file.
I checked Redis cluster, FTP server, it is normal and has files, create new
ListFTP, change listing strategy but not work. So I must restart nifi cluster.
It does not happen on other cluster.
I turn debug in processor but did not find any problem, only message "Returning
CLUSTER State: StandardStateMap[version=-1, values={}]".
The difference I can see between 2 cluster is the number of opened files.
> NiFi open too many library files on centos 7
>
>
> Key: NIFI-8541
> URL: https://issues.apache.org/jira/browse/NIFI-8541
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.11.4, 1.13.2
> Environment: CentOS Linux release 7.7.1908 (Core)
>Reporter: thang nguyen dinh
>Assignee: Joe Witt
>Priority: Blocker
> Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt,
> nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png,
> stop-nifi.png
>
>
> when i run nifi-1.11.4 without any configuration anh data-flow on centos 7,
> it open too many duplicate librarys, about 800K files. centos 6 does not have
> this issue.
> !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400!
> when stop nifi
> !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400!
>
> i have same issue with nifi-1.13.2
> !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400!
> almost of opened files is nifi library
> example:
> !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400!
>
> LINK lsof output and flow.xml.gz:
> [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (MINIFICPP-1559) VS2019 builds do not include all required DLLs in the msi package
Ferenc Gerlits created MINIFICPP-1559: - Summary: VS2019 builds do not include all required DLLs in the msi package Key: MINIFICPP-1559 URL: https://issues.apache.org/jira/browse/MINIFICPP-1559 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Affects Versions: 0.9.0 Reporter: Ferenc Gerlits Assignee: Ferenc Gerlits Attachments: image-2021-05-13-19-10-11-551.png If the msi package is built with Visual Studio 2019 (by adding the /2019 switch to win_build_vs.bat), then the resulting binary has a dependency on vcruntime140_1.dll, but this DLL is not included in the package. This DLL needs to be copied from the VS Redist directory into the package, too. !image-2021-05-13-19-10-11-551.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344003#comment-17344003 ] Joe Witt commented on NIFI-8541: I looked at the logs. They look perfect. If you do 'lsof -p ' the results suggest totally normal file counts. What is the problem you really need solved? Is something erroring? If so what? > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! > > LINK lsof output and flow.xml.gz: > [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 commented on pull request #5059: NIFI-8519 Adding HDFS support for NAR autoload
markap14 commented on pull request #5059: URL: https://github.com/apache/nifi/pull/5059#issuecomment-840691381 There is no write up. Just a general observation: it doesn't really make sense IMO to allow for the extensions in nifi to live in either local filesystem or HDFS. What does make sense is local filesystem or some other pluggable location. HDFS will be the only additional location at the point this PR is merged in but it makes sense that it should lay the groundwork for it to be pluggable is all that I was trying to convey. Thanks @simonbence will take a look. And I think that's fair enough to push that to NIFI-8602. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343996#comment-17343996 ] thang nguyen dinh commented on NIFI-8541: - I've add conf and log folder of nifi-1.13.2 at share-link: [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] I tried setup on some centos 7 servers and have same issue. I think you can easily reproduce it. Thank you. > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! > > LINK lsof output and flow.xml.gz: > [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] simonbence commented on pull request #5059: NIFI-8519 Adding HDFS support for NAR autoload
simonbence commented on pull request #5059: URL: https://github.com/apache/nifi/pull/5059#issuecomment-840682900 Hi @markap14 ! I incorporated your ideas, it really ended up better. The only exception is your last bullet point, which I extracted into a followup story https://issues.apache.org/jira/browse/NIFI-8602 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] patricker opened a new pull request #3511: NIFI-6175 Spark Livy - Improving Livy
patricker opened a new pull request #3511:
URL: https://github.com/apache/nifi/pull/3511
Description of PR
The Livy Session Controller is missing many of the options available, and
many of them I feel are critical for this service to be useful (queue? conf?
num of executors?)
Add in functionality to shutdown open sessions when service is disabled.
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343991#comment-17343991 ] Joe Witt commented on NIFI-8541: No. Definitely not normal. Can you please look at the nifi-bootstrap and nifi-app logs? Please share them. Right now I dont see anything actually wrong/obvious here. Just output which doesn't make sense. > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! > > LINK lsof output and flow.xml.gz: > [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8602) Managing older NARs within NAR autoloader
Simon Bence created NIFI-8602: - Summary: Managing older NARs within NAR autoloader Key: NIFI-8602 URL: https://issues.apache.org/jira/browse/NIFI-8602 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Simon Bence Assignee: Simon Bence Based on [~markap14]'s improvement idea on the original ticket: It's possible that over time users will continually add new versions of a NAR to the same directory. And over time this will result in loading massive amounts of NAR's into NiFi. Those that get removed will remain in NiFi. We will need some way to age off old NARs. Perhaps after performing a listing, we should detect any nar's that were previously downloaded from that NarProvider and not present in the Listing. Then, if any of those is not in use (i.e., no extensions are in the flow that use that bundle) then we can remove the bundle and its NarLoader. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343987#comment-17343987 ] thang nguyen dinh commented on NIFI-8541: - we're have 2 NiFi cluster * one has 5 server on centos 6 , lsof | wc -l show normal with about 12k opened file * one has 3 server on centos 7 and there are more than 800K opened file related to nifi, i don't understand I setup new standalone nifi which I show lsof to you, I think this issue is not relate to flow.xml.gz And if " It seems like the parent process id and the child process id are repeating in patterns as if tons and tons and tons of instances are being launched", is it normal? Thank you. > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! > > LINK lsof output and flow.xml.gz: > [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343983#comment-17343983 ] Joe Witt commented on NIFI-8541: the output of the broad lsof if you pick a single thing shows a repeating pattern. Note the first few columns. It seems like the parent process id and the child process id are repeating in patterns as if tons and tons and tons of instances are being launched. java 32691 admin mem REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 admin 170r REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 admin 2861r REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32695 admin mem REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32695 admin 170r REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32695 admin 2861r REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32696 admin mem REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32696 admin 170r REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32696 admin 2861r REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32697 admin mem REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32697 admin 170r REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32697 admin 2861r REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32698 admin mem REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32698 admin 170r REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar java 32691 32698 admin 2861r REG 253,0277351 2613849 /home/admin/nifi-1.11.4/work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/netty-buffer-4.1.42.Final.jar > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop n
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343979#comment-17343979 ] Joe Witt commented on NIFI-8541: and the flow definition shows there is nothing there...so you're just starting nifi and seeing how many open files there are? > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! > > LINK lsof output and flow.xml.gz: > [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343977#comment-17343977 ] Joe Witt commented on NIFI-8541: >From the 1.13.2 lsof output tied to the pid there is no indication of anything >abnormal. 6000 or so open files most of which are our libs. This is >fine/normal. Why do you believe there is a leak? > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! > > LINK lsof output and flow.xml.gz: > [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request #1073: MINIFICPP-1541 Move ODBC dependent SQL testing to the docker integration tests
lordgamez opened a new pull request #1073: URL: https://github.com/apache/nifi-minifi-cpp/pull/1073 - Refactor SQL processors to remove explicit `soci` library dependancy - Create MockDB for unit tests - Add option to use real SQLite database in unit tests - Add integration tests to test SQL processors with PostgreSQL database Jira ticket: https://issues.apache.org/jira/browse/MINIFICPP-1541 --- Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-8543) Incorporate MiNiFi Java functionality as a specialized assembly of NiFI
[ https://issues.apache.org/jira/browse/NIFI-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-8543: --- Component/s: MiNiFi > Incorporate MiNiFi Java functionality as a specialized assembly of NiFI > > > Key: NIFI-8543 > URL: https://issues.apache.org/jira/browse/NIFI-8543 > Project: Apache NiFi > Issue Type: Task > Components: MiNiFi >Reporter: Aldrin Piri >Assignee: Matt Burgess >Priority: Major > Fix For: 1.14.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > At its core the Java implementation of MiNiFi has largely been a core body of > NiFi core libraries in a separate assembly with some additional extension > points, namely those of configuration (via YAML) and configuration change > listeners. > Due to working with some of the internals of NiFi that are not exactly meant > for external consumption, there has been a certain impedance with each > successive release to make use of the latest and greatest. > This ticket is to investigate and consider the incorporation of MiNiFi Java > into the NiFi code base in a manner as highlighted above, extending/adapting > the core libraries, providing some additional extension points, and then > generating a custom assembly. > The idea is that in lieu of duplicating bits of code and providing > workarounds around some of the internal APIs we can have a more streamlined > build and keep these items in lockstep with the core NiFi libraries being > more aware of changes that MiNiFi is inherently dependent upon. > To our users, there should be little perceptible change. The core means of > interaction should remain while providing a similar footprint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8543) Incorporate MiNiFi Java functionality as a specialized assembly of NiFI
[ https://issues.apache.org/jira/browse/NIFI-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343956#comment-17343956 ] Matt Burgess commented on NIFI-8543: This used to be MINIFI-422 but has been moved to the NIFI Jira project (as will the rest of the MINIFI Jiras shortly) with a Component of MiNiFi. > Incorporate MiNiFi Java functionality as a specialized assembly of NiFI > > > Key: NIFI-8543 > URL: https://issues.apache.org/jira/browse/NIFI-8543 > Project: Apache NiFi > Issue Type: Task >Reporter: Aldrin Piri >Assignee: Matt Burgess >Priority: Major > Fix For: 1.14.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > At its core the Java implementation of MiNiFi has largely been a core body of > NiFi core libraries in a separate assembly with some additional extension > points, namely those of configuration (via YAML) and configuration change > listeners. > Due to working with some of the internals of NiFi that are not exactly meant > for external consumption, there has been a certain impedance with each > successive release to make use of the latest and greatest. > This ticket is to investigate and consider the incorporation of MiNiFi Java > into the NiFi code base in a manner as highlighted above, extending/adapting > the core libraries, providing some additional extension points, and then > generating a custom assembly. > The idea is that in lieu of duplicating bits of code and providing > workarounds around some of the internal APIs we can have a more streamlined > build and keep these items in lockstep with the core NiFi libraries being > more aware of changes that MiNiFi is inherently dependent upon. > To our users, there should be little perceptible change. The core means of > interaction should remain while providing a similar footprint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-5723) PutDatabaseRecord strips underbar character from column names
[
https://issues.apache.org/jira/browse/NIFI-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5723:
---
Component/s: MiNiFi
> PutDatabaseRecord strips underbar character from column names
> -
>
> Key: NIFI-5723
> URL: https://issues.apache.org/jira/browse/NIFI-5723
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI, MiNiFi
>Affects Versions: 1.7.1
> Environment: Red Hat Enterprise Linux Server release 7.5 (Maipo).
> Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 16 cores. 32 Gb RAM. OpenJDK
> Runtime Environment (build 1.8.0_181-b13)
>Reporter: Uwe Geercken
>Priority: Major
>
> In *PutDatabaseRecord.java* there is following code:
> private static String normalizeColumnName(final String colName, final boolean
> translateColumnNames) {
> return colName == null ? null : (translateColumnNames ?
> colName.toUpperCase().replace("_", "") : colName);
> }
> If translateColumnNames is set to true, then the underbar character is
> stripped from the column names.
> And it is also here:
> final ColumnDescription desc =
> tableSchema.getColumns().get(normalizeColumnName(fieldName,
> settings.translateFieldNames));
> if (desc == null && !settings.ignoreUnmappedFields) {
> throw new SQLDataException("Cannot map field '" +
> fieldName + "' to any column in the database");
> }
> I do not see a reason why this is done. An underbar is a common delimiter in
> database column names.
> Specifically, I have data comming from an Oracle database which is all
> uppercase and with the underbar character as the delimiter in the column
> names and due to the current code, I can not update my MySQL database using
> the same column names.
> Trying to solve the issue in other ways does not really work as well.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-5723) PutDatabaseRecord strips underbar character from column names
[
https://issues.apache.org/jira/browse/NIFI-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5723:
---
Component/s: (was: MiNiFi)
> PutDatabaseRecord strips underbar character from column names
> -
>
> Key: NIFI-5723
> URL: https://issues.apache.org/jira/browse/NIFI-5723
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.7.1
> Environment: Red Hat Enterprise Linux Server release 7.5 (Maipo).
> Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 16 cores. 32 Gb RAM. OpenJDK
> Runtime Environment (build 1.8.0_181-b13)
>Reporter: Uwe Geercken
>Priority: Major
>
> In *PutDatabaseRecord.java* there is following code:
> private static String normalizeColumnName(final String colName, final boolean
> translateColumnNames) {
> return colName == null ? null : (translateColumnNames ?
> colName.toUpperCase().replace("_", "") : colName);
> }
> If translateColumnNames is set to true, then the underbar character is
> stripped from the column names.
> And it is also here:
> final ColumnDescription desc =
> tableSchema.getColumns().get(normalizeColumnName(fieldName,
> settings.translateFieldNames));
> if (desc == null && !settings.ignoreUnmappedFields) {
> throw new SQLDataException("Cannot map field '" +
> fieldName + "' to any column in the database");
> }
> I do not see a reason why this is done. An underbar is a common delimiter in
> database column names.
> Specifically, I have data comming from an Oracle database which is all
> uppercase and with the underbar character as the delimiter in the column
> names and due to the current code, I can not update my MySQL database using
> the same column names.
> Trying to solve the issue in other ways does not really work as well.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8537) Components become invalid if referencing a disabled controller service, even if referencing property is not in scope due to dependencies
[ https://issues.apache.org/jira/browse/NIFI-8537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-8537: --- Fix Version/s: 1.14.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Components become invalid if referencing a disabled controller service, even > if referencing property is not in scope due to dependencies > > > Key: NIFI-8537 > URL: https://issues.apache.org/jira/browse/NIFI-8537 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.14.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > If a processor/controller service property (A) has a dependency on another > property (B), we do not perform validation of property A (i.e., the property > descriptor's validator is ignored). However, if Property A references a > Controller Service, and that Controller Service is either disabled or > invalid, we currently consider the referencing component invalid. This should > not be the case. > To replicate: > * Create an AvroSchemaRegistry Controller Service. Leave the Controller > Service disabled. > * Create a JsonTreeReader controller service. > * Configure the JsonTreeReader Controller Service. > * Set "Schema Access Strategy" to "Use 'Schema Name' Property" > * Set "Schema Registry" to the AvroSchemaRegistry created in the first step. > * Change "Schema Access Strategy" to "Infer Schema" > * Click Apply to apply the changes. > The JsonTreeReader service will be invalid due to the fact that > AvroSchemaRegistry is disabled. Not only should the Schema Registry property > not be considered in validation, it becomes very confusing now because the > validation error states: "'Schema Registry' validated against XYZ is invalid > because Controller Service with ID XYZ is invalid." But when configuring > JsonTreeReader, we don't even see the "Schema Registry" property. > The reference should not be validated unless its property's dependencies have > been satisfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8537) Components become invalid if referencing a disabled controller service, even if referencing property is not in scope due to dependencies
[ https://issues.apache.org/jira/browse/NIFI-8537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343953#comment-17343953 ] ASF subversion and git services commented on NIFI-8537: --- Commit 97ed0efbc23356297ecd2568d1d3b3305fc2a9e7 in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=97ed0ef ] NIFI-8537: Check that a property's dependencies are satisfied before checking if the referenced controller service valid/enabled Signed-off-by: Matthew Burgess This closes #5071 > Components become invalid if referencing a disabled controller service, even > if referencing property is not in scope due to dependencies > > > Key: NIFI-8537 > URL: https://issues.apache.org/jira/browse/NIFI-8537 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.14.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > If a processor/controller service property (A) has a dependency on another > property (B), we do not perform validation of property A (i.e., the property > descriptor's validator is ignored). However, if Property A references a > Controller Service, and that Controller Service is either disabled or > invalid, we currently consider the referencing component invalid. This should > not be the case. > To replicate: > * Create an AvroSchemaRegistry Controller Service. Leave the Controller > Service disabled. > * Create a JsonTreeReader controller service. > * Configure the JsonTreeReader Controller Service. > * Set "Schema Access Strategy" to "Use 'Schema Name' Property" > * Set "Schema Registry" to the AvroSchemaRegistry created in the first step. > * Change "Schema Access Strategy" to "Infer Schema" > * Click Apply to apply the changes. > The JsonTreeReader service will be invalid due to the fact that > AvroSchemaRegistry is disabled. Not only should the Schema Registry property > not be considered in validation, it becomes very confusing now because the > validation error states: "'Schema Registry' validated against XYZ is invalid > because Controller Service with ID XYZ is invalid." But when configuring > JsonTreeReader, we don't even see the "Schema Registry" property. > The reference should not be validated unless its property's dependencies have > been satisfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mattyb149 closed pull request #5071: NIFI-8537: Check that a property's dependencies are satisfied before …
mattyb149 closed pull request #5071: URL: https://github.com/apache/nifi/pull/5071 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #5071: NIFI-8537: Check that a property's dependencies are satisfied before …
mattyb149 commented on pull request #5071: URL: https://github.com/apache/nifi/pull/5071#issuecomment-840645120 +1 LGTM, verified the behavior is as expected. Thanks for the fix! Merging to main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #5075: NIFI-8536: PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port property
mattyb149 commented on pull request #5075:
URL: https://github.com/apache/nifi/pull/5075#issuecomment-840642467
This looks good to me but would like to see a unit test, you could
add/change the following in TestPrometheusRecordSink.initTask() for example:
```
final MockVariableRegistry variableRegistry = new MockVariableRegistry();
variableRegistry.setVariable(new VariableDescriptor("port"), "7077");
when(context.getProperty(PrometheusMetricsUtil.METRICS_ENDPOINT_PORT)).thenReturn(new
MockPropertyValue("${port}", null, variableRegistry));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] markap14 opened a new pull request #5076: NIFI-8542: When returning content via TriggerResult.readContent(FlowF…
markap14 opened a new pull request #5076:
URL: https://github.com/apache/nifi/pull/5076
…ile), ensure that we take into account the content claim offset and length
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-8542) The FlowFile content returned by Stateless TriggerResult.readContent may contain too much data
[
https://issues.apache.org/jira/browse/NIFI-8542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-8542:
-
Status: Patch Available (was: Open)
> The FlowFile content returned by Stateless TriggerResult.readContent may
> contain too much data
> --
>
> Key: NIFI-8542
> URL: https://issues.apache.org/jira/browse/NIFI-8542
> Project: Apache NiFi
> Issue Type: Bug
> Components: NiFi Stateless
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.14.0
>
>
> When using Stateless NiFi, the TriggerResult has a readContent(FlowFile)
> method that should return the contents of the FlowFile. However, if a
> FlowFile was created using {{ProcessSession.clone(FlowFile flowFile, long
> offset, long size)}} then the contents of the entire content claim are still
> returned, not just the contents between offset & offset + size. This means
> that any stateless flow where the output is from SplitText or SegmentContent
> will return too much content.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thang nguyen dinh updated NIFI-8541: Description: when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, it open too many duplicate librarys, about 800K files. centos 6 does not have this issue. !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! when stop nifi !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! i have same issue with nifi-1.13.2 !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! almost of opened files is nifi library example: !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! LINK lsof output and flow.xml.gz: [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] was: when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, it open too many duplicate librarys, about 800K files. centos 6 does not have this issue. !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! when stop nifi !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! i have same issue with nifi-1.13.2 !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! almost of opened files is nifi library example: !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! > > LINK lsof output and flow.xml.gz: > [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343939#comment-17343939 ] thang nguyen dinh commented on NIFI-8541: - i run nifi right after extract without any configuration link lsof and flow.xml.gz [https://drive.google.com/drive/folders/1V2Y_DHfiivJfGlYsgm68AelL3By3EBcB?usp=sharing] > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8461) Queue reports items but cannot list them
[ https://issues.apache.org/jira/browse/NIFI-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343938#comment-17343938 ] Matt Burgess commented on NIFI-8461: That's what I believe has been fixed by NIFI-8330, but it didn't seem to work for Francois on 1.13.2 but it works for me on main (to become 1.14.0). Maybe there's another change in main that (together) address the issue. > Queue reports items but cannot list them > > > Key: NIFI-8461 > URL: https://issues.apache.org/jira/browse/NIFI-8461 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 > Environment: Docker image apache/nifi :1.13.2, Windows 10 Pro, WSL 2 > (Ubuntu), x86_64 >Reporter: Kevin Aagaard >Assignee: Matt Burgess >Priority: Major > Labels: queue > Attachments: ExecuteScript_Queue_Issue.txt, NIFI-8461.zip, > image-2021-04-22-08-48-58-117.png, image-2021-04-22-08-49-40-904.png, > image-2021-04-27-14-23-07-849.png, image-2021-04-27-14-23-24-477.png > > > The following segment of workflow demonstrates the issue. The queue reports > that there are items within it, but they cannot be listed, even after > stopping the consumer (and producer). Since this is in a Docker Container, it > is unlikely an OS error. > > !image-2021-04-22-08-48-58-117.png! > > !image-2021-04-22-08-49-40-904.png! > I do not currently have a simplified example workflow to recreate the error, > but can work on it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8542) The FlowFile content returned by Stateless TriggerResult.readContent may contain too much data
Mark Payne created NIFI-8542:
Summary: The FlowFile content returned by Stateless
TriggerResult.readContent may contain too much data
Key: NIFI-8542
URL: https://issues.apache.org/jira/browse/NIFI-8542
Project: Apache NiFi
Issue Type: Bug
Components: NiFi Stateless
Reporter: Mark Payne
Assignee: Mark Payne
Fix For: 1.14.0
When using Stateless NiFi, the TriggerResult has a readContent(FlowFile) method
that should return the contents of the FlowFile. However, if a FlowFile was
created using {{ProcessSession.clone(FlowFile flowFile, long offset, long
size)}} then the contents of the entire content claim are still returned, not
just the contents between offset & offset + size. This means that any stateless
flow where the output is from SplitText or SegmentContent will return too much
content.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8535) PutDatabaseRecord should give a better message when the table cannot be found
[ https://issues.apache.org/jira/browse/NIFI-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-8535: --- Status: Patch Available (was: In Progress) > PutDatabaseRecord should give a better message when the table cannot be found > - > > Key: NIFI-8535 > URL: https://issues.apache.org/jira/browse/NIFI-8535 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently PutDatabaseRecord calls DatabaseMetaData.getColumns() to try and > match the columns from the specified table to the fields in the incoming > record(s). However if the table itself is not found, this method returns an > empty ResultSet, so it is not known whether the table does not exist or if it > exists with no columns. > PutDatabaseRecord should call DatabaseMetaData.getTables() if the column list > is empty, and give a more descriptive error message if the table is not > found. This can help the user determine whether there is a field/column > mismatch or a catalog/schema/table name mismatch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343917#comment-17343917 ] Joe Witt commented on NIFI-8541: And we need to see what the flow.xml.gz has and/or need a detailed written description of it. Also we need the output of 'lsof -p ' exclusively. If you want to also provide it with the greop that is fine but please share them uniquely. > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343916#comment-17343916 ] Joe Witt commented on NIFI-8541: Thanks - same output for 1.13.2 please. > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343913#comment-17343913 ] thang nguyen dinh edited comment on NIFI-8541 at 5/13/21, 2:56 PM: --- [~joewitt] "lsof -p nifi_pid" does not list all file nifi open. so I added the out put of cmd "lsof | grep 32691", 32691 is nifi pid. Thank you. sample 2000 line: [^lsof-20210513-32691-2000-sample.txt] full log: [https://drive.google.com/file/d/1t0mkVNTz45ODjZqY8bz14I7ihO-MiFnE/view?usp=sharing] was (Author: alistar): [~joewitt] "lsof -p nifi_pid" does not list all file nifi open. so I added the out put of cmd "lsof | grep 32691", 32691 is nifi pid. Thank you. full log: [https://drive.google.com/file/d/1t0mkVNTz45ODjZqY8bz14I7ihO-MiFnE/view?usp=sharing] > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thang nguyen dinh updated NIFI-8541: Attachment: lsof-20210513-32691-2000-sample.txt > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, lsof-20210513-32691-2000-sample.txt, > nifi-1.13.png, nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, > stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343913#comment-17343913 ] thang nguyen dinh commented on NIFI-8541: - [~joewitt] "lsof -p nifi_pid" does not list all file nifi open. so I added the out put of cmd "lsof | grep 32691", 32691 is nifi pid. Thank you. full log: [https://drive.google.com/file/d/1t0mkVNTz45ODjZqY8bz14I7ihO-MiFnE/view?usp=sharing] > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (MINIFICPP-1558) Remove s3 bucket objects test transiently fails
Gabor Gyimesi created MINIFICPP-1558: Summary: Remove s3 bucket objects test transiently fails Key: MINIFICPP-1558 URL: https://issues.apache.org/jira/browse/MINIFICPP-1558 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Reporter: Gabor Gyimesi Attachments: remove_object_from_s3_bucket_failure.log -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8524) PutDatabaseRecord fails when inserting or updating a CHAR column
[
https://issues.apache.org/jira/browse/NIFI-8524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343905#comment-17343905
]
Roger Sliva commented on NIFI-8524:
---
Some additional information on this - when I converted the target table (SQL
Server) to use NCHAR and NVARCHAR instead of CHAR and VARCHAR, the
PutDatabaseRecord succeeded!
The PutDatabaseRecord uses a default JsonTreeReader set to infer the schema.
The schema was defined at the time the JSON was built as:
{
"type": "record",
"name": "gprmc_pcpti_schema",
"fields" : [
\{"name": "uuid", "type": ["null", "string"]},
\{"name": "current_ts", "type": ["null", "string"]},
\{"name": "GPRMC_Sentence_ID", "type": ["null", "string"]},
\{"name": "GPRMC_Fix_Time", "type": ["null", "string"]},
\{"name": "GPRMC_Data_Status", "type": ["null", "string"]},
\{"name": "GPRMC_Latitude", "type": ["null", "string"]},
\{"name": "GPRMC_Latitude_Dir", "type": ["null", "string"]},
\{"name": "GPRMC_Longitude", "type": ["null", "string"]},
\{"name": "GPRMC_Longitude_Dir", "type": ["null", "string"]},
\{"name": "GPRMC_Ground_Speed", "type": ["null", "string"]},
\{"name": "GPRMC_Track_Made_Good", "type": ["null", "string"]},
\{"name": "GPRMC_UT_Date", "type": ["null", "string"]},
\{"name": "GPRMC_Mag_Var", "type": ["null", "string"]},
\{"name": "GPRMC_Mag_Var_Dir", "type": ["null", "string"]},
\{"name": "GPRMC_Checksum", "type": ["null", "string"]},
\{"name": "PCPTI_Sentence_ID", "type": ["null", "string"]},
\{"name": "PCPTI_Vehicle_ID", "type": ["null", "string"]},
\{"name": "PCPTI_Value1", "type": ["null", "string"]},
\{"name": "PCPTI_Checksum", "type": ["null", "string"]}
]
}
Here is an obfuscated record sample.
[\{"uuid":"5aadfbd4-2fd0-4b67-8fdf-9d40e3a015a8","current_ts":"2021-05-11
13:22:14.016","GPRMC_Sentence_ID":"GPRMC","GPRMC_Fix_Time":"202212.000","GPRMC_Data_Status":"A","GPRMC_Latitude":"xx24.51859","GPRMC_Latitude_Dir":"N","GPRMC_Longitude":"xx151.44189","GPRMC_Longitude_Dir":"W","GPRMC_Ground_Speed":"0.0","GPRMC_Track_Made_Good":"0.0","GPRMC_UT_Date":"110521","GPRMC_Mag_Var":"","GPRMC_Mag_Var_Dir":"","GPRMC_Checksum":"D*7D","PCPTI_Sentence_ID":"PCPTI","PCPTI_Vehicle_ID":"Vxxx","PCPTI_Value1":"202212","PCPTI_Checksum":"202212*2A"}]
Here is the original table schema (before changing CHAR to NCHAR, etc.)
CREATE TABLE Fire.GPS_All_Records(
uuid varchar(36) NULL,
current_ts datetime NULL,
GPGGA_Sentence_ID varchar(10) NULL,
GPGGA_Fix_Time varchar(20) NULL,
GPGGA_Latitude varchar(20) NULL,
GPGGA_Latitude_Dir char(1) NULL,
GPGGA_Longitude varchar(20) NULL,
GPGGA_Longitude_Dir char(1) NULL,
GPGGA_Fix_Quality varchar(20) NULL,
GPGGA_Number_of_Satellites varchar(20) NULL,
GPGGA_Horizontal_Dilution varchar(20) NULL,
GPGGA_MSL_Altitude varchar(20) NULL,
GPGGA_Altitude_Unit char(1) NULL,
GPGGA_MSL_Height varchar(20) NULL,
GPGGA_Height_Unit char(1) NULL,
GPGGA_DGPS_Station_ID varchar(20) NULL,
GPGGA_Checksum varchar(10) NULL,
GPVTG_Sentence_ID varchar(10) NULL,
GPVTG_Track_Made_Good varchar(20) NULL,
GPVTG_Relative_To_North varchar(10) NULL,
GPVTG_Track_Made_Good_Mag varchar(20) NULL,
GPVTG_Relative_To_Mag_North varchar(10) NULL,
GPVTG_Speed varchar(10) NULL,
GPVTG_Speed_Is_Knots char(1) NULL,
GPVTG_Ground_Speed varchar(10) NULL,
GPVTG_Ground_Speed_Is_kph char(1) NULL,
GPVTG_Checksum varchar(20) NULL,
GPRMC_Sentence_ID varchar(10) NULL,
GPRMC_Fix_Time varchar(20) NULL,
GPRMC_Data_Status varchar(10) NULL,
GPRMC_Latitude varchar(20) NULL,
GPRMC_Latitude_Dir char(1) NULL,
GPRMC_Longitude varchar(20) NULL,
GPRMC_Longitude_Dir char(1) NULL,
GPRMC_Ground_Speed varchar(20) NULL,
GPRMC_Track_Made_Good varchar(20) NULL,
GPRMC_UT_Date varchar(20) NULL,
GPRMC_Mag_Var varchar(20) NULL,
GPRMC_Mag_Var_Dir varchar(10) NULL,
GPRMC_Checksum varchar(20) NULL,
PCPTI_Sentence_ID varchar(10) NULL,
PCPTI_Vehicle_ID varchar(20) NOT NULL,
PCPTI_Value1 varchar(20) NULL,
PCPTI_Checksum varchar(20) NULL
) ON PRIMARY
> PutDatabaseRecord fails when inserting or updating a CHAR column
>
>
> Key: NIFI-8524
> URL: https://issues.apache.org/jira/browse/NIFI-8524
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.13.2
> Environment: RHEL 7.4 using OpenJDK 11 with mssql-jdbc-9.2.1.jre11.jar
>Reporter: Roger Sliva
>Priority: Major
>
> PutDatabaseRecord fails when inserting or updating into a SQL Server 12+
> table with a CHAR(1) column.
> com.microsoft.sqlserver.jdbc.SQLServerException: The conversion from UNKNOWN
> to CHAR is unsupported.
> This works in 1.11.4
> When the CHAR columns are converted to VARCHAR the fail does not occur.
>
> From nifi-app.log:
> 2021-05-05 09:43:13,938 ERROR [Timer-Driven Process Thread-10]
> o.a.n.p.standard.PutDatabaseRecord
>
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343901#comment-17343901 ] Joe Witt commented on NIFI-8541: Ok then we need the full output of the lsof -p output for the nifi pid. Please send that as text. Remove/replace whatever sensitive dirnames you need but we need the text. > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt reopened NIFI-8541: > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343898#comment-17343898 ] thang nguyen dinh commented on NIFI-8541: - Im using jdk1.8.0_281 list of java versions i have tried (same issue): openjdk-11.0.11.0.9-1 openjdk-1.8.0.292 jdk1.8.0_281 jdk1.8.0_261 jdk1.8.0_51 !nifi-ps-jdk-281.png! > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343897#comment-17343897 ] Joe Witt commented on NIFI-8541: we think this issue was in JDK 8 281 and 271 from the oracle distro > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thang nguyen dinh updated NIFI-8541: Attachment: nifi-ps-jdk-281.png > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, nifi-ps-jdk-281.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (MINIFICPP-1244) Support the Initial Start Position property in TailFile
[ https://issues.apache.org/jira/browse/MINIFICPP-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Gyimesi resolved MINIFICPP-1244. -- Resolution: Fixed > Support the Initial Start Position property in TailFile > --- > > Key: MINIFICPP-1244 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1244 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Ferenc Gerlits >Assignee: Gabor Gyimesi >Priority: Minor > Labels: selected-for-development > Fix For: 0.10.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > The TailFile processor in NiFi Java has a property called *Initial Start > Position*, with allowed values: > * *Beginning of Time*: Start with the oldest data that matches the Rolling > Filename Pattern and then begin reading from the File to Tail > * *Beginning of File*: Start with the beginning of the File to Tail. Do not > ingest any data that has already been rolled over > * *Current Time*: Start with the data at the end of the File to Tail. Do not > ingest any data that has already been rolled over or any data in the File to > Tail that has already been written. > MiNiFi C++ does not have this property, and always behaves as if Beginning of > File was selected. Add the *Initial Start Position* property to MiNiFi C++, > with the three allowed values and behaviors as above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] scottyaslan commented on pull request #5069: NIFI-6685: Add align and distribute UI actions
scottyaslan commented on pull request #5069: URL: https://github.com/apache/nifi/pull/5069#issuecomment-840576777 reviewing... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt resolved NIFI-8541. Assignee: Joe Witt Resolution: Information Provided This suggests that the system using NiFi where the file count continues to grow is using Java 8 update 271 I believe it is. Update the JDK version and ensure NiFi is using that and you should be fine again. If that does not solve it then we need the complete output of running lsof so we can get a better sense of what is in there. That said, the very likely problem is the buggy JDK version. It had to do with certain types of sockets leaking. It occurred in a very specific JDK and went away right after that. > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Assignee: Joe Witt >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (MINIFICPP-1556) Setting partial credentials in AWS credentials service may set other properties
[ https://issues.apache.org/jira/browse/MINIFICPP-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Gyimesi reassigned MINIFICPP-1556: Assignee: Gabor Gyimesi > Setting partial credentials in AWS credentials service may set other > properties > --- > > Key: MINIFICPP-1556 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1556 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Gabor Gyimesi >Assignee: Gabor Gyimesi >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > Setting only one of authentication key or secret key of AWS credentials can > lead to setting AWS credential file property to the previous auth key/secret > key value if the credentials file property is empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (MINIFICPP-1556) Setting partial credentials in AWS credentials service may set other properties
[ https://issues.apache.org/jira/browse/MINIFICPP-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Gyimesi resolved MINIFICPP-1556. -- Resolution: Fixed > Setting partial credentials in AWS credentials service may set other > properties > --- > > Key: MINIFICPP-1556 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1556 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Gabor Gyimesi >Assignee: Gabor Gyimesi >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > Setting only one of authentication key or secret key of AWS credentials can > lead to setting AWS credential file property to the previous auth key/secret > key value if the credentials file property is empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343866#comment-17343866 ] Pierre Villard commented on NIFI-8541: -- Version of Java? > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (MINIFICPP-1553) AWSCredentialsService should be able to refresh its credentials after expiry
[ https://issues.apache.org/jira/browse/MINIFICPP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Gyimesi resolved MINIFICPP-1553. -- Resolution: Fixed > AWSCredentialsService should be able to refresh its credentials after expiry > - > > Key: MINIFICPP-1553 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1553 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Gabor Gyimesi >Assignee: Gabor Gyimesi >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > The AWSCredentialService in MiNiFi only invokes the AWS credential libraries > within its onEnabled method and caches the credentials for later use. It > doesn't have a way to refresh the credential after it's been persisted in > memory, this leads to the credentials to expire after a while. Optimally > we'd want to be able to ensure the credential is refreshed (which can be > something that's property driven). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 commented on pull request #5071: NIFI-8537: Check that a property's dependencies are satisfied before …
markap14 commented on pull request #5071: URL: https://github.com/apache/nifi/pull/5071#issuecomment-84023 Thanks @exceptionfactory . I realized as soon as I pushed it that it was going to break and then I updated locally and forgot to push. Addressed now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thang nguyen dinh updated NIFI-8541: Description: when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, it open too many duplicate librarys, about 800K files. centos 6 does not have this issue. !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! when stop nifi !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! i have same issue with nifi-1.13.2 !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! almost of opened files is nifi library example: !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! was: when i run nifi-1.11.4 without any configuration anh data-flow in centos 7, it open too many duplicate librarys, about 800K files. centos 6 does not have this issue. !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! when stop nifi !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! i have same issue with nifi-1.13.2 !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! almost of opened files is nifi library example: !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow on centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8541) NiFi open too many library files on centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thang nguyen dinh updated NIFI-8541: Summary: NiFi open too many library files on centos 7 (was: NiFi open too many library files in centos 7) > NiFi open too many library files on centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow in centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] ottobackwards commented on pull request #5059: NIFI-8519 Adding HDFS support for NAR autoload
ottobackwards commented on pull request #5059: URL: https://github.com/apache/nifi/pull/5059#issuecomment-840550448 Absolutely, I had different goals. Is there something written up on "We want this to allow pluggability for any number of locations. We would also want to account for the fact that some systems may be offline, .."? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on pull request #1044: MINIFICPP-1526 Add ConsumeJournald to consume systemd journal messages
lordgamez commented on pull request #1044: URL: https://github.com/apache/nifi-minifi-cpp/pull/1044#issuecomment-840528703 When I try testing the processor manually on my local machine it fails with the following error: `[2021-05-13 14:00:02.434] [org::apache::nifi::minifi::core::ProcessGroup] [error] Failed to start processor d9606075-a35a-4509-b705-13e793e1b82b (ConsumeJournald): sd_journal_get_cursor: Cannot assign requested address` I could only find a similar [issue](https://github.com/timberio/vector/issues/1081) but I did not find a solution for it, so it still needs to be investigated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1066: MINIFICPP-1532: Create a processor to capture resource consumption me…
martinzink commented on a change in pull request #1066:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1066#discussion_r631751390
##
File path: extensions/pdh/tests/PerformanceDataCounterTests.cpp
##
@@ -0,0 +1,187 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "TestBase.h"
+#include "PDHCounters.h"
+#include "MemoryConsumptionCounter.h"
+#include "gsl/gsl-lite.hpp"
+
+using org::apache::nifi::minifi::processors::SinglePDHCounter;
+using org::apache::nifi::minifi::processors::PDHCounterArray;
+using org::apache::nifi::minifi::processors::PDHCounter;
+using org::apache::nifi::minifi::processors::MemoryConsumptionCounter;
+
+
+TEST_CASE("PDHCounterNameTests", "[pdhcounternametests]") {
+ std::unique_ptr test_counter =
PDHCounter::createPDHCounter("\\System\\Threads");
+ REQUIRE(nullptr != dynamic_cast (test_counter.get()));
+ REQUIRE("\\System\\Threads" == test_counter->getName());
+ REQUIRE("System" == test_counter->getObjectName());
+ REQUIRE("Threads" == test_counter->getCounterName());
+}
+
+TEST_CASE("PDHCounterArrayNameTests", "[pdhcounterarraytests]") {
+ std::unique_ptr test_counter_array =
PDHCounter::createPDHCounter("\\LogicalDisk(*)\\% Free Space");
+ REQUIRE(nullptr != dynamic_cast
(test_counter_array.get()));
+ REQUIRE("\\LogicalDisk(*)\\% Free Space" == test_counter_array->getName());
+ REQUIRE("LogicalDisk" == test_counter_array->getObjectName());
+ REQUIRE("% Free Space" == test_counter_array->getCounterName());
+}
+
+TEST_CASE("PDHCountersInvalidNameTests", "[pdhcountersinvalidnametests]") {
+ REQUIRE(nullptr == PDHCounter::createPDHCounter("Invalid Name"));
+ REQUIRE(nullptr == PDHCounter::createPDHCounter(""));
+ REQUIRE(nullptr == PDHCounter::createPDHCounter("Something\\Counter"));
+ REQUIRE(nullptr == PDHCounter::createPDHCounter("\\Too\\Many\\Separators"));
+ REQUIRE(nullptr == PDHCounter::createPDHCounter("Too\\Many\\Separators"));
+ REQUIRE(nullptr != PDHCounter::createPDHCounter("\\Valid\\Counter"));
+}
+
+class TestablePDHCounter : public SinglePDHCounter {
+ public:
+ explicit TestablePDHCounter(const std::string& query_name, bool is_double =
true) : SinglePDHCounter(query_name, is_double) {
+ }
+};
+
+class TestablePDHCounterArray : public PDHCounterArray {
+ public:
+ explicit TestablePDHCounterArray(const std::string& query_name, bool
is_double = true) : PDHCounterArray(query_name, is_double) {
+ }
+};
+
+TEST_CASE("PDHCountersAddingToQueryTests", "[pdhcountersaddingtoquerytests]") {
+ PDH_HQUERY pdh_query;
+ PdhOpenQueryA(nullptr, 0, _query);
+ gsl::finally([_query] { PdhCloseQuery(_query); });
Review comment:
yeah, you were right (it complies because pdh_query is ptr but its
location is also a ptr) fixed it in
[d536ded](https://github.com/apache/nifi-minifi-cpp/pull/1066/commits/d536deda68b57c0c18ee34ff32b6a843d819b8ef)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni closed pull request #1070: MINIFICPP-1556 Fix setting partial or invalid credentials in AWS credentials service
adamdebreceni closed pull request #1070: URL: https://github.com/apache/nifi-minifi-cpp/pull/1070 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1066: MINIFICPP-1532: Create a processor to capture resource consumption me…
fgerlits commented on a change in pull request #1066:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1066#discussion_r631731069
##
File path: extensions/pdh/tests/PerformanceDataCounterTests.cpp
##
@@ -0,0 +1,187 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "TestBase.h"
+#include "PDHCounters.h"
+#include "MemoryConsumptionCounter.h"
+#include "gsl/gsl-lite.hpp"
+
+using org::apache::nifi::minifi::processors::SinglePDHCounter;
+using org::apache::nifi::minifi::processors::PDHCounterArray;
+using org::apache::nifi::minifi::processors::PDHCounter;
+using org::apache::nifi::minifi::processors::MemoryConsumptionCounter;
+
+
+TEST_CASE("PDHCounterNameTests", "[pdhcounternametests]") {
+ std::unique_ptr test_counter =
PDHCounter::createPDHCounter("\\System\\Threads");
+ REQUIRE(nullptr != dynamic_cast (test_counter.get()));
+ REQUIRE("\\System\\Threads" == test_counter->getName());
+ REQUIRE("System" == test_counter->getObjectName());
+ REQUIRE("Threads" == test_counter->getCounterName());
+}
+
+TEST_CASE("PDHCounterArrayNameTests", "[pdhcounterarraytests]") {
+ std::unique_ptr test_counter_array =
PDHCounter::createPDHCounter("\\LogicalDisk(*)\\% Free Space");
+ REQUIRE(nullptr != dynamic_cast
(test_counter_array.get()));
+ REQUIRE("\\LogicalDisk(*)\\% Free Space" == test_counter_array->getName());
+ REQUIRE("LogicalDisk" == test_counter_array->getObjectName());
+ REQUIRE("% Free Space" == test_counter_array->getCounterName());
+}
+
+TEST_CASE("PDHCountersInvalidNameTests", "[pdhcountersinvalidnametests]") {
+ REQUIRE(nullptr == PDHCounter::createPDHCounter("Invalid Name"));
+ REQUIRE(nullptr == PDHCounter::createPDHCounter(""));
+ REQUIRE(nullptr == PDHCounter::createPDHCounter("Something\\Counter"));
+ REQUIRE(nullptr == PDHCounter::createPDHCounter("\\Too\\Many\\Separators"));
+ REQUIRE(nullptr == PDHCounter::createPDHCounter("Too\\Many\\Separators"));
+ REQUIRE(nullptr != PDHCounter::createPDHCounter("\\Valid\\Counter"));
+}
+
+class TestablePDHCounter : public SinglePDHCounter {
+ public:
+ explicit TestablePDHCounter(const std::string& query_name, bool is_double =
true) : SinglePDHCounter(query_name, is_double) {
+ }
+};
+
+class TestablePDHCounterArray : public PDHCounterArray {
+ public:
+ explicit TestablePDHCounterArray(const std::string& query_name, bool
is_double = true) : PDHCounterArray(query_name, is_double) {
+ }
+};
+
+TEST_CASE("PDHCountersAddingToQueryTests", "[pdhcountersaddingtoquerytests]") {
+ PDH_HQUERY pdh_query;
+ PdhOpenQueryA(nullptr, 0, _query);
+ gsl::finally([_query] { PdhCloseQuery(_query); });
Review comment:
I think this should be `PdhCloseQuery(pdh_query)` -- not clear why it
compiles with the `&`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1066: MINIFICPP-1532: Create a processor to capture resource consumption me…
fgerlits commented on a change in pull request #1066:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1066#discussion_r631728967
##
File path: extensions/pdh/PerformanceDataMonitor.cpp
##
@@ -0,0 +1,283 @@
+/**
+ * @file GenerateFlowFile.cpp
+ * GenerateFlowFile class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "PerformanceDataMonitor.h"
+#include "PDHCounters.h"
+#include "MemoryConsumptionCounter.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Relationship PerformanceDataMonitor::Success("success", "All files are
routed to success");
+
+core::Property PerformanceDataMonitor::PredefinedGroups(
+core::PropertyBuilder::createProperty("Predefined Groups")->
+withDescription("Comma separated list to which predefined groups should be
included")->
+withDefaultValue("")->build());
+
+core::Property PerformanceDataMonitor::CustomPDHCounters(
+core::PropertyBuilder::createProperty("Custom PDH Counters")->
+withDescription("Comma separated list of PDHCounters to collect from")->
+withDefaultValue("")->build());
+
+core::Property PerformanceDataMonitor::OutputFormatProperty(
+core::PropertyBuilder::createProperty("Output format")->
+withDescription("Format of the created flowfiles")->
+withAllowableValue(JSON_FORMAT_STR)->
+withAllowableValue(OPEN_TELEMETRY_FORMAT_STR)->
+withDefaultValue(JSON_FORMAT_STR)->build());
+
+PerformanceDataMonitor::~PerformanceDataMonitor() {
+ if (pdh_query_ != nullptr)
+PdhCloseQuery(pdh_query_);
+ for (PerformanceDataCounter* resource_consumption_counter :
resource_consumption_counters_) {
+delete resource_consumption_counter;
+ }
+}
+
+void PerformanceDataMonitor::onSchedule(const
std::shared_ptr& context, const
std::shared_ptr& sessionFactory) {
+ setupMembersFromProperties(context);
+
+ std::vector valid_counters;
+
+ for (PerformanceDataCounter* counter : resource_consumption_counters_) {
+PDHCounterBase* pdh_counter = dynamic_cast (counter);
+if (pdh_counter != nullptr) {
+ if (pdh_query_ == nullptr)
+PdhOpenQuery(NULL, NULL, _query_);
+ PDH_STATUS add_to_query_result = pdh_counter->addToQuery(pdh_query_);
+ if (add_to_query_result != ERROR_SUCCESS) {
+logger_->log_error(("Error adding " + pdh_counter->getName() + " to
query: " + std::to_string(add_to_query_result)).c_str());
+delete counter;
+ } else {
+valid_counters.push_back(counter);
+ }
+} else {
+ valid_counters.push_back(counter);
+}
+ }
+ resource_consumption_counters_ = valid_counters;
+ PdhCollectQueryData(pdh_query_);
Review comment:
yes, I think that would be useful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-8541) NiFi open too many library files in centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343804#comment-17343804 ] Duong Pham commented on NIFI-8541: -- I got the same > NiFi open too many library files in centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow in centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1066: MINIFICPP-1532: Create a processor to capture resource consumption me…
martinzink commented on a change in pull request #1066:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1066#discussion_r631726747
##
File path: extensions/pdh/PerformanceDataMonitor.cpp
##
@@ -0,0 +1,283 @@
+/**
+ * @file GenerateFlowFile.cpp
+ * GenerateFlowFile class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "PerformanceDataMonitor.h"
+#include "PDHCounters.h"
+#include "MemoryConsumptionCounter.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Relationship PerformanceDataMonitor::Success("success", "All files are
routed to success");
+
+core::Property PerformanceDataMonitor::PredefinedGroups(
+core::PropertyBuilder::createProperty("Predefined Groups")->
+withDescription("Comma separated list to which predefined groups should be
included")->
+withDefaultValue("")->build());
+
+core::Property PerformanceDataMonitor::CustomPDHCounters(
+core::PropertyBuilder::createProperty("Custom PDH Counters")->
+withDescription("Comma separated list of PDHCounters to collect from")->
+withDefaultValue("")->build());
+
+core::Property PerformanceDataMonitor::OutputFormatProperty(
+core::PropertyBuilder::createProperty("Output format")->
+withDescription("Format of the created flowfiles")->
+withAllowableValue(JSON_FORMAT_STR)->
+withAllowableValue(OPEN_TELEMETRY_FORMAT_STR)->
+withDefaultValue(JSON_FORMAT_STR)->build());
+
+PerformanceDataMonitor::~PerformanceDataMonitor() {
+ if (pdh_query_ != nullptr)
+PdhCloseQuery(pdh_query_);
+ for (PerformanceDataCounter* resource_consumption_counter :
resource_consumption_counters_) {
+delete resource_consumption_counter;
+ }
+}
+
+void PerformanceDataMonitor::onSchedule(const
std::shared_ptr& context, const
std::shared_ptr& sessionFactory) {
+ setupMembersFromProperties(context);
+
+ std::vector valid_counters;
+
+ for (PerformanceDataCounter* counter : resource_consumption_counters_) {
+PDHCounterBase* pdh_counter = dynamic_cast (counter);
+if (pdh_counter != nullptr) {
+ if (pdh_query_ == nullptr)
+PdhOpenQuery(NULL, NULL, _query_);
+ PDH_STATUS add_to_query_result = pdh_counter->addToQuery(pdh_query_);
+ if (add_to_query_result != ERROR_SUCCESS) {
+logger_->log_error(("Error adding " + pdh_counter->getName() + " to
query: " + std::to_string(add_to_query_result)).c_str());
+delete counter;
+ } else {
+valid_counters.push_back(counter);
+ }
+} else {
+ valid_counters.push_back(counter);
+}
+ }
+ resource_consumption_counters_ = valid_counters;
+ PdhCollectQueryData(pdh_query_);
Review comment:
now the query gets created everytime in onSchedule, so in theory it
shouldnt be nullptr
but if for somereason it is we could check the return of the
PdhCollectQueryData (here in onschedule and in ontrigger aswell) to log out the
errors and yield?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-8541) NiFi open too many library files in centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thang nguyen dinh updated NIFI-8541: Attachment: stop-nifi.png dupliate-lib.png nifi-1.13.png nifi-open-too-many-libs-centos7.png > NiFi open too many library files in centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Priority: Blocker > Attachments: dupliate-lib.png, nifi-1.13.png, > nifi-open-too-many-libs-centos7.png, stop-nifi.png > > > when i run nifi-1.11.4 without any configuration anh data-flow in centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1066: MINIFICPP-1532: Create a processor to capture resource consumption me…
fgerlits commented on a change in pull request #1066:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1066#discussion_r631722874
##
File path: extensions/pdh/PerformanceDataMonitor.cpp
##
@@ -0,0 +1,283 @@
+/**
+ * @file GenerateFlowFile.cpp
+ * GenerateFlowFile class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "PerformanceDataMonitor.h"
+#include "PDHCounters.h"
+#include "MemoryConsumptionCounter.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Relationship PerformanceDataMonitor::Success("success", "All files are
routed to success");
+
+core::Property PerformanceDataMonitor::PredefinedGroups(
+core::PropertyBuilder::createProperty("Predefined Groups")->
+withDescription("Comma separated list to which predefined groups should be
included")->
+withDefaultValue("")->build());
+
+core::Property PerformanceDataMonitor::CustomPDHCounters(
+core::PropertyBuilder::createProperty("Custom PDH Counters")->
+withDescription("Comma separated list of PDHCounters to collect from")->
+withDefaultValue("")->build());
+
+core::Property PerformanceDataMonitor::OutputFormatProperty(
+core::PropertyBuilder::createProperty("Output format")->
+withDescription("Format of the created flowfiles")->
+withAllowableValue(JSON_FORMAT_STR)->
+withAllowableValue(OPEN_TELEMETRY_FORMAT_STR)->
+withDefaultValue(JSON_FORMAT_STR)->build());
+
+PerformanceDataMonitor::~PerformanceDataMonitor() {
+ if (pdh_query_ != nullptr)
+PdhCloseQuery(pdh_query_);
+ for (PerformanceDataCounter* resource_consumption_counter :
resource_consumption_counters_) {
+delete resource_consumption_counter;
+ }
+}
+
+void PerformanceDataMonitor::onSchedule(const
std::shared_ptr& context, const
std::shared_ptr& sessionFactory) {
+ setupMembersFromProperties(context);
+
+ std::vector valid_counters;
+
+ for (PerformanceDataCounter* counter : resource_consumption_counters_) {
+PDHCounterBase* pdh_counter = dynamic_cast (counter);
+if (pdh_counter != nullptr) {
+ if (pdh_query_ == nullptr)
+PdhOpenQuery(NULL, NULL, _query_);
+ PDH_STATUS add_to_query_result = pdh_counter->addToQuery(pdh_query_);
+ if (add_to_query_result != ERROR_SUCCESS) {
+logger_->log_error(("Error adding " + pdh_counter->getName() + " to
query: " + std::to_string(add_to_query_result)).c_str());
+delete counter;
+ } else {
+valid_counters.push_back(counter);
+ }
+} else {
+ valid_counters.push_back(counter);
+}
+ }
+ resource_consumption_counters_ = valid_counters;
+ PdhCollectQueryData(pdh_query_);
Review comment:
I can't see the check; is `PdhCollectQueryData()` going to throw with a
reasonable error message when called with a null argument?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-8541) NiFi open too many library files in centos 7
[ https://issues.apache.org/jira/browse/NIFI-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thang nguyen dinh updated NIFI-8541: Priority: Blocker (was: Critical) > NiFi open too many library files in centos 7 > > > Key: NIFI-8541 > URL: https://issues.apache.org/jira/browse/NIFI-8541 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Priority: Blocker > > when i run nifi-1.11.4 without any configuration anh data-flow in centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-8540) NiFi open too many library files in centos 7
[ https://issues.apache.org/jira/browse/NIFI-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thang nguyen dinh resolved NIFI-8540. - Resolution: Duplicate > NiFi open too many library files in centos 7 > > > Key: NIFI-8540 > URL: https://issues.apache.org/jira/browse/NIFI-8540 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4, 1.13.2 > Environment: CentOS Linux release 7.7.1908 (Core) >Reporter: thang nguyen dinh >Priority: Critical > > when i run nifi-1.11.4 without any configuration anh data-flow in centos 7, > it open too many duplicate librarys, about 800K files. centos 6 does not have > this issue. > !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! > when stop nifi > !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! > > i have same issue with nifi-1.13.2 > !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! > almost of opened files is nifi library > example: > !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8541) NiFi open too many library files in centos 7
thang nguyen dinh created NIFI-8541: --- Summary: NiFi open too many library files in centos 7 Key: NIFI-8541 URL: https://issues.apache.org/jira/browse/NIFI-8541 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.13.2, 1.11.4 Environment: CentOS Linux release 7.7.1908 (Core) Reporter: thang nguyen dinh when i run nifi-1.11.4 without any configuration anh data-flow in centos 7, it open too many duplicate librarys, about 800K files. centos 6 does not have this issue. !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! when stop nifi !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! i have same issue with nifi-1.13.2 !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! almost of opened files is nifi library example: !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8540) NiFi open too many library files in centos 7
thang nguyen dinh created NIFI-8540: --- Summary: NiFi open too many library files in centos 7 Key: NIFI-8540 URL: https://issues.apache.org/jira/browse/NIFI-8540 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.13.2, 1.11.4 Environment: CentOS Linux release 7.7.1908 (Core) Reporter: thang nguyen dinh when i run nifi-1.11.4 without any configuration anh data-flow in centos 7, it open too many duplicate librarys, about 800K files. centos 6 does not have this issue. !https://community.cloudera.com/t5/image/serverpage/image-id/31161iAD2A4726623B08EE/image-size/medium?v=1.0=400! when stop nifi !https://community.cloudera.com/t5/image/serverpage/image-id/31165i8D6E7C394AE26C6F/image-size/medium?v=1.0=400! i have same issue with nifi-1.13.2 !https://community.cloudera.com/t5/image/serverpage/image-id/31163i2901E83C543D3691/image-size/medium?v=1.0=400! almost of opened files is nifi library example: !https://community.cloudera.com/t5/image/serverpage/image-id/31164i401AE378E2B259E9/image-size/medium?v=1.0=400! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8539) ConsumeKafkaRecord_2_0 1.13.0: Flowfile does not match data from Kafka
[ https://issues.apache.org/jira/browse/NIFI-8539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Nikolaev updated NIFI-8539: --- Summary: ConsumeKafkaRecord_2_0 1.13.0: Flowfile does not match data from Kafka (was: ConsumeKafkaRecord_2_0 1.13.0: ) > ConsumeKafkaRecord_2_0 1.13.0: Flowfile does not match data from Kafka > > > Key: NIFI-8539 > URL: https://issues.apache.org/jira/browse/NIFI-8539 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.0 >Reporter: Nikolay Nikolaev >Priority: Major > Attachments: 38f1d0f9-33c9-4cf1-978a-9060274615a5.avro, data in > Kafka.json, see in Avro.PNG, see in Kafka.PNG, test_flow.png > > > Steps to reproduce the bug: > * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 > 1.13.0" and write output file with PutFile > !test_flow.png! > * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] > contains "raw"-data from kafka topic, and > [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. > We can see, that section "after" is different in this files, even though it > should be the same record > In Kafka: > !see in Kafka.PNG! > In Flowfile: > !see in Avro.PNG! > The error does not always appear, but only in some records. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8539) ConsumeKafkaRecord_2_0 1.13.0:
[ https://issues.apache.org/jira/browse/NIFI-8539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Nikolaev updated NIFI-8539: --- Description: Steps to reproduce the bug: * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 1.13.0" and write output file with PutFile !test_flow.png! * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] contains "raw"-data from kafka topic, and [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. We can see, that section "after" is different in this files, even though it should be the same record In Kafka: !see in Kafka.PNG! In Flowfile: !see in Avro.PNG! The error does not always appear, but only in some records. was: Steps to reproduce the bug: * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 1.13.0" and write output file with PutFile ( !test_flow.png|thumbnail! ) * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] contains "raw"-data from kafka topic, and [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. We can see, that section "after" is different in this files, even though it should be the same record In Kafka - !see in Kafka.PNG|thumbnail! In Flowfile - !see in Avro.PNG|thumbnail! The error does not always appear, but only in some records. > ConsumeKafkaRecord_2_0 1.13.0: > --- > > Key: NIFI-8539 > URL: https://issues.apache.org/jira/browse/NIFI-8539 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.0 >Reporter: Nikolay Nikolaev >Priority: Major > Attachments: 38f1d0f9-33c9-4cf1-978a-9060274615a5.avro, data in > Kafka.json, see in Avro.PNG, see in Kafka.PNG, test_flow.png > > > Steps to reproduce the bug: > * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 > 1.13.0" and write output file with PutFile > !test_flow.png! > * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] > contains "raw"-data from kafka topic, and > [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. > We can see, that section "after" is different in this files, even though it > should be the same record > In Kafka: > !see in Kafka.PNG! > In Flowfile: > !see in Avro.PNG! > The error does not always appear, but only in some records. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8539) ConsumeKafkaRecord_2_0 1.13.0:
[ https://issues.apache.org/jira/browse/NIFI-8539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Nikolaev updated NIFI-8539: --- Description: Steps to reproduce the bug: * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 1.13.0" and write output file with PutFile ( !test_flow.png|thumbnail! ) * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] contains "raw"-data from kafka topic, and [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. We can see, that section "after" is different in this files, even though it should be the same record In Kafka - !see in Kafka.PNG|thumbnail! In Flowfile - !see in Avro.PNG|thumbnail! The error does not always appear, but only in some records. was: Steps to reproduce the bug: * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 1.13.0" and write output file with PutFile ( !test_flow.png! ) * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] contains "raw"-data from kafka topic, and [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. We can see, that section "after" is different in this files, even though it should be the same record In Kafka - !see in Kafka.PNG! In Flowfile - !see in Avro.PNG! The error does not always appear, but only in some records. > ConsumeKafkaRecord_2_0 1.13.0: > --- > > Key: NIFI-8539 > URL: https://issues.apache.org/jira/browse/NIFI-8539 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.0 >Reporter: Nikolay Nikolaev >Priority: Major > Attachments: 38f1d0f9-33c9-4cf1-978a-9060274615a5.avro, data in > Kafka.json, see in Avro.PNG, see in Kafka.PNG, test_flow.png > > > Steps to reproduce the bug: > * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 > 1.13.0" and write output file with PutFile ( !test_flow.png|thumbnail! ) > * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] > contains "raw"-data from kafka topic, and > [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. > We can see, that section "after" is different in this files, even though it > should be the same record > In Kafka - !see in Kafka.PNG|thumbnail! > In Flowfile - !see in Avro.PNG|thumbnail! > The error does not always appear, but only in some records. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8539) ConsumeKafkaRecord_2_0 1.13.0:
[ https://issues.apache.org/jira/browse/NIFI-8539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Nikolaev updated NIFI-8539: --- Description: Steps to reproduce the bug: * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 1.13.0" and write output file with PutFile ( !test_flow.png! ) * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] contains "raw"-data from kafka topic, and [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. We can see, that section "after" is different in this files, even though it should be the same record In Kafka - !see in Kafka.PNG! In Flowfile - !see in Avro.PNG! The error does not always appear, but only in some records. was: Steps to reproduce the bug: * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 1.13.0" and write output file with PutFile ( !test_flow.png! ) * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] contains "raw"-data from kafka topic, and [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. We can see, that section "after" is different in this files, even though it should be the same record In Kafka - !see in Avro.PNG! In Flowfile - !see in Avro.PNG! The error does not always appear, but only in some records. > ConsumeKafkaRecord_2_0 1.13.0: > --- > > Key: NIFI-8539 > URL: https://issues.apache.org/jira/browse/NIFI-8539 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.0 >Reporter: Nikolay Nikolaev >Priority: Major > Attachments: 38f1d0f9-33c9-4cf1-978a-9060274615a5.avro, data in > Kafka.json, see in Avro.PNG, see in Kafka.PNG, test_flow.png > > > Steps to reproduce the bug: > * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 > 1.13.0" and write output file with PutFile ( !test_flow.png! ) > * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] > contains "raw"-data from kafka topic, and > [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. > We can see, that section "after" is different in this files, even though it > should be the same record > In Kafka - !see in Kafka.PNG! > In Flowfile - !see in Avro.PNG! > The error does not always appear, but only in some records. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8539) ConsumeKafkaRecord_2_0 1.13.0:
[ https://issues.apache.org/jira/browse/NIFI-8539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Nikolaev updated NIFI-8539: --- Description: Steps to reproduce the bug: * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 1.13.0" and write output file with PutFile ( !test_flow.png! ) * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] contains "raw"-data from kafka topic, and [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. We can see, that section "after" is different in this files, even though it should be the same record In Kafka - !see in Avro.PNG! In Flowfile - !see in Avro.PNG! The error does not always appear, but only in some records. was: Steps to reproduce the bug: * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 1.13.0" and write output file with PutFile (see picture) * Periodically, some flowfiles get incorrect. Example: file contains "raw"-data from kafka topic, and file is output flowfile (and > ConsumeKafkaRecord_2_0 1.13.0: > --- > > Key: NIFI-8539 > URL: https://issues.apache.org/jira/browse/NIFI-8539 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.0 >Reporter: Nikolay Nikolaev >Priority: Major > Attachments: 38f1d0f9-33c9-4cf1-978a-9060274615a5.avro, data in > Kafka.json, see in Avro.PNG, see in Kafka.PNG, test_flow.png > > > Steps to reproduce the bug: > * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 > 1.13.0" and write output file with PutFile ( !test_flow.png! ) > * Periodically, some flowfiles get incorrect. Example: [^data in Kafka.json] > contains "raw"-data from kafka topic, and > [^38f1d0f9-33c9-4cf1-978a-9060274615a5.avro] is output flowfile. > We can see, that section "after" is different in this files, even though it > should be the same record > In Kafka - !see in Avro.PNG! > In Flowfile - !see in Avro.PNG! > The error does not always appear, but only in some records. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8539) ConsumeKafkaRecord_2_0 1.13.0:
Nikolay Nikolaev created NIFI-8539: -- Summary: ConsumeKafkaRecord_2_0 1.13.0: Key: NIFI-8539 URL: https://issues.apache.org/jira/browse/NIFI-8539 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.13.0 Reporter: Nikolay Nikolaev Attachments: 38f1d0f9-33c9-4cf1-978a-9060274615a5.avro, data in Kafka.json, see in Avro.PNG, see in Kafka.PNG, test_flow.png Steps to reproduce the bug: * build a simple flow: read data from Kafka with "ConsumeKafkaRecord_2_0 1.13.0" and write output file with PutFile (see picture) * Periodically, some flowfiles get incorrect. Example: file contains "raw"-data from kafka topic, and file is output flowfile (and -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] adamdebreceni opened a new pull request #1072: MINIFICPP-1482 - Fix C2RequestClassTest transient failure
adamdebreceni opened a new pull request #1072: URL: https://github.com/apache/nifi-minifi-cpp/pull/1072 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (MINIFICPP-1482) C2RequestClassTest transient failure
[ https://issues.apache.org/jira/browse/MINIFICPP-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Debreceni reassigned MINIFICPP-1482: - Assignee: Adam Debreceni > C2RequestClassTest transient failure > > > Key: MINIFICPP-1482 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1482 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Gabor Gyimesi >Assignee: Adam Debreceni >Priority: Minor > Labels: MiNiFi-CPP-Hygiene > Attachments: C2RequestClassTest-failure-ubuntu1604.log > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] adamdebreceni opened a new pull request #1071: MINIFICPP-1548 - Use the same rocksdb instance for multiple repositories
adamdebreceni opened a new pull request #1071: URL: https://github.com/apache/nifi-minifi-cpp/pull/1071 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
