[jira] [Commented] (NIFI-6115) Support custom lookup SQL at DatabaseRecordLookupService
[
https://issues.apache.org/jira/browse/NIFI-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249518#comment-17249518
]
Ajeesh B commented on NIFI-6115:
This will be a cool feature.
> Support custom lookup SQL at DatabaseRecordLookupService
>
>
> Key: NIFI-6115
> URL: https://issues.apache.org/jira/browse/NIFI-6115
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Koji Kawamura
>Priority: Major
>
> NIFI-6082 added DatabaseRecordLookupService. Further, we could add
> CUSTOM_LOOKUP_SQL to support custom where conditions and order by ... etc.
> Design:
> * If CUSTOM_LOOKUP_SQL is specified, TABLE_NAME and LOOKUP_KEY_COLUMN are
> not required
> * User would specify a custom query like {{select firstName, lastName from
> users where companyName = ${companyName} and department = ${department} order
> by job_responsibility}}
> * Then DatabaseRecordLookupService can understand what coordinates are
> required by using VariableImpact.isAffected. ProcessGroup can be a reference
> for how to do this.
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java#L2975]
> * Then, the ControllerService replaces EL part into NamedParameter. Should
> note on the doc that EL function can not be used here. If needed, it should
> be done at Processor's dynamic properties.
> * The generated SQL would look like {{select firstName, lastName from users
> where companyName = :companyName and department = :department order by
> job_responsibility}}
> * Then user would configure LookupRecord with two dynamic properties:
> ** {{companyName}}: {{/companyName}}
> ** {{departmentName}}: {{/departmentName}}
> This way, user can lookup record very flexibly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8019) SSL Enabled Protocol test failures when TLSv1 and TLSv1.1 disabled in java.security
[ https://issues.apache.org/jira/browse/NIFI-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Gough updated NIFI-8019: --- Fix Version/s: 1.13.0 Resolution: Fixed Status: Resolved (was: Patch Available) > SSL Enabled Protocol test failures when TLSv1 and TLSv1.1 disabled in > java.security > --- > > Key: NIFI-8019 > URL: https://issues.apache.org/jira/browse/NIFI-8019 > Project: Apache NiFi > Issue Type: Bug > Components: Security >Affects Versions: 1.12.1 > Environment: Fedora 33 OpenJDK 11.0.9 >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 1.13.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > The SslContextFactoryTest in nifi-security-utils and other test classes > evaluate the array of enabled protocols during various unit tests after > constructing an SSLContext. This unit test and others contain a static array > of expected protocols that include TLSv1 and TLSv1.1. > Recent versions of Java 8 and 11 continue to allow these protocols, however, > Fedora 33 introduced changes to the default cryptographic policies that > disable TLSv1 and TLSv1.1. The following Fedora Wiki page describes the > changes: > [https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2] > The Fedora 33 _crypto-policies_ RPM includes the following policy file: > /usr/share/crypto-policies/DEFAULT/java.txt > The Java policy includes TLSv1 and TLSv1.1 in the property for > jdk.tls.disabledAlgorithms. This policy is included at runtime due to the > java.security policy enabling security.useSystemPropertiesFile. > The SslContextFactoryTest and other tests that evaluate enabled SSL protocols > should be updated to dynamically determine which protocols to expect using > the SSLContext.getDefaultSSLParameters().getProtocols() method. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8019) SSL Enabled Protocol test failures when TLSv1 and TLSv1.1 disabled in java.security
[ https://issues.apache.org/jira/browse/NIFI-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249482#comment-17249482 ] ASF subversion and git services commented on NIFI-8019: --- Commit 28ca7478d6c209f7029670dd4ced5ed9206ecc9f in nifi's branch refs/heads/main from exceptionfactory [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=28ca747 ] NIFI-8019 Added TlsPlatform to provide runtime TLS protocol configuration NIFI-8019 Renamed getDefaultProtocols() to getSupportedProtocols() Signed-off-by: Nathan Gough This closes #4673. > SSL Enabled Protocol test failures when TLSv1 and TLSv1.1 disabled in > java.security > --- > > Key: NIFI-8019 > URL: https://issues.apache.org/jira/browse/NIFI-8019 > Project: Apache NiFi > Issue Type: Bug > Components: Security >Affects Versions: 1.12.1 > Environment: Fedora 33 OpenJDK 11.0.9 >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > The SslContextFactoryTest in nifi-security-utils and other test classes > evaluate the array of enabled protocols during various unit tests after > constructing an SSLContext. This unit test and others contain a static array > of expected protocols that include TLSv1 and TLSv1.1. > Recent versions of Java 8 and 11 continue to allow these protocols, however, > Fedora 33 introduced changes to the default cryptographic policies that > disable TLSv1 and TLSv1.1. The following Fedora Wiki page describes the > changes: > [https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2] > The Fedora 33 _crypto-policies_ RPM includes the following policy file: > /usr/share/crypto-policies/DEFAULT/java.txt > The Java policy includes TLSv1 and TLSv1.1 in the property for > jdk.tls.disabledAlgorithms. This policy is included at runtime due to the > java.security policy enabling security.useSystemPropertiesFile. > The SslContextFactoryTest and other tests that evaluate enabled SSL protocols > should be updated to dynamically determine which protocols to expect using > the SSLContext.getDefaultSSLParameters().getProtocols() method. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8019) SSL Enabled Protocol test failures when TLSv1 and TLSv1.1 disabled in java.security
[ https://issues.apache.org/jira/browse/NIFI-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249481#comment-17249481 ] ASF subversion and git services commented on NIFI-8019: --- Commit 28ca7478d6c209f7029670dd4ced5ed9206ecc9f in nifi's branch refs/heads/main from exceptionfactory [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=28ca747 ] NIFI-8019 Added TlsPlatform to provide runtime TLS protocol configuration NIFI-8019 Renamed getDefaultProtocols() to getSupportedProtocols() Signed-off-by: Nathan Gough This closes #4673. > SSL Enabled Protocol test failures when TLSv1 and TLSv1.1 disabled in > java.security > --- > > Key: NIFI-8019 > URL: https://issues.apache.org/jira/browse/NIFI-8019 > Project: Apache NiFi > Issue Type: Bug > Components: Security >Affects Versions: 1.12.1 > Environment: Fedora 33 OpenJDK 11.0.9 >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > The SslContextFactoryTest in nifi-security-utils and other test classes > evaluate the array of enabled protocols during various unit tests after > constructing an SSLContext. This unit test and others contain a static array > of expected protocols that include TLSv1 and TLSv1.1. > Recent versions of Java 8 and 11 continue to allow these protocols, however, > Fedora 33 introduced changes to the default cryptographic policies that > disable TLSv1 and TLSv1.1. The following Fedora Wiki page describes the > changes: > [https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2] > The Fedora 33 _crypto-policies_ RPM includes the following policy file: > /usr/share/crypto-policies/DEFAULT/java.txt > The Java policy includes TLSv1 and TLSv1.1 in the property for > jdk.tls.disabledAlgorithms. This policy is included at runtime due to the > java.security policy enabling security.useSystemPropertiesFile. > The SslContextFactoryTest and other tests that evaluate enabled SSL protocols > should be updated to dynamically determine which protocols to expect using > the SSLContext.getDefaultSSLParameters().getProtocols() method. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] thenatog closed pull request #4673: NIFI-8019 Added TlsPlatform to provide runtime default TLS Protocols
thenatog closed pull request #4673: URL: https://github.com/apache/nifi/pull/4673 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4673: NIFI-8019 Added TlsPlatform to provide runtime default TLS Protocols
thenatog commented on pull request #4673: URL: https://github.com/apache/nifi/pull/4673#issuecomment-744887599 +1 LGTM, will merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] sjyang18 commented on pull request #4286: NIFI-7386: Azurite emulator support
sjyang18 commented on pull request #4286: URL: https://github.com/apache/nifi/pull/4286#issuecomment-744844252 I rebased my branch on top of main and resolved the conflict. @jfrazee, Would you take a look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4673: NIFI-8019 Added TlsPlatform to provide runtime default TLS Protocols
exceptionfactory commented on a change in pull request #4673:
URL: https://github.com/apache/nifi/pull/4673#discussion_r542822592
##
File path:
nifi-commons/nifi-security-utils-api/src/main/java/org/apache/nifi/security/util/TlsPlatform.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.security.util;
+
+import static java.util.Collections.unmodifiableSet;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLParameters;
+import java.security.NoSuchAlgorithmException;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * Transport Layer Security Platform provides runtime protocol configuration
information
+ */
+public class TlsPlatform {
+private static final Pattern PROTOCOL_VERSION =
Pattern.compile("TLSv(\\d+\\.?\\d*)");
+
+private static final int FIRST_GROUP = 1;
+
+private static final List LEGACY_PROTOCOLS =
Arrays.asList("TLSv1", "TLSv1.1");
+
+private static final SortedMap SORTED_PROTOCOLS =
getDefaultSslContextProtocols();
+
+private static final Set DEFAULT_PROTOCOLS = unmodifiableSet(new
TreeSet<>(SORTED_PROTOCOLS.values()).descendingSet());
+
+private static final Set PREFERRED_PROTOCOLS = unmodifiableSet(
+DEFAULT_PROTOCOLS.stream()
+.filter(protocol -> !LEGACY_PROTOCOLS.contains(protocol))
+.collect(Collectors.toSet())
+);
+
+/**
+ * Get Default Protocols based on Java Security configuration
+ *
+ * @return Set of Transport Layer Security Protocol names
+ */
+public static Set getDefaultProtocols() {
Review comment:
That sounds good, I pushed an update renaming the method to
getSupportedProtocols.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] thenatog commented on a change in pull request #4673: NIFI-8019 Added TlsPlatform to provide runtime default TLS Protocols
thenatog commented on a change in pull request #4673:
URL: https://github.com/apache/nifi/pull/4673#discussion_r542779005
##
File path:
nifi-commons/nifi-security-utils-api/src/main/java/org/apache/nifi/security/util/TlsPlatform.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.security.util;
+
+import static java.util.Collections.unmodifiableSet;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLParameters;
+import java.security.NoSuchAlgorithmException;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * Transport Layer Security Platform provides runtime protocol configuration
information
+ */
+public class TlsPlatform {
+private static final Pattern PROTOCOL_VERSION =
Pattern.compile("TLSv(\\d+\\.?\\d*)");
+
+private static final int FIRST_GROUP = 1;
+
+private static final List LEGACY_PROTOCOLS =
Arrays.asList("TLSv1", "TLSv1.1");
+
+private static final SortedMap SORTED_PROTOCOLS =
getDefaultSslContextProtocols();
+
+private static final Set DEFAULT_PROTOCOLS = unmodifiableSet(new
TreeSet<>(SORTED_PROTOCOLS.values()).descendingSet());
+
+private static final Set PREFERRED_PROTOCOLS = unmodifiableSet(
+DEFAULT_PROTOCOLS.stream()
+.filter(protocol -> !LEGACY_PROTOCOLS.contains(protocol))
+.collect(Collectors.toSet())
+);
+
+/**
+ * Get Default Protocols based on Java Security configuration
+ *
+ * @return Set of Transport Layer Security Protocol names
+ */
+public static Set getDefaultProtocols() {
Review comment:
Yeah I think getSupportedProtocols works, with a comment that the
protocols returned are _all_ the protocols supported by the JVM.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8094) Support BCFKS Keystore Type
David Handermann created NIFI-8094:
--
Summary: Support BCFKS Keystore Type
Key: NIFI-8094
URL: https://issues.apache.org/jira/browse/NIFI-8094
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework, Extensions, Security
Affects Versions: 1.12.1
Reporter: David Handermann
Assignee: David Handermann
The [Bouncy Castle FIPS Key
Store|https://cryptosense.com/blog/bouncycastle-keystore-security/] (BCFKS)
format supports storage of certificates and private keys using AES-CCM and
PBKDF2 algorithms, providing greater security than the standard JKS and PKCS12
implementations. Support for BCFKS can be implemented using Bouncy Castle
security provider libraries that are already leveraged throughout the system.
Initial support should include the ability to specify BCFKS as the key store
and trust store type in standard properties files as well as the ability to
select BCFKS in implementations of the SSLContextService.
Extension components that do not use {{SSLContextService.createSSLContext()}}
may need additional work, which should be addressed in related issues following
this implementation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] markobean commented on pull request #4225: NIFI-7387: ComponentLog accepts Object varargs instead of array
markobean commented on pull request #4225: URL: https://github.com/apache/nifi/pull/4225#issuecomment-744693725 I tested this by building nifi in the feature branch, and installing (1.13.0-SNAPSHOT.) I also placed the 1.12.1 version of two NARs in the lib directory: nifi-standard-nar, nifi-update-attribute-nar. In confirmed that identical log messages were issued by both versions of SplitText and UpdateAttribute processors. I performed this test in both Java 8 and 11. Also, the odd log statement verification in ParseEvtxTest was removed completely. I do not see any remaining issues. Please merge. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-8088) Remove PKCS12 Deprecation Warning in KeyStoreUtils
[ https://issues.apache.org/jira/browse/NIFI-8088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-8088: --- Status: Patch Available (was: In Progress) > Remove PKCS12 Deprecation Warning in KeyStoreUtils > -- > > Key: NIFI-8088 > URL: https://issues.apache.org/jira/browse/NIFI-8088 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.12.1 >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Labels: pkcs12, security > Time Spent: 10m > Remaining Estimate: 0h > > KeyStoreUtils.getTrustStore() logs a warning message indicating the PKCS12 > trust stores are deprecated in favor of JKS. JEP 229 changed the default key > store type to PKCS12 starting with Java 9. The Site to Site Client, GetHTTP, > and PostHTTP Processors use the KeyStoreUtils.getTrustStore() method. The > method itself checks the trust store type, logs the warning for PKCS12 and > delegates to KeyStoreUtils.getKeyStore(). > References to KeyStoreUtils.getTrustStore() should be replaced with > KeyStoreUtils.getKeyStore() and the getTrustStore() method should be removed > to avoid confusing log warnings in deployments where PKCS12 trust stores are > used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] thenatog commented on pull request #4687: NIFI-1930 Updated ListenHTTP to set TLS included protocols
thenatog commented on pull request #4687: URL: https://github.com/apache/nifi/pull/4687#issuecomment-744669663 Merged This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog closed pull request #4687: NIFI-1930 Updated ListenHTTP to set TLS included protocols
thenatog closed pull request #4687: URL: https://github.com/apache/nifi/pull/4687 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-1930) ListenHTTP does not use SSLContextService protocol
[ https://issues.apache.org/jira/browse/NIFI-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Gough updated NIFI-1930: --- Fix Version/s: 1.13.0 Resolution: Fixed Status: Resolved (was: Patch Available) > ListenHTTP does not use SSLContextService protocol > -- > > Key: NIFI-1930 > URL: https://issues.apache.org/jira/browse/NIFI-1930 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.6.1, 1.12.1 >Reporter: Mark Bean >Assignee: David Handermann >Priority: Major > Labels: TLS, security > Fix For: 1.13.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The ListenHTTP processor ignores the protocol property of the > SSLContextService. Even when the service specifies a specific protocol, > ListenHTTP will negotiate and honor alternate protocols. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-1930) ListenHTTP does not use SSLContextService protocol
[ https://issues.apache.org/jira/browse/NIFI-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249253#comment-17249253 ] ASF subversion and git services commented on NIFI-1930: --- Commit 5bcb5abc8680d02f50c235f824050d8a044de1b7 in nifi's branch refs/heads/main from exceptionfactory [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5bcb5ab ] NIFI-1930 Updated ListenHTTP to set TLS included protocols based on SSLContextService NIFI-1930 Removed unused import in SSLContextServiceTest Signed-off-by: Nathan Gough This closes #4673. > ListenHTTP does not use SSLContextService protocol > -- > > Key: NIFI-1930 > URL: https://issues.apache.org/jira/browse/NIFI-1930 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.6.1, 1.12.1 >Reporter: Mark Bean >Assignee: David Handermann >Priority: Major > Labels: TLS, security > Time Spent: 50m > Remaining Estimate: 0h > > The ListenHTTP processor ignores the protocol property of the > SSLContextService. Even when the service specifies a specific protocol, > ListenHTTP will negotiate and honor alternate protocols. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-1930) ListenHTTP does not use SSLContextService protocol
[ https://issues.apache.org/jira/browse/NIFI-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249252#comment-17249252 ] ASF subversion and git services commented on NIFI-1930: --- Commit 5bcb5abc8680d02f50c235f824050d8a044de1b7 in nifi's branch refs/heads/main from exceptionfactory [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5bcb5ab ] NIFI-1930 Updated ListenHTTP to set TLS included protocols based on SSLContextService NIFI-1930 Removed unused import in SSLContextServiceTest Signed-off-by: Nathan Gough This closes #4673. > ListenHTTP does not use SSLContextService protocol > -- > > Key: NIFI-1930 > URL: https://issues.apache.org/jira/browse/NIFI-1930 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.6.1, 1.12.1 >Reporter: Mark Bean >Assignee: David Handermann >Priority: Major > Labels: TLS, security > Time Spent: 50m > Remaining Estimate: 0h > > The ListenHTTP processor ignores the protocol property of the > SSLContextService. Even when the service specifies a specific protocol, > ListenHTTP will negotiate and honor alternate protocols. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] thenatog closed pull request #4673: NIFI-8019 Added TlsPlatform to provide runtime default TLS Protocols
thenatog closed pull request #4673: URL: https://github.com/apache/nifi/pull/4673 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4687: NIFI-1930 Updated ListenHTTP to set TLS included protocols
thenatog commented on pull request #4687: URL: https://github.com/apache/nifi/pull/4687#issuecomment-744658973 +1 LGTM, will merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-4558) Populate default keystore/truststore types in SSLContextService
[
https://issues.apache.org/jira/browse/NIFI-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249209#comment-17249209
]
David Handermann commented on NIFI-4558:
This feature would be useful, but instead of defaulting to JKS, it seems better
to use the KeyStore.getDefaultType() method to match the JVM runtime
configuration. This would result in JKS on Java 8 and PKCS12 on Java 9 and
greater. If that is too confusing, recommend defaulting to PKCS12 as that is
the standard for current versions of Java.
> Populate default keystore/truststore types in SSLContextService
> ---
>
> Key: NIFI-4558
> URL: https://issues.apache.org/jira/browse/NIFI-4558
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Andy LoPresto
>Assignee: Nathan Gough
>Priority: Trivial
> Labels: controller_services, jks, keystore, pkcs12, security,
> ssl, tls, truststore, ux
>
> The keystore and truststore type is almost always JKS as opposed to PKCS12
> when creating SSL controller services. Both {{StandardSSLContextService}} and
> {{StandardRestrictedSSLContextService}} should have those fields
> autopopulated to JKS, saving 2-4 clicks per instantiation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #959: MINIFICPP-1424 Stabilize several flaky test suites
lordgamez commented on a change in pull request #959:
URL: https://github.com/apache/nifi-minifi-cpp/pull/959#discussion_r542602723
##
File path: libminifi/src/io/tls/TLSServerSocket.cpp
##
@@ -87,13 +87,13 @@ void
TLSServerSocket::registerCallback(std::function accept_function, st
* Initializes the socket
* @return result of the creation operation.
*/
-void TLSServerSocket::registerCallback(std::function accept_function,
std::function*, int *)> handler) {
- fx = [this](std::function accept_function,
std::function*, int *)> handler) {
+void TLSServerSocket::registerCallback(std::function accept_function,
std::function*, int *)> handler,
std::chrono::milliseconds timeout) {
+ fx = [this](std::function accept_function,
std::function*, int *)> handler,
std::chrono::milliseconds timeout) {
int ret = 0;
std::vector fds;
int size;
while (accept_function()) {
- int fd = select_descriptor(3000);
+ int fd = select_descriptor(timeout.count());
Review comment:
Fixed in
[bfc8394](https://github.com/apache/nifi-minifi-cpp/pull/959/commits/bfc83949f4c3549d5a85fc0250068eea7e7801d1)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #959: MINIFICPP-1424 Stabilize several flaky test suites
fgerlits commented on a change in pull request #959:
URL: https://github.com/apache/nifi-minifi-cpp/pull/959#discussion_r542574442
##
File path: libminifi/src/io/tls/TLSServerSocket.cpp
##
@@ -87,13 +87,13 @@ void
TLSServerSocket::registerCallback(std::function accept_function, st
* Initializes the socket
* @return result of the creation operation.
*/
-void TLSServerSocket::registerCallback(std::function accept_function,
std::function*, int *)> handler) {
- fx = [this](std::function accept_function,
std::function*, int *)> handler) {
+void TLSServerSocket::registerCallback(std::function accept_function,
std::function*, int *)> handler,
std::chrono::milliseconds timeout) {
+ fx = [this](std::function accept_function,
std::function*, int *)> handler,
std::chrono::milliseconds timeout) {
int ret = 0;
std::vector fds;
int size;
while (accept_function()) {
- int fd = select_descriptor(3000);
+ int fd = select_descriptor(timeout.count());
Review comment:
this is a narrowing conversion from at least 45 bits (signed) to 16 bits
(unsigned); it could be made safe (and warning-free) with `gsl::narrow`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #959: MINIFICPP-1424 Stabilize several flaky test suites
fgerlits commented on a change in pull request #959:
URL: https://github.com/apache/nifi-minifi-cpp/pull/959#discussion_r542574442
##
File path: libminifi/src/io/tls/TLSServerSocket.cpp
##
@@ -87,13 +87,13 @@ void
TLSServerSocket::registerCallback(std::function accept_function, st
* Initializes the socket
* @return result of the creation operation.
*/
-void TLSServerSocket::registerCallback(std::function accept_function,
std::function*, int *)> handler) {
- fx = [this](std::function accept_function,
std::function*, int *)> handler) {
+void TLSServerSocket::registerCallback(std::function accept_function,
std::function*, int *)> handler,
std::chrono::milliseconds timeout) {
+ fx = [this](std::function accept_function,
std::function*, int *)> handler,
std::chrono::milliseconds timeout) {
int ret = 0;
std::vector fds;
int size;
while (accept_function()) {
- int fd = select_descriptor(3000);
+ int fd = select_descriptor(timeout.count());
Review comment:
this is a narrowing conversion from at least 45 bits to 16 bits; could
be made safe (and warning-free) with `gsl::narrow`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-7991) Flow Configuration History displays "annotation data not found/available" from "Advanced" changes.
[ https://issues.apache.org/jira/browse/NIFI-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-7991: --- Status: Patch Available (was: Open) > Flow Configuration History displays "annotation data not found/available" > from "Advanced" changes. > -- > > Key: NIFI-7991 > URL: https://issues.apache.org/jira/browse/NIFI-7991 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Tim Smith >Assignee: Tim Smith >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > On making changes through UpdateAttributes Advanced tab, discovered that > "AnnotationData" is presented in "Flow Configuration History" as "annotation > data not found/available" in both previous and new values by > ProcessorAuditor. UpdateAttribute represents the annotation data as XML. > ProcessorAuditor, at a minimum, should attempt to represent new and previous > values as the difference between the XML nodes or as String values, if not in > XML format. Default value presented is not useful. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #959: MINIFICPP-1424 Stabilize several flaky test suites
lordgamez commented on a change in pull request #959:
URL: https://github.com/apache/nifi-minifi-cpp/pull/959#discussion_r542554296
##
File path: libminifi/test/persistence-tests/PersistenceTests.cpp
##
@@ -326,10 +334,12 @@ TEST_CASE("Persisted flowFiles are updated on
modification", "[TestP1]") {
ff_repository->start();
// wait for FlowFileRepository to start and notify the owners of
// the resurrected FlowFiles
-std::this_thread::sleep_for(std::chrono::milliseconds{100});
-
std::set> expired;
-auto file = flow.output->poll(expired);
+std::shared_ptr file = nullptr;
+for (auto i = 0; i < 20 && !file; ++i) {
Review comment:
Fixed in
[a20314b](https://github.com/apache/nifi-minifi-cpp/pull/959/commits/a20314b080d4fded9d71d61d68d6be616032eb4f)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #959: MINIFICPP-1424 Stabilize several flaky test suites
lordgamez commented on a change in pull request #959:
URL: https://github.com/apache/nifi-minifi-cpp/pull/959#discussion_r542553948
##
File path: libminifi/include/io/tls/TLSServerSocket.h
##
@@ -53,7 +53,7 @@ class TLSServerSocket : public BaseServerSocket, public
TLSSocket {
/**
* Registers a call back and starts the read for the server socket.
*/
- void registerCallback(std::function accept_function,
std::function*, int *)> handler);
+ void registerCallback(std::function accept_function,
std::function*, int *)> handler, uint16_t timeout =
3000);
Review comment:
Fixed in
[e35f799](https://github.com/apache/nifi-minifi-cpp/pull/959/commits/e35f799294095be494056f415a24b7350192e61d)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8093) Possible race condition when deleting oldest provenance file
Matt Burgess created NIFI-8093: -- Summary: Possible race condition when deleting oldest provenance file Key: NIFI-8093 URL: https://issues.apache.org/jira/browse/NIFI-8093 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Matt Burgess Saw this in some recent NiFi logs: 2020-12-14 11:42:53,770 WARN [Provenance Repository Maintenance-2-thread-1] o.a.n.p.store.WriteAheadStorePartition Provenance Event Store Partition[directory=./provenance_repository] Failed to delete oldest event file ./provenance_repository/37169.prov. This file should be cleaned up manually. 2020-12-14 11:42:53,796 INFO [Provenance Repository Maintenance-2-thread-1] o.a.n.p.store.WriteAheadStorePartition Provenance Event Store Partition[directory=./provenance_repository] Deleted ./provenance_repository/37169.prov.gz event file (120.69 MB) due to storage limits It appears there may be a race condition where NiFi is attempting to delete the oldest provenance file while it is being compressed. After compression completes, the .gz file does get successfully deleted (so no manual intervention is required). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8092) Fix ITPutS3Object test failure due to Outposts storage class
[ https://issues.apache.org/jira/browse/NIFI-8092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Turcsanyi updated NIFI-8092: -- Status: Patch Available (was: Open) > Fix ITPutS3Object test failure due to Outposts storage class > > > Key: NIFI-8092 > URL: https://issues.apache.org/jira/browse/NIFI-8092 > Project: Apache NiFi > Issue Type: Test > Components: Extensions >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > "Outposts" storage class cannot be tested on AWS cloud infrastructure. It > should be skipped. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-7856:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.13.0
>
> Attachments: 1683472.prov, NIFI-7856.xml, ls.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249100#comment-17249100
]
ASF subversion and git services commented on NIFI-7856:
---
Commit a73cd6a610f2e4a43e82bb26e2e4f983b9bfa1a5 in nifi's branch
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a73cd6a ]
NIFI-7856: If a Provenance Event File is ready to be rolled over due to the
maximum amount of time having elapsed, avoid rolling over unless there is at
least one event written to the event file. Otherwise, we can have multiple
RecordWriters / RecordWriterLeases pointing to the same file. This can result
in data being overwritten, as well as failing to compress the event file upon
rollover. Also added significant DEBUG/TRACE level logging.
Signed-off-by: Matthew Burgess
This closes #4580
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.13.0
>
> Attachments: 1683472.prov, NIFI-7856.xml, ls.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] mattyb149 closed pull request #4580: NIFI-7856: If a Provenance Event File is ready to be rolled over due …
mattyb149 closed pull request #4580: URL: https://github.com/apache/nifi/pull/4580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #4580: NIFI-7856: If a Provenance Event File is ready to be rolled over due …
mattyb149 commented on pull request #4580: URL: https://github.com/apache/nifi/pull/4580#issuecomment-744568800 +1 LGTM, reproduced the issue then verified it works with this PR. Thanks for the fix! Merging to main This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] turcsanyip opened a new pull request #4728: NIFI-8092: Fix ITPutS3Object test failure due to Outposts storage class
turcsanyip opened a new pull request #4728: URL: https://github.com/apache/nifi/pull/4728 https://issues.apache.org/jira/browse/NIFI-8092 In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with **NIFI-** where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically `main`)? - [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._ ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] Have you verified that the full build is successful on JDK 8? - [ ] Have you verified that the full build is successful on JDK 11? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`? - [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`? - [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-8092) Fix ITPutS3Object test failure due to Outposts storage class
[ https://issues.apache.org/jira/browse/NIFI-8092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249096#comment-17249096 ] ASF subversion and git services commented on NIFI-8092: --- Commit 8499b2c24561bec1fc0482cbe421413a3225ed1f in nifi's branch refs/heads/NIFI-8092 from Peter Turcsanyi [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8499b2c ] NIFI-8092: Fix ITPutS3Object test failure due to Outposts storage class > Fix ITPutS3Object test failure due to Outposts storage class > > > Key: NIFI-8092 > URL: https://issues.apache.org/jira/browse/NIFI-8092 > Project: Apache NiFi > Issue Type: Test >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Minor > > "Outposts" storage class cannot be tested on AWS cloud infrastructure. It > should be skipped. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8092) Fix ITPutS3Object test failure due to Outposts storage class
[ https://issues.apache.org/jira/browse/NIFI-8092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Turcsanyi updated NIFI-8092: -- Component/s: Extensions > Fix ITPutS3Object test failure due to Outposts storage class > > > Key: NIFI-8092 > URL: https://issues.apache.org/jira/browse/NIFI-8092 > Project: Apache NiFi > Issue Type: Test > Components: Extensions >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Minor > > "Outposts" storage class cannot be tested on AWS cloud infrastructure. It > should be skipped. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8092) Fix ITPutS3Object test failure due to Outposts storage class
[ https://issues.apache.org/jira/browse/NIFI-8092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Turcsanyi updated NIFI-8092: -- Summary: Fix ITPutS3Object test failure due to Outposts storage class (was: Fix ITPutS3Object test failure due the Outposts storage class) > Fix ITPutS3Object test failure due to Outposts storage class > > > Key: NIFI-8092 > URL: https://issues.apache.org/jira/browse/NIFI-8092 > Project: Apache NiFi > Issue Type: Test >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Minor > > "Outposts" storage class cannot be tested on AWS cloud infrastructure. It > should be skipped. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8092) Fix ITPutS3Object test failure due the Outposts storage class
Peter Turcsanyi created NIFI-8092: - Summary: Fix ITPutS3Object test failure due the Outposts storage class Key: NIFI-8092 URL: https://issues.apache.org/jira/browse/NIFI-8092 Project: Apache NiFi Issue Type: Test Reporter: Peter Turcsanyi Assignee: Peter Turcsanyi "Outposts" storage class cannot be tested on AWS cloud infrastructure. It should be skipped. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4508: NIFI-6576 add basic auth to confluent schema registry service
exceptionfactory commented on a change in pull request #4508:
URL: https://github.com/apache/nifi/pull/4508#discussion_r542516691
##
File path:
nifi-nar-bundles/nifi-confluent-platform-bundle/nifi-confluent-schema-registry-service/src/main/java/org/apache/nifi/confluent/schemaregistry/ConfluentSchemaRegistry.java
##
@@ -112,6 +114,32 @@
.required(true)
.build();
+static final PropertyDescriptor PROP_BASIC_AUTH_USERNAME = new
PropertyDescriptor.Builder()
+.name("Authentication Username")
+.displayName("Authentication Username")
+.description("The username to be used by the client to authenticate
against the Remote URL. Cannot include control characters (0-31), ':', or DEL
(127).")
+.required(false)
+
.addValidator(StandardValidators.createRegexMatchingValidator(Pattern.compile("^[\\x20-\\x39\\x3b-\\x7e\\x80-\\xff]+$")))
+.build();
+
+static final PropertyDescriptor PROP_BASIC_AUTH_PASSWORD = new
PropertyDescriptor.Builder()
+.name("Authentication Password")
+.displayName("Authentication Password")
+.description("The password to be used by the client to authenticate
against the Remote URL.")
+.required(false)
+.sensitive(true)
+
.addValidator(StandardValidators.createRegexMatchingValidator(Pattern.compile("^[\\x20-\\x7e\\x80-\\xff]+$")))
+.build();
+
+static final PropertyDescriptor PROP_AUTH_TYPE = new
PropertyDescriptor.Builder()
Review comment:
Leaving the property in place and adding an allowable value of _NONE_
sounds like a good approach.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #959: MINIFICPP-1424 Stabilize several flaky test suites
fgerlits commented on a change in pull request #959:
URL: https://github.com/apache/nifi-minifi-cpp/pull/959#discussion_r542496017
##
File path: libminifi/test/persistence-tests/PersistenceTests.cpp
##
@@ -326,10 +334,12 @@ TEST_CASE("Persisted flowFiles are updated on
modification", "[TestP1]") {
ff_repository->start();
// wait for FlowFileRepository to start and notify the owners of
// the resurrected FlowFiles
-std::this_thread::sleep_for(std::chrono::milliseconds{100});
-
std::set> expired;
-auto file = flow.output->poll(expired);
+std::shared_ptr file = nullptr;
+for (auto i = 0; i < 20 && !file; ++i) {
Review comment:
you could use `verifyEventHappenedInPollTime` (in
IntegrationTestUtils.h) for this; also in two other cases in RepoTests
##
File path: libminifi/include/io/tls/TLSServerSocket.h
##
@@ -53,7 +53,7 @@ class TLSServerSocket : public BaseServerSocket, public
TLSSocket {
/**
* Registers a call back and starts the read for the server socket.
*/
- void registerCallback(std::function accept_function,
std::function*, int *)> handler);
+ void registerCallback(std::function accept_function,
std::function*, int *)> handler, uint16_t timeout =
3000);
Review comment:
using `std::chrono::milliseconds` would be clearer than `uint16_t`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8091) Clear state for all processors within processor group
Martin created NIFI-8091: Summary: Clear state for all processors within processor group Key: NIFI-8091 URL: https://issues.apache.org/jira/browse/NIFI-8091 Project: Apache NiFi Issue Type: New Feature Reporter: Martin Some processors may have a state, for instance QueryDatabaseTable, and from time to time you may have to clear state. If you have a processor group with many processors you may need to clear state of many processors which can be tedious. It would be nice if you could right click a processor group and select clear state for all processors. Even better would be if you could right click a processor group and select from a list of processors which one you want to clear state -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8090) Force run processor / processor group
Martin created NIFI-8090: Summary: Force run processor / processor group Key: NIFI-8090 URL: https://issues.apache.org/jira/browse/NIFI-8090 Project: Apache NiFi Issue Type: New Feature Reporter: Martin If you schedule a processor using Timer driven scheduling strategy, you can stop / start a processor or processor group to force it to start now. But if you have scheduled processors using CRON driven scheduling strategy, and you for some reason need to run it _now_, you have to change to Timer driven strategy, stop/start and revert to CRON driven schedule when it is finished. It would be nice to be able to right click a processor or processor group, and select force run / force start to run it once. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] tpalfy commented on a change in pull request #4721: NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP
tpalfy commented on a change in pull request #4721:
URL: https://github.com/apache/nifi/pull/4721#discussion_r542506658
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFileTransfer.java
##
@@ -69,7 +72,10 @@
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.defaultValue(".")
.build();
-
+public static final PropertyDescriptor FILE_TANSFER_LISTING_STRATEGY = new
PropertyDescriptor.Builder()
Review comment:
Indeed, fixing it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a change in pull request #4721: NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP
tpalfy commented on a change in pull request #4721:
URL: https://github.com/apache/nifi/pull/4721#discussion_r542505450
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFileTransfer.java
##
@@ -138,4 +144,19 @@ protected boolean isListingResetNecessary(final
PropertyDescriptor property) {
protected abstract FileTransfer getFileTransfer(final ProcessContext
context);
protected abstract String getProtocolName();
+
+protected void validateAdjustedTimeWindow(ValidationContext
validationContext, Collection results) {
+if (
+
BY_ADJUSTED_TIME_WINDOW.getValue().equals(validationContext.getProperty(LISTING_STRATEGY).getValue())
Review comment:
Yeah, I'll add that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-8089) Run Schedule to support Parameter Context
[ https://issues.apache.org/jira/browse/NIFI-8089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin updated NIFI-8089: - Description: It would be nice to be able to set run schedules using parameter context. Use case: assume you have implemented some form of DevOps, and you are running 2 or more nifi deployments (dev/prod) with the same data flows versioned through NiFi Registry. You can use parameter context to query different databases etc. It would be nice to be able to set Run Schedule as parameter as well, in case you want different run schedules in dev and prod was: It would be nice to be able to set run schedules using parameter context. One use case is this, assume you have implemented some form of DevOps, and you are running 2 or more nifi deployments (dev/prod) with the same data flows versioned through NiFi Registry. You can use parameter context to query different databases etc. It would be nice to have run schedule as parameter as well, in case you want different run schedules in dev and prod > Run Schedule to support Parameter Context > -- > > Key: NIFI-8089 > URL: https://issues.apache.org/jira/browse/NIFI-8089 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Martin >Priority: Minor > > It would be nice to be able to set run schedules using parameter context. > Use case: assume you have implemented some form of DevOps, and you are > running 2 or more nifi deployments (dev/prod) with the same data flows > versioned through NiFi Registry. You can use parameter context to query > different databases etc. It would be nice to be able to set Run Schedule as > parameter as well, in case you want different run schedules in dev and prod -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8089) Run Schedule to support Parameter Context
[ https://issues.apache.org/jira/browse/NIFI-8089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin updated NIFI-8089: - Priority: Minor (was: Major) > Run Schedule to support Parameter Context > -- > > Key: NIFI-8089 > URL: https://issues.apache.org/jira/browse/NIFI-8089 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Martin >Priority: Minor > > It would be nice to be able to set run schedules using parameter context. > One use case is this, assume you have implemented some form of DevOps, and > you are running 2 or more nifi deployments (dev/prod) with the same data > flows versioned through NiFi Registry. You can use parameter context to query > different databases etc. It would be nice to have run schedule as parameter > as well, in case you want different run schedules in dev and prod -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8089) Run Schedule to support Parameter Context
Martin created NIFI-8089: Summary: Run Schedule to support Parameter Context Key: NIFI-8089 URL: https://issues.apache.org/jira/browse/NIFI-8089 Project: Apache NiFi Issue Type: New Feature Reporter: Martin It would be nice to be able to set run schedules using parameter context. One use case is this, assume you have implemented some form of DevOps, and you are running 2 or more nifi deployments (dev/prod) with the same data flows versioned through NiFi Registry. You can use parameter context to query different databases etc. It would be nice to have run schedule as parameter as well, in case you want different run schedules in dev and prod -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] exceptionfactory opened a new pull request #4727: NIFI-8088 Removed deprecation warning log for PKCS12 trust stores
exceptionfactory opened a new pull request #4727:
URL: https://github.com/apache/nifi/pull/4727
Description of PR
NIFI-8088 Removes KeyStoreUtils.getTrustStore() method which logged a
deprecation warning for PKCS12 trust stores. JEP 229 changed the default key
store type to PKCS12 starting in Java 9. The getTrustStore() method checked
the trust store type and delegated to the getKeyStore() method, so this PR
refactors replaces references to getTrustStore() with getKeyStore() to provide
the same functionality.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [X] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [X] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [X] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a change in pull request #4721: NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP
tpalfy commented on a change in pull request #4721:
URL: https://github.com/apache/nifi/pull/4721#discussion_r542471394
##
File path:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java
##
@@ -207,6 +225,39 @@
.defaultValue(BY_TIMESTAMPS.getValue())
.build();
+public static final PropertyDescriptor TIME_ADJUSTMENT = new Builder()
+.name("time-adjustment")
+.displayName("Time Adjustment")
+.description("If the system hosting the files is in a different time
zone than NiFi, either it's timezone or the numerical difference should be set
here." +
+" If a timezone is specified, NiFi tries to calculate the time
difference." +
+" If a numeric value is set, its value can be either a single
integer (milliseconds) or in HH:mm/HH:mm:ss format." +
+" EXAMPLE: NiFi is hosted in UTC, File Server is hosted in EST. In
this case 'Time Adjustment' value should be -05:00:00 or 1800." +
Review comment:
I think File Server is adequate. The main point is handling time zone
difference. The whole concept is realistically applicable only with remotely
hosted files.
Edit: To clarify: the new strategy theoretically could be applied to local
file systems as well. "File Server" is only mentioned in the example though,
which focuses on a scenario where there _is_ a time zone difference. In that
scenario "File Server" is adequate. (In other places I use the term "system
hosting the files".)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory edited a comment on pull request #4508: NIFI-6576 add basic auth to confluent schema registry service
exceptionfactory edited a comment on pull request #4508: URL: https://github.com/apache/nifi/pull/4508#issuecomment-744524769 @mark-weghorst Thanks for responding to comments and making updates. Regarding the dependsOn properties, the current version of the Developer Guide has some helpful documentation under [Validating Processor Properties](https://github.com/apache/nifi/blob/d773521ee01f3a34fdd21365a86c8de7b82077a6/nifi-docs/src/main/asciidoc/developer-guide.adoc#validating-processor-properties). Following the example described, the `Username` and `Password` properties described could include a `dependsOn(PROP_AUTHENTICATION_TYPE, "BASIC")` builder method which would make them dependent on selecting `BASIC` as the Authentication Type property. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #4508: NIFI-6576 add basic auth to confluent schema registry service
exceptionfactory commented on pull request #4508: URL: https://github.com/apache/nifi/pull/4508#issuecomment-744524769 @mark-weghorst Thanks for responding to comments and making updates. Regarding the dependsOn properties, the current version of the Developer Guide has some helpful documentation under [Validating Processor Properties.](https://github.com/apache/nifi/blob/d773521ee01f3a34fdd21365a86c8de7b82077a6/nifi-docs/src/main/asciidoc/developer-guide.adoc#validating-processor-properties). Following the example described, the `Username` and `Password` properties described could include a `dependsOn(PROP_AUTHENTICATION_TYPE, "BASIC")` builder method which would make them dependent on selecting `BASIC` as the Authentication Type property. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a change in pull request #4721: NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP
tpalfy commented on a change in pull request #4721:
URL: https://github.com/apache/nifi/pull/4721#discussion_r542483971
##
File path:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java
##
@@ -207,6 +225,39 @@
.defaultValue(BY_TIMESTAMPS.getValue())
.build();
+public static final PropertyDescriptor TIME_ADJUSTMENT = new Builder()
+.name("time-adjustment")
+.displayName("Time Adjustment")
+.description("If the system hosting the files is in a different time
zone than NiFi, either it's timezone or the numerical difference should be set
here." +
+" If a timezone is specified, NiFi tries to calculate the time
difference." +
+" If a numeric value is set, its value can be either a single
integer (milliseconds) or in HH:mm/HH:mm:ss format." +
+" EXAMPLE: NiFi is hosted in UTC, File Server is hosted in EST. In
this case 'Time Adjustment' value should be -05:00:00 or 1800." +
+" If the locations were reversed i.e. NiFi is hosted in EST, File
Server is hosted in UTC, the value should be 05:00:00 or 1800." +
+" NOTE: Any mid-year changes (due to daylight saving for example)
requires manual re-adjustment in this case."
+)
+.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.required(false)
+.addValidator(new Validator() {
+Pattern signed_integer_or_signed_HHmm_or_HHmmss =
Pattern.compile("-?(\\d{2}:\\d{2}(:\\d{2})?)|-?\\d+");
Review comment:
I think it's actually better for anyone who reads to code to see the
validation expression right here in the validator.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a change in pull request #4721: NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP
tpalfy commented on a change in pull request #4721:
URL: https://github.com/apache/nifi/pull/4721#discussion_r542481674
##
File path:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java
##
@@ -431,11 +482,157 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
} else if (BY_ENTITIES.equals(listingStrategy)) {
listByTrackingEntities(context, session);
+} else if (BY_ADJUSTED_TIME_WINDOW.equals(listingStrategy)) {
+listByAdjustedSlidingTimeWindow(context, session);
+
} else {
throw new ProcessException("Unknown listing strategy: " +
listingStrategy);
}
}
+public void listByAdjustedSlidingTimeWindow(final ProcessContext context,
final ProcessSession session) throws ProcessException {
+if (this.lastListedLatestEntryTimestampMillis == null ||
justElectedPrimaryNode) {
+try {
+final StateMap stateMap =
context.getStateManager().getState(getStateScope(context));
+
Optional.ofNullable(stateMap.get(LATEST_LISTED_ENTRY_TIMESTAMP_KEY))
+.map(Long::parseLong)
+.ifPresent(lastTimestamp ->
this.lastListedLatestEntryTimestampMillis = lastTimestamp);
+
+justElectedPrimaryNode = false;
+} catch (final IOException ioe) {
+getLogger().error("Failed to retrieve timestamp of last
listing from the State Manager. Will not perform listing until this is
accomplished.");
+context.yield();
+return;
+}
+}
+
+long lowerBoundInclusiveTimestamp =
Optional.ofNullable(this.lastListedLatestEntryTimestampMillis).orElse(0L);
+long upperBoundExclusiveTimestamp;
+
+long currentTime = getCurrentTime();
+
+final TreeMap> orderedEntries = new TreeMap<>();
+try {
+List entityList = performListing(context,
lowerBoundInclusiveTimestamp);
+
+boolean targetSystemHasMilliseconds = false;
+boolean targetSystemHasSeconds = false;
+for (final T entity : entityList) {
+final long entityTimestampMillis = entity.getTimestamp();
+if (!targetSystemHasMilliseconds) {
+targetSystemHasMilliseconds = entityTimestampMillis % 1000
> 0;
+}
+if (!targetSystemHasSeconds) {
+targetSystemHasSeconds = entityTimestampMillis % 60_000 >
0;
+}
+}
+
+// Determine target system time precision.
+String specifiedPrecision =
context.getProperty(TARGET_SYSTEM_TIMESTAMP_PRECISION).getValue();
+if (StringUtils.isBlank(specifiedPrecision)) {
+// If TARGET_SYSTEM_TIMESTAMP_PRECISION is not supported by
the Processor, then specifiedPrecision can be null, instead of its default
value.
+specifiedPrecision = getDefaultTimePrecision();
+}
+final TimeUnit targetSystemTimePrecision
+=
PRECISION_AUTO_DETECT.getValue().equals(specifiedPrecision)
+? targetSystemHasMilliseconds ? TimeUnit.MILLISECONDS
: targetSystemHasSeconds ? TimeUnit.SECONDS : TimeUnit.MINUTES
+: PRECISION_MILLIS.getValue().equals(specifiedPrecision) ?
TimeUnit.MILLISECONDS
+: PRECISION_SECONDS.getValue().equals(specifiedPrecision)
? TimeUnit.SECONDS : TimeUnit.MINUTES;
+final Long listingLagMillis =
LISTING_LAG_MILLIS.get(targetSystemTimePrecision);
+
+upperBoundExclusiveTimestamp =
getAdjustedCurrentTimestamp(context, currentTime) - listingLagMillis;
+
+if (getLogger().isTraceEnabled()) {
+getLogger().trace("interval: " + lowerBoundInclusiveTimestamp
+ " - " + upperBoundExclusiveTimestamp);
+getLogger().trace("entityList: " +
entityList.stream().map(entity -> entity.getName() + "_" +
entity.getTimestamp()).collect(Collectors.joining(", ")));
+}
+entityList
+.stream()
+.filter(entity -> entity.getTimestamp() >=
lowerBoundInclusiveTimestamp)
+.filter(entity -> entity.getTimestamp() <
upperBoundExclusiveTimestamp)
+.forEach(entity -> orderedEntries
+.computeIfAbsent(entity.getTimestamp(), __ -> new
ArrayList<>())
+.add(entity)
+);
+if (getLogger().isTraceEnabled()) {
+getLogger().trace("orderedEntries: " +
+orderedEntries.values().stream()
+.flatMap(List::stream)
+.map(entity -> entity.getName() + "_" +
entity.getTimestamp())
+.collect(Collectors.joining(", "))
+);
+
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4508: NIFI-6576 add basic auth to confluent schema registry service
exceptionfactory commented on a change in pull request #4508:
URL: https://github.com/apache/nifi/pull/4508#discussion_r542479942
##
File path:
nifi-nar-bundles/nifi-confluent-platform-bundle/nifi-confluent-schema-registry-service/src/main/java/org/apache/nifi/confluent/schemaregistry/ConfluentSchemaRegistry.java
##
@@ -112,6 +114,32 @@
.required(true)
.build();
+static final PropertyDescriptor PROP_BASIC_AUTH_USERNAME = new
PropertyDescriptor.Builder()
+.name("Authentication Username")
+.displayName("Authentication Username")
+.description("The username to be used by the client to authenticate
against the Remote URL. Cannot include control characters (0-31), ':', or DEL
(127).")
+.required(false)
+
.addValidator(StandardValidators.createRegexMatchingValidator(Pattern.compile("^[\\x20-\\x39\\x3b-\\x7e\\x80-\\xff]+$")))
+.build();
+
+static final PropertyDescriptor PROP_BASIC_AUTH_PASSWORD = new
PropertyDescriptor.Builder()
+.name("Authentication Password")
+.displayName("Authentication Password")
+.description("The password to be used by the client to authenticate
against the Remote URL.")
+.required(false)
+.sensitive(true)
+
.addValidator(StandardValidators.createRegexMatchingValidator(Pattern.compile("^[\\x20-\\x7e\\x80-\\xff]+$")))
+.build();
+
+static final PropertyDescriptor PROP_AUTH_TYPE = new
PropertyDescriptor.Builder()
+.name("Authentication Type")
+.displayName("Authentication Type")
+.description("Basic authentication will use the 'Authentication
Username' " +
+"and 'Authentication Password' property values. See Confluent
Schema Registry documentation for more details.")
+.required(false)
+.allowableValues("BASIC")
Review comment:
The [Confluent Server
documentation](https://docs.confluent.io/platform/current/kafka-rest/production-deployment/confluent-server/security.html#http-basic-authentication)
lists BASIC and NONE as options for the authentication method for certain
clients. Following that approach and having a default value of NONE, in
combination with an AuthenticationType enum, would help make the configuration
properties and implementation approach clearer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a change in pull request #4721: NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP
tpalfy commented on a change in pull request #4721:
URL: https://github.com/apache/nifi/pull/4721#discussion_r542476852
##
File path:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java
##
@@ -431,11 +482,157 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
} else if (BY_ENTITIES.equals(listingStrategy)) {
listByTrackingEntities(context, session);
+} else if (BY_ADJUSTED_TIME_WINDOW.equals(listingStrategy)) {
+listByAdjustedSlidingTimeWindow(context, session);
+
} else {
throw new ProcessException("Unknown listing strategy: " +
listingStrategy);
}
}
+public void listByAdjustedSlidingTimeWindow(final ProcessContext context,
final ProcessSession session) throws ProcessException {
+if (this.lastListedLatestEntryTimestampMillis == null ||
justElectedPrimaryNode) {
+try {
+final StateMap stateMap =
context.getStateManager().getState(getStateScope(context));
+
Optional.ofNullable(stateMap.get(LATEST_LISTED_ENTRY_TIMESTAMP_KEY))
+.map(Long::parseLong)
+.ifPresent(lastTimestamp ->
this.lastListedLatestEntryTimestampMillis = lastTimestamp);
+
+justElectedPrimaryNode = false;
+} catch (final IOException ioe) {
+getLogger().error("Failed to retrieve timestamp of last
listing from the State Manager. Will not perform listing until this is
accomplished.");
+context.yield();
+return;
+}
+}
+
+long lowerBoundInclusiveTimestamp =
Optional.ofNullable(this.lastListedLatestEntryTimestampMillis).orElse(0L);
+long upperBoundExclusiveTimestamp;
+
+long currentTime = getCurrentTime();
+
+final TreeMap> orderedEntries = new TreeMap<>();
+try {
+List entityList = performListing(context,
lowerBoundInclusiveTimestamp);
+
+boolean targetSystemHasMilliseconds = false;
+boolean targetSystemHasSeconds = false;
+for (final T entity : entityList) {
+final long entityTimestampMillis = entity.getTimestamp();
+if (!targetSystemHasMilliseconds) {
+targetSystemHasMilliseconds = entityTimestampMillis % 1000
> 0;
+}
+if (!targetSystemHasSeconds) {
+targetSystemHasSeconds = entityTimestampMillis % 60_000 >
0;
+}
+}
+
+// Determine target system time precision.
+String specifiedPrecision =
context.getProperty(TARGET_SYSTEM_TIMESTAMP_PRECISION).getValue();
+if (StringUtils.isBlank(specifiedPrecision)) {
+// If TARGET_SYSTEM_TIMESTAMP_PRECISION is not supported by
the Processor, then specifiedPrecision can be null, instead of its default
value.
+specifiedPrecision = getDefaultTimePrecision();
+}
+final TimeUnit targetSystemTimePrecision
+=
PRECISION_AUTO_DETECT.getValue().equals(specifiedPrecision)
+? targetSystemHasMilliseconds ? TimeUnit.MILLISECONDS
: targetSystemHasSeconds ? TimeUnit.SECONDS : TimeUnit.MINUTES
+: PRECISION_MILLIS.getValue().equals(specifiedPrecision) ?
TimeUnit.MILLISECONDS
+: PRECISION_SECONDS.getValue().equals(specifiedPrecision)
? TimeUnit.SECONDS : TimeUnit.MINUTES;
+final Long listingLagMillis =
LISTING_LAG_MILLIS.get(targetSystemTimePrecision);
+
+upperBoundExclusiveTimestamp =
getAdjustedCurrentTimestamp(context, currentTime) - listingLagMillis;
+
+if (getLogger().isTraceEnabled()) {
+getLogger().trace("interval: " + lowerBoundInclusiveTimestamp
+ " - " + upperBoundExclusiveTimestamp);
+getLogger().trace("entityList: " +
entityList.stream().map(entity -> entity.getName() + "_" +
entity.getTimestamp()).collect(Collectors.joining(", ")));
+}
+entityList
+.stream()
+.filter(entity -> entity.getTimestamp() >=
lowerBoundInclusiveTimestamp)
+.filter(entity -> entity.getTimestamp() <
upperBoundExclusiveTimestamp)
+.forEach(entity -> orderedEntries
+.computeIfAbsent(entity.getTimestamp(), __ -> new
ArrayList<>())
+.add(entity)
+);
+if (getLogger().isTraceEnabled()) {
+getLogger().trace("orderedEntries: " +
+orderedEntries.values().stream()
+.flatMap(List::stream)
+.map(entity -> entity.getName() + "_" +
entity.getTimestamp())
+.collect(Collectors.joining(", "))
+);
+
[GitHub] [nifi] tpalfy commented on a change in pull request #4721: NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP
tpalfy commented on a change in pull request #4721:
URL: https://github.com/apache/nifi/pull/4721#discussion_r542476249
##
File path:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java
##
@@ -431,11 +482,157 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
} else if (BY_ENTITIES.equals(listingStrategy)) {
listByTrackingEntities(context, session);
+} else if (BY_ADJUSTED_TIME_WINDOW.equals(listingStrategy)) {
+listByAdjustedSlidingTimeWindow(context, session);
+
} else {
throw new ProcessException("Unknown listing strategy: " +
listingStrategy);
}
}
+public void listByAdjustedSlidingTimeWindow(final ProcessContext context,
final ProcessSession session) throws ProcessException {
+if (this.lastListedLatestEntryTimestampMillis == null ||
justElectedPrimaryNode) {
+try {
+final StateMap stateMap =
context.getStateManager().getState(getStateScope(context));
+
Optional.ofNullable(stateMap.get(LATEST_LISTED_ENTRY_TIMESTAMP_KEY))
+.map(Long::parseLong)
+.ifPresent(lastTimestamp ->
this.lastListedLatestEntryTimestampMillis = lastTimestamp);
+
+justElectedPrimaryNode = false;
+} catch (final IOException ioe) {
+getLogger().error("Failed to retrieve timestamp of last
listing from the State Manager. Will not perform listing until this is
accomplished.");
+context.yield();
+return;
+}
+}
+
+long lowerBoundInclusiveTimestamp =
Optional.ofNullable(this.lastListedLatestEntryTimestampMillis).orElse(0L);
+long upperBoundExclusiveTimestamp;
+
+long currentTime = getCurrentTime();
+
+final TreeMap> orderedEntries = new TreeMap<>();
+try {
+List entityList = performListing(context,
lowerBoundInclusiveTimestamp);
+
+boolean targetSystemHasMilliseconds = false;
+boolean targetSystemHasSeconds = false;
+for (final T entity : entityList) {
Review comment:
I considered it and decided to leave it as it. I rarely do this but in
this case it would be either too risky or would require significant effort to
add additional proper test coverage.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4508: NIFI-6576 add basic auth to confluent schema registry service
exceptionfactory commented on a change in pull request #4508:
URL: https://github.com/apache/nifi/pull/4508#discussion_r542475418
##
File path:
nifi-nar-bundles/nifi-confluent-platform-bundle/nifi-confluent-schema-registry-service/src/main/java/org/apache/nifi/confluent/schemaregistry/client/RestSchemaRegistryClient.java
##
@@ -68,14 +69,21 @@
private static final String SCHEMA_REGISTRY_CONTENT_TYPE =
"application/vnd.schemaregistry.v1+json";
-public RestSchemaRegistryClient(final List baseUrls, final int
timeoutMillis, final SSLContext sslContext, final ComponentLog logger) {
+public RestSchemaRegistryClient(final List baseUrls, final String
authType, final String authUser,
+final String authPass, final int
timeoutMillis, final SSLContext sslContext, final ComponentLog logger) {
this.baseUrls = new ArrayList<>(baseUrls);
final ClientConfig clientConfig = new ClientConfig();
clientConfig.property(ClientProperties.CONNECT_TIMEOUT, timeoutMillis);
clientConfig.property(ClientProperties.READ_TIMEOUT, timeoutMillis);
client = WebUtils.createClient(clientConfig, sslContext);
+if (!authUser.isEmpty() && !authType.isEmpty()) {
+if (authType.equals("BASIC")) {
Review comment:
The [Confluent Server
documentation](https://docs.confluent.io/platform/current/kafka-rest/production-deployment/confluent-server/security.html)
mentions HTTP Basic and mutual TLS, as well as additional methods for
SASL-based authentication between components. Maintaining the parameter makes
sense given that background.
Rather than relying on a particular string value, however, adding an
AuthenticationType enum with the values of BASIC and NONE would help make the
implementation clearer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request #959: MINIFICPP-1424 Stabilize several flaky test suites
lordgamez opened a new pull request #959: URL: https://github.com/apache/nifi-minifi-cpp/pull/959 This PR addresses several issues in tests, addressed by each commit, so it is advised to review on a per-commit basis. Fixes included for the following test suites (more information in the commit messages): - FileUtilsTests - Fix for last_write_time anomalies - RepoTests and PersistanceTests - Separate checkpoint usage when running in parallel - SecureSocketGetTCP tests - Fix timing issues - TailFileTests - Fix random generation and timing problems - GetTCPTests - Added missing ReconnectInterval property - Fixed recv error checking on Windows to properly try to reconnect and adjusted reconnection property values - Flow-Tests - Replace state manager provider with UnorderedMapPersistableKeyValueStoreService to avoid parallel running issues - DBProvenanceRepositoryTests - Remove an unreliable check - Fedora build - Added check for fastest mirror and retry mechanism for curl timeout problems - FlowControllerTests - Adjusted timeouts --- Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a change in pull request #4721: NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP
tpalfy commented on a change in pull request #4721:
URL: https://github.com/apache/nifi/pull/4721#discussion_r542471394
##
File path:
nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java
##
@@ -207,6 +225,39 @@
.defaultValue(BY_TIMESTAMPS.getValue())
.build();
+public static final PropertyDescriptor TIME_ADJUSTMENT = new Builder()
+.name("time-adjustment")
+.displayName("Time Adjustment")
+.description("If the system hosting the files is in a different time
zone than NiFi, either it's timezone or the numerical difference should be set
here." +
+" If a timezone is specified, NiFi tries to calculate the time
difference." +
+" If a numeric value is set, its value can be either a single
integer (milliseconds) or in HH:mm/HH:mm:ss format." +
+" EXAMPLE: NiFi is hosted in UTC, File Server is hosted in EST. In
this case 'Time Adjustment' value should be -05:00:00 or 1800." +
Review comment:
I think File Server is adequate. The main point is handling time zone
difference. The whole concept is realistically applicable only with remotely
hosted files.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
adamdebreceni commented on a change in pull request #958: URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542465018 ## File path: libminifi/include/controllers/NetworkPrioritizerService.h ## @@ -121,6 +126,7 @@ class NetworkPrioritizerService : public core::controller::ControllerService, pu bool verify_interfaces_; private: + std::shared_ptr clock_; Review comment: info(not ment to argue either for or against) : if we want to reference a member of a `shared_ptr` pointee, and not accidentally mess up, we can share the lifetime of the pointee while pointing to the member, so the "container" is kept alive even if we are only using its member This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
szaszm commented on a change in pull request #958:
URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542450733
##
File path: libminifi/include/utils/TimeUtil.h
##
@@ -53,6 +53,29 @@ inline uint64_t getTimeNano() {
return
std::chrono::duration_cast(std::chrono::system_clock::now().time_since_epoch()).count();
}
+/**
+ * Mockable clock classes
+ */
+class Clock {
+ public:
+ virtual ~Clock() = default;
+ virtual std::chrono::milliseconds timeSinceEpoch() const = 0;
Review comment:
Currently I see the `Clock` class hierarchy as just related functions in
disguise. This change would allow us to get rid of the classes and replace them
with free functions (e.g. `get_time`, `get_steady_clock_time` and a lambda with
a captured local for the manual case), which is closer to the truth IMO.
I think functions are simpler than classes (one operation vs multiple
operations + data), so for my code, I would opt for that approach, but I'm
totally fine with you not sharing my viewpoint and keeping the code as is.
For the readability argument, I'm arguing that the usage we describe is
retrieving the time, and that can be described by naming the functions and the
type-erased member something along the lines of `get_time_`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (MINIFICPP-1424) Stabilize flaky test cases
[ https://issues.apache.org/jira/browse/MINIFICPP-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Gyimesi reassigned MINIFICPP-1424: Assignee: Gabor Gyimesi > Stabilize flaky test cases > -- > > Key: MINIFICPP-1424 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1424 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Gabor Gyimesi >Assignee: Gabor Gyimesi >Priority: Major > Labels: MiNiFi-CPP-Hygiene > Time Spent: 20m > Remaining Estimate: 0h > > Multiple test cases have been failing intermittently in the CI environment. > The root causes need to be identified and fixed to have a stable and reliable > test suite for the project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1424) Stabilize several flaky test suites
[ https://issues.apache.org/jira/browse/MINIFICPP-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Gyimesi updated MINIFICPP-1424: - Summary: Stabilize several flaky test suites (was: Stabilize flaky test cases) > Stabilize several flaky test suites > --- > > Key: MINIFICPP-1424 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1424 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Gabor Gyimesi >Assignee: Gabor Gyimesi >Priority: Major > Labels: MiNiFi-CPP-Hygiene > Time Spent: 20m > Remaining Estimate: 0h > > Multiple test cases have been failing intermittently in the CI environment. > The root causes need to be identified and fixed to have a stable and reliable > test suite for the project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (MINIFICPP-1433) Price analysis
Mtabcom created MINIFICPP-1433:
--
Summary: Price analysis
Key: MINIFICPP-1433
URL: https://issues.apache.org/jira/browse/MINIFICPP-1433
Project: Apache NiFi MiNiFi C++
Issue Type: Bug
Reporter: Mtabcom
Attachments: تكرتشارت نت
{quote}[link title|http://example.com]
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
fgerlits commented on a change in pull request #958:
URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542436009
##
File path: libminifi/include/utils/TimeUtil.h
##
@@ -53,6 +53,29 @@ inline uint64_t getTimeNano() {
return
std::chrono::duration_cast(std::chrono::system_clock::now().time_since_epoch()).count();
}
+/**
+ * Mockable clock classes
+ */
+class Clock {
+ public:
+ virtual ~Clock() = default;
+ virtual std::chrono::milliseconds timeSinceEpoch() const = 0;
Review comment:
Yes, that would be another way of doing it, but I'm not sure it would be
simpler. Do you think it would be so much simpler that it's worth rewriting
this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
fgerlits commented on a change in pull request #958: URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542434970 ## File path: libminifi/include/controllers/NetworkPrioritizerService.h ## @@ -121,6 +126,7 @@ class NetworkPrioritizerService : public core::controller::ControllerService, pu bool verify_interfaces_; private: + std::shared_ptr clock_; Review comment: The service and the unit test. If only the service owned it, the unit test would need to have a naked pointer (or reference) to the clock, and rely on the order of the operations to make sure it remains valid -- I have considered this, but did not like it. Since the `clock_` field is only created once, and never copied, there isn't much of an overhead. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
adamdebreceni commented on a change in pull request #958:
URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542433430
##
File path: libminifi/include/utils/TimeUtil.h
##
@@ -53,6 +53,29 @@ inline uint64_t getTimeNano() {
return
std::chrono::duration_cast(std::chrono::system_clock::now().time_since_epoch()).count();
}
+/**
+ * Mockable clock classes
+ */
+class Clock {
+ public:
+ virtual ~Clock() = default;
+ virtual std::chrono::milliseconds timeSinceEpoch() const = 0;
Review comment:
regarding the `std::function<...>`, I would keep the `Clock` class, as
IMO it better describes its usage
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
adamdebreceni commented on a change in pull request #958:
URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542423177
##
File path: libminifi/include/utils/TimeUtil.h
##
@@ -53,6 +53,29 @@ inline uint64_t getTimeNano() {
return
std::chrono::duration_cast(std::chrono::system_clock::now().time_since_epoch()).count();
}
+/**
+ * Mockable clock classes
+ */
+class Clock {
+ public:
+ virtual ~Clock() = default;
+ virtual std::chrono::milliseconds timeSinceEpoch() const = 0;
Review comment:
also: I wouldn't introduce a `SystemClock` class until it is actually
needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
adamdebreceni commented on a change in pull request #958: URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542420260 ## File path: libminifi/include/controllers/NetworkPrioritizerService.h ## @@ -121,6 +126,7 @@ class NetworkPrioritizerService : public core::controller::ControllerService, pu bool verify_interfaces_; private: + std::shared_ptr clock_; Review comment: in the tests the clock is being advanced externally (if this is the only reason, we should create a TestAccessor, and switch to `unique_ptr`) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on a change in pull request #911: MINIFICPP-1373 - Add proper FlowFile::getAttribute getter [QoL][mini-PR]
hunyadi-dev commented on a change in pull request #911:
URL: https://github.com/apache/nifi-minifi-cpp/pull/911#discussion_r542421625
##
File path: libminifi/include/core/FlowFile.h
##
@@ -129,7 +130,9 @@ class FlowFile : public core::Connectable, public
ReferenceContainer {
* @param value value to set
* @return result of finding key
*/
- bool getAttribute(std::string key, std::string& value) const;
+ bool getAttribute(const std::string& key, std::string& value) const;
+
+ utils::optional>
getAttribute(const std::string& key) const;
Review comment:
I am not arguing for safety, to plut it blunt I just do not like
unnecessary copies.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
szaszm commented on a change in pull request #958:
URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542421731
##
File path: libminifi/include/utils/TimeUtil.h
##
@@ -53,6 +53,29 @@ inline uint64_t getTimeNano() {
return
std::chrono::duration_cast(std::chrono::system_clock::now().time_since_epoch()).count();
}
+/**
+ * Mockable clock classes
+ */
+class Clock {
+ public:
+ virtual ~Clock() = default;
+ virtual std::chrono::milliseconds timeSinceEpoch() const = 0;
Review comment:
This could be type erased with
`std::function` instead of a class hierarchy for
simplicity.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
adamdebreceni commented on a change in pull request #958: URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542420260 ## File path: libminifi/include/controllers/NetworkPrioritizerService.h ## @@ -121,6 +126,7 @@ class NetworkPrioritizerService : public core::controller::ControllerService, pu bool verify_interfaces_; private: + std::shared_ptr clock_; Review comment: in the tests the clock is being advanced externally This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
szaszm commented on a change in pull request #958: URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542418714 ## File path: libminifi/include/controllers/NetworkPrioritizerService.h ## @@ -121,6 +126,7 @@ class NetworkPrioritizerService : public core::controller::ControllerService, pu bool verify_interfaces_; private: + std::shared_ptr clock_; Review comment: Why shared_ptr, what are the other owners? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
adamdebreceni commented on a change in pull request #958:
URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542405642
##
File path: libminifi/test/unit/NetworkPrioritizerServiceTests.cpp
##
@@ -41,129 +47,105 @@ TEST_CASE("TestPrioritizerOneInterface", "[test1]") {
}
TEST_CASE("TestPrioritizerOneInterfaceMaxPayload", "[test2]") {
- auto controller =
std::make_shared("TestService");
- std::shared_ptr configuration =
std::make_shared();
+ auto controller = createNetworkPrioritizerService("TestService");
controller->initialize();
controller->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0,eth1");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"1 B");
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxPayload,
"1 B");
+
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"1 kB");
+
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxPayload,
"10 B");
controller->onEnable();
- // can't because we've triggered the max payload
- REQUIRE("" == controller->getInterface(5).getInterface());
+
+ REQUIRE("eth0" == controller->getInterface(5).getInterface());
+ REQUIRE("" == controller->getInterface(20).getInterface()); // larger than
max payload
+ REQUIRE("eth0" == controller->getInterface(5).getInterface());
}
TEST_CASE("TestPrioritizerOneInterfaceMaxThroughput", "[test3]") {
- auto controller =
std::make_shared("TestService");
- std::shared_ptr configuration =
std::make_shared();
+ auto clock = std::make_shared();
+ auto controller = createNetworkPrioritizerService("TestService", clock);
controller->initialize();
controller->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0,eth1");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
controller->onEnable();
- // can't because we've triggered the max payload
REQUIRE("eth0" == controller->getInterface(5).getInterface());
REQUIRE("eth0" == controller->getInterface(5).getInterface());
- REQUIRE("" == controller->getInterface(5).getInterface());
- std::this_thread::sleep_for(std::chrono::milliseconds(10));
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
+ REQUIRE("" == controller->getInterface(5).getInterface()); // max
throughput reached
+ clock->advance(std::chrono::milliseconds{10}); // wait for more tokens to
be generated
+ REQUIRE("eth0" == controller->getInterface(5).getInterface()); // now we
can send again
}
TEST_CASE("TestPriorotizerMultipleInterfaces", "[test4]") {
-
LogTestController::getInstance().setTrace();
-
- auto controller =
std::make_shared("TestService");
- auto controller2 =
std::make_shared("TestService2");
- auto controller3 =
std::make_shared("TestService3");
+ auto clock = std::make_shared();
+ auto parent_controller = createNetworkPrioritizerService("TestService",
clock);
std::shared_ptr configuration =
std::make_shared();
- controller->initialize();
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
+ parent_controller->initialize();
+
parent_controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
- controller3->initialize();
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0");
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
- controller3->onEnable();
-
- controller2->initialize();
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth1");
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
- controller2->onEnable();
- std::vector > services;
- services.push_back(controller2);
- services.push_back(controller3);
- controller->setLinkedControllerServices(services);
- controller->onEnable();
- // can't because we've triggered the max payload
- REQUIRE("eth1" == controller->getInterface(5).getInterface());
- REQUIRE("eth1" == controller->getInterface(5).getInterface());
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
-}
+ auto controller0 = createNetworkPrioritizerService("TestService_eth0",
clock);
+ controller0->initialize();
+
contro
[jira] [Created] (NIFI-8088) Remove PKCS12 Deprecation Warning in KeyStoreUtils
David Handermann created NIFI-8088: -- Summary: Remove PKCS12 Deprecation Warning in KeyStoreUtils Key: NIFI-8088 URL: https://issues.apache.org/jira/browse/NIFI-8088 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.12.1 Reporter: David Handermann Assignee: David Handermann KeyStoreUtils.getTrustStore() logs a warning message indicating the PKCS12 trust stores are deprecated in favor of JKS. JEP 229 changed the default key store type to PKCS12 starting with Java 9. The Site to Site Client, GetHTTP, and PostHTTP Processors use the KeyStoreUtils.getTrustStore() method. The method itself checks the trust store type, logs the warning for PKCS12 and delegates to KeyStoreUtils.getKeyStore(). References to KeyStoreUtils.getTrustStore() should be replaced with KeyStoreUtils.getKeyStore() and the getTrustStore() method should be removed to avoid confusing log warnings in deployments where PKCS12 trust stores are used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
fgerlits commented on a change in pull request #958:
URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542401133
##
File path: libminifi/test/unit/NetworkPrioritizerServiceTests.cpp
##
@@ -41,129 +47,105 @@ TEST_CASE("TestPrioritizerOneInterface", "[test1]") {
}
TEST_CASE("TestPrioritizerOneInterfaceMaxPayload", "[test2]") {
- auto controller =
std::make_shared("TestService");
- std::shared_ptr configuration =
std::make_shared();
+ auto controller = createNetworkPrioritizerService("TestService");
controller->initialize();
controller->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0,eth1");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"1 B");
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxPayload,
"1 B");
+
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"1 kB");
+
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxPayload,
"10 B");
controller->onEnable();
- // can't because we've triggered the max payload
- REQUIRE("" == controller->getInterface(5).getInterface());
+
+ REQUIRE("eth0" == controller->getInterface(5).getInterface());
+ REQUIRE("" == controller->getInterface(20).getInterface()); // larger than
max payload
+ REQUIRE("eth0" == controller->getInterface(5).getInterface());
}
TEST_CASE("TestPrioritizerOneInterfaceMaxThroughput", "[test3]") {
- auto controller =
std::make_shared("TestService");
- std::shared_ptr configuration =
std::make_shared();
+ auto clock = std::make_shared();
+ auto controller = createNetworkPrioritizerService("TestService", clock);
controller->initialize();
controller->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0,eth1");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
controller->onEnable();
- // can't because we've triggered the max payload
REQUIRE("eth0" == controller->getInterface(5).getInterface());
REQUIRE("eth0" == controller->getInterface(5).getInterface());
- REQUIRE("" == controller->getInterface(5).getInterface());
- std::this_thread::sleep_for(std::chrono::milliseconds(10));
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
+ REQUIRE("" == controller->getInterface(5).getInterface()); // max
throughput reached
+ clock->advance(std::chrono::milliseconds{10}); // wait for more tokens to
be generated
+ REQUIRE("eth0" == controller->getInterface(5).getInterface()); // now we
can send again
}
TEST_CASE("TestPriorotizerMultipleInterfaces", "[test4]") {
-
LogTestController::getInstance().setTrace();
-
- auto controller =
std::make_shared("TestService");
- auto controller2 =
std::make_shared("TestService2");
- auto controller3 =
std::make_shared("TestService3");
+ auto clock = std::make_shared();
+ auto parent_controller = createNetworkPrioritizerService("TestService",
clock);
std::shared_ptr configuration =
std::make_shared();
- controller->initialize();
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
+ parent_controller->initialize();
+
parent_controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
- controller3->initialize();
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0");
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
- controller3->onEnable();
-
- controller2->initialize();
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth1");
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
- controller2->onEnable();
- std::vector > services;
- services.push_back(controller2);
- services.push_back(controller3);
- controller->setLinkedControllerServices(services);
- controller->onEnable();
- // can't because we've triggered the max payload
- REQUIRE("eth1" == controller->getInterface(5).getInterface());
- REQUIRE("eth1" == controller->getInterface(5).getInterface());
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
-}
+ auto controller0 = createNetworkPrioritizerService("TestService_eth0",
clock);
+ controller0->initialize();
+
controller0
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
fgerlits commented on a change in pull request #958:
URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542401133
##
File path: libminifi/test/unit/NetworkPrioritizerServiceTests.cpp
##
@@ -41,129 +47,105 @@ TEST_CASE("TestPrioritizerOneInterface", "[test1]") {
}
TEST_CASE("TestPrioritizerOneInterfaceMaxPayload", "[test2]") {
- auto controller =
std::make_shared("TestService");
- std::shared_ptr configuration =
std::make_shared();
+ auto controller = createNetworkPrioritizerService("TestService");
controller->initialize();
controller->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0,eth1");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"1 B");
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxPayload,
"1 B");
+
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"1 kB");
+
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxPayload,
"10 B");
controller->onEnable();
- // can't because we've triggered the max payload
- REQUIRE("" == controller->getInterface(5).getInterface());
+
+ REQUIRE("eth0" == controller->getInterface(5).getInterface());
+ REQUIRE("" == controller->getInterface(20).getInterface()); // larger than
max payload
+ REQUIRE("eth0" == controller->getInterface(5).getInterface());
}
TEST_CASE("TestPrioritizerOneInterfaceMaxThroughput", "[test3]") {
- auto controller =
std::make_shared("TestService");
- std::shared_ptr configuration =
std::make_shared();
+ auto clock = std::make_shared();
+ auto controller = createNetworkPrioritizerService("TestService", clock);
controller->initialize();
controller->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0,eth1");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
controller->onEnable();
- // can't because we've triggered the max payload
REQUIRE("eth0" == controller->getInterface(5).getInterface());
REQUIRE("eth0" == controller->getInterface(5).getInterface());
- REQUIRE("" == controller->getInterface(5).getInterface());
- std::this_thread::sleep_for(std::chrono::milliseconds(10));
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
+ REQUIRE("" == controller->getInterface(5).getInterface()); // max
throughput reached
+ clock->advance(std::chrono::milliseconds{10}); // wait for more tokens to
be generated
+ REQUIRE("eth0" == controller->getInterface(5).getInterface()); // now we
can send again
}
TEST_CASE("TestPriorotizerMultipleInterfaces", "[test4]") {
-
LogTestController::getInstance().setTrace();
-
- auto controller =
std::make_shared("TestService");
- auto controller2 =
std::make_shared("TestService2");
- auto controller3 =
std::make_shared("TestService3");
+ auto clock = std::make_shared();
+ auto parent_controller = createNetworkPrioritizerService("TestService",
clock);
std::shared_ptr configuration =
std::make_shared();
- controller->initialize();
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
+ parent_controller->initialize();
+
parent_controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
- controller3->initialize();
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0");
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
- controller3->onEnable();
-
- controller2->initialize();
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth1");
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
- controller2->onEnable();
- std::vector > services;
- services.push_back(controller2);
- services.push_back(controller3);
- controller->setLinkedControllerServices(services);
- controller->onEnable();
- // can't because we've triggered the max payload
- REQUIRE("eth1" == controller->getInterface(5).getInterface());
- REQUIRE("eth1" == controller->getInterface(5).getInterface());
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
-}
+ auto controller0 = createNetworkPrioritizerService("TestService_eth0",
clock);
+ controller0->initialize();
+
controller0
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #958: MINIFICPP-1432 Remove the timing-sensitivity of NetworkPrioritizerServiceTests
adamdebreceni commented on a change in pull request #958:
URL: https://github.com/apache/nifi-minifi-cpp/pull/958#discussion_r542346927
##
File path: libminifi/test/unit/NetworkPrioritizerServiceTests.cpp
##
@@ -41,129 +47,105 @@ TEST_CASE("TestPrioritizerOneInterface", "[test1]") {
}
TEST_CASE("TestPrioritizerOneInterfaceMaxPayload", "[test2]") {
- auto controller =
std::make_shared("TestService");
- std::shared_ptr configuration =
std::make_shared();
+ auto controller = createNetworkPrioritizerService("TestService");
controller->initialize();
controller->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0,eth1");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"1 B");
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxPayload,
"1 B");
+
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"1 kB");
+
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxPayload,
"10 B");
controller->onEnable();
- // can't because we've triggered the max payload
- REQUIRE("" == controller->getInterface(5).getInterface());
+
+ REQUIRE("eth0" == controller->getInterface(5).getInterface());
+ REQUIRE("" == controller->getInterface(20).getInterface()); // larger than
max payload
+ REQUIRE("eth0" == controller->getInterface(5).getInterface());
}
TEST_CASE("TestPrioritizerOneInterfaceMaxThroughput", "[test3]") {
- auto controller =
std::make_shared("TestService");
- std::shared_ptr configuration =
std::make_shared();
+ auto clock = std::make_shared();
+ auto controller = createNetworkPrioritizerService("TestService", clock);
controller->initialize();
controller->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0,eth1");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
controller->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
controller->onEnable();
- // can't because we've triggered the max payload
REQUIRE("eth0" == controller->getInterface(5).getInterface());
REQUIRE("eth0" == controller->getInterface(5).getInterface());
- REQUIRE("" == controller->getInterface(5).getInterface());
- std::this_thread::sleep_for(std::chrono::milliseconds(10));
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
+ REQUIRE("" == controller->getInterface(5).getInterface()); // max
throughput reached
+ clock->advance(std::chrono::milliseconds{10}); // wait for more tokens to
be generated
+ REQUIRE("eth0" == controller->getInterface(5).getInterface()); // now we
can send again
}
TEST_CASE("TestPriorotizerMultipleInterfaces", "[test4]") {
-
LogTestController::getInstance().setTrace();
-
- auto controller =
std::make_shared("TestService");
- auto controller2 =
std::make_shared("TestService2");
- auto controller3 =
std::make_shared("TestService3");
+ auto clock = std::make_shared();
+ auto parent_controller = createNetworkPrioritizerService("TestService",
clock);
std::shared_ptr configuration =
std::make_shared();
- controller->initialize();
-
controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
+ parent_controller->initialize();
+
parent_controller->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
- controller3->initialize();
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth0");
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller3->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
- controller3->onEnable();
-
- controller2->initialize();
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::NetworkControllers,
"eth1");
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::VerifyInterfaces,
"false");
-
controller2->setProperty(minifi::controllers::NetworkPrioritizerService::MaxThroughput,
"10 B");
- controller2->onEnable();
- std::vector > services;
- services.push_back(controller2);
- services.push_back(controller3);
- controller->setLinkedControllerServices(services);
- controller->onEnable();
- // can't because we've triggered the max payload
- REQUIRE("eth1" == controller->getInterface(5).getInterface());
- REQUIRE("eth1" == controller->getInterface(5).getInterface());
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
- REQUIRE("eth0" == controller->getInterface(5).getInterface());
-}
+ auto controller0 = createNetworkPrioritizerService("TestService_eth0",
clock);
+ controller0->initialize();
+
contro
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #911: MINIFICPP-1373 - Add proper FlowFile::getAttribute getter [QoL][mini-PR]
szaszm commented on a change in pull request #911:
URL: https://github.com/apache/nifi-minifi-cpp/pull/911#discussion_r542328156
##
File path: libminifi/include/core/FlowFile.h
##
@@ -129,7 +130,9 @@ class FlowFile : public core::Connectable, public
ReferenceContainer {
* @param value value to set
* @return result of finding key
*/
- bool getAttribute(std::string key, std::string& value) const;
+ bool getAttribute(const std::string& key, std::string& value) const;
+
+ utils::optional>
getAttribute(const std::string& key) const;
Review comment:
Note: An optional of reference handlers should in general be replaced by
a non-owning pointer. They have the same semantics, but tooling is better
equipped to handle pointers.
`auto` generally means safe value semantics, and it becoming a "serious
misuse" is another footgun that reference types currently have to live with.
How is this safer otherwise, than returning a const ref or non-owning pointer
to const?
(I'm continuing because I want to either convince you that this is better or
understand your standpoint and why I'm wrong.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #953: MINIFICPP-1421 - Correct C2JstackTest synchronization
arpadboda closed pull request #953: URL: https://github.com/apache/nifi-minifi-cpp/pull/953 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-1424) Stabilize flaky test cases
[ https://issues.apache.org/jira/browse/MINIFICPP-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Gerlits updated MINIFICPP-1424: -- Labels: MiNiFi-CPP-Hygiene (was: ) > Stabilize flaky test cases > -- > > Key: MINIFICPP-1424 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1424 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Gabor Gyimesi >Priority: Major > Labels: MiNiFi-CPP-Hygiene > Time Spent: 20m > Remaining Estimate: 0h > > Multiple test cases have been failing intermittently in the CI environment. > The root causes need to be identified and fixed to have a stable and reliable > test suite for the project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7835) SFTP processors (maybe other too) proxy doesn't use credentials
[ https://issues.apache.org/jira/browse/NIFI-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248860#comment-17248860 ] Pierrick LEONARD commented on NIFI-7835: Hello, Thanks for the information for fuse-ssh. Unfortunately I have to many flows to start to change the way they are working today :). I will wait for a resolution. > SFTP processors (maybe other too) proxy doesn't use credentials > --- > > Key: NIFI-7835 > URL: https://issues.apache.org/jira/browse/NIFI-7835 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.12.0, 1.11.4 > Environment: CentOS 7, Java 8 and 11, standalone Nifi >Reporter: Wiktor Kubicki >Priority: Major > > Hello, > we tried use socks proxy configuration (with authorization) for > listSFTP/fetchSFTP/getSFTP using processor configuration or Proxy > Configuration Service. Each time we has got Getting java.net.SocketException: > Malformed. > We did tcp dump and saw, that the user/password is not provided - user is > same as system user rather than this one from configuration. > I've easly recreated issue on clean CentOS with java 8 and 11. Same problem > was on NiFi 1.11.4 and 1.12.0. > I think the problem is due of using sshj, but can be wrong. Very similar > issue is https://issues.apache.org/jira/browse/NIFI-7749 - probably with the > same cause. > > My workaround for sftp is using fuse-sftp and mount remote server as local > share. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1432) NetworkPrioritizerServiceTests are sensitive to timing
[
https://issues.apache.org/jira/browse/MINIFICPP-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Gerlits updated MINIFICPP-1432:
--
Labels: MiNiFi-CPP-Hygiene (was: )
> NetworkPrioritizerServiceTests are sensitive to timing
> --
>
> Key: MINIFICPP-1432
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1432
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Ferenc Gerlits
>Assignee: Ferenc Gerlits
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The throughput of {{NetworkPrioritizerService}} depends on timing, as a
> certain number of tokens are added to the "token bucket" every millisecond,
> and packages are allowed to go through an interface only if the interface has
> enough tokens in its bucket.
> Make the clock-dependency of {{NetworkPrioritizerService}} explicit and
> mockable, and mock it out in the {{NetworkPrioritizerServiceTests}} unit
> tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on a change in pull request #953: MINIFICPP-1421 - Disable C2JstackTest
hunyadi-dev commented on a change in pull request #953:
URL: https://github.com/apache/nifi-minifi-cpp/pull/953#discussion_r542233286
##
File path: extensions/http-curl/tests/C2JstackTest.cpp
##
@@ -18,35 +18,46 @@
#undef NDEBUG
#include
+#include
#include "TestBase.h"
#include "HTTPIntegrationBase.h"
#include "HTTPHandlers.h"
#include "utils/IntegrationTestUtils.h"
class VerifyC2DescribeJstack : public VerifyC2Describe {
public:
+ explicit VerifyC2DescribeJstack(const std::atomic_bool&
acknowledgement_checked) : VerifyC2Describe(),
acknowledgement_checked_(acknowledgement_checked) {}
void runAssertions() override {
-using org::apache::nifi::minifi::utils::verifyLogLinePresenceInPollTime;
-
assert(verifyLogLinePresenceInPollTime(std::chrono::milliseconds(wait_time_),
"SchedulingAgent"));
+// This check was previously only confirming the presence of log sinks.
+// See: https://issues.apache.org/jira/browse/MINIFICPP-1421
+using org::apache::nifi::minifi::utils::verifyEventHappenedInPollTime;
+
assert(verifyEventHappenedInPollTime(std::chrono::milliseconds(wait_time_), [&]
{ return acknowledgement_checked_.load(); }));
}
+ protected:
+ const std::atomic_bool& acknowledgement_checked_;
};
class DescribeJstackHandler : public HeartbeatHandler {
public:
+ explicit DescribeJstackHandler(std::atomic_bool& acknowledgement_checked) :
HeartbeatHandler(), acknowledgement_checked_(acknowledgement_checked) {}
void handleHeartbeat(const rapidjson::Document&, struct mg_connection *
conn) override {
sendHeartbeatResponse("DESCRIBE", "jstack", "889398", conn);
}
void handleAcknowledge(const rapidjson::Document& root) override {
assert(root.HasMember("Flowcontroller threadpool #0"));
+acknowledgement_checked_.store(true);
}
+ protected:
+ std::atomic_bool& acknowledgement_checked_;
};
int main(int argc, char **argv) {
const cmd_args args = parse_cmdline_args(argc, argv, "heartbeat");
- VerifyC2DescribeJstack harness;
+ std::atomic_bool acknowledgement_checked{ false };
Review comment:
By checked I meant the assertion in already happened. I dont mind
renaming it, so consider this corrected :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on a change in pull request #953: MINIFICPP-1421 - Disable C2JstackTest
hunyadi-dev commented on a change in pull request #953:
URL: https://github.com/apache/nifi-minifi-cpp/pull/953#discussion_r542232128
##
File path: extensions/http-curl/tests/C2JstackTest.cpp
##
@@ -18,35 +18,46 @@
#undef NDEBUG
#include
+#include
#include "TestBase.h"
#include "HTTPIntegrationBase.h"
#include "HTTPHandlers.h"
#include "utils/IntegrationTestUtils.h"
class VerifyC2DescribeJstack : public VerifyC2Describe {
public:
+ explicit VerifyC2DescribeJstack(const std::atomic_bool&
acknowledgement_checked) : VerifyC2Describe(),
acknowledgement_checked_(acknowledgement_checked) {}
void runAssertions() override {
-using org::apache::nifi::minifi::utils::verifyLogLinePresenceInPollTime;
-
assert(verifyLogLinePresenceInPollTime(std::chrono::milliseconds(wait_time_),
"SchedulingAgent"));
+// This check was previously only confirming the presence of log sinks.
+// See: https://issues.apache.org/jira/browse/MINIFICPP-1421
Review comment:
It is probably no longer useful anymore, I thought this was going to be
a temporary fix, and people checking on it would find the attached jira useful,
but I decided to close the ticket after this PR is merged.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (MINIFICPP-1432) NetworkPrioritizerServiceTests are sensitive to timing
[
https://issues.apache.org/jira/browse/MINIFICPP-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Gerlits updated MINIFICPP-1432:
--
Description:
The throughput of {{NetworkPrioritizerService}} depends on timing, as a certain
number of tokens are added to the "token bucket" every millisecond, and
packages are allowed to go through an interface only if the interface has
enough tokens in its bucket.
Make the clock-dependency of {{NetworkPrioritizerService}} explicit and
mockable, and mock it out in the {{NetworkPrioritizerServiceTests}} unit tests.
was:
The throughput of {{NetworkPrioritizerService}} depends on timing, as a certain
number of tokens are added to the "token bucket" every second, and packages are
allowed to go through an interface only if the interface has enough tokens in
its bucket.
Make the clock-dependency of {{NetworkPrioritizerService}} explicit and
mockable, and mock it out in the {{NetworkPrioritizerServiceTests}} unit tests.
> NetworkPrioritizerServiceTests are sensitive to timing
> --
>
> Key: MINIFICPP-1432
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1432
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Ferenc Gerlits
>Assignee: Ferenc Gerlits
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The throughput of {{NetworkPrioritizerService}} depends on timing, as a
> certain number of tokens are added to the "token bucket" every millisecond,
> and packages are allowed to go through an interface only if the interface has
> enough tokens in its bucket.
> Make the clock-dependency of {{NetworkPrioritizerService}} explicit and
> mockable, and mock it out in the {{NetworkPrioritizerServiceTests}} unit
> tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on a change in pull request #911: MINIFICPP-1373 - Add proper FlowFile::getAttribute getter [QoL][mini-PR]
hunyadi-dev commented on a change in pull request #911:
URL: https://github.com/apache/nifi-minifi-cpp/pull/911#discussion_r542227113
##
File path: libminifi/include/core/FlowFile.h
##
@@ -129,7 +130,9 @@ class FlowFile : public core::Connectable, public
ReferenceContainer {
* @param value value to set
* @return result of finding key
*/
- bool getAttribute(std::string key, std::string& value) const;
+ bool getAttribute(const std::string& key, std::string& value) const;
+
+ utils::optional>
getAttribute(const std::string& key) const;
Review comment:
One needs to call `.value().get()` or `->get()` on an optional of
reference handlers. It indeed would be an issue if someone seriously misused
`auto` for holding the return value, but otherwise this is even safer than
returning a const ref.
https://godbolt.org/z/jbhf3n
Will change this to **return a copy** :cry:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #953: MINIFICPP-1421 - Disable C2JstackTest
fgerlits commented on a change in pull request #953:
URL: https://github.com/apache/nifi-minifi-cpp/pull/953#discussion_r542223028
##
File path: extensions/http-curl/tests/C2JstackTest.cpp
##
@@ -18,35 +18,46 @@
#undef NDEBUG
#include
+#include
#include "TestBase.h"
#include "HTTPIntegrationBase.h"
#include "HTTPHandlers.h"
#include "utils/IntegrationTestUtils.h"
class VerifyC2DescribeJstack : public VerifyC2Describe {
public:
+ explicit VerifyC2DescribeJstack(const std::atomic_bool&
acknowledgement_checked) : VerifyC2Describe(),
acknowledgement_checked_(acknowledgement_checked) {}
void runAssertions() override {
-using org::apache::nifi::minifi::utils::verifyLogLinePresenceInPollTime;
-
assert(verifyLogLinePresenceInPollTime(std::chrono::milliseconds(wait_time_),
"SchedulingAgent"));
+// This check was previously only confirming the presence of log sinks.
+// See: https://issues.apache.org/jira/browse/MINIFICPP-1421
Review comment:
I don't think this comment is useful; if someone is interested in the
history of this function, they can use `git blame`.
##
File path: extensions/http-curl/tests/C2JstackTest.cpp
##
@@ -18,35 +18,46 @@
#undef NDEBUG
#include
+#include
#include "TestBase.h"
#include "HTTPIntegrationBase.h"
#include "HTTPHandlers.h"
#include "utils/IntegrationTestUtils.h"
class VerifyC2DescribeJstack : public VerifyC2Describe {
public:
+ explicit VerifyC2DescribeJstack(const std::atomic_bool&
acknowledgement_checked) : VerifyC2Describe(),
acknowledgement_checked_(acknowledgement_checked) {}
void runAssertions() override {
-using org::apache::nifi::minifi::utils::verifyLogLinePresenceInPollTime;
-
assert(verifyLogLinePresenceInPollTime(std::chrono::milliseconds(wait_time_),
"SchedulingAgent"));
+// This check was previously only confirming the presence of log sinks.
+// See: https://issues.apache.org/jira/browse/MINIFICPP-1421
+using org::apache::nifi::minifi::utils::verifyEventHappenedInPollTime;
+
assert(verifyEventHappenedInPollTime(std::chrono::milliseconds(wait_time_), [&]
{ return acknowledgement_checked_.load(); }));
}
+ protected:
+ const std::atomic_bool& acknowledgement_checked_;
};
class DescribeJstackHandler : public HeartbeatHandler {
public:
+ explicit DescribeJstackHandler(std::atomic_bool& acknowledgement_checked) :
HeartbeatHandler(), acknowledgement_checked_(acknowledgement_checked) {}
void handleHeartbeat(const rapidjson::Document&, struct mg_connection *
conn) override {
sendHeartbeatResponse("DESCRIBE", "jstack", "889398", conn);
}
void handleAcknowledge(const rapidjson::Document& root) override {
assert(root.HasMember("Flowcontroller threadpool #0"));
+acknowledgement_checked_.store(true);
}
+ protected:
+ std::atomic_bool& acknowledgement_checked_;
};
int main(int argc, char **argv) {
const cmd_args args = parse_cmdline_args(argc, argv, "heartbeat");
- VerifyC2DescribeJstack harness;
+ std::atomic_bool acknowledgement_checked{ false };
Review comment:
`acknowledgement_received` may be a better name for this, as we set it
to `true` when the ack is received, not when we check that it has been received
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on a change in pull request #901: MINIFICPP-1288 - Refactor YamlConfiguration::parsePropertiesNodeYaml
hunyadi-dev commented on a change in pull request #901:
URL: https://github.com/apache/nifi-minifi-cpp/pull/901#discussion_r542202475
##
File path: libminifi/src/core/yaml/YamlConfiguration.cpp
##
@@ -770,106 +770,122 @@ void YamlConfiguration::parsePortYaml(YAML::Node
*portNode, core::ProcessGroup *
}
}
-void YamlConfiguration::parsePropertiesNodeYaml(YAML::Node *propertiesNode,
std::shared_ptr processor, const std::string
&component_name,
-const std::string
&yaml_section) {
- // Treat generically as a YAML node so we can perform inspection on entries
to ensure they are populated
- logger_->log_trace("Entered %s", component_name);
- for (YAML::const_iterator propsIter = propertiesNode->begin(); propsIter !=
propertiesNode->end(); ++propsIter) {
-std::string propertyName = propsIter->first.as();
-YAML::Node propertyValueNode = propsIter->second;
-logger_->log_trace("Encountered %s", propertyName);
-if (!propertyValueNode.IsNull() && propertyValueNode.IsDefined()) {
- if (propertyValueNode.IsSequence()) {
-for (auto iter : propertyValueNode) {
- if (iter.IsDefined()) {
-YAML::Node nodeVal = iter.as();
-YAML::Node propertiesNode = nodeVal["value"];
-// must insert the sequence in differently.
-std::string rawValueString = propertiesNode.as();
-logger_->log_debug("Found %s=%s", propertyName, rawValueString);
-if (!processor->updateProperty(propertyName, rawValueString)) {
- std::shared_ptr proc =
std::dynamic_pointer_cast(processor);
- if (proc != 0) {
-logger_->log_warn("Received property %s with value %s but is
not one of the properties for %s. "
- "Attempting to add as dynamic property.",
- propertyName, rawValueString,
proc->getName());
-if (!processor->setDynamicProperty(propertyName,
rawValueString)) {
- logger_->log_warn("Unable to set the dynamic property %s
with value %s", propertyName.c_str(), rawValueString.c_str());
-} else {
- logger_->log_warn("Dynamic property %s with value %s set",
propertyName.c_str(), rawValueString.c_str());
-}
- }
-}
- }
-}
- } else {
-core::Property myProp(propertyName, "", "");
-processor->getProperty(propertyName, myProp);
-PropertyValue defaultValue;
-defaultValue = myProp.getDefaultValue();
-auto defaultType = defaultValue.getTypeInfo();
-PropertyValue coercedValue = defaultValue;
-
-// coerce the types. upon failure we will either exit or use the
default value.
-// we do this here ( in addition to the PropertyValue class ) to get
the earliest
-// possible YAML failure.
-try {
- if (defaultType == typeid(std::string)) {
-auto typedValue = propertyValueNode.as();
-coercedValue = typedValue;
- } else if (defaultType == typeid(int64_t)) {
-auto typedValue = propertyValueNode.as();
-coercedValue = typedValue;
- } else if (defaultType == typeid(uint64_t)) {
-try {
- auto typedValue = propertyValueNode.as();
- coercedValue = typedValue;
-} catch (...) {
- auto typedValue = propertyValueNode.as();
- coercedValue = typedValue;
-}
- } else if (defaultType == typeid(int)) {
-auto typedValue = propertyValueNode.as();
-coercedValue = typedValue;
- } else if (defaultType == typeid(bool)) {
-auto typedValue = propertyValueNode.as();
-coercedValue = typedValue;
- } else {
-auto typedValue = propertyValueNode.as();
-coercedValue = typedValue;
- }
-} catch (...) {
- std::string eof;
- bool exit_on_failure = false;
- if
(configuration_->get(Configure::nifi_flow_configuration_file_exit_failure,
eof)) {
-utils::StringUtils::StringToBool(eof, exit_on_failure);
- }
- logger_->log_error("Invalid conversion for field %s. Value %s",
myProp.getName(), propertyValueNode.as());
- if (exit_on_failure) {
-std::cerr << "Invalid conversion for " << myProp.getName() << " to
" << defaultType.name() << std::endl;
+void YamlConfiguration::parsePropertyValueSequence(const std::string&
propertyName, const YAML::Node& propertyValueNode,
std::shared_ptr processor,
+const std::string &yaml_section) {
+ for (auto iter : propertyValueNode) {
+if (iter.IsDefined()) {
+ YAML::Node nodeVal = iter.as();
+ YAML::Node propertiesNode = nodeVal["value"];
+ // must insert the sequence in differently.
+ std::string rawValueStr
