[jira] [Commented] (NIFI-5443) Improve cluster configuration for dynamic scaling
[
https://issues.apache.org/jira/browse/NIFI-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559232#comment-16559232
]
Prashanth Venkatesan commented on NIFI-5443:
[~alopresto] There is no blocker... Currently as a workaround in our docker
scripts, we introduced one boolean flag to control updation of authorizers.xml.
So, in case of scale-out cases, we pass that flag as "false" so that
*authorizers.xml* won't be populated. This seems like currently made our
dynamic scaling bit smoother.
> Improve cluster configuration for dynamic scaling
> -
>
> Key: NIFI-5443
> URL: https://issues.apache.org/jira/browse/NIFI-5443
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Priority: Critical
> Labels: cluster, docker, kubernetes, rkt, scale, security
>
> Currently, NiFi is designed for static clusters, with frequent references in
> configuration files to a priori knowledge of node hostnames, ports, etc.
> Efforts should be taken to make NiFi easier to dynamically scale. This can
> involve containerization improvements via Docker/rkt, deployment improvements
> via Kubernetes, and abstraction of the configuration values needed to stand
> up the cluster. A node should be able to join the cluster, and, given the
> correct keystore and truststore, immediately communicate with other existing
> nodes in the cluster without requiring direct configuration changes to them,
> or a restart of any node.
> * {{authorizers.xml}}
> * node identities
> * permissions ({{RW}} on {{/proxy}})
> * ZooKeeper configuration
> * etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5443) Improve cluster configuration for dynamic scaling
[
https://issues.apache.org/jira/browse/NIFI-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556775#comment-16556775
]
Prashanth Venkatesan commented on NIFI-5443:
[~alopresto] Scaling in running NiFi secure cluster expects empty
authorizers.xml from new node to join the cluster. Say, node-0 & node-1
running as secure nifi cluster. I modified policies and flow in that cluster.
Now, if new fresh node-3 to join the cluster, we need to pass *empty
authorizers.xml.* Is there any other elegant way of handling this because we
need to handle this in docker container?
> Improve cluster configuration for dynamic scaling
> -
>
> Key: NIFI-5443
> URL: https://issues.apache.org/jira/browse/NIFI-5443
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Priority: Critical
> Labels: cluster, docker, kubernetes, rkt, scale, security
>
> Currently, NiFi is designed for static clusters, with frequent references in
> configuration files to a priori knowledge of node hostnames, ports, etc.
> Efforts should be taken to make NiFi easier to dynamically scale. This can
> involve containerization improvements via Docker/rkt, deployment improvements
> via Kubernetes, and abstraction of the configuration values needed to stand
> up the cluster. A node should be able to join the cluster, and, given the
> correct keystore and truststore, immediately communicate with other existing
> nodes in the cluster without requiring direct configuration changes to them,
> or a restart of any node.
> * {{authorizers.xml}}
> * node identities
> * permissions ({{RW}} on {{/proxy}})
> * ZooKeeper configuration
> * etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5370) Cluster request replication failing with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544903#comment-16544903
]
Prashanth Venkatesan commented on NIFI-5370:
[~alopresto] I understand your point on adding new user via NiFi UI/API. I
experimented this in NiFi VM cluster. But I am facing the below general
problem.
Assume I have initially 2 nodes (say nifi-node-1 & nifi-node-2), I create certs
using _"tls-toolkit.sh -n 'nifi-node-1,nifi-node-2' "._ **Now I am scaling out
to 3 nodes (say nifi-node-3) , now if regenerate the certs for 3 nodes using
tls-toolkit , then new nodes can't validate the other 2 nodes in cluster giving
*"java.security.cert.CertPathValidatorException: Path does not chain with any
of the trust anchors".* Do you have any resolution for handling this scenario
without wildcarded certs?
> Cluster request replication failing with wildcard certs
> ---
>
> Key: NIFI-5370
> URL: https://issues.apache.org/jira/browse/NIFI-5370
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, cluster, security, tls, wildcard
> Fix For: 1.8.0, 1.7.1
>
>
> From the users mailing list:
> {quote}
> Team,
>
> NiFi secured cluster throws below error with wildcarded self-signed
> standalone certificate. Just a brief background, we are deploying nifi in
> Kubernetes where we have to use wildcarded certificates. Till nifi 1.6.0, it
> was working fine.
> Also I tried bringing up NiFi in linux VM in secured cluster mode with
> wildcarded certs, I am getting same error.
>
> Toolkit command to generate certs:
> bin/tls-toolkit.sh standalone -n
> '*.mynifi-nifi-headless.default.svc.cluster.local’ -C 'CN=admin, OU=NIFI' -o
>
>
> Logs:
> 2018-07-02 12:40:32,369 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local:8443 due to
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> 2018-07-02 12:40:32,370 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> at
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:316)
>
> Please help me in resolving this.
>
> Note: Same certificates is working for single mode setup.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5370) Cluster request replication failing with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16538093#comment-16538093
]
Prashanth Venkatesan commented on NIFI-5370:
[~alopresto] Reason behind going towards wildcarded certs was to handle the
dynamic scaling easily especially in containerised environment(say DCOS,
Kubernetes, etc). To my knowledge in NiFi, if we are using uniquely identified
certificates we have to add 'Initial User Identity' and 'Node Identity' in
*authorizers.xml* file for every new node in cluster. So if we are scaling out
we have to update the authorizers.xml file in all nodes that results in restart
of existing nodes. Also in-case of multi node cluster, managing multiple
uniquely identified certificates is bit difficult.
> Cluster request replication failing with wildcard certs
> ---
>
> Key: NIFI-5370
> URL: https://issues.apache.org/jira/browse/NIFI-5370
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, cluster, security, tls, wildcard
>
> From the users mailing list:
> {quote}
> Team,
>
> NiFi secured cluster throws below error with wildcarded self-signed
> standalone certificate. Just a brief background, we are deploying nifi in
> Kubernetes where we have to use wildcarded certificates. Till nifi 1.6.0, it
> was working fine.
> Also I tried bringing up NiFi in linux VM in secured cluster mode with
> wildcarded certs, I am getting same error.
>
> Toolkit command to generate certs:
> bin/tls-toolkit.sh standalone -n
> '*.mynifi-nifi-headless.default.svc.cluster.local’ -C 'CN=admin, OU=NIFI' -o
>
>
> Logs:
> 2018-07-02 12:40:32,369 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local:8443 due to
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> 2018-07-02 12:40:32,370 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> at
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:316)
>
> Please help me in resolving this.
>
> Note: Same certificates is working for single mode setup.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (NIFI-5370) Cluster request replication failing with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533587#comment-16533587
]
Prashanth Venkatesan edited comment on NIFI-5370 at 7/5/18 12:27 PM:
-
[~alopresto] - From
[NiFiHostVerifier|https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/NiFiHostnameVerifier.java]
, i can infer that it will either validated when CN equals hostname or SAN
should contain hostname. Hence wildcarded certs without SAN is not verified.
But in my case, i need to use wildcard certificates.
was (Author: prashanv):
Just want to add few more point to this issue. [~alopresto] - From
[NiFiHostVerifier|https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/NiFiHostnameVerifier.java]
, i can infer that it will either validated when CN equals hostname or SAN
should contain hostname. Hence wildcarded certs without SAN is not verified.
But in my case, i need to use wildcard certificates.
> Cluster request replication failing with wildcard certs
> ---
>
> Key: NIFI-5370
> URL: https://issues.apache.org/jira/browse/NIFI-5370
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, cluster, security, tls, wildcard
>
> From the users mailing list:
> {quote}
> Team,
>
> NiFi secured cluster throws below error with wildcarded self-signed
> standalone certificate. Just a brief background, we are deploying nifi in
> Kubernetes where we have to use wildcarded certificates. Till nifi 1.6.0, it
> was working fine.
> Also I tried bringing up NiFi in linux VM in secured cluster mode with
> wildcarded certs, I am getting same error.
>
> Toolkit command to generate certs:
> bin/tls-toolkit.sh standalone -n
> '*.mynifi-nifi-headless.default.svc.cluster.local’ -C 'CN=admin, OU=NIFI' -o
>
>
> Logs:
> 2018-07-02 12:40:32,369 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local:8443 due to
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> 2018-07-02 12:40:32,370 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> at
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:316)
>
> Please help me in resolving this.
>
> Note: Same certificates is working for single mode setup.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (NIFI-5370) Cluster request replication failing with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533587#comment-16533587
]
Prashanth Venkatesan edited comment on NIFI-5370 at 7/5/18 12:26 PM:
-
Just want to add few more point to this issue. [~alopresto] - From
[NiFiHostVerifier|https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/NiFiHostnameVerifier.java]
, i can infer that it will either validated when CN equals hostname or SAN
should contain hostname. Hence wildcarded certs without SAN is not verified.
But in my case, i need to use wildcard certificates.
was (Author: prashanv):
Just want to add few more point to this issue. [~alopresto] - From
[[NiFiHostVerifier|https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/NiFiHostnameVerifier.java]|https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/NiFiHostnameVerifier.java]
, i can infer that it will either validated when CN equals hostname or SAN
should contain hostname. Hence wildcarded certs without SAN is not verified.
But in my case, i need to use wildcard certificates.
> Cluster request replication failing with wildcard certs
> ---
>
> Key: NIFI-5370
> URL: https://issues.apache.org/jira/browse/NIFI-5370
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, cluster, security, tls, wildcard
>
> From the users mailing list:
> {quote}
> Team,
>
> NiFi secured cluster throws below error with wildcarded self-signed
> standalone certificate. Just a brief background, we are deploying nifi in
> Kubernetes where we have to use wildcarded certificates. Till nifi 1.6.0, it
> was working fine.
> Also I tried bringing up NiFi in linux VM in secured cluster mode with
> wildcarded certs, I am getting same error.
>
> Toolkit command to generate certs:
> bin/tls-toolkit.sh standalone -n
> '*.mynifi-nifi-headless.default.svc.cluster.local’ -C 'CN=admin, OU=NIFI' -o
>
>
> Logs:
> 2018-07-02 12:40:32,369 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local:8443 due to
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> 2018-07-02 12:40:32,370 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> at
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:316)
>
> Please help me in resolving this.
>
> Note: Same certificates is working for single mode setup.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5370) Cluster request replication failing with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533587#comment-16533587
]
Prashanth Venkatesan commented on NIFI-5370:
Just want to add few more point to this issue. [~alopresto] - From
[[NiFiHostVerifier|https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/NiFiHostnameVerifier.java]|https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/NiFiHostnameVerifier.java]
, i can infer that it will either validated when CN equals hostname or SAN
should contain hostname. Hence wildcarded certs without SAN is not verified.
But in my case, i need to use wildcard certificates.
> Cluster request replication failing with wildcard certs
> ---
>
> Key: NIFI-5370
> URL: https://issues.apache.org/jira/browse/NIFI-5370
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, cluster, security, tls, wildcard
>
> From the users mailing list:
> {quote}
> Team,
>
> NiFi secured cluster throws below error with wildcarded self-signed
> standalone certificate. Just a brief background, we are deploying nifi in
> Kubernetes where we have to use wildcarded certificates. Till nifi 1.6.0, it
> was working fine.
> Also I tried bringing up NiFi in linux VM in secured cluster mode with
> wildcarded certs, I am getting same error.
>
> Toolkit command to generate certs:
> bin/tls-toolkit.sh standalone -n
> '*.mynifi-nifi-headless.default.svc.cluster.local’ -C 'CN=admin, OU=NIFI' -o
>
>
> Logs:
> 2018-07-02 12:40:32,369 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local:8443 due to
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> 2018-07-02 12:40:32,370 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> at
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:316)
>
> Please help me in resolving this.
>
> Note: Same certificates is working for single mode setup.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5327) NetFlow Processors
Prashanth Venkatesan created NIFI-5327: -- Summary: NetFlow Processors Key: NIFI-5327 URL: https://issues.apache.org/jira/browse/NIFI-5327 Project: Apache NiFi Issue Type: New Feature Components: Core Framework Affects Versions: 1.6.0 Reporter: Prashanth Venkatesan As network traffic data scopes for the big data use case, would like NiFi to have processors to support parsing of those protocols. Netflow is a protocol introduced by Cisco that provides the ability to collect IP network traffic as it enters or exits an interface and is described in detail in here: [https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html] Currently, I have created the following processor: *ParseNetflowv5*: Parses the ingress netflowv5 bytes and ingest as either NiFi flowfile attributes or as a JSON content. This also sends one-time-template. Further ahead, we can add many processor specific to network protocols in this nar bundle. I will create a pull request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
