[jira] [Updated] (NIFI-13724) Nifi Registry Events Include Additional Metadata
[ https://issues.apache.org/jira/browse/NIFI-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dye357 updated NIFI-13724: -- Fix Version/s: (was: 1.28.0) (was: 2.0.0-M5) > Nifi Registry Events Include Additional Metadata > > > Key: NIFI-13724 > URL: https://issues.apache.org/jira/browse/NIFI-13724 > Project: Apache NiFi > Issue Type: New Feature > Components: NiFi Registry >Affects Versions: 1.27.0, 2.0.0-M4 >Reporter: Dye357 >Assignee: Dye357 >Priority: Minor > > Currently the Nifi Registry EventFactory adds a few pieces of metadata to the > EventType - when writing a custom extension for Nifi Registry a developer is > limited in the amount of information they have about the event – this ticket > would expand the scope of metadata included in each event to include the > Name, Description and Timestamps. > As a work-around we were using the nifi-registry-client to use the provided > IDs to enrich our dataset however we've run into problems: > 1. The nifi registry client is broken when running inside an Nifi Registry > extension due to JavaX/Jakarta classpath issues. (Separate ticket) > 2. This the most important of the two points – if we have a Bucket Deleted or > Flow Deleted event only having the ID is useless because we can no longer > query the API to get the name and description for the Entity in question > which has led to some auditing/accountability concerns. > I'm putting together a PR to include the additional specified metadata in the > EventFactory. The PR should be merge-able on 1.x and 2.x -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12605) Stateless Nifi Artifactory Extension Client
Dye357 created NIFI-12605: - Summary: Stateless Nifi Artifactory Extension Client Key: NIFI-12605 URL: https://issues.apache.org/jira/browse/NIFI-12605 Project: Apache NiFi Issue Type: New Feature Components: NiFi Stateless Reporter: Dye357 Currently Stateless Nifi supports pulling extensions via the Nexus Extension Client - this ticket would add another implementation of the Extension Client interface to support pulling artifacts from Artifactory. Yes, Technically the nexus client will work against an Artifactory as long as the repository type is Maven2 however the nexus client as-is does not support additional features offered by Artifactory such as Authentication with an Auth Token. Proposed implementation would be an Artifactory Extension Client, the client would support authenticating using an auth token (via configuration) for pulling bundles. Additionally the interface would be modified to add an enabled flag to the Extension Client interface such that we can disable extension clients that we do-not want to run. The Artifactory extension client to be disabled by default so the current out of the box behavior's remains unchanged. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11471) Component Timeouts are configurable in Stateless Nifi
Dye357 created NIFI-11471:
-
Summary: Component Timeouts are configurable in Stateless Nifi
Key: NIFI-11471
URL: https://issues.apache.org/jira/browse/NIFI-11471
Project: Apache NiFi
Issue Type: Improvement
Components: NiFi Stateless
Reporter: Dye357
Currently in Stateless Nifi each component is required to start within a
statically defined timeout. In some cases (IE controller services which pull
down web resources) this timeout is too short and causes the entire Stateless
Nifi instance to fail starting.
{code:java}
private static final long COMPONENT_ENABLE_TIMEOUT_MILLIS =
TimeUnit.SECONDS.toMillis(10); {code}
Suggested improvement is to make this timeout configurable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11239) Stateless-Nifi not logging performance counters
[ https://issues.apache.org/jira/browse/NIFI-11239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dye357 updated NIFI-11239: -- Status: Open (was: Patch Available) Upon further review we found the code is working as expected and our test-case was flawed. https://apachenifi.slack.com/archives/C0L9VCD47/p1678380820708399?thread_ts=1678379275.505629&cid=C0L9VCD47 > Stateless-Nifi not logging performance counters > --- > > Key: NIFI-11239 > URL: https://issues.apache.org/jira/browse/NIFI-11239 > Project: Apache NiFi > Issue Type: Bug > Components: NiFi Stateless >Affects Versions: 1.20.0 >Reporter: Dye357 >Assignee: Stephanie Ambrose >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Description: It appears at some point work was done to log the performance > counters for flows running in stateless-nifi. I've tested on 1.19.1 and > 1.20.0 and found that the functionality is broken, the headers for the > logging print however no component counts displayed: > > - > 2023-03-02 10:14:03,503 INFO [Background Tasks] > o.a.n.c.reporting.LogComponentStatuses Counters: > -- > | Counter Context | Counter Name > | Counter Value | Increase/sec | > -- > -- > > Further investigating the code reveals that the LogComponentStatues class is > being passed a FlowCounterContext which is effectively empty. It looks like > StatelessRepositoryContext.java is not overriding the requisite abstract > method to inject the peromanceCounters into the FlowCounterContext. > > Acceptance Criteria: The flow performance counters are correctly logged when > logCounters() is invokes in LogComponentStatuses.java. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11239) Stateless-Nifi not logging performance counters
Dye357 created NIFI-11239: - Summary: Stateless-Nifi not logging performance counters Key: NIFI-11239 URL: https://issues.apache.org/jira/browse/NIFI-11239 Project: Apache NiFi Issue Type: Bug Components: NiFi Stateless Affects Versions: 1.20.0 Reporter: Dye357 Assignee: Stephanie Ambrose Description: It appears at some point work was done to log the performance counters for flows running in stateless-nifi. I've tested on 1.19.1 and 1.20.0 and found that the functionality is broken, the headers for the logging print however no component counts displayed: - 2023-03-02 10:14:03,503 INFO [Background Tasks] o.a.n.c.reporting.LogComponentStatuses Counters: -- | Counter Context | Counter Name | Counter Value | Increase/sec | -- -- Further investigating the code reveals that the LogComponentStatues class is being passed a FlowCounterContext which is effectively empty. It looks like StatelessRepositoryContext.java is not overriding the requisite abstract method to inject the peromanceCounters into the FlowCounterContext. Acceptance Criteria: The flow performance counters are correctly logged when logCounters() is invokes in LogComponentStatuses.java. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11231) Stateless Nifi Supports Secure Parameter Context Variables
[ https://issues.apache.org/jira/browse/NIFI-11231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dye357 reassigned NIFI-11231: - Assignee: Stephanie Ambrose > Stateless Nifi Supports Secure Parameter Context Variables > -- > > Key: NIFI-11231 > URL: https://issues.apache.org/jira/browse/NIFI-11231 > Project: Apache NiFi > Issue Type: Bug > Components: NiFi Stateless >Affects Versions: 1.20.0 >Reporter: Dye357 >Assignee: Stephanie Ambrose >Priority: Major > Fix For: 1.21.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently Secure Parameter Context Variables are not supported in Stateless > Nifi - this is due to StatelessFlowManager being stubbed to throw a "Not > Implemented" exception when providing a process group configured with a > secure Parameter Context variable. > Desired functionality: Secure Parameter Context Variables can be used with > stateless Nifi. If a process group contains a Secure Parameter Context > Variable the context is used to hydrate all parameters within the provided > flow including any marked sensitive. > Acceptance Criteria: Process Groups using secure parameters can be used in > stateless Nifi. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11231) Stateless Nifi Supports Secure Parameter Context Variables
[ https://issues.apache.org/jira/browse/NIFI-11231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dye357 updated NIFI-11231: -- Summary: Stateless Nifi Supports Secure Parameter Context Variables (was: Stateless Nifi Supports Secure Parameter Contexts) > Stateless Nifi Supports Secure Parameter Context Variables > -- > > Key: NIFI-11231 > URL: https://issues.apache.org/jira/browse/NIFI-11231 > Project: Apache NiFi > Issue Type: Bug > Components: NiFi Stateless >Affects Versions: 1.20.0 >Reporter: Dye357 >Priority: Major > Fix For: 1.21.0 > > > Currently Secure Parameter Context Variables are not supported in Stateless > Nifi - this is due to StatelessFlowManager being stubbed to throw a "Not > Implemented" exception when providing a process group configured with a > secure Parameter Context variable. > Desired functionality: Secure Parameter Context Variables can be used with > stateless Nifi. If a process group contains a Secure Parameter Context > Variable the context is used to hydrate all parameters within the provided > flow including any marked sensitive. > Acceptance Criteria: Process Groups using secure parameters can be used in > stateless Nifi. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11231) Stateless Nifi Supports Secure Parameter Contexts
Dye357 created NIFI-11231: - Summary: Stateless Nifi Supports Secure Parameter Contexts Key: NIFI-11231 URL: https://issues.apache.org/jira/browse/NIFI-11231 Project: Apache NiFi Issue Type: Bug Components: NiFi Stateless Affects Versions: 1.20.0 Reporter: Dye357 Fix For: 1.21.0 Currently Secure Parameter Context Variables are not supported in Stateless Nifi - this is due to StatelessFlowManager being stubbed to throw a "Not Implemented" exception when providing a process group configured with a secure Parameter Context variable. Desired functionality: Secure Parameter Context Variables can be used with stateless Nifi. If a process group contains a Secure Parameter Context Variable the context is used to hydrate all parameters within the provided flow including any marked sensitive. Acceptance Criteria: Process Groups using secure parameters can be used in stateless Nifi. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-9568) nifi-jolt-transform-json-ui includes unnecessary assets due to wildcard
Dye357 created NIFI-9568: Summary: nifi-jolt-transform-json-ui includes unnecessary assets due to wildcard Key: NIFI-9568 URL: https://issues.apache.org/jira/browse/NIFI-9568 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.15.2 Reporter: Dye357 Fix For: 1.16.0 The Nifi-Jolt-Transform-json-UI War file gets built with unnecessary files from build environment. This is due to an wildcard being used to bring anything in: **/*> on line 148 of pom.xml. In our use-case this is causing our development configurations to show up in the war file. Recommend using a tighter allow-list so that only *.css and *.js files are included in the build. Acceptance Criteria: Only js and css assets are included in the nifi-jolt-transform-ui war file, non-relevant files such as package.json & build environment files (intellij .idea) are no-longer included in the war file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9568) nifi-jolt-transform-json-ui includes unnecessary assets due to wildcard
[ https://issues.apache.org/jira/browse/NIFI-9568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17475565#comment-17475565 ] Dye357 commented on NIFI-9568: -- I will supply a merge request for this change. > nifi-jolt-transform-json-ui includes unnecessary assets due to wildcard > --- > > Key: NIFI-9568 > URL: https://issues.apache.org/jira/browse/NIFI-9568 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.15.2 >Reporter: Dye357 >Priority: Minor > Fix For: 1.16.0 > > Original Estimate: 12h > Remaining Estimate: 12h > > The Nifi-Jolt-Transform-json-UI War file gets built with unnecessary files > from build environment. This is due to an wildcard being used to bring > anything in: **/*> on line 148 of pom.xml. In our use-case > this is causing our development configurations to show up in the war file. > Recommend using a tighter allow-list so that only *.css and *.js files are > included in the build. > Acceptance Criteria: Only js and css assets are included in the > nifi-jolt-transform-ui war file, non-relevant files such as package.json & > build environment files (intellij .idea) are no-longer included in the war > file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFIREG-205) NiFi Registry DB gets out of sync with git repository, no apprent remediation
[ https://issues.apache.org/jira/browse/NIFIREG-205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677397#comment-16677397 ] Dye357 commented on NIFIREG-205: Thanks for fixing this!! > NiFi Registry DB gets out of sync with git repository, no apprent remediation > - > > Key: NIFIREG-205 > URL: https://issues.apache.org/jira/browse/NIFIREG-205 > Project: NiFi Registry > Issue Type: Bug > Environment: Centos 7.5 >Reporter: Dye357 >Assignee: Koji Kawamura >Priority: Major > Fix For: 0.4.0 > > > I've observed a couple issues with the GitFlowPersistenceAdapter: > # When adding a new process group to NIFREG If for any reason the git > repository is in a "dirty" (untracked file) state the adding of the process > group fails. However an entry is still created in the DB with a version of 0. > Once in this state you cannot delete the flow from NIFIREG and you cannot > restart version control from nifi with the same name. I assume the only way > to fix this is to manually go into the DB and delete the record. > # When using Remote To Push, if the push fails the same behavior in #1 is > exhibited. It's not reasonable to expect that a push will always succeed. The > remote git repository could be offline for maintenance etc... > Steps to reproduce: > # Start nifi registry with an empty db and clean git repo. > # add an untracked file to the git repo but do-not commit it. > # Start a processgroup under version control. > # Expect Failure in Nifi UI > # Expect Exception in Log saying untracked files in git repo. > # Delete flow from nifi-registry using Actions -> Delete. > # Expect Failure case, recieve error deleting flow message. > # Refresh nifi-registry UI - flow is still present, version is 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFIREG-205) NiFi Registry DB gets out of sync with git repository, no apprent remediation
Dye357 created NIFIREG-205: -- Summary: NiFi Registry DB gets out of sync with git repository, no apprent remediation Key: NIFIREG-205 URL: https://issues.apache.org/jira/browse/NIFIREG-205 Project: NiFi Registry Issue Type: Bug Environment: Centos 7.5 Reporter: Dye357 I've observed a couple issues with the GitFlowPersistenceAdapter: # When adding a new process group to NIFREG If for any reason the git repository is in a "dirty" (untracked file) state the adding of the process group fails. However an entry is still created in the DB with a version of 0. Once in this state you cannot delete the flow from NIFIREG and you cannot restart version control from nifi with the same name. I assume the only way to fix this is to manually go into the DB and delete the record. # When using Remote To Push, if the push fails the same behavior in #1 is exhibited. It's not reasonable to expect that a push will always succeed. The remote git repository could be offline for maintenance etc... Steps to reproduce: # Start nifi registry with an empty db and clean git repo. # add an untracked file to the git repo but do-not commit it. # Start a processgroup under version control. # Expect Failure in Nifi UI # Expect Exception in Log saying untracked files in git repo. # Delete flow from nifi-registry using Actions -> Delete. # Expect Failure case, recieve error deleting flow message. # Refresh nifi-registry UI - flow is still present, version is 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4904) PutElasticSearch5 should support higher than elasticsearch 5.0.0
Dye357 created NIFI-4904:
Summary: PutElasticSearch5 should support higher than
elasticsearch 5.0.0
Key: NIFI-4904
URL: https://issues.apache.org/jira/browse/NIFI-4904
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 1.5.0
Environment: Ubuntu
Reporter: Dye357
Currently the PutElasticSearch5 component is using the following transport
artifact
org.elasticsearch.client
transport
${es.version}
Where es.version is 5.0.1. Upgrading to the highest 5.x dependency would enable
this component to be compatible with later 5.x versions of elastic search as
well as early versions of elastic search 6.x.
Here is Nifi 1.5.0 connecting to ES 6.2.1 on port 9300:
[2018-02-23T01:41:04,162][WARN ][o.e.t.n.Netty4Transport ] [uQSW8O8] exception
caught on transport layer [NettyTcpChannel\{localAddress=/127.0.0.1:9300,
remoteAddress=/127.0.0.1:57457}], closing connection
java.lang.IllegalStateException: Received message from unsupported version:
[5.0.0] minimal compatible version is: [5.6.0]
at
org.elasticsearch.transport.TcpTransport.ensureVersionCompatibility(TcpTransport.java:1430)
~[elasticsearch-6.2.1.jar:6.2.1]
at
org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1377)
~[elasticsearch-6.2.1.jar:6.2.1]
at
org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64)
~[transport-netty4-6.2.1.jar:6.2.1]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.util.concurrent.SingleThre
