[jira] [Comment Edited] (NIFI-12685) NiFi Cluster - java.net.SocketTimeoutException: timeout
[
https://issues.apache.org/jira/browse/NIFI-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812750#comment-17812750
]
Giovanni edited comment on NIFI-12685 at 1/31/24 3:27 PM:
--
After further analysis I found a recurring error on the primary node:
{code:java}
2024-01-31 15:14:21,158 ERROR [Load-Balanced Client Thread-3]
o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to
tor1nifi1.skl.one:8443 for load balancing {code}
It was due to iptables that was blocking port 6342
(nifi.cluster.load.balance.port) on every node. Adding the rule to iptables
resolved the issue.
was (Author: JIRAUSER300198):
After further analysis I found a recurring error on the primary node:
2024-01-31 15:14:21,158 ERROR [Load-Balanced Client Thread-3]
o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to
tor1nifi1.skl.one:8443 for load balancing
It was due to iptables that was blocking port 6342
(nifi.cluster.load.balance.port) on every node. Adding the rule to iptables
resolved the issue.
> NiFi Cluster - java.net.SocketTimeoutException: timeout
>
>
> Key: NIFI-12685
> URL: https://issues.apache.org/jira/browse/NIFI-12685
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.22.0
>Reporter: Giovanni
>Priority: Blocker
> Attachments: image-2024-01-30-10-19-30-651.png
>
>
> Hi,
> I have a 3 nodes cluster and every time I try to edit a workflow one of the
> nodes goes timeout and UI return the following error:
> !image-2024-01-30-10-19-30-651.png!
>
> At the same time one of the nodes reports the following log in the
> nifi-app.log:
> {code:java}
> 2024-01-30 09:11:28,489 WARN [Replicate Request Thread-496]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> java.net.SocketTimeoutException: timeout
> at okio.SocketAsyncTimeout.newTimeoutException(JvmOkio.kt:147)
> at okio.AsyncTimeout.access$newTimeoutException(AsyncTimeout.kt:158)
> at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:337)
> at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:427)
> at
> okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:320)
> at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
> at
> okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:180)
> at
> okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110)
> at
> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
> at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
> at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
> at
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
> at
> okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
> at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
> at
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:136)
> at
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:130)
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:645)
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:869)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.net.SocketException: Socket closed
> at
> java.base/java.net.SocketInputStream.read(SocketInputStream.java:183)
>
[jira] [Resolved] (NIFI-12685) NiFi Cluster - java.net.SocketTimeoutException: timeout
[
https://issues.apache.org/jira/browse/NIFI-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni resolved NIFI-12685.
-
Resolution: Not A Bug
After further analysis I found a recurring error on the primary node:
2024-01-31 15:14:21,158 ERROR [Load-Balanced Client Thread-3]
o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to
tor1nifi1.skl.one:8443 for load balancing
It was due to iptables that was blocking port 6342
(nifi.cluster.load.balance.port) on every node. Adding the rule to iptables
resolved the issue.
> NiFi Cluster - java.net.SocketTimeoutException: timeout
>
>
> Key: NIFI-12685
> URL: https://issues.apache.org/jira/browse/NIFI-12685
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.22.0
>Reporter: Giovanni
>Priority: Blocker
> Attachments: image-2024-01-30-10-19-30-651.png
>
>
> Hi,
> I have a 3 nodes cluster and every time I try to edit a workflow one of the
> nodes goes timeout and UI return the following error:
> !image-2024-01-30-10-19-30-651.png!
>
> At the same time one of the nodes reports the following log in the
> nifi-app.log:
> {code:java}
> 2024-01-30 09:11:28,489 WARN [Replicate Request Thread-496]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> java.net.SocketTimeoutException: timeout
> at okio.SocketAsyncTimeout.newTimeoutException(JvmOkio.kt:147)
> at okio.AsyncTimeout.access$newTimeoutException(AsyncTimeout.kt:158)
> at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:337)
> at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:427)
> at
> okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:320)
> at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
> at
> okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:180)
> at
> okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110)
> at
> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
> at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
> at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
> at
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
> at
> okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
> at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
> at
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:136)
> at
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:130)
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:645)
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:869)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.net.SocketException: Socket closed
> at
> java.base/java.net.SocketInputStream.read(SocketInputStream.java:183)
> at
> java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
> at
> java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
> at
> java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
> at
> java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
> at
> java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1455)
[jira] [Created] (NIFI-12685) NiFi Cluster - java.net.SocketTimeoutException: timeout
Giovanni created NIFI-12685:
---
Summary: NiFi Cluster - java.net.SocketTimeoutException: timeout
Key: NIFI-12685
URL: https://issues.apache.org/jira/browse/NIFI-12685
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.22.0
Reporter: Giovanni
Attachments: image-2024-01-30-10-19-30-651.png
Hi,
I have a 3 nodes cluster and every time I try to edit a workflow one of the
nodes goes timeout and UI return the following error:
!image-2024-01-30-10-19-30-651.png!
At the same time one of the nodes reports the following log in the nifi-app.log:
{code:java}
2024-01-30 09:11:28,489 WARN [Replicate Request Thread-496]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator
java.net.SocketTimeoutException: timeout
at okio.SocketAsyncTimeout.newTimeoutException(JvmOkio.kt:147)
at okio.AsyncTimeout.access$newTimeoutException(AsyncTimeout.kt:158)
at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:337)
at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:427)
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:320)
at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
at
okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:180)
at
okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110)
at
okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at
okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at
okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at
okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at
okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at
okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
at
org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:136)
at
org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:130)
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:645)
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:869)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.SocketException: Socket closed
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:183)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at
java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
at
java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
at
java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
at
java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1455)
at
java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066)
at okio.InputStreamSource.read(JvmOkio.kt:94)
at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:125)
... 26 common frames omitted {code}
As I thought it was a performance issue, I doubled each nodes resources but the
error is still occurring.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12528) Funnels bug
[ https://issues.apache.org/jira/browse/NIFI-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-12528: Description: I've notice a bug in our working cluster of 4 nodes. When two funnels are connected, with no other connections as themselves, like in attached image, is not possible to delete them. After trying to modify one of their connections, the Node where the action is performed crashes, and will not turn on unless the entire cluster gets restarted. was: I've notice a bug in our working cluster of 4 nodes. When two funnels are connected, with no other connections as themselves, like in attached image, is not possible to delete them. After trying to modify one of their connections, the Node where the action is performed crashes, and will not turn on unless the entire cluster restarts. > Funnels bug > --- > > Key: NIFI-12528 > URL: https://issues.apache.org/jira/browse/NIFI-12528 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.23.2 > Environment: NiFi cluster >Reporter: Giovanni >Priority: Critical > Attachments: image001.png > > > I've notice a bug in our working cluster of 4 nodes. > When two funnels are connected, with no other connections as themselves, like > in attached image, is not possible to delete them. > After trying to modify one of their connections, the Node where the action is > performed crashes, and will not turn on unless the entire cluster gets > restarted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12528) Funnels bug
Giovanni created NIFI-12528: --- Summary: Funnels bug Key: NIFI-12528 URL: https://issues.apache.org/jira/browse/NIFI-12528 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.23.2 Environment: NiFi cluster Reporter: Giovanni Attachments: image001.png I've notice a bug in our working cluster of 4 nodes. When two funnels are connected, with no other connections as themselves, like in attached image, is not possible to delete them. After trying to modify one of their connections, the Node where the action is performed crashes, and will not turn on unless the entire cluster restarts. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12442) Adding support for RocksDB
[ https://issues.apache.org/jira/browse/NIFI-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795080#comment-17795080 ] Giovanni commented on NIFI-12442: - Dear David, As you suggested, I've implemented the DistributedMapCacheClient interface, while still keeping the old components within the project. There's a pre-release version, feel free to try it. When creating the services, you can add dynamic properties to add RocksOptions to the RocksDb instance. I suggest you to use the option: "setCreateIfMissing", "true", to create automatically the RocksDb instance if not existing. I'm up to other suggestions. Soon I'll release it to maven central. Here's the link to the pre-release: [Pre-release v1.0.1|https://github.com/kommpn/nifi-rocksdb-manager/releases/tag/v1.0.1] > Adding support for RocksDB > -- > > Key: NIFI-12442 > URL: https://issues.apache.org/jira/browse/NIFI-12442 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.23.2 >Reporter: Giovanni >Priority: Minor > Original Estimate: 336h > Remaining Estimate: 336h > > I would like to suggest the creation of 3 new components: > The first one is a service, which opens an existing RocksDb or, eventually, > using RocksOptions, create it from scratch. It will manage all the open > options (classic, read/write, only read, secondary). > The second one, is a RocksDbReader, that uses the service to communicate with > the RocksDb in order to retrieve informations through a lookup. It can save > the searched content inside an attribute or inside the flowFile content. It > will be capable of using both APIs, such as simple "db.get" and via > RocksIterator. > The last one, is a RocksDbWriter, that uses the same service as the reader, > but can write values inside the RocksDb, both from flowFile attribute or > flowFile content, using properties to determine the key to use. > > Feel free to express your opinions, if you think this will be useful or > useless. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-12442) Adding support for RocksDB
[ https://issues.apache.org/jira/browse/NIFI-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795080#comment-17795080 ] Giovanni edited comment on NIFI-12442 at 12/10/23 1:59 PM: --- Dear David, As you suggested, I've implemented the DistributedMapCacheClient interface, while still keeping the old components within the project. There's a pre-release version, feel free to try it. When creating the services, you can add dynamic properties to add RocksOptions to the RocksDb instance. I suggest you to use the option: "setCreateIfMissing", "true", to create automatically the RocksDb instance if not existing. I'm up to other suggestions. Soon I'll release it to maven central. Here's the link to the pre-release with the NAR file in it: [Pre-release v1.0.1|https://github.com/kommpn/nifi-rocksdb-manager/releases/tag/v1.0.1] was (Author: JIRAUSER301883): Dear David, As you suggested, I've implemented the DistributedMapCacheClient interface, while still keeping the old components within the project. There's a pre-release version, feel free to try it. When creating the services, you can add dynamic properties to add RocksOptions to the RocksDb instance. I suggest you to use the option: "setCreateIfMissing", "true", to create automatically the RocksDb instance if not existing. I'm up to other suggestions. Soon I'll release it to maven central. Here's the link to the pre-release: [Pre-release v1.0.1|https://github.com/kommpn/nifi-rocksdb-manager/releases/tag/v1.0.1] > Adding support for RocksDB > -- > > Key: NIFI-12442 > URL: https://issues.apache.org/jira/browse/NIFI-12442 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.23.2 >Reporter: Giovanni >Priority: Minor > Original Estimate: 336h > Remaining Estimate: 336h > > I would like to suggest the creation of 3 new components: > The first one is a service, which opens an existing RocksDb or, eventually, > using RocksOptions, create it from scratch. It will manage all the open > options (classic, read/write, only read, secondary). > The second one, is a RocksDbReader, that uses the service to communicate with > the RocksDb in order to retrieve informations through a lookup. It can save > the searched content inside an attribute or inside the flowFile content. It > will be capable of using both APIs, such as simple "db.get" and via > RocksIterator. > The last one, is a RocksDbWriter, that uses the same service as the reader, > but can write values inside the RocksDb, both from flowFile attribute or > flowFile content, using properties to determine the key to use. > > Feel free to express your opinions, if you think this will be useful or > useless. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12442) Adding support for RocksDB
[ https://issues.apache.org/jira/browse/NIFI-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794129#comment-17794129 ] Giovanni commented on NIFI-12442: - Dear [~exceptionfactory], I've created a simple repository with a pre-release. Feel free to give me feedbacks. https://github.com/kommpn/nifi-rocksdb-manager > Adding support for RocksDB > -- > > Key: NIFI-12442 > URL: https://issues.apache.org/jira/browse/NIFI-12442 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.23.2 >Reporter: Giovanni >Priority: Minor > Original Estimate: 336h > Remaining Estimate: 336h > > I would like to suggest the creation of 3 new components: > The first one is a service, which opens an existing RocksDb or, eventually, > using RocksOptions, create it from scratch. It will manage all the open > options (classic, read/write, only read, secondary). > The second one, is a RocksDbReader, that uses the service to communicate with > the RocksDb in order to retrieve informations through a lookup. It can save > the searched content inside an attribute or inside the flowFile content. It > will be capable of using both APIs, such as simple "db.get" and via > RocksIterator. > The last one, is a RocksDbWriter, that uses the same service as the reader, > but can write values inside the RocksDb, both from flowFile attribute or > flowFile content, using properties to determine the key to use. > > Feel free to express your opinions, if you think this will be useful or > useless. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12442) Adding support for RocksDB
[ https://issues.apache.org/jira/browse/NIFI-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792522#comment-17792522 ] Giovanni commented on NIFI-12442: - Thanks for your considerations. Surely the advantage is the speed of calculation. A data flow possibly enrich data with half a million checks on 30 seconds with 2 clustered nodes (in my private experience, with my local clustered components). It's way efficient than other solutions. Obviously is up to the user to choose which one solution fits better for its purpose. At FS space cost and possibly RAM (depending of which RocksOptions the user choose), it could delivery a fast output. I understand the problematic relative to the maintainability and compatibility could be a problem. As you said, it's probably a better solution to create an external library, that receive constant updates, that a possible RocksDb chooser user could use. Thank you, at this point I think that this Ticket could be closed. > Adding support for RocksDB > -- > > Key: NIFI-12442 > URL: https://issues.apache.org/jira/browse/NIFI-12442 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.23.2 >Reporter: Giovanni >Priority: Minor > Original Estimate: 336h > Remaining Estimate: 336h > > I would like to suggest the creation of 3 new components: > The first one is a service, which opens an existing RocksDb or, eventually, > using RocksOptions, create it from scratch. It will manage all the open > options (classic, read/write, only read, secondary). > The second one, is a RocksDbReader, that uses the service to communicate with > the RocksDb in order to retrieve informations through a lookup. It can save > the searched content inside an attribute or inside the flowFile content. It > will be capable of using both APIs, such as simple "db.get" and via > RocksIterator. > The last one, is a RocksDbWriter, that uses the same service as the reader, > but can write values inside the RocksDb, both from flowFile attribute or > flowFile content, using properties to determine the key to use. > > Feel free to express your opinions, if you think this will be useful or > useless. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12442) Adding support for RocksDB
[ https://issues.apache.org/jira/browse/NIFI-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17791717#comment-17791717 ] Giovanni commented on NIFI-12442: - Thanks David for answering this question. The use case that you proposed is exactly the one I was thinking of. Working on multiple nodes, data will be written and read in all nodes FS, maintaining databases consistency. A potential use case can be the one where a Rocksdb is created in a separate instance, loaded inside NiFi fs via S3, and read/updated via data flow, with updated data, and then re-uploaded to S3. This is a simple use case, where a user has total management power on his RocksDb. Then data inside them can be used to enrich data retrieved from the internet. As you said, this could probably not be much viable in the official repo. But can still be useful in certain cases. > Adding support for RocksDB > -- > > Key: NIFI-12442 > URL: https://issues.apache.org/jira/browse/NIFI-12442 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.23.2 >Reporter: Giovanni >Priority: Minor > Original Estimate: 336h > Remaining Estimate: 336h > > I would like to suggest the creation of 3 new components: > The first one is a service, which opens an existing RocksDb or, eventually, > using RocksOptions, create it from scratch. It will manage all the open > options (classic, read/write, only read, secondary). > The second one, is a RocksDbReader, that uses the service to communicate with > the RocksDb in order to retrieve informations through a lookup. It can save > the searched content inside an attribute or inside the flowFile content. It > will be capable of using both APIs, such as simple "db.get" and via > RocksIterator. > The last one, is a RocksDbWriter, that uses the same service as the reader, > but can write values inside the RocksDb, both from flowFile attribute or > flowFile content, using properties to determine the key to use. > > Feel free to express your opinions, if you think this will be useful or > useless. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12442) Adding support for RocksDB
Giovanni created NIFI-12442: --- Summary: Adding support for RocksDB Key: NIFI-12442 URL: https://issues.apache.org/jira/browse/NIFI-12442 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.23.2 Reporter: Giovanni I would like to suggest the creation of 3 new components: The first one is a service, which opens an existing RocksDb or, eventually, using RocksOptions, create it from scratch. It will manage all the open options (classic, read/write, only read, secondary). The second one, is a RocksDbReader, that uses the service to communicate with the RocksDb in order to retrieve informations through a lookup. It can save the searched content inside an attribute or inside the flowFile content. It will be capable of using both APIs, such as simple "db.get" and via RocksIterator. The last one, is a RocksDbWriter, that uses the same service as the reader, but can write values inside the RocksDb, both from flowFile attribute or flowFile content, using properties to determine the key to use. Feel free to express your opinions, if you think this will be useful or useless. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11764) flowfile storage size mismatch
Giovanni created NIFI-11764:
---
Summary: flowfile storage size mismatch
Key: NIFI-11764
URL: https://issues.apache.org/jira/browse/NIFI-11764
Project: Apache NiFi
Issue Type: Improvement
Reporter: Giovanni
Attachments: image-2023-06-29-10-24-00-879.png,
image-2023-06-29-10-25-06-665.png, image-2023-06-29-10-27-17-702.png,
image-2023-06-29-10-28-45-547.png, image-2023-06-29-10-30-07-266.png,
image-2023-06-29-10-31-33-526.png
Hi,
Nifi is reporting a wrong value for the flowfile storage used space.
If I check the dashboard it reports 528.05 MB on all nodes:
!image-2023-06-29-10-24-00-879.png|width=574,height=206!
The values are also confirmed by the nodes status history:
!image-2023-06-29-10-25-06-665.png|width=584,height=431!
However my monitoring tool reports 52KB only:
!image-2023-06-29-10-27-17-702.png|width=997,height=165!
These low values are confirmed on the hosts themselves:
!image-2023-06-29-10-28-45-547.png|width=485,height=179!
!image-2023-06-29-10-30-07-266.png|width=484,height=178!
!image-2023-06-29-10-31-33-526.png|width=485,height=176!
The flowfile repository settings are the same on all nodes:
* nifi1
{code:java}
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=/var/nifi/flowfile_repo/data
nifi.flowfile.repository.checkpoint.interval=20 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=truenifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=2{code}
* nifi2
{code:java}
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=/var/nifi/flowfile_repo/data
nifi.flowfile.repository.checkpoint.interval=20 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=truenifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=2 {code}
* nifi3
{code:java}
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=/var/nifi/flowfile_repo/data
nifi.flowfile.repository.checkpoint.interval=20 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=truenifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=2 {code}
I also monitored the disk usage in realtime and it reports less than 100MB on
each node for a very few seconds. However still far from the steady values
reported in the GUI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737509#comment-17737509 ] Giovanni commented on NIFI-11530: - I got it. Thanks for your help [~joewitt] > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, > image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png, > image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png, > jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni resolved NIFI-11530. - Resolution: Not A Bug > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, > image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png, > image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png, > jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[
https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737221#comment-17737221
]
Giovanni edited comment on NIFI-11530 at 6/26/23 4:06 PM:
--
After I configured the volumes I also set the provenance max size to 9GB:
{code:java}
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=/var/nifi/provenance_repo/data
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=9 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false {code}
So 8.98GB could be a normal size according to the setting or should I expect
the repo size to be lower?
was (Author: JIRAUSER300198):
After I configured the volumes I also set the provenance max size to 9GB:
{code:java}
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=/var/nifi/provenance_repo/data
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=9 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false {code}
So 8.98GB could be a normal size according to the setting or should I expect
the repo size to be lower?
> Disk full even with nifi.content.repository.archive.max.usage.percentage set
> to 50%
> ---
>
> Key: NIFI-11530
> URL: https://issues.apache.org/jira/browse/NIFI-11530
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.20.0
> Environment: Ubuntu 20.04.5 LTS
>Reporter: Giovanni
>Priority: Major
> Attachments: 20230605_disk_usage.jpg, content_archive.jpg,
> flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png,
> image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png,
> image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png,
> jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg,
> provenance_archive.jpg
>
>
> Nifi primary node reports disk full causing all nodes to stop working.
> Restarting nifi service does not resolve.
> Restarting the VM does not resolve.
> The only way to fix is to clean te content_repository dir:
> rm -rf ./nifi/content_repository/*
>
> Unfortunately I have no logs of the issue ongoing.
>
> UPDATE:
> I'm having the problem again.
> Every archive size is more than 50% on each node, with 70%+ peak on
> coordinator node (see attachments).
> I'm also attaching nifi-app.log this time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[
https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737221#comment-17737221
]
Giovanni commented on NIFI-11530:
-
After I configured the volumes I also set the provenance max size to 9GB:
{code:java}
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=/var/nifi/provenance_repo/data
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=9 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false {code}
So 8.98GB could be a normal size according to the setting or should I expect
the repo size to be lower?
> Disk full even with nifi.content.repository.archive.max.usage.percentage set
> to 50%
> ---
>
> Key: NIFI-11530
> URL: https://issues.apache.org/jira/browse/NIFI-11530
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.20.0
> Environment: Ubuntu 20.04.5 LTS
>Reporter: Giovanni
>Priority: Major
> Attachments: 20230605_disk_usage.jpg, content_archive.jpg,
> flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png,
> image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png,
> image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png,
> jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg,
> provenance_archive.jpg
>
>
> Nifi primary node reports disk full causing all nodes to stop working.
> Restarting nifi service does not resolve.
> Restarting the VM does not resolve.
> The only way to fix is to clean te content_repository dir:
> rm -rf ./nifi/content_repository/*
>
> Unfortunately I have no logs of the issue ongoing.
>
> UPDATE:
> I'm having the problem again.
> Every archive size is more than 50% on each node, with 70%+ peak on
> coordinator node (see attachments).
> I'm also attaching nifi-app.log this time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-11530: Attachment: image-2023-06-26-11-16-48-186.png > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, > image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png, > image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png, > jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737060#comment-17737060 ] Giovanni commented on NIFI-11530: - Update: I reconfigured the repository to have a logical volume for each: !image-2023-06-26-11-13-19-289.png! The overall performance are improved. However the provenance repository is still problematic: !image-2023-06-26-11-15-00-379.png! The other repos are ok though: !image-2023-06-26-11-16-21-096.png! !image-2023-06-26-11-16-35-791.png! !image-2023-06-26-11-16-48-186.png! > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, > image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png, > image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png, > jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-11530: Attachment: image-2023-06-26-11-16-21-096.png > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, > image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png, > image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png, > jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-11530: Attachment: image-2023-06-26-11-16-35-791.png > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, > image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png, > image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png, > jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-11530: Attachment: image-2023-06-26-11-15-00-379.png > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, > image-2023-06-26-11-15-00-379.png, jvm.jpg, nifi1-app.log, nifi2-app.log, > nifi3-app.log, nifi_bug.jpg, provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-11530: Attachment: image-2023-06-26-11-13-19-289.png > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, jvm.jpg, > nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17730426#comment-17730426 ] Giovanni edited comment on NIFI-11530 at 6/8/23 7:40 AM: - Hi Joe, so what is the recommended configuration? Should separate logical volumes be enough? Also 1 volume per repository? Consider my nifi setup is a 3 nodes cluster. was (Author: JIRAUSER300198): Hi Joe, so what is the recommended configuration? Consider my nifi setup is a 3 nodes cluster. > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, > nifi_bug.jpg, provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17730426#comment-17730426 ] Giovanni commented on NIFI-11530: - Hi Joe, so what is the recommended configuration? Consider my nifi setup is a 3 nodes cluster. > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, > nifi_bug.jpg, provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[
https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729762#comment-17729762
]
Giovanni commented on NIFI-11530:
-
Hi Joe,
thanks for your reply.
Here are my settings:
{code:java}
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=./flowfile_repository
nifi.flowfile.repository.checkpoint.interval=20 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=true
nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=2{code}
{code:java}
# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=50 KB
nifi.content.repository.directory.default=./content_repository
nifi.content.repository.archive.max.retention.period=7 days
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/{code}
{code:java}
# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=10 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
# Comma-separated list of fields. Fields that are not indexed will not be
searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some
examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when
searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving
a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be
truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=10{code}
In response to your point, I confirm each node has only a single disk.
> Disk full even with nifi.content.repository.archive.max.usage.percentage set
> to 50%
> ---
>
> Key: NIFI-11530
> URL: https://issues.apache.org/jira/browse/NIFI-11530
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.20.0
> Environment: Ubuntu 20.04.5 LTS
>Reporter: Giovanni
>Priority: Major
> Attachments: 20230605_disk_usage.jpg, content_archive.jpg,
> flowfile_archive.jpg, jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log,
> nifi_bug.jpg, provenance_archive.jpg
>
>
> Nifi primary node reports disk full causing all nodes to stop working.
> Restarting nifi service does not resolve.
> Restarting the VM does not resolve.
> The only way to fix is to clean te content_repository dir:
> rm -rf ./nifi/content_repository/*
>
> Unfortunately I have no logs of the issue ongoing.
>
> UPDATE:
> I'm having the problem again.
> Every archive size is more than 50% on each node, with 70%+ peak on
> coordinator node (see attachments).
> I'm also attaching nifi-app.log this time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-11530: Description: Nifi primary node reports disk full causing all nodes to stop working. Restarting nifi service does not resolve. Restarting the VM does not resolve. The only way to fix is to clean te content_repository dir: rm -rf ./nifi/content_repository/* Unfortunately I have no logs of the issue ongoing. UPDATE: I'm having the problem again. Every archive size is more than 50% on each node, with 70%+ peak on coordinator node (see attachments). I'm also attaching nifi-app.log this time. was: Nifi primary node reports disk full causing all nodes to stop working. Restarting nifi service does not resolve. Restarting the VM does not resolve. The only way to fix is to clean te content_repository dir: rm -rf ./nifi/content_repository/* Unfortunately I have no logs of the issue ongoing. UPDATE: I'm having the problem again. Every archive size is more than 50% on each node, with 70%+ peak on coordinator node (see attachments). I'm also attaching nifi-app.log this time. > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, > nifi_bug.jpg, provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-11530: Description: Nifi primary node reports disk full causing all nodes to stop working. Restarting nifi service does not resolve. Restarting the VM does not resolve. The only way to fix is to clean te content_repository dir: rm -rf ./nifi/content_repository/* Unfortunately I have no logs of the issue ongoing. UPDATE: I'm having the problem again. Every archive size is more than 50% on each node, with 70%+ peak on coordinator node (see attachments). I'm also attaching nifi-app.log this time. was: Nifi primary node reports disk full causing all nodes to stop working. Restarting nifi service does not resolve. Restarting the VM does not resolve. The only way to fix is to clean te content_repository dir: rm -rf ./nifi/content_repository/* Unfortunately I have no logs of the issue ongoing. UPDATE: I'm having the problem again. Every archive size is more than 50% on each node, with 70%+ peak on coordinator node (see attachments). I'm also attaching nifi-app.log that is reporting the following error: > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, > nifi_bug.jpg, provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-11530: Attachment: 20230605_disk_usage.jpg > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, > nifi_bug.jpg, provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log that is reporting the following error: -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-11530: Description: Nifi primary node reports disk full causing all nodes to stop working. Restarting nifi service does not resolve. Restarting the VM does not resolve. The only way to fix is to clean te content_repository dir: rm -rf ./nifi/content_repository/* Unfortunately I have no logs of the issue ongoing. UPDATE: I'm having the problem again. Every archive size is more than 50% on each node, with 70%+ peak on coordinator node (see attachments). I'm also attaching nifi-app.log that is reporting the following error: was: Nifi primary node reports disk full causing all nodes to stop working. Restarting nifi service does not resolve. Restarting the VM does not resolve. The only way to fix is to clean te content_repository dir: rm -rf ./nifi/content_repository/* Unfortunately I have no logs of the issue ongoing. > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: content_archive.jpg, flowfile_archive.jpg, jvm.jpg, > nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log that is reporting the following error: -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-11530: Attachment: nifi1-app.log nifi2-app.log nifi3-app.log > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: content_archive.jpg, flowfile_archive.jpg, jvm.jpg, > nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni updated NIFI-11530: Attachment: content_archive.jpg flowfile_archive.jpg jvm.jpg provenance_archive.jpg > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: content_archive.jpg, flowfile_archive.jpg, jvm.jpg, > nifi_bug.jpg, provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
Giovanni created NIFI-11530: --- Summary: Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50% Key: NIFI-11530 URL: https://issues.apache.org/jira/browse/NIFI-11530 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.20.0 Environment: Ubuntu 20.04.5 LTS Reporter: Giovanni Attachments: nifi_bug.jpg Nifi primary node reports disk full causing all nodes to stop working. Restarting nifi service does not resolve. Restarting the VM does not resolve. The only way to fix is to clean te content_repository dir: rm -rf ./nifi/content_repository/* Unfortunately I have no logs of the issue ongoing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10869) ExtractText processor - RegEx captured twice
[ https://issues.apache.org/jira/browse/NIFI-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luigi De Giovanni updated NIFI-10869: - Priority: Minor (was: Major) > ExtractText processor - RegEx captured twice > > > Key: NIFI-10869 > URL: https://issues.apache.org/jira/browse/NIFI-10869 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.18.0 >Reporter: Luigi De Giovanni >Priority: Minor > Attachments: image (1).png, image (2).png, image.png > > > Hi, > There is an issue with the ExtractText processor, in the attempt of creating > FlowFile attributes from FlowFile contents. > When creating a custom property with a regex value that does not contain > named groups, the value captured in the group is added as 2 different > attributes. > E.G. > ||Property Name||Property Value||Expected FlowFile Attribute||Actual outcome|| > |param.alpha||param.alpha=alpha|param.alpha=alpha > param.alpha.1=alpha| > Reading the documentation, this might even be an expected behaviour, but if > so, it is preferable to have the captured value only as a single attribute, > without duplication. > Please see the attachment for an example. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10869) ExtractText processor - RegEx captured twice
[ https://issues.apache.org/jira/browse/NIFI-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luigi De Giovanni updated NIFI-10869: - Issue Type: Improvement (was: Bug) > ExtractText processor - RegEx captured twice > > > Key: NIFI-10869 > URL: https://issues.apache.org/jira/browse/NIFI-10869 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.18.0 >Reporter: Luigi De Giovanni >Priority: Major > Attachments: image (1).png, image (2).png, image.png > > > Hi, > There is an issue with the ExtractText processor, in the attempt of creating > FlowFile attributes from FlowFile contents. > When creating a custom property with a regex value that does not contain > named groups, the value captured in the group is added as 2 different > attributes. > E.G. > ||Property Name||Property Value||Expected FlowFile Attribute||Actual outcome|| > |param.alpha||param.alpha=alpha|param.alpha=alpha > param.alpha.1=alpha| > Reading the documentation, this might even be an expected behaviour, but if > so, it is preferable to have the captured value only as a single attribute, > without duplication. > Please see the attachment for an example. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10869) ExtractText processor - RegEx captured twice
[ https://issues.apache.org/jira/browse/NIFI-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luigi De Giovanni updated NIFI-10869: - Description: Hi, There is an issue with the ExtractText processor, in the attempt of creating FlowFile attributes from FlowFile contents. When creating a custom property with a regex value that does not contain named groups, the value captured in the group is added as 2 different attributes. E.G. ||Property Name||Property Value||Expected FlowFile Attribute||Actual outcome|| |param.alpha||param.alpha=alpha|param.alpha=alpha param.alpha.1=alpha| Reading the documentation, this might even be an expected behaviour, but if so, it is preferable to have the captured value only as a single attribute, without duplication. Please see the attachment for an example. was: Hi, There is an issue with the ExtractText processor, in the attempt of creating FlowFile attributes from FlowFile contents. When creating a custom property with a regex value that does not contain named groups, the value captured in the group is added as 2 different attributes. E.G. ||Property Name||Property Value||Expected FlowFile Attribute||Actual outcome|| |request.param.alpha||request.param.alpha=alpha|request.param.alpha=alpha request.param.alpha.1=alpha| Reading the documentation, this might even be an expected behaviour, but if so, it is preferable to have the captured value only as a single attribute, without duplication. Please see the attachment for an example. > ExtractText processor - RegEx captured twice > > > Key: NIFI-10869 > URL: https://issues.apache.org/jira/browse/NIFI-10869 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.18.0 >Reporter: Luigi De Giovanni >Priority: Major > Attachments: image (1).png, image (2).png, image.png > > > Hi, > There is an issue with the ExtractText processor, in the attempt of creating > FlowFile attributes from FlowFile contents. > When creating a custom property with a regex value that does not contain > named groups, the value captured in the group is added as 2 different > attributes. > E.G. > ||Property Name||Property Value||Expected FlowFile Attribute||Actual outcome|| > |param.alpha||param.alpha=alpha|param.alpha=alpha > param.alpha.1=alpha| > Reading the documentation, this might even be an expected behaviour, but if > so, it is preferable to have the captured value only as a single attribute, > without duplication. > Please see the attachment for an example. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10869) ExtractText processor - RegEx captured twice
[ https://issues.apache.org/jira/browse/NIFI-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luigi De Giovanni updated NIFI-10869: - Description: Hi, There is an issue with the ExtractText processor, in the attempt of creating FlowFile attributes from FlowFile contents. When creating a custom property with a regex value that does not contain named groups, the value captured in the group is added as 2 different attributes. E.G. ||Property Name||Property Value||Expected FlowFile Attribute||Actual outcome|| |request.param.alpha||request.param.alpha=alpha|request.param.alpha=alpha request.param.alpha.1=alpha| Reading the documentation, this might even be an expected behaviour, but if so, it is preferable to have the captured value only as a single attribute, without duplication. Please see the attachment for an example. was: Hi, There is an issue with the ExtractText processor, in the attempt of creating FlowFile attributes from FlowFile contents. When creating a custom property with a regex value that does not contain named groups, the value captured in the group is added as 2 different attributes. E.G. ||Property Name||Property Value||Expected FlowFile Attribute||Actual outcome|| |request.param.alpha||request.param.alpha=alpha|request.param.alpha=alpha request.param.alpha.1=alpha| Reading the documentation, this might even be an expected behaviour, but if so, it is preferable to have the captured value only as a single attribute, without duplication. Please see the attachment for a real example > ExtractText processor - RegEx captured twice > > > Key: NIFI-10869 > URL: https://issues.apache.org/jira/browse/NIFI-10869 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.18.0 >Reporter: Luigi De Giovanni >Priority: Minor > Attachments: image (1).png, image (2).png, image.png > > > Hi, > There is an issue with the ExtractText processor, in the attempt of creating > FlowFile attributes from FlowFile contents. > When creating a custom property with a regex value that does not contain > named groups, the value captured in the group is added as 2 different > attributes. > E.G. > ||Property Name||Property Value||Expected FlowFile Attribute||Actual outcome|| > |request.param.alpha| alpha>|request.param.alpha=alpha|request.param.alpha=alpha > request.param.alpha.1=alpha| > Reading the documentation, this might even be an expected behaviour, but if > so, it is preferable to have the captured value only as a single attribute, > without duplication. > Please see the attachment for an example. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10869) ExtractText processor - RegEx captured twice
[ https://issues.apache.org/jira/browse/NIFI-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luigi De Giovanni updated NIFI-10869: - Priority: Major (was: Minor) > ExtractText processor - RegEx captured twice > > > Key: NIFI-10869 > URL: https://issues.apache.org/jira/browse/NIFI-10869 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.18.0 >Reporter: Luigi De Giovanni >Priority: Major > Attachments: image (1).png, image (2).png, image.png > > > Hi, > There is an issue with the ExtractText processor, in the attempt of creating > FlowFile attributes from FlowFile contents. > When creating a custom property with a regex value that does not contain > named groups, the value captured in the group is added as 2 different > attributes. > E.G. > ||Property Name||Property Value||Expected FlowFile Attribute||Actual outcome|| > |request.param.alpha| alpha>|request.param.alpha=alpha|request.param.alpha=alpha > request.param.alpha.1=alpha| > Reading the documentation, this might even be an expected behaviour, but if > so, it is preferable to have the captured value only as a single attribute, > without duplication. > Please see the attachment for an example. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10869) ExtractText processor - RegEx captured twice
Luigi De Giovanni created NIFI-10869: Summary: ExtractText processor - RegEx captured twice Key: NIFI-10869 URL: https://issues.apache.org/jira/browse/NIFI-10869 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.18.0 Reporter: Luigi De Giovanni Attachments: image (1).png, image (2).png, image.png Hi, There is an issue with the ExtractText processor, in the attempt of creating FlowFile attributes from FlowFile contents. When creating a custom property with a regex value that does not contain named groups, the value captured in the group is added as 2 different attributes. E.G. ||Property Name||Property Value||Expected FlowFile Attribute||Actual outcome|| |request.param.alpha||request.param.alpha=alpha|request.param.alpha=alpha request.param.alpha.1=alpha| Reading the documentation, this might even be an expected behaviour, but if so, it is preferable to have the captured value only as a single attribute, without duplication. Please see the attachment for a real example -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10869) ExtractText processor - RegEx captured twice
[ https://issues.apache.org/jira/browse/NIFI-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luigi De Giovanni updated NIFI-10869: - Priority: Minor (was: Major) > ExtractText processor - RegEx captured twice > > > Key: NIFI-10869 > URL: https://issues.apache.org/jira/browse/NIFI-10869 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.18.0 >Reporter: Luigi De Giovanni >Priority: Minor > Attachments: image (1).png, image (2).png, image.png > > > Hi, > There is an issue with the ExtractText processor, in the attempt of creating > FlowFile attributes from FlowFile contents. > When creating a custom property with a regex value that does not contain > named groups, the value captured in the group is added as 2 different > attributes. > E.G. > ||Property Name||Property Value||Expected FlowFile Attribute||Actual outcome|| > |request.param.alpha| alpha>|request.param.alpha=alpha|request.param.alpha=alpha > request.param.alpha.1=alpha| > Reading the documentation, this might even be an expected behaviour, but if > so, it is preferable to have the captured value only as a single attribute, > without duplication. > Please see the attachment for a real example > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-3620) InvokeHttp should be able to send multipart forms
[ https://issues.apache.org/jira/browse/NIFI-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637755#comment-17637755 ] Luigi De Giovanni commented on NIFI-3620: - Hi, it would be great if this could be fixed. I have experienced the same issue recently and I managed to work around it. I am pasting below some info I shared on the slack channel. "I am dealing with an API which takes only POST requests and needs parameters passed in the body, as multipart/form-data .The InvokeHTTP processor doesn't work - with the API I am dealing with - if I define the request parameters as additional properties, like post:form:. This is because the form-data is attached by the processor to the URL. However, since the InvokeHTTP processor takes the contents of the input FlowFile as request body, I worked around that by creating the multipart/form-data text in a GenerateFlowFile or ReplaceText processor, with proper mime type and boundary. Passing this to the InvokeHTTP without custom attributes makes that API react correctly." The trick is basically in formatting the text of the body as an actual multipart/form mime type. I reproduced it by checking the standard specification for multipart/form-data available publicly. It would be great if the processor was able to do this formatting by itself. Also, multipart/form-data shouldn't be attached to the URL, but remain in the request body only. For any question, feel free > InvokeHttp should be able to send multipart forms > - > > Key: NIFI-3620 > URL: https://issues.apache.org/jira/browse/NIFI-3620 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Andre F de Miranda >Assignee: Andre F de Miranda >Priority: Major > > Currently InvokeHTTP is unable to send multipart forms, meaning it cannot be > used to a number of REST APIs endpoints. > This has been identified by users > https://lists.apache.org/thread.html/32654510792f62ab855a785265c5ad5747cc0490da225fd2d44ac10a@%3Cdev.nifi.apache.org%3E -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (NIFI-5901) Write JSON record in database
[ https://issues.apache.org/jira/browse/NIFI-5901 ]
Luigi De Giovanni deleted comment on NIFI-5901:
-
was (Author: JIRAUSER298268):
It would be great to know at least a workaround for this case, if this feature
is not being implemented.
JOLT transformations don't seem to work, handling the JSON as string, as all
the field names lose the double quotes.
> Write JSON record in database
> -
>
> Key: NIFI-5901
> URL: https://issues.apache.org/jira/browse/NIFI-5901
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Flo Rance
>Assignee: Mike Thomsen
>Priority: Minor
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> It would be good to be able to store a whole json record in databases that
> implement it (e.g. postgresql). This would require to define the field in the
> shema as json/jsonb and then let PutDatabaseRecord inserts the json value in
> the json/jsonb field.
> At the moment, it's possible to store a json/jsonb through Postgresql JDBC
> using the Java sql type 'OTHER':
> Object data = "\{...}"; // the JSON document
> PreparedStatement.setObject(1, data, java.sql.Types.OTHER);
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-5901) Write JSON record in database
[
https://issues.apache.org/jira/browse/NIFI-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631131#comment-17631131
]
Luigi De Giovanni commented on NIFI-5901:
-
It would be great to know at least a workaround for this case, if this feature
is not being implemented.
JOLT transformations don't seem to work, handling the JSON as string, as all
the field names lose the double quotes.
> Write JSON record in database
> -
>
> Key: NIFI-5901
> URL: https://issues.apache.org/jira/browse/NIFI-5901
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Flo Rance
>Assignee: Mike Thomsen
>Priority: Minor
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> It would be good to be able to store a whole json record in databases that
> implement it (e.g. postgresql). This would require to define the field in the
> shema as json/jsonb and then let PutDatabaseRecord inserts the json value in
> the json/jsonb field.
> At the moment, it's possible to store a json/jsonb through Postgresql JDBC
> using the Java sql type 'OTHER':
> Object data = "\{...}"; // the JSON document
> PreparedStatement.setObject(1, data, java.sql.Types.OTHER);
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
