Re: VolatileContentRepository removal

[Matthieu Ré]

Hi Mike, David,

Thanks for your answers and your clarity !

To sum up our use case, our team has set up two cloud-based NiFi
("stateful") clusters. The second one deals with a big amount of
record-based data, to drag data from sources (Kafka, files and databases)
to other systems (OpenSearch and internal tools) performing a lot of costly
transformations (in the last 5 minutes for instance, my prod instance
indicates 300G of read, 170G of write, within 400 processors). This cluster
deals with data we can recover from a process, so in this use case we are
data-loss tolerant. The 12 NiFi nodes of the cluster (as well as a 3-nodes
ZK cluster, and 1 NiFi Registry) are running on private-cloud VM instances.
Their operations on disks are not as performant as it could be on physical
machines, and are restricted to a certain amount of IOps that we can easily
reach with disk-based Repositories ; that is the main reason why we tried
the VolatileContentRepository. Since 1.13.1, we use a version of NiFi that
we build with our custom bundles and the fix that was linked in NIFI-8760
and for now it is working fine for us. We had some memory leaks on custom
processors but after correction and limitation on queue sizes we never
encountered OOM on the Volatile again.
Don't hesitate to ask for more information or precision on the use case, or
for any advice !

About the OOM on RAM-disk approach, the last time we tried it was on
1.14.0. In the next few weeks we will try to migrate to 1.16.0, I'll be
glad to investigate if we still experience OOMs and report it if we do. And
if we don't, it could be a great solution for us to replace the
VolatileContentRepository. I have to add that I just discovered the NiFi
Stateless engine and especially the ExecuteStateless processor and the
repository you mentioned, with some refacto we could reduce the number of
queues in between ExecuteStateless processors that would induce FlowFiles
to be written on disks. We will investigate if this is enough for our nodes
to be below the threshold of authorized IOps per volume.

So thanks to you I understand that the Volatile has some weaknesses and
could have great alternatives. If all these alternatives fail, I would be
glad to investigate for 1.17 the possibility to promote the
ByteArrayContentRepository to the main framework.

Thank you!
Matthieu

Le mer. 30 mars 2022 à 15:18, David Handermann 
a écrit :

> Hi Matthieu,
>
> Thanks for raising this question for discussion. Other maintainers may be
> able to provide additional background, but part of the reason for removing
> the VolatileContentRepository implementation was that there were some more
> fundamental problems with the implementation. Although various framework
> updates included patching the implementation along the way, the repository
> was not maintained on a regular basis, which resulted in it being broken
> for several releases.
>
> As Mike said, it would be helpful to share more details about your use
> case, and also to hear more about whether you still experience memory
> issues with the file system repository in current releases.
>
> On a related note, NiFi Stateless includes a new in-memory content
> repository named ByteArrayContentRepository [1]. It is currently packaged
> in the NiFi Stateless bundle, but it might be possible to consider
> promoting it to the framework level, if there is value in a non-persistent
> content repository going forward.
>
> Regards,
> David Handermann
>
> [1]
> https://github.com/apache/nifi/blob/main/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/repository/ByteArrayContentRepository.java
>
> On Wed, Mar 30, 2022 at 7:45 AM Mike Thomsen 
> wrote:
>
>> We've been moving away from supporting it for a while, and I think it
>> comes down to a lot of both factors when you consider the time
>> involved in getting good patches and reviewing them. That said, until
>> 1.17 is released, I think there's room for community members like you
>> and your team to work with us on fixing the gaps that made a strong
>> case for removing it.
>>
>> I think I saw in your ticket that you provided patches through Jira.
>> My recommendation would be to do a feature branch that reverts the
>> removal, applies your patches and submit it as a PR on GitHub. Then
>> request a review. Obviously, there's no guarantees there because it's
>> based on folks' time and energy to do a review, but that would be the
>> right process at least to move your request forward.
>>
>> In the long run, I think it would be a lot better for you to share
>> your use case with us and to see if there's a better route ahead for
>> your team and NiFi. Sounds like an interesting use case, so it would
>

VolatileContentRepository removal

[Matthieu Ré]

Hi everyone,

We wanted to talk about this ticket
https://issues.apache.org/jira/browse/NIFI-8760 and the
VolatileContentRepository... I understood that we weren't many to still use
this Repository, but in our use case with a very limited cloud environment
with strict IOps regulations, it fitted perfectly and we managed several To
of data per day efficiently.

We tested other repositories, even a FileSystemContentRepo with RAM based
disk that did not match the case since we experimented numerous OOMs with
the same amount of RAM mounted.

I provided a patch to fix it, that should be applied after 1.13.0 and a
refactor of Claims handling, waiting for a discussion about it. Now I read
that it should disappear in 1.17.0 :(

Is it due to a technical limitation for further features ? Or is it  too
costly to maintain it ?

Thanks! Regards,
Matthieu

Re: [NiFi-8760] Processors fail to process flowfiles with VolatileContentRepository

[Matthieu Ré]

Thank you very much for your answers !

That's a surprise ! The VolatileContentRepository seemed to answer
perfectly our need to treat a big amount of data with low resources and
especially low I/O on mounted disks, with non critical data and potential
data loss authorized.

I just tried your solution @Mark mounting a tmpfs and FileSystemRepository
(on 1.11.4), but it seems like for the same amount of data and same RAM
space used, the VolatileContentRepository used a constant <5% of space,
while the FileSystemRepository was using a very unstable amount of space,
frequently running out of space. (I must add that we don't store any
archive, nifi.content.repository.archive.enabled=false). Maybe am I missing
a configuration that consumes a lot of space with the FileSystemRepository ?

Stateless NiFi sounds very interesting ! Just had a look at pvillard's demo
(https://github.com/pvillard31/nifi-stateless-demo) and the framework's
readme (
https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-stateless),
but do you have any more resources about it ? I would like to understand a
little bit more what differs from the standard framework and how it can fit
our use case.

Have a nice day,
Matthieu


Le ven. 23 juil. 2021 à 17:20, Joe Witt  a écrit :

> It seems like any use case that we previously thought VolatileContentRepo
> would be good for now we'd say Stateless NiFi is a dramatically better
> approach.
>
> We need to doc this better but the capability is there now for sure.
>
> On Fri, Jul 23, 2021 at 8:13 AM Mark Payne  wrote:
>
>> Matthieu,
>>
>> I would highly recommend against using VolatileContentRepository. You’re
>> the first one I’ve heard of using it in a few years. Typically, the
>> FileSystemRepository is sufficient. If you truly want to run with the
>> content in RAM I would recommend creating a RAM Disk and pointing the
>> FileSystemRepository to that.
>>
>> Thanks
>> -Mark
>>
>>
>> On Jul 21, 2021, at 10:31 AM, Matthieu Ré  wrote:
>>
>> Hi Chris, thank you for your quick response
>>
>> I tried the flow with 1.13.2 and 1.13.1, and 1.14.0 just before the first
>> RC and it still had the problem, so I am not sure if this is related to the
>> session handling you pointed out, that has been fixed in 1.13.2
>>
>> Le mer. 21 juil. 2021 à 16:22, Chris Sampson 
>> a écrit :
>>
>>> 1.13.1 was known to have problems with session handling - see the
>>> Release Note "lowlights" for 1.13.1 [1]
>>>
>>> It is recommended to upgrade to version 1.13.2 (or the latest 1.14.0).
>>> If you can't upgrade then 1.13.0 would be better than 1.13.1.
>>>
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1
>>>
>>> ---
>>> *Chris Sampson*
>>> IT Consultant
>>> chris.samp...@naimuri.com
>>> <https://www.naimuri.com/>
>>>
>>>
>>> On Wed, 21 Jul 2021 at 15:14, Matthieu Ré  wrote:
>>>
>>>> Hi all,
>>>>
>>>> Currently using NiFi 1.11.4, we face a blocking issue trying to switch
>>>> to NiFi 1.13.1+ due to the VolatileContentRepository : some processors we
>>>> use (and probably others that we didn't try) were not able to process
>>>> flowfiles, such as MargeRecord, QueryRecord or SplitJson (logs are in the 
>>>> Jira
>>>> ticket NiFi-8760 <https://issues.apache.org/jira/browse/NIFI-8760>).
>>>>
>>>> I wanted to know if any of you guys are able to reproduce the issue,
>>>> and if this is not a misconfiguration from our side. The nifi.properties
>>>> and flow.xml.gz used are available in the ticket. If I am not missing
>>>> anything, we could identify that the issue could come from this commit
>>>> <https://github.com/apache/nifi/commit/528fce2407d092d4ced1a58fcc14d0bc6e660b89>
>>>>  since
>>>> it appeared with the 1.13.1 and the flow is working fine with 1.13.0.
>>>>
>>>> Open to contribute as much as I can if you confirm that this is not due
>>>> to a misconfiguration..
>>>>
>>>> Thanks !
>>>> Matthieu
>>>>
>>>
>>
>> --
>>
>> Matthieu RÉ
>> Data Scientist - Machine Learning Engineer - Dassault Systèmes
>>
>> ENSIIE, M2 AIC (Université Paris-Saclay)
>>
>> Tel: 0631609755
>>
>> Email: re.matth...@gmail.com
>>
>>
>>

Re: [NiFi-8760] Processors fail to process flowfiles with VolatileContentRepository

[Matthieu Ré]

Hi Chris, thank you for your quick response

I tried the flow with 1.13.2 and 1.13.1, and 1.14.0 just before the first
RC and it still had the problem, so I am not sure if this is related to the
session handling you pointed out, that has been fixed in 1.13.2

Le mer. 21 juil. 2021 à 16:22, Chris Sampson  a
écrit :

> 1.13.1 was known to have problems with session handling - see the Release
> Note "lowlights" for 1.13.1 [1]
>
> It is recommended to upgrade to version 1.13.2 (or the latest 1.14.0). If
> you can't upgrade then 1.13.0 would be better than 1.13.1.
>
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
> <https://www.naimuri.com/>
>
>
> On Wed, 21 Jul 2021 at 15:14, Matthieu Ré  wrote:
>
>> Hi all,
>>
>> Currently using NiFi 1.11.4, we face a blocking issue trying to switch to
>> NiFi 1.13.1+ due to the VolatileContentRepository : some processors we use
>> (and probably others that we didn't try) were not able to process
>> flowfiles, such as MargeRecord, QueryRecord or SplitJson (logs are in the 
>> Jira
>> ticket NiFi-8760 <https://issues.apache.org/jira/browse/NIFI-8760>).
>>
>> I wanted to know if any of you guys are able to reproduce the issue, and
>> if this is not a misconfiguration from our side. The nifi.properties and
>> flow.xml.gz used are available in the ticket. If I am not missing anything,
>> we could identify that the issue could come from this commit
>> <https://github.com/apache/nifi/commit/528fce2407d092d4ced1a58fcc14d0bc6e660b89>
>>  since
>> it appeared with the 1.13.1 and the flow is working fine with 1.13.0.
>>
>> Open to contribute as much as I can if you confirm that this is not due
>> to a misconfiguration..
>>
>> Thanks !
>> Matthieu
>>
>

-- 

Matthieu RÉ
Data Scientist - Machine Learning Engineer - Dassault Systèmes

ENSIIE, M2 AIC (Université Paris-Saclay)

Tel: 0631609755

Email: re.matth...@gmail.com

[NiFi-8760] Processors fail to process flowfiles with VolatileContentRepository

[Matthieu Ré]

Hi all,

Currently using NiFi 1.11.4, we face a blocking issue trying to switch to
NiFi 1.13.1+ due to the VolatileContentRepository : some processors we use
(and probably others that we didn't try) were not able to process
flowfiles, such as MargeRecord, QueryRecord or SplitJson (logs are in the Jira
ticket NiFi-8760 ).

I wanted to know if any of you guys are able to reproduce the issue, and if
this is not a misconfiguration from our side. The nifi.properties and
flow.xml.gz used are available in the ticket. If I am not missing anything,
we could identify that the issue could come from this commit

since
it appeared with the 1.13.1 and the flow is working fine with 1.13.0.

Open to contribute as much as I can if you confirm that this is not due to
a misconfiguration..

Thanks !
Matthieu