Hello all,
Who could help me to determine the cpu/memory need for nifi instance to fetch the data from Postgresql hosted in google ?
We got this error :
==> Error : executesql.error.message
Ran out of memory retrieving query results.
The procesor ExecuteSQL has this config : Set Au
It should be memory efficient so I think this is likely a configuration
aspect of your processor. Can you share the configuration for all
properties?
As a side note: if NiFi ran out of memory, you'd always want to restart it
because you are never sure what's the state of the JVM after an OOME.
Le
Hello.
I also think the problem is more about the processor, I guess ExecuteSQL.
Should play with batch configuration and commit flag to commit intermediate
FlowFile.
The out of memory exception makes me believe the full table is retrieved,
and if it is huge the FlowFile content is very large.
Aan: users@nifi.apache.org
Onderwerp: Re: Hardware requirement for NIFI instance
You don't often get email from
lapinoujou...@gmail.com<mailto:lapinoujou...@gmail.com>. Learn why this is
important<https://aka.ms/LearnAboutSenderIdentification>
Hello.
I also think the problem is more a
sers@nifi.apache.org
Subject: RE: Hardware requirement for NIFI instance
You don't often get email from
isha.lam...@virtualsciences.nl<mailto:isha.lam...@virtualsciences.nl>. Learn
why this is important<https://aka.ms/LearnAboutSenderIdentification>
We've had this occur when e
If I remember correctly, the default Fetch Size for Postgresql is to
get all the rows at once, which can certainly cause the problem.
Perhaps try setting Fetch Size to something like 1000 or so and see if
that alleviates the problem.
Regards,
Matt
On Thu, Jan 4, 2024 at 8:48 AM Etienne Jouvin wr
ery.fetchtime
180996
executesql.resultset.index
0
executesql.row.count
14961077
File Size
1.62 GB
Regards
Minh
Envoyé: jeudi 4 janvier 2024 à 17:18
De: "Matt Burgess"
À: users@nifi.apache.org
Objet: Re: Hardware requirement for NIFI instance
If I remember correctly, the def
ery.fetchtime
180996
executesql.resultset.index
0
executesql.row.count
14961077
File Size
1.62 GB
Regards
Minh
Envoyé: jeudi 4 janvier 2024 à 17:18
De: "Matt Burgess"
À: users@nifi.apache.org
Objet: Re: Hardware requirement for NIFI instance
If I remember correct
s
>
>
> *Envoyé:* jeudi 4 janvier 2024 à 19:01
> *De:* e-soci...@gmx.fr
> *À:* users@nifi.apache.org
> *Cc:* users@nifi.apache.org
> *Objet:* Re: Hardware requirement for NIFI instance
>
> Hello all,
>
> Thanks a lot for the reply.
>
> So for more details.
>
&
s possible to merge avro files to one avro file ?
>>
>> Regards
>>
>>
>> Envoyé: jeudi 4 janvier 2024 à 19:01
>> De: e-soci...@gmx.fr
>> À: users@nifi.apache.org
>> Cc: users@nifi.apache.org
>> Objet: Re: Hardware requirement for NIFI instance
important thing for us it has only one avro file by table.
>>
>> So it is possible to merge avro files to one avro file ?
>>
>> Regards
>>
>>
>> Envoyé: jeudi 4 janvier 2024 à 19:01
>> De: e-soci...@gmx.fr
>> À: users@nifi.apache.org
&g
perfectly
FAILED TEST "OUT OF MEMORY" if the properties are :
> 1 executeSQL with "select * from table"
Output Batch Size : 0
Fetch Size : 0
Regards
Envoyé: vendredi 5 janvier 2024 à 08:12
De: "Matt Burgess"
À: users@nifi.apache.org
Objet: Re: Hardware requi
AILED TEST "OUT OF MEMORY" if the properties are :
>
> > 1 executeSQL with "select * from table"
> Output Batch Size : 0
> Fetch Size : 0
> Regards
>
>
> *Envoyé:* vendredi 5 janvier 2024 à 08:12
> *De:* "Matt Burgess"
> *À:* users@nif
he Web UI works perfectly
>
>
> FAILED TEST "OUT OF MEMORY" if the properties are :
>
> > 1 executeSQL with "select * from table"
> Output Batch Size : 0
> Fetch Size : 0
> Regards
>
>
> *Envoyé:* vendredi 5 janvier 2024 à 08:12
> *De:* &qu
Envoyé: vendredi 5 janvier 2024 à 15:52
De: "Mark Payne"
À: "users@nifi.apache.org"
Objet: Re: Hardware requirement for NIFI instance
Thanks for following up. That actually makes sense. I don’t think Output Batch Size will play a very big role here. But Fetch Size, if I unders
15 matches
Mail list logo
