Re: java.lang.OutOfMemoryError: Java heap space : NIFI 1.23.2
Hello Joe, Thanks for the response. But I found the solution. Like the table size is around 7,6GB (avro) for 161000 rows And I setup Fetch size : 20 so the processor ExecuteSQL try to fetch all datas. Like the heap size is setup to 8GB I have reduced the "Fetch Size" to 1 and it is worked. Regards Envoyé: mardi 19 mars 2024 à 14:53 De: "Joe Witt" À: users@nifi.apache.org Objet: Re: java.lang.OutOfMemoryError: Java heap space : NIFI 1.23.2 Hello The key output is java.lang.OutOfMemoryError: Java heap space Review batch property options to limit response sizes in the database calls. Thanks On Tue, Mar 19, 2024 at 6:15 AMwrote: Hello I got the executeSQL processor does the sql command "select * from public.table1" It is a postgresql database. Here the end of properties of processor. Max Wait Time 0 seconds Normalize Table/Column Names false Use Avro Logical Types false Compression Format NONE Default Decimal Precision 10 Default Decimal Scale 0 Max Rows Per Flow File 0 Output Batch Size 0 Fetch Size: 20 Set Auto Commit :false The same sql command works in nifi 1.16.3 with the same configuration I don't know why it is failed. Thanks Need you help .. there is a strange error : 2024-03-19 12:58:37,683 ERROR [Load-Balanced Client Thread-6] org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task java.lang.OutOfMemoryError: Java heap space 2024-03-19 12:58:37,684 ERROR [Load-Balanced Client Thread-2] org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task java.lang.OutOfMemoryError: Java heap space at java.base/java.util.concurrent.CopyOnWriteArrayList.iterator(CopyOnWriteArrayList.java:1024) at java.base/java.util.concurrent.CopyOnWriteArraySet.iterator(CopyOnWriteArraySet.java:389) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:54) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2024-03-19 12:58:48,940 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2024-03-19 12:58:49,347 ERROR [Timer-Driven Process Thread-9] org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task java.lang.OutOfMemoryError: Java heap space 2024-03-19 12:58:49,351 INFO [NiFi Web Server-5835] org.apache.nifi.web.server.RequestLog 138.21.169.37 - CN=admin.plants, OU=NIFI [19/Mar/2024:12:58:48 +] "GET /nifi-api/flow/cluster/summary HTTP/1.1" 200 104 "https://nifi-01:9091/nifi/?processGroupId=0f426c92-018e-1000--3fca1e11=" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" 2024-03-19 12:58:48,559 ERROR [Load-Balanced Client Thread-3] org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task java.lang.OutOfMemoryError: Java heap space 2024-03-19 12:58:49,745 INFO [NiFi Web Server-5419] org.apache.nifi.web.server.RequestLog 138.21.169.37 - - [19/Mar/2024:12:58:49 +] "POST /rb_bf28073qyu?type=js3=v_4_srv_8_sn_B1E58A3741D2949DD454A88FF8A4BAF3_perc_10_ol_0_mul_1_app-3A4e195de4d0714591_1_app-3A44074a8878754fd3_1_app-3Ad1603d0792f56d4b_1_app-3A8092cfc902bb1761_1_rcs-3Acss_1=8=post=QKHPLOMPDHAQNPHMHHTUOHHKWJHFRNCG-0=1710851904127=https%3A%2F%2Fnifi-01%3A9091%2Fnifi%2F%3FprocessGroupId%3D0f426c92-018e-1000--3fca1e11%26componentIds%3D=3=d1603d0792f56d4b=1268897524=7xpdnw1j=1 HTTP/1.1" 200 109 "https://nifi-01:9091/nifi/?processGroupId=0f426c92-018e-1000--3fca1e11=" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" 2024-03-19 12:58:57,419 ERROR [Timer-Driven Process Thread-5] org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task java.lang.OutOfMemoryError: Java heap space 2024-03-19 12:58:50,209 WARN [NiFi Web Server-5689] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine leader for role 'Cluster Coordinator'; returning null org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/clu_quality_2/leaders/Cluster Coordinator at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at
Re: java.lang.OutOfMemoryError: Java heap space : NIFI 1.23.2
Specifically set Fetch Size to something like 1000, by default setting Fetch Size to zero will cause Postgres to fetch the entire ResultSet into memory [1]. We should probably change that default to avoid problems like this and with other drivers (for example, Oracle's default is 10 rows which is often inadequate). Regards, Matt [1] https://franckpachot.medium.com/oracle-postgres-jdbc-fetch-size-3012d494712 On Tue, Mar 19, 2024 at 9:56 AM Joe Witt wrote: > Hello > > The key output is > > java.lang.OutOfMemoryError: Java heap space > > Review batch property options to limit response sizes in the database > calls. > > Thanks > > > On Tue, Mar 19, 2024 at 6:15 AM wrote: > >> Hello >> >> I got the executeSQL processor does the sql command "select * from >> public.table1" >> It is a postgresql database. >> >> Here the end of properties of processor. >> >> Max Wait Time 0 seconds >> Normalize Table/Column Names false >> Use Avro Logical Types false >> Compression Format NONE >> Default Decimal Precision 10 >> Default Decimal Scale 0 >> Max Rows Per Flow File 0 >> Output Batch Size 0 >> Fetch Size: 20 >> Set Auto Commit :false >> >> >> The same sql command works in nifi 1.16.3 with the same configuration >> >> I don't know why it is failed. Thanks >> >> Need you help .. there is a strange error : >> >> 2024-03-19 12:58:37,683 ERROR [Load-Balanced Client Thread-6] >> org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task >> java.lang.OutOfMemoryError: Java heap space >> 2024-03-19 12:58:37,684 ERROR [Load-Balanced Client Thread-2] >> org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task >> java.lang.OutOfMemoryError: Java heap space >> at >> java.base/java.util.concurrent.CopyOnWriteArrayList.iterator(CopyOnWriteArrayList.java:1024) >> at >> java.base/java.util.concurrent.CopyOnWriteArraySet.iterator(CopyOnWriteArraySet.java:389) >> at >> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:54) >> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >> at >> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) >> at >> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >> at >> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >> at java.base/java.lang.Thread.run(Thread.java:834) >> 2024-03-19 12:58:48,940 INFO [main-EventThread] >> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED >> 2024-03-19 12:58:49,347 ERROR [Timer-Driven Process Thread-9] >> org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task >> java.lang.OutOfMemoryError: Java heap space >> 2024-03-19 12:58:49,351 INFO [NiFi Web Server-5835] >> org.apache.nifi.web.server.RequestLog 138.21.169.37 - CN=admin.plants, >> OU=NIFI [19/Mar/2024:12:58:48 +] "GET /nifi-api/flow/cluster/summary >> HTTP/1.1" 200 104 " >> https://nifi-01:9091/nifi/?processGroupId=0f426c92-018e-1000--3fca1e11=; >> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, >> like Gecko) Chrome/105.0.0.0 Safari/537.36" >> 2024-03-19 12:58:48,559 ERROR [Load-Balanced Client Thread-3] >> org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task >> java.lang.OutOfMemoryError: Java heap space >> 2024-03-19 12:58:49,745 INFO [NiFi Web Server-5419] >> org.apache.nifi.web.server.RequestLog 138.21.169.37 - - >> [19/Mar/2024:12:58:49 +] "POST >> /rb_bf28073qyu?type=js3=v_4_srv_8_sn_B1E58A3741D2949DD454A88FF8A4BAF3_perc_10_ol_0_mul_1_app-3A4e195de4d0714591_1_app-3A44074a8878754fd3_1_app-3Ad1603d0792f56d4b_1_app-3A8092cfc902bb1761_1_rcs-3Acss_1=8=post=QKHPLOMPDHAQNPHMHHTUOHHKWJHFRNCG-0=1710851904127=https%3A%2F%2Fnifi-01%3A9091%2Fnifi%2F%3FprocessGroupId%3D0f426c92-018e-1000--3fca1e11%26componentIds%3D=3=d1603d0792f56d4b=1268897524=7xpdnw1j=1 >> HTTP/1.1" 200 109 " >> https://nifi-01:9091/nifi/?processGroupId=0f426c92-018e-1000--3fca1e11=; >> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, >> like Gecko) Chrome/105.0.0.0 Safari/537.36" >> 2024-03-19 12:58:57,419 ERROR [Timer-Driven Process Thread-5] >> org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task >> java.lang.OutOfMemoryError: Java heap space >> 2024-03-19 12:58:50,209 WARN [NiFi Web Server-5689] >> o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine leader for >> role 'Cluster Coordinator'; returning null >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /nifi/clu_quality_2/leaders/Cluster >> Coordinator >> at >>
Re: java.lang.OutOfMemoryError: Java heap space : NIFI 1.23.2
Hello The key output is java.lang.OutOfMemoryError: Java heap space Review batch property options to limit response sizes in the database calls. Thanks On Tue, Mar 19, 2024 at 6:15 AM wrote: > Hello > > I got the executeSQL processor does the sql command "select * from > public.table1" > It is a postgresql database. > > Here the end of properties of processor. > > Max Wait Time 0 seconds > Normalize Table/Column Names false > Use Avro Logical Types false > Compression Format NONE > Default Decimal Precision 10 > Default Decimal Scale 0 > Max Rows Per Flow File 0 > Output Batch Size 0 > Fetch Size: 20 > Set Auto Commit :false > > > The same sql command works in nifi 1.16.3 with the same configuration > > I don't know why it is failed. Thanks > > Need you help .. there is a strange error : > > 2024-03-19 12:58:37,683 ERROR [Load-Balanced Client Thread-6] > org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task > java.lang.OutOfMemoryError: Java heap space > 2024-03-19 12:58:37,684 ERROR [Load-Balanced Client Thread-2] > org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task > java.lang.OutOfMemoryError: Java heap space > at > java.base/java.util.concurrent.CopyOnWriteArrayList.iterator(CopyOnWriteArrayList.java:1024) > at > java.base/java.util.concurrent.CopyOnWriteArraySet.iterator(CopyOnWriteArraySet.java:389) > at > org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:54) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > 2024-03-19 12:58:48,940 INFO [main-EventThread] > o.a.c.f.state.ConnectionStateManager State change: SUSPENDED > 2024-03-19 12:58:49,347 ERROR [Timer-Driven Process Thread-9] > org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task > java.lang.OutOfMemoryError: Java heap space > 2024-03-19 12:58:49,351 INFO [NiFi Web Server-5835] > org.apache.nifi.web.server.RequestLog 138.21.169.37 - CN=admin.plants, > OU=NIFI [19/Mar/2024:12:58:48 +] "GET /nifi-api/flow/cluster/summary > HTTP/1.1" 200 104 " > https://nifi-01:9091/nifi/?processGroupId=0f426c92-018e-1000--3fca1e11=; > "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, > like Gecko) Chrome/105.0.0.0 Safari/537.36" > 2024-03-19 12:58:48,559 ERROR [Load-Balanced Client Thread-3] > org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task > java.lang.OutOfMemoryError: Java heap space > 2024-03-19 12:58:49,745 INFO [NiFi Web Server-5419] > org.apache.nifi.web.server.RequestLog 138.21.169.37 - - > [19/Mar/2024:12:58:49 +] "POST > /rb_bf28073qyu?type=js3=v_4_srv_8_sn_B1E58A3741D2949DD454A88FF8A4BAF3_perc_10_ol_0_mul_1_app-3A4e195de4d0714591_1_app-3A44074a8878754fd3_1_app-3Ad1603d0792f56d4b_1_app-3A8092cfc902bb1761_1_rcs-3Acss_1=8=post=QKHPLOMPDHAQNPHMHHTUOHHKWJHFRNCG-0=1710851904127=https%3A%2F%2Fnifi-01%3A9091%2Fnifi%2F%3FprocessGroupId%3D0f426c92-018e-1000--3fca1e11%26componentIds%3D=3=d1603d0792f56d4b=1268897524=7xpdnw1j=1 > HTTP/1.1" 200 109 " > https://nifi-01:9091/nifi/?processGroupId=0f426c92-018e-1000--3fca1e11=; > "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, > like Gecko) Chrome/105.0.0.0 Safari/537.36" > 2024-03-19 12:58:57,419 ERROR [Timer-Driven Process Thread-5] > org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task > java.lang.OutOfMemoryError: Java heap space > 2024-03-19 12:58:50,209 WARN [NiFi Web Server-5689] > o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine leader for > role 'Cluster Coordinator'; returning null > org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /nifi/clu_quality_2/leaders/Cluster > Coordinator > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:54) > at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2480) > at > org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:243) > at > org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:232) > at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:94) > at >
java.lang.OutOfMemoryError: Java heap space : NIFI 1.23.2
Hello I got the executeSQL processor does the sql command "select * from public.table1" It is a postgresql database. Here the end of properties of processor. Max Wait Time 0 seconds Normalize Table/Column Names false Use Avro Logical Types false Compression Format NONE Default Decimal Precision 10 Default Decimal Scale 0 Max Rows Per Flow File 0 Output Batch Size 0 Fetch Size: 20 Set Auto Commit :false The same sql command works in nifi 1.16.3 with the same configuration I don't know why it is failed. Thanks Need you help .. there is a strange error : 2024-03-19 12:58:37,683 ERROR [Load-Balanced Client Thread-6] org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task java.lang.OutOfMemoryError: Java heap space 2024-03-19 12:58:37,684 ERROR [Load-Balanced Client Thread-2] org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task java.lang.OutOfMemoryError: Java heap space at java.base/java.util.concurrent.CopyOnWriteArrayList.iterator(CopyOnWriteArrayList.java:1024) at java.base/java.util.concurrent.CopyOnWriteArraySet.iterator(CopyOnWriteArraySet.java:389) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:54) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2024-03-19 12:58:48,940 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2024-03-19 12:58:49,347 ERROR [Timer-Driven Process Thread-9] org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task java.lang.OutOfMemoryError: Java heap space 2024-03-19 12:58:49,351 INFO [NiFi Web Server-5835] org.apache.nifi.web.server.RequestLog 138.21.169.37 - CN=admin.plants, OU=NIFI [19/Mar/2024:12:58:48 +] "GET /nifi-api/flow/cluster/summary HTTP/1.1" 200 104 "https://nifi-01:9091/nifi/?processGroupId=0f426c92-018e-1000--3fca1e11=" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" 2024-03-19 12:58:48,559 ERROR [Load-Balanced Client Thread-3] org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task java.lang.OutOfMemoryError: Java heap space 2024-03-19 12:58:49,745 INFO [NiFi Web Server-5419] org.apache.nifi.web.server.RequestLog 138.21.169.37 - - [19/Mar/2024:12:58:49 +] "POST /rb_bf28073qyu?type=js3=v_4_srv_8_sn_B1E58A3741D2949DD454A88FF8A4BAF3_perc_10_ol_0_mul_1_app-3A4e195de4d0714591_1_app-3A44074a8878754fd3_1_app-3Ad1603d0792f56d4b_1_app-3A8092cfc902bb1761_1_rcs-3Acss_1=8=post=QKHPLOMPDHAQNPHMHHTUOHHKWJHFRNCG-0=1710851904127=https%3A%2F%2Fnifi-01%3A9091%2Fnifi%2F%3FprocessGroupId%3D0f426c92-018e-1000--3fca1e11%26componentIds%3D=3=d1603d0792f56d4b=1268897524=7xpdnw1j=1 HTTP/1.1" 200 109 "https://nifi-01:9091/nifi/?processGroupId=0f426c92-018e-1000--3fca1e11=" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" 2024-03-19 12:58:57,419 ERROR [Timer-Driven Process Thread-5] org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task java.lang.OutOfMemoryError: Java heap space 2024-03-19 12:58:50,209 WARN [NiFi Web Server-5689] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine leader for role 'Cluster Coordinator'; returning null org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/clu_quality_2/leaders/Cluster Coordinator at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2480) at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:243) at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:232) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:94) at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:229) at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:220) at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:42) at