[ https://issues.apache.org/jira/browse/RATIS-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902479#comment-16902479 ]
Josh Elser commented on RATIS-485: ---------------------------------- Haha, ok, so {{./client.sh filestore loadgen --numFiles 1024 --size 1024}} will give you a good time, lol. {noformat} 2019-08-07 17:03:40 DEBUG RaftClient:344 - client-6214A651F1DB: suggested new leader: null. Failed RaftClientRequest:client-6214A651F1DB->n1@group-6F7570313233, cid=0, seq=1*, RW, org.apache.ratis.examples.filestore.FileStoreClient$$Lambda$42/1815546035@6fb2ce1e with java.io.IOException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2019-08-07 17:03:40 DEBUG RaftClient:363 - client-6214A651F1DB: oldLeader=n1, curLeader=n1, newLeader=n2 2019-08-07 17:03:40 DEBUG RaftClient:371 - group-6F7570313233 client-6214A651F1DB: client change Leader from n1 to n2 ex=java.io.IOException 2019-08-07 17:03:41 DEBUG RaftClient:344 - client-6214A651F1DB: suggested new leader: null. Failed RaftClientRequest:client-6214A651F1DB->n2@group-6F7570313233, cid=0, seq=1*, RW, org.apache.ratis.examples.filestore.FileStoreClient$$Lambda$42/1815546035@6fb2ce1e with java.io.IOException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2019-08-07 17:03:41 DEBUG RaftClient:363 - client-6214A651F1DB: oldLeader=n2, curLeader=n2, newLeader=n1 2019-08-07 17:03:41 DEBUG RaftClient:371 - group-6F7570313233 client-6214A651F1DB: client change Leader from n2 to n1 ex=java.io.IOException 2019-08-07 17:04:03 WARN TimeoutScheduler:55 - Failed to task #0: java.lang.OutOfMemoryError: unable to create new native thread 2019-08-07 17:04:03 WARN TimeoutScheduler:55 - Failed to task #0: java.lang.OutOfMemoryError: unable to create new native thread {noformat} Shortly after this, my mac crashed hard :) Seems like something in the ratis-grpc binding might be leaking sockets? > Load Generator OOMs if Ratis Unavailable > ---------------------------------------- > > Key: RATIS-485 > URL: https://issues.apache.org/jira/browse/RATIS-485 > Project: Ratis > Issue Type: Bug > Components: examples > Reporter: Clay B. > Priority: Trivial > Attachments: loadgen.log > > > Running the load generator without a Ratis cluster (e.g. spurious node IPs) > results in an OOM. > If one has a single Ratis server it tries seemingly indefinitely: > {code:java} > vagrant@ratis-server:~/incubator-ratis$ > ./ratis-examples/src/main/bin/client.sh filestore loadgen --size 1048576 > --numFiles 100 --peers n0:127.0.0.1:1{code} > If one has two Ratis servers it OOMs: > {code:java} > vagrant@ratis-server:~/incubator-ratis$ > ./ratis-examples/src/main/bin/client.sh filestore loadgen --size 1048576 > --numFiles 100 --peers n0:127.0.0.1:1,n1:127.0.0.1:2 > [...] > 1/787867107@5e5792a0 with java.util.concurrent.CompletionException: > java.io.IOException: > org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io > exception > 2019-02-14 07:47:22 DEBUG RaftClient:417 - client-272A2E13A5DD: suggested new > leader: null. Failed > RaftClientRequest:client-272A2E13A5DD->n1@group-6F7570313233, cid=0, seq=0 > RW, > org.apache.ratis.examples.filestore.FileStoreClient$$Lambda$41/787867107@5e5792a0 > with java.io.IOException: > org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io > exception > 2019-02-14 07:47:22 DEBUG RaftClient:437 - client-272A2E13A5DD: change Leader > from n1 to n0 > 2019-02-14 07:47:22 DEBUG RaftClient:291 - schedule attempt #10740 with > policy RetryForeverNoSleep for > RaftClientRequest:client-272A2E13A5DD->n1@group-6F7570313233, cid=0, seq=0 > RW, > org.apache.ratis.examples.filestore.FileStoreClient$$Lambda$41/787867107@5e5792a0 > 2019-02-14 07:47:22 DEBUG RaftClient:323 - client-272A2E13A5DD: send* > RaftClientRequest:client-272A2E13A5DD->n0@group-6F7570313233, cid=0, seq=0 > RW, > org.apache.ratis.examples.filestore.FileStoreClient$$Lambda$41/787867107@5e5792a0 > 2019-02-14 07:47:22 DEBUG RaftClient:338 - client-272A2E13A5DD: Failed > RaftClientRequest:client-272A2E13A5DD->n0@group-6F7570313233, cid=0, seq=0 > RW, > org.apache.ratis.examples.filestore.FileStoreClient$$Lambda$41/787867107@5e5792a0 > with java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: > unable to create new native thread > Exception in thread "main" java.util.concurrent.CompletionException: > java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.ratis.client.impl.RaftClientImpl.lambda$sendRequestAsync$14(RaftClientImpl.java:349) > at > java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) > at > java.util.concurrent.CompletableFuture.uniExceptionallyStage(CompletableFuture.java:884) > at > java.util.concurrent.CompletableFuture.exceptionally(CompletableFuture.java:2196) > at > org.apache.ratis.client.impl.RaftClientImpl.sendRequestAsync(RaftClientImpl.java:334) > at > org.apache.ratis.client.impl.RaftClientImpl.sendRequestWithRetryAsync(RaftClientImpl.java:286) > at > org.apache.ratis.util.SlidingWindow$Client.sendOrDelayRequest(SlidingWindow.java:243) > at > org.apache.ratis.util.SlidingWindow$Client.retry(SlidingWindow.java:259) > at > org.apache.ratis.client.impl.RaftClientImpl.lambda$null$10(RaftClientImpl.java:293) > at > org.apache.ratis.util.TimeoutScheduler.lambda$onTimeout$0(TimeoutScheduler.java:85) > at > org.apache.ratis.util.TimeoutScheduler.lambda$onTimeout$1(TimeoutScheduler.java:104) > at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:50) > at org.apache.ratis.util.LogUtils$1.run(LogUtils.java:91) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:717) > at > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) > at > java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603) > at > java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334) > at > java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) > at > org.apache.ratis.util.TimeoutScheduler.schedule(TimeoutScheduler.java:117) > at > org.apache.ratis.util.TimeoutScheduler.onTimeout(TimeoutScheduler.java:104) > at > org.apache.ratis.util.TimeoutScheduler.onTimeout(TimeoutScheduler.java:82) > at > org.apache.ratis.util.TimeoutScheduler.onTimeout(TimeoutScheduler.java:134) > at > org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.onNext(GrpcClientProtocolClient.java:234) > at > org.apache.ratis.grpc.client.GrpcClientRpc.sendRequestAsync(GrpcClientRpc.java:71) > at > org.apache.ratis.client.impl.RaftClientImpl.sendRequestAsync(RaftClientImpl.java:324) > ... 15 more > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)