Mirza Aliev created IGNITE-16562: ------------------------------------ Summary: ItSortAggregateTest.initTestData is flaky Key: IGNITE-16562 URL: https://issues.apache.org/jira/browse/IGNITE-16562 Project: Ignite Issue Type: Bug Reporter: Mirza Aliev Attachments: _Integration_Tests_Module_Runner_1655.log
ItSortAggregateTest#initTestData could not be completed because of spamming {noformat} 2022-02-15 12:34:59:608 +0300 [ERROR][%ItSortAggregateTest_null_2%Raft-Group-Client-8][RaftGroupServiceImpl] Failed to refresh a leader [groupId=a6328a6b-7da8-4a8f-a1b5-a27c75c6007b_part_9] java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:502) at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.lambda$accept$1(RaftGroupServiceImpl.java:544) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.util.concurrent.TimeoutException {noformat} Log contains suspicions warnings from scalecube on a very beginning {noformat} [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650003] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650004] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650005] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650006] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650007] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650008] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650009] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650010] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650011] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650012] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650013] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650014] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650015] Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] Skipping to add/update member: {m: default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [12:34:57]W: [ {noformat} Seems that this might be the reason why some groups cannot elect leaders as far as we get {{Failed to refresh a leader}} for some partitions groups, but we cannot get leader elected message like {{Node <a6328a6b-7da8-4a8f-a1b5-a27c75c6007b_part_4/172.17.0.6:3344> become leader of group}} for some partitions -- This message was sent by Atlassian Jira (v8.20.1#820001)