[
https://issues.apache.org/jira/browse/IGNITE-23817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor updated IGNITE-23817:
--------------------------
Description:
*Steps to reproduce:*
# Start 3 nodes (each node is CMG, each node
{color:#067d17}"-Xms4096m"{color}, {color:#067d17}"-Xmx4096m"{color}), each on
separate host. Each host vCPU: 4, Memory: 32GB.
# Create 50 tables with 5 columns in 10 threads.
# Assert 50 tables are present in system view.
# Insert 1 row into each.
# Assert rows content is correct in 10 threads.
# Repeat steps 2-5 while amount of tables is 1000.
*Expected:*
1000 tables are created.
*Actual:*
Exception during 500-549 tables creation at step 4.
{code:java}
org.opentest4j.AssertionFailedError: org.opentest4j.AssertionFailedError:
Execute: Insert row into tables 500 - 549 in 10 threads ==> Unexpected
exception thrown: java.util.concurrent.ExecutionException:
java.sql.SQLException: Send with retry timed out [retryCount = 2, groupId =
1023_part_11, traceId = 2b26c51f-5d03-4ac5-ba12-144cdc1b9e22, request =
org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.partition.replicator.network.command.UpdateCommandImpl),
originCommand = null].
org.opentest4j.AssertionFailedError: Execute: Insert row into tables 500 - 549
in 10 threads ==> Unexpected exception thrown:
java.util.concurrent.ExecutionException: java.sql.SQLException: Send with retry
timed out [retryCount = 2, groupId = 1023_part_11, traceId =
2b26c51f-5d03-4ac5-ba12-144cdc1b9e22, request =
org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.partition.replicator.network.command.UpdateCommandImpl),
originCommand = null].
at
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
at
app//org.junit.jupiter.api.AssertDoesNotThrow.createAssertionFailedError(AssertDoesNotThrow.java:84)
at
app//org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:53)
at
app//org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:40)
at
app//org.junit.jupiter.api.Assertions.assertDoesNotThrow(Assertions.java:3183)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$invokeTasks$14(TablesAmountCapacityBaseTest.java:381)
at app//io.qameta.allure.Allure.lambda$step$0(Allure.java:113)
at app//io.qameta.allure.Allure.lambda$step$1(Allure.java:127)
at app//io.qameta.allure.Allure.step(Allure.java:181)
at app//io.qameta.allure.Allure.step(Allure.java:125)
at app//io.qameta.allure.Allure.step(Allure.java:112)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.step(TablesAmountCapacityBaseTest.java:271)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.invokeTasks(TablesAmountCapacityBaseTest.java:376)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.insertRowInTablesParallel(TablesAmountCapacityBaseTest.java:183)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.testTablesAmount(TablesAmountCapacityBaseTest.java:92)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityMultiNodeTest.create1000EmptyTablesAmountOfColumnsEachAndMakeSimpleQueries(TablesAmountCapacityMultiNodeTest.java:97)
at [email protected]/java.lang.reflect.Method.invoke(Method.java:580)
at
app//io.qameta.allure.junit5.AllureJunit5.interceptTestTemplateMethod(AllureJunit5.java:59)
at [email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at [email protected]/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.ExecutionException: java.sql.SQLException: Send
with retry timed out [retryCount = 2, groupId = 1023_part_11, traceId =
2b26c51f-5d03-4ac5-ba12-144cdc1b9e22, request =
org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.partition.replicator.network.command.UpdateCommandImpl),
originCommand = null].
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$invokeTasks$13(TablesAmountCapacityBaseTest.java:383)
at
org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:49)
... 19 more
Caused by: java.sql.SQLException: Send with retry timed out [retryCount = 2,
groupId = 1023_part_11, traceId = 2b26c51f-5d03-4ac5-ba12-144cdc1b9e22, request
=
org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.partition.replicator.network.command.UpdateCommandImpl),
originCommand = null].
at
org.apache.ignite.internal.jdbc.proto.IgniteQueryErrorCode.createJdbcSqlException(IgniteQueryErrorCode.java:57)
at
org.apache.ignite.internal.jdbc.JdbcStatement.execute0(JdbcStatement.java:154)
at
org.apache.ignite.internal.jdbc.JdbcPreparedStatement.executeWithArguments(JdbcPreparedStatement.java:765)
at
org.apache.ignite.internal.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:173)
at
org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$insertRowInTablesParallel$4(TablesAmountCapacityBaseTest.java:175)
... 4 more{code}
Server logs are in attachment.
was:
*Steps to reproduce:*
# Start 3 nodes (each node is CMG, each node
{color:#067d17}"-Xms4096m"{color}, {color:#067d17}"-Xmx4096m"{color}), each on
separate host. Each host vCPU: 4, Memory: 32GB.
# Create 50 tables with 5 columns in 10 threads.
# Assert 50 tables are present in system view.
# Insert 1 row into each.
# Assert rows content is correct in 10 threads.
# Repeat steps 2-5 while amount of tables is 1000.
*Expected:*
1000 tables are created.
*Actual:*
Exception during 500-549 tables creation at step 4.
{code:java}
org.opentest4j.AssertionFailedError: org.opentest4j.AssertionFailedError:
Execute: Create tables 450 - 499 ==> Unexpected exception thrown:
java.util.concurrent.ExecutionException: java.sql.SQLException: Send with retry
timed out [retryCount = 4, groupId = metastorage_group, traceId =
4cc53c17-e23b-43ea-be7d-8e7f3ea4e772, request =
org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.metastorage.command.MultiInvokeCommandImpl),
originCommand = null].
org.opentest4j.AssertionFailedError: Execute: Create tables 450 - 499 ==>
Unexpected exception thrown: java.util.concurrent.ExecutionException:
java.sql.SQLException: Send with retry timed out [retryCount = 4, groupId =
metastorage_group, traceId = 4cc53c17-e23b-43ea-be7d-8e7f3ea4e772, request =
org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.metastorage.command.MultiInvokeCommandImpl),
originCommand = null].
at
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
at
app//org.junit.jupiter.api.AssertDoesNotThrow.createAssertionFailedError(AssertDoesNotThrow.java:84)
at
app//org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:53)
at
app//org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:40)
at
app//org.junit.jupiter.api.Assertions.assertDoesNotThrow(Assertions.java:3183)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$invokeTasks$14(TablesAmountCapacityBaseTest.java:381)
at app//io.qameta.allure.Allure.lambda$step$0(Allure.java:113)
at app//io.qameta.allure.Allure.lambda$step$1(Allure.java:127)
at app//io.qameta.allure.Allure.step(Allure.java:181)
at app//io.qameta.allure.Allure.step(Allure.java:125)
at app//io.qameta.allure.Allure.step(Allure.java:112)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.step(TablesAmountCapacityBaseTest.java:271)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.invokeTasks(TablesAmountCapacityBaseTest.java:376)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.createTablesParallel(TablesAmountCapacityBaseTest.java:129)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.testTablesAmount(TablesAmountCapacityBaseTest.java:88)
at
app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityMultiNodeTest.create1000EmptyTablesAmountOfColumnsEachAndMakeSimpleQueries(TablesAmountCapacityMultiNodeTest.java:97)
at [email protected]/java.lang.reflect.Method.invoke(Method.java:580)
at
app//io.qameta.allure.junit5.AllureJunit5.interceptTestTemplateMethod(AllureJunit5.java:59)
at [email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at [email protected]/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.ExecutionException: java.sql.SQLException: Send
with retry timed out [retryCount = 4, groupId = metastorage_group, traceId =
4cc53c17-e23b-43ea-be7d-8e7f3ea4e772, request =
org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.metastorage.command.MultiInvokeCommandImpl),
originCommand = null].
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$invokeTasks$13(TablesAmountCapacityBaseTest.java:383)
at
org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:49)
... 19 more
Caused by: java.sql.SQLException: Send with retry timed out [retryCount = 4,
groupId = metastorage_group, traceId = 4cc53c17-e23b-43ea-be7d-8e7f3ea4e772,
request =
org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.metastorage.command.MultiInvokeCommandImpl),
originCommand = null].
at
org.apache.ignite.internal.jdbc.proto.IgniteQueryErrorCode.createJdbcSqlException(IgniteQueryErrorCode.java:57)
at
org.apache.ignite.internal.jdbc.JdbcStatement.execute0(JdbcStatement.java:154)
at
org.apache.ignite.internal.jdbc.JdbcStatement.executeUpdate(JdbcStatement.java:177)
at
org.gridgain.qaaas.sql.generator.TableGeneratorHelper.createTable(TableGeneratorHelper.java:98)
at
org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$createTablesParallel$0(TablesAmountCapacityBaseTest.java:121)
... 4 more {code}
Server logs are in attachment.
> Tables creation in 10 threads throws "Send with retry timed out"
> ----------------------------------------------------------------
>
> Key: IGNITE-23817
> URL: https://issues.apache.org/jira/browse/IGNITE-23817
> Project: Ignite
> Issue Type: Bug
> Components: persistence
> Affects Versions: 3.0.0-beta1
> Environment: 3 nodes (each node is CMG, each node
> {color:#067d17}"-Xms4096m"{color}, {color:#067d17}"-Xmx4096m"{color}), each
> on separate host. Each host vCPU: 4, Memory: 32GB.
> Reporter: Igor
> Priority: Major
> Labels: ignite-3
> Attachments:
> 20241202_034128_450000_TablesAmountCapacityMultiNodeTest_cluster.zip
>
>
> *Steps to reproduce:*
> # Start 3 nodes (each node is CMG, each node
> {color:#067d17}"-Xms4096m"{color}, {color:#067d17}"-Xmx4096m"{color}), each
> on separate host. Each host vCPU: 4, Memory: 32GB.
> # Create 50 tables with 5 columns in 10 threads.
> # Assert 50 tables are present in system view.
> # Insert 1 row into each.
> # Assert rows content is correct in 10 threads.
> # Repeat steps 2-5 while amount of tables is 1000.
> *Expected:*
> 1000 tables are created.
> *Actual:*
> Exception during 500-549 tables creation at step 4.
>
> {code:java}
> org.opentest4j.AssertionFailedError: org.opentest4j.AssertionFailedError:
> Execute: Insert row into tables 500 - 549 in 10 threads ==> Unexpected
> exception thrown: java.util.concurrent.ExecutionException:
> java.sql.SQLException: Send with retry timed out [retryCount = 2, groupId =
> 1023_part_11, traceId = 2b26c51f-5d03-4ac5-ba12-144cdc1b9e22, request =
> org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.partition.replicator.network.command.UpdateCommandImpl),
> originCommand = null].
> org.opentest4j.AssertionFailedError: Execute: Insert row into tables 500 -
> 549 in 10 threads ==> Unexpected exception thrown:
> java.util.concurrent.ExecutionException: java.sql.SQLException: Send with
> retry timed out [retryCount = 2, groupId = 1023_part_11, traceId =
> 2b26c51f-5d03-4ac5-ba12-144cdc1b9e22, request =
> org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.partition.replicator.network.command.UpdateCommandImpl),
> originCommand = null].
> at
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
> at
> app//org.junit.jupiter.api.AssertDoesNotThrow.createAssertionFailedError(AssertDoesNotThrow.java:84)
> at
> app//org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:53)
> at
> app//org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:40)
> at
> app//org.junit.jupiter.api.Assertions.assertDoesNotThrow(Assertions.java:3183)
> at
> app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$invokeTasks$14(TablesAmountCapacityBaseTest.java:381)
> at app//io.qameta.allure.Allure.lambda$step$0(Allure.java:113)
> at app//io.qameta.allure.Allure.lambda$step$1(Allure.java:127)
> at app//io.qameta.allure.Allure.step(Allure.java:181)
> at app//io.qameta.allure.Allure.step(Allure.java:125)
> at app//io.qameta.allure.Allure.step(Allure.java:112)
> at
> app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.step(TablesAmountCapacityBaseTest.java:271)
> at
> app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.invokeTasks(TablesAmountCapacityBaseTest.java:376)
> at
> app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.insertRowInTablesParallel(TablesAmountCapacityBaseTest.java:183)
> at
> app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.testTablesAmount(TablesAmountCapacityBaseTest.java:92)
> at
> app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityMultiNodeTest.create1000EmptyTablesAmountOfColumnsEachAndMakeSimpleQueries(TablesAmountCapacityMultiNodeTest.java:97)
> at [email protected]/java.lang.reflect.Method.invoke(Method.java:580)
> at
> app//io.qameta.allure.junit5.AllureJunit5.interceptTestTemplateMethod(AllureJunit5.java:59)
> at [email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:317)
> at
> [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at [email protected]/java.lang.Thread.run(Thread.java:1583)
> Caused by: java.util.concurrent.ExecutionException: java.sql.SQLException:
> Send with retry timed out [retryCount = 2, groupId = 1023_part_11, traceId =
> 2b26c51f-5d03-4ac5-ba12-144cdc1b9e22, request =
> org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.partition.replicator.network.command.UpdateCommandImpl),
> originCommand = null].
> at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$invokeTasks$13(TablesAmountCapacityBaseTest.java:383)
> at
> org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:49)
> ... 19 more
> Caused by: java.sql.SQLException: Send with retry timed out [retryCount = 2,
> groupId = 1023_part_11, traceId = 2b26c51f-5d03-4ac5-ba12-144cdc1b9e22,
> request =
> org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.partition.replicator.network.command.UpdateCommandImpl),
> originCommand = null].
> at
> org.apache.ignite.internal.jdbc.proto.IgniteQueryErrorCode.createJdbcSqlException(IgniteQueryErrorCode.java:57)
> at
> org.apache.ignite.internal.jdbc.JdbcStatement.execute0(JdbcStatement.java:154)
> at
> org.apache.ignite.internal.jdbc.JdbcPreparedStatement.executeWithArguments(JdbcPreparedStatement.java:765)
> at
> org.apache.ignite.internal.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:173)
> at
> org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$insertRowInTablesParallel$4(TablesAmountCapacityBaseTest.java:175)
> ... 4 more{code}
> Server logs are in attachment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
