[
https://issues.apache.org/jira/browse/IGNITE-19655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729675#comment-17729675
]
Konstantin Orlov commented on IGNITE-19655:
-------------------------------------------
{quote}
the SQL engine does not seem to care about the fact that the node has already
left.
{quote}
It's true. But SQL doesn't have to.
SQL engine uses TopologyService (TS) to get all live members of a cluster. So
it's responsibility of TS to keep track of members of the cluster and provide
valid state for those who asking for it.
I run mentioned test locally and found rather peculiar log sequence:
{noformat}
2023-06-06 12:25:50:581 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Node joined
[node=ClusterNode [id=6f399661-6f3f-4d8a-ae79-ec2ad758c80a,
name=itrst_sitdnbsiwsaiitf_1, address=192.168.8.104:3345, nodeMetadata=null]]
2023-06-06 12:25:50:582 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Topology snapshot
[nodes=[itrst_sitdnbsiwsaiitf_1, itrst_sitdnbsiwsaiitf_0]]
2023-06-06 12:25:51:558 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Topology snapshot
[nodes=[itrst_sitdnbsiwsaiitf_1, itrst_sitdnbsiwsaiitf_0]]
2023-06-06 12:25:51:904 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Node joined
[node=ClusterNode [id=c015b0a9-aa4b-4a27-b42b-ba611b46978b,
name=itrst_sitdnbsiwsaiitf_2, address=192.168.8.104:3346, nodeMetadata=null]]
2023-06-06 12:25:51:904 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Topology snapshot
[nodes=[itrst_sitdnbsiwsaiitf_2, itrst_sitdnbsiwsaiitf_1,
itrst_sitdnbsiwsaiitf_0]]
2023-06-06 12:25:52:870 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Topology snapshot
[nodes=[itrst_sitdnbsiwsaiitf_2, itrst_sitdnbsiwsaiitf_1,
itrst_sitdnbsiwsaiitf_0]]
2023-06-06 12:25:58:668 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Topology snapshot
[nodes=[itrst_sitdnbsiwsaiitf_2, itrst_sitdnbsiwsaiitf_1,
itrst_sitdnbsiwsaiitf_0]]
2023-06-06 12:26:06:408 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Node joined
[node=ClusterNode [id=21182b57-f6fa-4129-94fe-c2f78a06296c,
name=itrst_sitdnbsiwsaiitf_2, address=192.168.8.104:3346, nodeMetadata=null]]
2023-06-06 12:26:06:408 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Topology snapshot
[nodes=[itrst_sitdnbsiwsaiitf_2, itrst_sitdnbsiwsaiitf_1,
itrst_sitdnbsiwsaiitf_0]]
2023-06-06 12:26:07:204 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Topology snapshot
[nodes=[itrst_sitdnbsiwsaiitf_2, itrst_sitdnbsiwsaiitf_1,
itrst_sitdnbsiwsaiitf_0]]
2023-06-06 12:26:07:261 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Node left
[member=ClusterNode [id=c015b0a9-aa4b-4a27-b42b-ba611b46978b,
name=itrst_sitdnbsiwsaiitf_2, address=192.168.8.104:3346, nodeMetadata=null]]
2023-06-06 12:26:07:303 +0300
[INFO][sc-cluster-3344-1][ScaleCubeTopologyService] Topology snapshot
[nodes=[itrst_sitdnbsiwsaiitf_2, itrst_sitdnbsiwsaiitf_1,
itrst_sitdnbsiwsaiitf_0]]
2023-06-06 12:26:29:460 +0300 [INFO][sc-cluster-3344-1][Transport] Stopping
[address=192.168.8.104:3344]
2023-06-06 12:26:29:461 +0300 [INFO][sc-cluster-3344-1][Transport] Stopped
[address=192.168.8.104:3344]
{noformat}
In the log, the node {{itrst_sitdnbsiwsaiitf_2}} is joined twice: first time
with ID={{c015...}}, and then with ID={{2118...}}. Then Node left was fired for
this node with ID={{c015...}}. Since this node had been re-registered with new
ID, this event just ignored leaving topology snapshot without change:
{{Topology snapshot [nodes=[itrst_sitdnbsiwsaiitf_2, itrst_sitdnbsiwsaiitf_1,
itrst_sitdnbsiwsaiitf_0]]}}.
I believe this is root cause of tests flakiness.
> Distributed Sql keeps mapping query fragments to a node that has already left
> -----------------------------------------------------------------------------
>
> Key: IGNITE-19655
> URL: https://issues.apache.org/jira/browse/IGNITE-19655
> Project: Ignite
> Issue Type: Bug
> Reporter: Roman Puchkovskiy
> Assignee: Maksim Zhuravkov
> Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> There are two test failures:
> [https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/7271211?expandCode+Inspection=true&expandBuildProblemsSection=true&hideProblemsFromDependencies=false&expandBuildTestsSection=true&hideTestsFromDependencies=false]
> and
> [https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/7272905?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandCode+Inspection=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true]
>
> (org.apache.ignite.internal.raftsnapshot.ItTableRaftSnapshotsTest.entriesKeepAppendedAfterSnapshotInstallation
> and
> org.apache.ignite.internal.raftsnapshot.ItTableRaftSnapshotsTest.snapshotInstallTimeoutDoesNotBreakSubsequentInstallsWhenSecondAttemptIsIdenticalToFirst,
> correspondingly).
> In both cases, the test code creates a table with 3 replicas on a cluster of
> 3 nodes, then it stops the last node and tries to make an insert using one of
> the 2 remaining nodes. The RAFT majority (2 of 3) is still preserved, so the
> insert should succeed. It's understood that the insert might be issued before
> the remaining nodes understand that the third node has left, so we have a
> retry mechanism in place, it makes up to 5 attempts for almost 8 seconds (in
> total).
> But in both the failed runs, each of 5 attempts failed because a fragment of
> the INSERT query was mapped to the missing node. This seems to be a bad luck
> (as the tests pass most of the time, fail rate is about 2.5%), but anyway:
> the SQL engine does not seem to care about the fact that the node has already
> left.
> Probably, the SQL engine should track the Logical Topology events and avoid
> mapping query fragments to the missing nodes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
