[ https://issues.apache.org/jira/browse/CALCITE-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987023#comment-16987023 ]
Ruben Q L commented on CALCITE-3535: ------------------------------------ With some invaluable help from [~zabetak], it seems that we have finally discovered the issue. The root cause of the problem is that the nested loop join implementation in EnumerableDefaults#nestedLoopJoin *does not close all its resources*. Specifically, it uses some enumerators that are never closed. This means, when running the tests, these un-closed enumerators hold some resources, e.g. Connection objects, which are never closed, i.e. released, and that causes the deadlock when trying to get a new Connection from the Connection pool in further tests. We can see now this issue because, due to the change in EnumerableJoinRule, nested loop join algorithm is more used than before, e.g. an inner Join with condition=true was executed previously via a HashJoin, and now with a NestedLoopJoin. Since more nested loop joins are executed in the tests, more un-closed resources are generated, until we eventually reach the problem. To avoid mixing things, I will commit the fix in EnumerableDefaults#nestedLoopJoin in a separate tickect. By change, I happen to have an open ticket which is somehow related to this topic: "CALCITE-3546 - Improve EnumerableDefaults nested loop join". That ticket provides a new, more optimized nested loop join algorithm, so the issue will probably be mitigated. However, the old (and "resource-greedy") algorithm still remains (the new optimized algorithm does not support right / full join); so I will commit the fix to close resources in there. > EnumerableJoinRule: remove unnecessary Filter on top of INNER Join > ------------------------------------------------------------------ > > Key: CALCITE-3535 > URL: https://issues.apache.org/jira/browse/CALCITE-3535 > Project: Calcite > Issue Type: Task > Components: core > Affects Versions: 1.21.0 > Reporter: Ruben Q L > Assignee: Ruben Q L > Priority: Minor > Labels: pull-request-available > Fix For: 1.22.0 > > Attachments: threadDump.txt > > Time Spent: 20m > Remaining Estimate: 0h > > With the implementation of CALCITE-2973, now EnumerableHashJoin supports all > type of conditions (not just equi joins). However, there is still one > [TODO|https://github.com/apache/calcite/blob/07c1efbb2b42a0e27f9d01968d357d9c1e17cd51/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableJoinRule.java#L89] > in EnumerableHashJoinRule that, in case of an INNER Join, creates a Filter > with the non-equi conditions on top of the EnumerableHashJoin (created only > with the equi-conditions), this filter is not really needed, since now > EnumerableHashJoinRule can support the full condition: equi and non-equi > items. -- This message was sent by Atlassian Jira (v8.3.4#803005)