[
https://issues.apache.org/jira/browse/CALCITE-5099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538758#comment-17538758
]
itxiangkui commented on CALCITE-5099:
-------------------------------------
I think I have sorted out the conditions and process of the entire bug trigger:
【environment】
1. sqlline interactive environment (latest version), the server uses Avatical's
LocalService to open a port
2. Calcite's sql is applied in Jdbc connection scenarios with multiple
backends, for example, the backend uses JdbcSchema to connect to Mysql (the
table data of multiple mysql instances is used for join operation)
3. calcite version 1.30.0
[Bug trigger process]
1. The client of sqlline submits a sql, such as "select
mysql1.d1.t1,mysql2.d2.t2 where t1.id=t2.id"
2. After LocalService receives the request, it starts to iterate over the data
(and then returns a FetchResponse(Meta.Frame...) object)
3. The fetch process of Meta.Frame will call the
ResultSetEnumerable.enumeratorBasedOnStatement method
4. The third step will return an "iterator", which is actually
Linq4j.EnumeratorIterator
It also implements IteratorAble and AutoCloseable. The close method of this
interface will get the connection of the statement, and then close it to
prevent resource leakage.
5. There is code in this fetch method
{code:java}
MetaImpl.collect(signature.cursorFactory,
LimitIterator.of(iterator, fetchMaxRowCount),
new ArrayList<List<Object>>()) {code}
LimitIterator is a wrapper around iterator, this new iterator discards the
features of the AutoCloseable() interface, and here's where the problem
starts...
6. There is the following code in the method of MetaImpl.collect
{code:java}
public static List<List<Object>> collect(CursorFactory cursorFactory,
Iterable<Object> iterable, final List<List<Object>> list) {
switch(cursorFactory.style) {
case OBJECT:
for (Object o : iterable) {
list.add(Collections.singletonList(o));
}
return list; {code}
After finally getting the list, this iterator will never have the opportunity
to close it again...
7. The connection pool is used at the main entrance of Calcite to execute Sql,
and the connection given by the connection pool is leaked...
Refer to the key code snippet:
{code:java}
@Override public Frame fetch(StatementHandle h, long offset,
int fetchMaxRowCount) throws NoSuchStatementException {
final CalciteConnectionImpl calciteConnection = getConnection();
CalciteServerStatement stmt = calciteConnection.server.getStatement(h);
final Signature signature = requireNonNull(stmt.getSignature(),
() -> "stmt.getSignature() is null for " + stmt);
final Iterator<Object> iterator;
Iterator<Object> stmtResultSet = stmt.getResultSet();
if (stmtResultSet == null) {
final Iterable<Object> iterable =
_createIterable(h, signature, null, null);
iterator = iterable.iterator();
stmt.setResultSet(iterator);
} else {
iterator = stmtResultSet;
}
final List rows =
MetaImpl.collect(signature.cursorFactory,
LimitIterator.of(iterator, fetchMaxRowCount),
new ArrayList<List<Object>>());
boolean done = fetchMaxRowCount == 0 || rows.size() < fetchMaxRowCount;
@SuppressWarnings("unchecked") List<Object> rows1 = (List<Object>) rows;
return new Meta.Frame(offset, done, rows1);
}
{code}
> ResultSetEnumerable.enumeratorBasedOnStatement cause the backend connection
> leak
> ---------------------------------------------------------------------------------
>
> Key: CALCITE-5099
> URL: https://issues.apache.org/jira/browse/CALCITE-5099
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.30.0
> Reporter: itxiangkui
> Priority: Major
> Labels: connection, connection-pooling, leak
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
