[ https://issues.apache.org/jira/browse/CALCITE-5690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720683#comment-17720683 ]
Julian Hyde commented on CALCITE-5690: -------------------------------------- Does {{RelDataTypeHolder}} claim to be thread-safe? Is its implementation actually thread-safe? Does the code using it need to be thread-safe, and if so, did it make a mistake using a non-thread-safe class? Sorry to answer questions with questions, but the problem here seems to be that some code is making invalid assumptions and/or some code is making invalid claims. The solution is to answer those questions. > ConcurrentModificationException during validation of table with > DynamicRecordType > --------------------------------------------------------------------------------- > > Key: CALCITE-5690 > URL: https://issues.apache.org/jira/browse/CALCITE-5690 > Project: Calcite > Issue Type: Bug > Affects Versions: 1.32.0 > Reporter: Abhishek Singh Chouhan > Priority: Major > > When multiple threads are doing validation on a Table with > DynamicRecordTypeImpl, we run into a ConcurrentModificationException. One of > the instances where this happens is when two validations are happening in > parallel > Thread1 - > SELECT DS_GET_QUANTILE(DS_QUANTILES_SKETCH(COL1), 0.9) FROM TABLE1 GROUP BY > COL1, COL2, COL3 > Thread2 - SELECT DS_GET_QUANTILE(DS_QUANTILES_SKETCH(COL1), 0.9) FROM TABLE1 > GROUP BY COL1, COL2 > While expanding the groupby in thread1 we intern the type at multiple places, > eg. DelegatingScope#fullyQualify -> SqlValidatorScope#rowType -> > RelDataTypeFactoryImpl#createTypeWithNullability -> canonize > During the validation in thread1 we cache DynamicRecordTypeImpl with Col1 and > Col2 as the fields. > The race condition happens when thread2 gets a hold of this(abovementioned) > cached DynamicRecordTypeImpl during its validation and is iterating over the > fields eg during SqlValidatorImpl.validateGroupItem and thread 1 then goes > and inserts Col3 field in the holder. > {code:java} > java.util.ConcurrentModificationException > at > java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043) > at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997) > at > org.apache.calcite.rel.type.RelDataTypeHolder.getFieldOrInsert(RelDataTypeHolder.java:56) > at > org.apache.calcite.rel.type.DynamicRecordTypeImpl.getField(DynamicRecordTypeImpl.java:59) > at > org.apache.calcite.sql.validate.SqlNameMatchers$BaseMatcher.field(SqlNameMatchers.java:126) > at > org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6455) > at > org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6360) > at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:324) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1869) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1854) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateGroupItem(SqlValidatorImpl.java:4393) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateGroupClause(SqlValidatorImpl.java:4366) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3701) > at > org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:64) > at > org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1107) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1078) > at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:248) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1053) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:759) > {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)