Abhishek Singh Chouhan created CALCITE-5690: -----------------------------------------------
Summary: ConcurrentModificationException during validation of table with DynamicRecordType Key: CALCITE-5690 URL: https://issues.apache.org/jira/browse/CALCITE-5690 Project: Calcite Issue Type: Bug Affects Versions: 1.32.0 Reporter: Abhishek Singh Chouhan When multiple threads are doing validation on a Table with DynamicRecordTypeImpl, we run into a ConcurrentModificationException. One of the instances where this happens is when two validations are happening in parallel Thread1 - SELECT DS_GET_QUANTILE(DS_QUANTILES_SKETCH(COL1), 0.9) FROM TABLE1 GROUP BY COL1, COL2, COL3 Thread2 - SELECT DS_GET_QUANTILE(DS_QUANTILES_SKETCH(COL1), 0.9) FROM TABLE1 GROUP BY COL1, COL2 While expanding the groupby in thread1 we intern the type at multiple places, eg. DelegatingScope#fullyQualify -> SqlValidatorScope#rowType -> RelDataTypeFactoryImpl#createTypeWithNullability -> canonize During the validation in thread1 we cache DynamicRecordTypeImpl with Col1 and Col2 as the fields. The race condition happens when thread2 gets a hold of this(abovementioned) cached DynamicRecordTypeImpl during its validation and is iterating over the fields eg during SqlValidatorImpl.validateGroupItem and thread 1 then goes and inserts Col3 field in the holder. {code:java} java.util.ConcurrentModificationException at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043) at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997) at org.apache.calcite.rel.type.RelDataTypeHolder.getFieldOrInsert(RelDataTypeHolder.java:56) at org.apache.calcite.rel.type.DynamicRecordTypeImpl.getField(DynamicRecordTypeImpl.java:59) at org.apache.calcite.sql.validate.SqlNameMatchers$BaseMatcher.field(SqlNameMatchers.java:126) at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6455) at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6360) at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:324) at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1869) at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1854) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateGroupItem(SqlValidatorImpl.java:4393) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateGroupClause(SqlValidatorImpl.java:4366) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3701) at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:64) at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1107) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1078) at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:248) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1053) at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:759) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)