[ https://issues.apache.org/jira/browse/CALCITE-5690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723258#comment-17723258 ]
Julian Hyde commented on CALCITE-5690: -------------------------------------- Your 'here' links got corrupted, but I think I got your point. It probably makes sense to stop interning {{DynamicRecordTypeImpl}}, but not for the reasons you say. In your case, you seem to be using the same {{Table}} object and its embedded {{RelDataTypeHolder}} from two different queries, in different threads. As separate statements, those queries must have distinct type factories. And yet the {{Table}} is returning the same type object to both statements. There is a contradiction: that type object cannot belong to two different type factories. The solution, I think, is to remove {{RelDataTypeFactory}} from {{RelDataTypeHolder}} and make its {{fields}} field thread-safe. With other types, even though they are immutable, we should not assume that they are thread-safe; they probably are, but that wasn't a design goal. If there was some concrete goal - e.g. multi-threaded execution of VolcanoPlanner to optimize a particular query faster - then we should declare that as an explicit goal and write the necessary tests to ensure that it is met. > ConcurrentModificationException during validation of table with > DynamicRecordType > --------------------------------------------------------------------------------- > > Key: CALCITE-5690 > URL: https://issues.apache.org/jira/browse/CALCITE-5690 > Project: Calcite > Issue Type: Bug > Affects Versions: 1.32.0 > Reporter: Abhishek Singh Chouhan > Priority: Major > > When multiple threads are doing validation on a Table with > DynamicRecordTypeImpl, we run into a ConcurrentModificationException. One of > the instances where this happens is when two validations are happening in > parallel > Thread1 - > SELECT DS_GET_QUANTILE(DS_QUANTILES_SKETCH(COL1), 0.9) FROM TABLE1 GROUP BY > COL1, COL2, COL3 > Thread2 - SELECT DS_GET_QUANTILE(DS_QUANTILES_SKETCH(COL1), 0.9) FROM TABLE1 > GROUP BY COL1, COL2 > While expanding the groupby in thread1 we intern the type at multiple places, > eg. DelegatingScope#fullyQualify -> SqlValidatorScope#rowType -> > RelDataTypeFactoryImpl#createTypeWithNullability -> canonize > During the validation in thread1 we cache DynamicRecordTypeImpl with Col1 and > Col2 as the fields. > The race condition happens when thread2 gets a hold of this(abovementioned) > cached DynamicRecordTypeImpl during its validation and is iterating over the > fields eg during SqlValidatorImpl.validateGroupItem and thread 1 then goes > and inserts Col3 field in the holder. > {code:java} > java.util.ConcurrentModificationException > at > java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043) > at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997) > at > org.apache.calcite.rel.type.RelDataTypeHolder.getFieldOrInsert(RelDataTypeHolder.java:56) > at > org.apache.calcite.rel.type.DynamicRecordTypeImpl.getField(DynamicRecordTypeImpl.java:59) > at > org.apache.calcite.sql.validate.SqlNameMatchers$BaseMatcher.field(SqlNameMatchers.java:126) > at > org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6455) > at > org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6360) > at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:324) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1869) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1854) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateGroupItem(SqlValidatorImpl.java:4393) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateGroupClause(SqlValidatorImpl.java:4366) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3701) > at > org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:64) > at > org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1107) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1078) > at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:248) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1053) > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:759) > {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)