[jira] [Created] (IMPALA-8243) ConcurrentModificationException in Catalog stress tests
bharath v created IMPALA-8243: - Summary: ConcurrentModificationException in Catalog stress tests Key: IMPALA-8243 URL: https://issues.apache.org/jira/browse/IMPALA-8243 Project: IMPALA Issue Type: Bug Components: Catalog Affects Versions: Impala 3.1.0 Reporter: bharath v Assignee: bharath v Following is the full stack from the Catalog server logs. {noformat} 14:09:29.474424 14829 jni-util.cc:256] java.util.ConcurrentModificationException java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) java.util.ArrayList$Itr.next(ArrayList.java:851) org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1449) org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1278) org.apache.hadoop.hive.metastore.api.StorageDescriptor.write(StorageDescriptor.java:1144) org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:1062) org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:919) org.apache.hadoop.hive.metastore.api.Partition.write(Partition.java:815) org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:862) org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:759) org.apache.impala.thrift.TPartialPartitionInfo.write(TPartialPartitionInfo.java:665) org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:731) org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:624) org.apache.impala.thrift.TPartialTableInfo.write(TPartialTableInfo.java:543) org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:977) org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:857) org.apache.impala.thrift.TGetPartialCatalogObjectResponse.write(TGetPartialCatalogObjectResponse.java:739) org.apache.thrift.TSerializer.serialize(TSerializer.java:79) org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:233) {noformat} It looks like the bug is in the following piece of code. {noformat} /** * Returns a Hive-compatible partition object that may be used in calls to the * metastore. */ public org.apache.hadoop.hive.metastore.api.Partition toHmsPartition() { if (cachedMsPartitionDescriptor_ == null) return null; Preconditions.checkNotNull(table_.getNonPartitionFieldSchemas()); // Update the serde library class based on the currently used file format. org.apache.hadoop.hive.metastore.api.StorageDescriptor storageDescriptor = new org.apache.hadoop.hive.metastore.api.StorageDescriptor( table_.getNonPartitionFieldSchemas(), <= Reference to the actual field schema list. getLocation(), cachedMsPartitionDescriptor_.sdInputFormat, cachedMsPartitionDescriptor_.sdOutputFormat, cachedMsPartitionDescriptor_.sdCompressed, {noformat} It appears we are leaking a reference to {{nonPartFieldSchemas_}} in to the thrift object and once the thread leaves the lock scope, some other thread (load() for ex:) can potentially change the source list and the serialization code could throw {{ConcurrentModificationException}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8243) ConcurrentModificationException in Catalog stress tests
bharath v created IMPALA-8243: - Summary: ConcurrentModificationException in Catalog stress tests Key: IMPALA-8243 URL: https://issues.apache.org/jira/browse/IMPALA-8243 Project: IMPALA Issue Type: Bug Components: Catalog Affects Versions: Impala 3.1.0 Reporter: bharath v Assignee: bharath v Following is the full stack from the Catalog server logs. {noformat} 14:09:29.474424 14829 jni-util.cc:256] java.util.ConcurrentModificationException java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) java.util.ArrayList$Itr.next(ArrayList.java:851) org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1449) org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1278) org.apache.hadoop.hive.metastore.api.StorageDescriptor.write(StorageDescriptor.java:1144) org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:1062) org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:919) org.apache.hadoop.hive.metastore.api.Partition.write(Partition.java:815) org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:862) org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:759) org.apache.impala.thrift.TPartialPartitionInfo.write(TPartialPartitionInfo.java:665) org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:731) org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:624) org.apache.impala.thrift.TPartialTableInfo.write(TPartialTableInfo.java:543) org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:977) org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:857) org.apache.impala.thrift.TGetPartialCatalogObjectResponse.write(TGetPartialCatalogObjectResponse.java:739) org.apache.thrift.TSerializer.serialize(TSerializer.java:79) org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:233) {noformat} It looks like the bug is in the following piece of code. {noformat} /** * Returns a Hive-compatible partition object that may be used in calls to the * metastore. */ public org.apache.hadoop.hive.metastore.api.Partition toHmsPartition() { if (cachedMsPartitionDescriptor_ == null) return null; Preconditions.checkNotNull(table_.getNonPartitionFieldSchemas()); // Update the serde library class based on the currently used file format. org.apache.hadoop.hive.metastore.api.StorageDescriptor storageDescriptor = new org.apache.hadoop.hive.metastore.api.StorageDescriptor( table_.getNonPartitionFieldSchemas(), <= Reference to the actual field schema list. getLocation(), cachedMsPartitionDescriptor_.sdInputFormat, cachedMsPartitionDescriptor_.sdOutputFormat, cachedMsPartitionDescriptor_.sdCompressed, {noformat} It appears we are leaking a reference to {{nonPartFieldSchemas_}} in to the thrift object and once the thread leaves the lock scope, some other thread (load() for ex:) can potentially change the source list and the serialization code could throw {{ConcurrentModificationException}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org