[ https://issues.apache.org/jira/browse/IMPALA-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong updated IMPALA-2648: ---------------------------------- Component/s: Catalog > catalogd crashes when serialized messages are over 2 GB > ------------------------------------------------------- > > Key: IMPALA-2648 > URL: https://issues.apache.org/jira/browse/IMPALA-2648 > Project: IMPALA > Issue Type: Bug > Components: Catalog > Affects Versions: Impala 2.3.0 > Reporter: Silvius Rus > Assignee: Tianyi Wang > Priority: Critical > Labels: compute-stats, crash, downgraded > > We've seen a catalogd crash triggered by loading the metadata for a table > with about 20K partitions and 77 columns that has incremental stats. It > looks like the serialized message is over 2GB which is the Java max array > size. Ideally we should catch this exception and fail the query that needs > this table's metadata with an appropriate message. > {code} > I1107 06:47:56.641507 30252 jni-util.cc:177] java.lang.OutOfMemoryError: > Requested array size exceeds VM limit > at java.util.Arrays.copyOf(Arrays.java:2271) > at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) > at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) > at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) > at > org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145) > at > org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:187) > at > com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1831) > at > com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1543) > at com.cloudera.impala.thrift.THdfsPartition.write(THdfsPartition.java:1389) > at > com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:1123) > at > com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:969) > at com.cloudera.impala.thrift.THdfsTable.write(THdfsTable.java:848) > at > com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1628) > at > com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1395) > at com.cloudera.impala.thrift.TTable.write(TTable.java:1209) > at > com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1241) > at > com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1098) > at com.cloudera.impala.thrift.TCatalogObject.write(TCatalogObject.java:938) > at > com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:487) > at > com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:421) > at > com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse.write(TGetAllCatalogObjectsResponse.java:365) > at org.apache.thrift.TSerializer.serialize(TSerializer.java:79) > at > com.cloudera.impala.service.JniCatalog.getCatalogObjects(JniCatalog.java:110) > {code} > You can identify this issue by looking at the metastore database. Here is > how to see the size of the incremental stats for table id 12345. The table > with this value of 624 MB of incremental stats led to the catalogd crash > shown above. > {code} > postgres=# select pg_size_pretty(sum(length("PARTITION_PARAMS"."PARAM_KEY") + > length("PARTITION_PARAMS"."PARAM_VALUE"))) from "PARTITIONS", > "PARTITION_PARAMS" where "PARTITIONS"."TBL_ID"=12345 and > "PARTITIONS"."PART_ID" = "PARTITION_PARAMS"."PART_ID" and > "PARTITION_PARAMS"."PARAM_KEY" LIKE 'impala_intermediate%'; > pg_size_pretty > ---------------- > 624 MB > (1 row) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org