Pinal Shah created ATLAS-4866: --------------------------------- Summary: Export/Import: Skip adding processed guids in audit while importing huge zip Key: ATLAS-4866 URL: https://issues.apache.org/jira/browse/ATLAS-4866 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Pinal Shah Assignee: Pinal Shah
Atlas uses HBase as its store for audit repository. After import, atlas store the audit entity with the import information along with all the processed entitiy guids. When size of the entity is greater than "hbase.client.keyvalue.maxsize" property then audit entity creation fails with exception. {code:java} Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException: Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0] with size 23826581 exceeds limit of 10485760 bytes at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) : 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185 at org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422) at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51) {code} Hence in this case, storing processed entities guids is skipped while creating ExportImportAuditEntry -- This message was sent by Atlassian Jira (v8.20.10#820010)