[
https://issues.apache.org/jira/browse/ATLAS-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal Shah updated ATLAS-4866:
------------------------------
Description:
*Background:*
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
*Issue:*
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
{code:java}
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
{code}
*Solution:*
Hence in this case, storing processed entities guids is skipped while creating
ExportImportAuditEntry when size of entity goes beyond the value stored in
bytes below
{code:java}
atlas.hbase.client.keyvalue.maxsize{code}
was:
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
{code:java}
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
{code}
Hence in this case, storing processed entities guids is skipped while creating
ExportImportAuditEntry
> Export/Import: Skip adding processed guids in audit while importing huge zip
> ------------------------------------------------------------------------------
>
> Key: ATLAS-4866
> URL: https://issues.apache.org/jira/browse/ATLAS-4866
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Pinal Shah
> Assignee: Pinal Shah
> Priority: Major
>
> *Background:*
> Atlas uses HBase as its store for audit repository.
> After import, atlas store the audit entity with the import information along
> with all the processed entitiy guids.
> *Issue:*
> When size of the entity is greater than "hbase.client.keyvalue.maxsize"
> property then audit entity creation fails with exception.
> {code:java}
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: org.apache.hadoop.hbase.DoNotRetryIOException:
> Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
> with size 23826581 exceeds limit of 10485760 bytes
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
> : 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
> at
> org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
> at
> org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
> {code}
> *Solution:*
> Hence in this case, storing processed entities guids is skipped while
> creating ExportImportAuditEntry when size of entity goes beyond the value
> stored in bytes below
> {code:java}
> atlas.hbase.client.keyvalue.maxsize{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
