[
https://issues.apache.org/jira/browse/ATLAS-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal Shah updated ATLAS-4866:
------------------------------
Description:
*Background:*
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
*Issue:*
When large sized export zipped file is imported, import gives below error,
internally import gets succesful, but fail creating audit.
_{"errorCode":"ATLAS-500-00-001","errorMessage":"org.janusgraph.core.JanusGraphException:
Could not commit transaction due to exception during
persistence","errorCause":"Could not commit transaction due to exception during
persistence"}_
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
{code:java}
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
{code}
*Solution:*
Hence in this case, storing processed entities guids is skipped while creating
ExportImportAuditEntry when size of entity goes beyond the value stored in
bytes below
{code:java}
atlas.hbase.client.keyvalue.maxsize{code}
was:
*Background:*
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
*Issue:*
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
{code:java}
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
{code}
*Solution:*
Hence in this case, storing processed entities guids is skipped while creating
ExportImportAuditEntry when size of entity goes beyond the value stored in
bytes below
{code:java}
atlas.hbase.client.keyvalue.maxsize{code}
> Export/Import: Skip adding processed guids in audit while importing huge zip
> ------------------------------------------------------------------------------
>
> Key: ATLAS-4866
> URL: https://issues.apache.org/jira/browse/ATLAS-4866
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Pinal Shah
> Assignee: Pinal Shah
> Priority: Major
>
> *Background:*
> Atlas uses HBase as its store for audit repository.
> After import, atlas store the audit entity with the import information along
> with all the processed entitiy guids.
> *Issue:*
> When large sized export zipped file is imported, import gives below error,
> internally import gets succesful, but fail creating audit.
> _{"errorCode":"ATLAS-500-00-001","errorMessage":"org.janusgraph.core.JanusGraphException:
> Could not commit transaction due to exception during
> persistence","errorCause":"Could not commit transaction due to exception
> during persistence"}_
> When size of the entity is greater than "hbase.client.keyvalue.maxsize"
> property then audit entity creation fails with exception.
> {code:java}
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: org.apache.hadoop.hbase.DoNotRetryIOException:
> Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
> with size 23826581 exceeds limit of 10485760 bytes
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
> : 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
> at
> org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
> at
> org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
> {code}
> *Solution:*
> Hence in this case, storing processed entities guids is skipped while
> creating ExportImportAuditEntry when size of entity goes beyond the value
> stored in bytes below
> {code:java}
> atlas.hbase.client.keyvalue.maxsize{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
