[
https://issues.apache.org/jira/browse/HDDS-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819731#comment-17819731
]
Tsz-wo Sze edited comment on HDDS-9342 at 3/12/24 5:53 PM:
-----------------------------------------------------------
For fixing the rocksdb caused by this problem, we may
# backup OM rocksdb, and then
# use the rocksdb_ldb tool (provided by Rocksdb) to update TRANSACTIONINFO:
{code:java}
$rocksdb_ldb --db=/var/lib/hadoop-ozone/om/data/om.db/ scan
--column_family=transactionInfoTable
#TRANSACTIONINFO ==> 11#282615
$rocksdb_ldb --db=/var/lib/hadoop-ozone/om/data/om.db/ put "#TRANSACTIONINFO"
"11#573145" --column_family=transactionInfoTable
OK
$rocksdb_ldb --db=/var/lib/hadoop-ozone/om/data/om.db/ scan
--column_family=transactionInfoTable
#TRANSACTIONINFO ==> 11#573145
{code}
See also
[https://github.com/facebook/rocksdb/wiki/RocksDB-Troubleshooting-Guide]
*{color:#FF0000}IMPROTANT{color}:* If TRANSACTIONINFO is not updated to the
highest index, we may see the same error "IllegalArgumentException: Trying to
set updateID" again when restarting OM and the exception message will contain a
higher index. We MUST restore the OM rocksdb backed up at Step 1 before
retrying Step 2 with the higher index.
was (Author: szetszwo):
For fixing the rocksdb caused by this problem, we may
# backup OM rocksdb, and then
# use the rocksdb_ldb tool (provided by Rocksdb) to update TRANSACTIONINFO:
{code:java}
$rocksdb_ldb --db=/var/lib/hadoop-ozone/om/data/om.db/ scan
--column_family=transactionInfoTable
#TRANSACTIONINFO ==> 11#282615
$rocksdb_ldb --db=/var/lib/hadoop-ozone/om/data/om.db/ put "#TRANSACTIONINFO"
"11#573145" --column_family=transactionInfoTable
OK
$rocksdb_ldb --db=/var/lib/hadoop-ozone/om/data/om.db/ scan
--column_family=transactionInfoTable
#TRANSACTIONINFO ==> 11#573145
{code}
See also
[https://github.com/facebook/rocksdb/wiki/RocksDB-Troubleshooting-Guide]
*{color:#FF0000}IMPROTANT{color}:* If TRANSACTIONINFO is not updated to the
highest index, we may see the same error "IllegalArgumentException: Trying to
set updateID" again when restarting OM and the exception message will contain a
higher index. We MUST restore the OM rocksdb backed up at Step 1 before
retrying Step 2 with the higher index.
> OM restart failed due to transactionLogIndex smaller than current updateID
> --------------------------------------------------------------------------
>
> Key: HDDS-9342
> URL: https://issues.apache.org/jira/browse/HDDS-9342
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM, OM HA
> Affects Versions: 1.3.0
> Reporter: Hongbing Wang
> Assignee: Sammi Chen
> Priority: Blocker
> Labels: pull-request-available
> Attachments: HDDS-9342_reproduce_20231217.patch,
> HDDS-9342_testUpdateId.patch, HDDS-9342_testUpdateId_reproduce.patch,
> clipboard_image_1700795744614.png, om.shutdown-20230922.log
>
>
> OM restart failed, log as follow:
> create failed:
> {noformat}
> java.lang.IllegalArgumentException: Trying to set updateID to 2901863625
> which is not greater than the current value of 2901863627 for
> OMKeyInfo{volume='vol-xxx', bucket='xxx', key='user/xxx/platform/xxx',
> dataSize='268435456', creationTime='1695088210914',
> objectID='-9223371293977687808', parentID='0', replication='RATIS/THREE',
> fileChecksum='null}
> at
> org.apache.hadoop.ozone.om.helpers.WithObjectID.setUpdateID(WithObjectID.java:105)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.prepareFileInfo(OMKeyRequest.java:665)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.prepareKeyInfo(OMKeyRequest.java:623)
> at
> org.apache.hadoop.ozone.om.request.file.OMFileCreateRequest.validateAndUpdateCache(OMFileCreateRequest.java:255)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:311)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRouterRequestHandler.handleWriteRequest(OzoneManagerRouterRequestHandler.java:806)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:535)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:326)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> rename failed:
> {noformat}
> java.lang.IllegalArgumentException: Trying to set updateID to 2901863669
> which is not greater than the current value of 3076345041 for
> OMKeyInfo{volume='vol-xxx', bucket='xxx', key='checkative/xxx',
> dataSize='23124', creationTime='1695380440059',
> objectID='-9223371249310446848', parentID='0', replication='RATIS/THREE',
> fileChecksum='null}
> at
> org.apache.hadoop.ozone.om.helpers.WithObjectID.setUpdateID(WithObjectID.java:105)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRenameRequest.validateAndUpdateCache(OMKeyRenameRequest.java:190)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:311)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRouterRequestHandler.handleWriteRequest(OzoneManagerRouterRequestHandler.java:806)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:535)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:326)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
