Mukul Kumar Singh created HDFS-12337:
----------------------------------------
Summary: Ozone: Concurrent RocksDB open calls fail because of "No
locks available"
Key: HDFS-12337
URL: https://issues.apache.org/jira/browse/HDFS-12337
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Affects Versions: HDFS-7240
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
Fix For: HDFS-7240
HDFS-12216 fixes the issue with the static container cache and re-using the
same container port on datanode restart. However TestKeys still fails after
HDFS-12216 is fixed.
The test is now failing because Concurrent RocksDB open calls fail. In the
current code BlockDeleting service and Dispatcher tries to open the db
concurrently.
This jira will also fix the keepPort property for Ratis Container port and set
the reuse address property for XceiverServerRatis correctly.
{code}
2017-08-22 16:51:34,453 [BlockDeletingService#1] INFO utils.RocksDBStore
(RocksDBStore.java:<init>(64)) - opening db file
/Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
2017-08-22 16:51:34,460 [nioEventLoopGroup-9-1] INFO logging.LoggingHandler
(Slf4JLogger.java:info(101)) - [id: 0x8822cd3d, /0.0.0.0:57044] RECEIVED: [id:
0x61367e6f, /127.0.0.1:57173 => /127.0.0
.1:57044]
2017-08-22 16:51:34,461 [nioEventLoopGroup-10-1] INFO utils.RocksDBStore
(RocksDBStore.java:<init>(64)) - opening db file
/Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
2017-08-22 16:51:34,465 [nioEventLoopGroup-10-1] INFO utils.RocksDBStore
(RocksDBStore.java:<init>(67)) - Failed init RocksDB, db path :
/Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.dbexception
org.rocksdb.RocksDBException: lock
/Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db/LOCK:
No locks available
2017-08-22 16:51:34,465 [BlockDeletingService#1] INFO
background.BlockDeletingService (BlockDeletingService.java:call(203)) -
The elapsed time of task@70a576ee for deleting blocks: 12ms.
2017-08-22 16:51:34,474 [nioEventLoopGroup-10-1] INFO impl.Dispatcher
(ContainerUtils.java:logAndReturnError(129)) - Operation: GetKey : Trace
ID: 73f19131-f63b-459a-8f09-9a3db893a296 : Message:
621a3b15-b9fc-4d49-a6e6-29d4c40cc91f : Result: UNABLE_TO_READ_METADATA_DB
2017-08-22 16:51:34,475 [Thread-382] INFO exceptions.OzoneExceptionMapper
(OzoneExceptionMapper.java:toResponse(39)) ozone
c2a23759-c76f-49ea-b574-f0802a4e5b75/c0df3a48-f75b-4b5e-b1bd-c189ce698056/13b3d486-3d7a-49e4-bc9d-1ef63e674548
hdfs 73f19131-f63b-459a-8f09-9a3db893a296 - Returning exception. ex:
{"httpCode":500,"shortMessage":"internalServerError","resource":"hdfs","message":"621a3b15-b9fc-4d49-a6e6-29d4c40cc91f","requestID":"73f19131-f63b-459a-8f09-9a3db893a296","hostName":"hw13605.local"}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
